2024-09-21T02:16:07.905 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T02:16:07.929 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T02:16:08.023 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420 branch: wip-pdonnell-testing-20240920.212106-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7913420' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240920.212106-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: mds: mds_export_ephemeral_random_max: 0.1 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 1843221895adccfb3b692060251dd6a0183bc410 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 1843221895adccfb3b692060251dd6a0183bc410 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240920.212106-debug sha1: 1843221895adccfb3b692060251dd6a0183bc410 owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7864 sha1: 1843221895adccfb3b692060251dd6a0183bc410 sleep_before_teardown: 0 subset: 67/512 suite: fs suite_branch: wip-pdonnell-testing-20240920.212106-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1843221895adccfb3b692060251dd6a0183bc410 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBByPOU8gzT0VWh+GKu0ORl/44RhpDcQm16erW0DRrSVqwkWN1dFidHcDgWgIrLYShuotDxIR/JQu5T90tQSSyJU= smithi066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFnR/Hblvgkt3Q7so+5HGqEbVY7EX5Sz4bZm+P9xlPLMZHZ+G3qZoRpJaq6iGngLFltxz4luGseFTLn4D2+EVWA= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNEBA2Uei+i3QqUvGEx9WbZ+28AiIp0gnyE7+ZsExlTWi6yAfvdhhysYnyFr9dofGoT2nxI42ARGZhXREA8svaA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_23:48:22 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T02:16:08.024 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa; will attempt to use it 2024-09-21T02:16:08.024 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks 2024-09-21T02:16:08.025 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T02:16:08.026 INFO:teuthology.task.internal:Checking packages... 2024-09-21T02:16:08.045 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1843221895adccfb3b692060251dd6a0183bc410' 2024-09-21T02:16:08.045 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T02:16:08.045 INFO:teuthology.packaging:ref: None 2024-09-21T02:16:08.045 INFO:teuthology.packaging:tag: None 2024-09-21T02:16:08.045 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240920.212106-debug 2024-09-21T02:16:08.045 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:16:08.045 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240920.212106-debug 2024-09-21T02:16:08.186 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5120.g18432218 2024-09-21T02:16:08.188 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T02:16:08.194 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T02:16:08.195 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T02:16:08.200 INFO:teuthology.task.internal:Saving configuration 2024-09-21T02:16:08.218 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T02:16:08.225 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T02:16:08.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 02:10:21.885876', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBByPOU8gzT0VWh+GKu0ORl/44RhpDcQm16erW0DRrSVqwkWN1dFidHcDgWgIrLYShuotDxIR/JQu5T90tQSSyJU='} 2024-09-21T02:16:08.271 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 02:10:21.884647', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFnR/Hblvgkt3Q7so+5HGqEbVY7EX5Sz4bZm+P9xlPLMZHZ+G3qZoRpJaq6iGngLFltxz4luGseFTLn4D2+EVWA='} 2024-09-21T02:16:08.295 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 02:10:21.883108', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNEBA2Uei+i3QqUvGEx9WbZ+28AiIp0gnyE7+ZsExlTWi6yAfvdhhysYnyFr9dofGoT2nxI42ARGZhXREA8svaA='} 2024-09-21T02:16:08.295 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T02:16:08.301 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-21T02:16:08.301 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-21T02:16:08.301 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-21T02:16:08.301 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T02:16:08.379 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f656a0cc160>, signals=[15]) 2024-09-21T02:16:08.380 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T02:16:08.386 INFO:teuthology.task.internal:Opening connections... 2024-09-21T02:16:08.386 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T02:16:08.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:16:08.462 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2024-09-21T02:16:08.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:16:08.551 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-09-21T02:16:08.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:16:08.630 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T02:16:08.637 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-09-21T02:16:08.655 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-09-21T02:16:08.655 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-09-21T02:16:08.713 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-09-21T02:16:08.713 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T02:16:08.714 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T02:16:08.715 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-09-21T02:16:08.735 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2024-09-21T02:16:08.755 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2024-09-21T02:16:08.755 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2024-09-21T02:16:08.814 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream" 2024-09-21T02:16:08.814 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9" 2024-09-21T02:16:08.814 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2024-09-21T02:16:08.814 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2024-09-21T02:16:08.814 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9" 2024-09-21T02:16:08.814 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9" 2024-09-21T02:16:08.814 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T02:16:08.815 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2024-09-21T02:16:08.815 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon" 2024-09-21T02:16:08.815 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T02:16:08.815 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/" 2024-09-21T02:16:08.815 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T02:16:08.815 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T02:16:08.815 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T02:16:08.815 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2024-09-21T02:16:08.844 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-09-21T02:16:08.862 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-09-21T02:16:08.862 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-09-21T02:16:08.921 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-09-21T02:16:08.921 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-09-21T02:16:08.921 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-09-21T02:16:08.921 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-09-21T02:16:08.921 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-09-21T02:16:08.921 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-09-21T02:16:08.921 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T02:16:08.921 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-09-21T02:16:08.922 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-09-21T02:16:08.922 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T02:16:08.922 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-09-21T02:16:08.922 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T02:16:08.922 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T02:16:08.922 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T02:16:08.922 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-09-21T02:16:08.952 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T02:16:08.962 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T02:16:08.969 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T02:16:08.969 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-09-21T02:16:08.971 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2024-09-21T02:16:08.974 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-09-21T02:16:08.990 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T02:16:08.996 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T02:16:08.997 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-09-21T02:16:09.028 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2024-09-21T02:16:09.032 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-09-21T02:16:09.132 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T02:16:09.211 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T02:16:09.224 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T02:16:09.224 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T02:16:09.225 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T02:16:09.225 DEBUG:teuthology.orchestra.run.smithi066:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T02:16:09.226 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T02:16:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:16:09.243 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-09-21T02:16:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:16:09.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:16:09.244 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-09-21T02:16:09.244 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-21T02:16:09.300 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-503.el9.x86_64 2024-09-21T02:16:09.300 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-503.el9.x86_64 2024-09-21T02:16:09.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-09-21T02:16:09.302 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-503.el9.x86_64 2024-09-21T02:16:09.302 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-503.el9.x86_64 2024-09-21T02:16:09.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-09-21T02:16:09.303 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-503.el9.x86_64 2024-09-21T02:16:09.303 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.14.0-503.el9.x86_64 2024-09-21T02:16:09.303 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum install -y kernel 2024-09-21T02:16:12.939 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.3 MB 00:01 2024-09-21T02:16:13.035 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.3 MB 00:00 2024-09-21T02:16:14.815 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-21T02:16:14.835 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.3 MB 00:03 2024-09-21T02:16:15.262 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-09-21T02:16:16.612 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-21T02:16:19.225 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-21T02:16:20.659 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 110 kB/s | 19 kB 00:00 2024-09-21T02:16:21.715 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-21T02:16:21.896 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-21T02:16:23.020 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.5 MB 00:02 2024-09-21T02:16:24.026 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:01 2024-09-21T02:16:24.454 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 19 kB 00:00 2024-09-21T02:16:25.056 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-21T02:16:25.391 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-21T02:16:27.552 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-21T02:16:29.724 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T02:16:29.725 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T02:16:29.814 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-21T02:16:29.823 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-21T02:16:29.823 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-09-21T02:16:29.823 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-21T02:16:29.823 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-21T02:16:29.823 INFO:teuthology.orchestra.run.smithi066.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T02:16:29.823 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-21T02:16:29.824 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T02:16:29.824 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T02:16:29.824 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T02:16:29.824 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:16:29.824 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-21T02:16:29.824 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-21T02:16:29.824 INFO:teuthology.orchestra.run.smithi066.stdout:Install 4 Packages 2024-09-21T02:16:29.824 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:16:29.825 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 79 M 2024-09-21T02:16:29.825 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 126 M 2024-09-21T02:16:29.825 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-21T02:16:30.170 INFO:teuthology.orchestra.run.smithi066.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 426 kB/s | 36 kB 00:00 2024-09-21T02:16:30.628 INFO:teuthology.orchestra.run.smithi066.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-09-21T02:16:30.755 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-09-21T02:16:30.814 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-21T02:16:31.153 INFO:teuthology.orchestra.run.smithi066.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 32 MB/s | 35 MB 00:01 2024-09-21T02:16:31.629 INFO:teuthology.orchestra.run.smithi066.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-21T02:16:31.629 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-21T02:16:31.629 INFO:teuthology.orchestra.run.smithi066.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-21T02:16:32.337 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-21T02:16:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T02:16:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T02:16:32.738 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-21T02:16:32.738 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-21T02:16:32.756 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T02:16:32.756 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T02:16:32.808 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-21T02:16:32.819 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-21T02:16:32.819 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-21T02:16:32.819 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-21T02:16:32.819 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-21T02:16:32.819 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-09-21T02:16:32.820 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:16:32.821 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 79 M 2024-09-21T02:16:32.821 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 126 M 2024-09-21T02:16:32.821 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-21T02:16:32.862 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-09-21T02:16:32.878 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-09-21T02:16:32.878 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-09-21T02:16:32.878 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-09-21T02:16:32.878 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-09-21T02:16:32.878 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T02:16:32.878 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:16:32.879 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 79 M 2024-09-21T02:16:32.880 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 126 M 2024-09-21T02:16:32.880 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-09-21T02:16:33.023 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 233 kB/s | 36 kB 00:00 2024-09-21T02:16:33.184 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 127 kB/s | 36 kB 00:00 2024-09-21T02:16:33.760 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-21T02:16:33.760 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-21T02:16:33.849 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-09-21T02:16:34.218 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-09-21T02:16:34.449 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-21T02:16:34.757 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 18 MB/s | 35 MB 00:01 2024-09-21T02:16:34.758 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-21T02:16:34.758 INFO:teuthology.orchestra.run.smithi190.stdout:Total 41 MB/s | 79 MB 00:01 2024-09-21T02:16:35.434 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-21T02:16:35.786 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-21T02:16:35.786 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-21T02:16:36.342 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-21T02:16:36.741 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-21T02:16:36.741 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-21T02:16:36.785 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 8.9 MB/s | 35 MB 00:03 2024-09-21T02:16:37.015 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T02:16:37.095 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:16:37.477 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 6.7 MB/s | 29 MB 00:04 2024-09-21T02:16:37.477 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-09-21T02:16:37.477 INFO:teuthology.orchestra.run.smithi029.stdout:Total 17 MB/s | 79 MB 00:04 2024-09-21T02:16:38.191 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-09-21T02:16:38.606 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-09-21T02:16:38.606 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-09-21T02:16:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-21T02:16:39.576 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-09-21T02:16:39.576 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-09-21T02:16:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T02:16:39.916 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:16:39.958 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:16:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:16:42.126 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-09-21T02:16:42.675 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:16:42.803 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:16:42.877 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T02:16:42.992 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:16:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:16:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:16:45.761 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:16:45.969 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:16:47.618 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:16:47.736 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:16:50.584 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:16:51.099 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:16:51.159 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:16:52.591 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:16:56.297 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:30.808 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:30.849 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:30.926 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:30.969 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:31.216 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:31.216 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T02:17:31.216 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:17:31.216 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:17:31.353 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:31.353 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T02:17:31.353 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:17:31.353 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:17:32.118 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:32.118 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:17:32.118 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-21T02:17:32.119 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.119 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.119 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.119 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.119 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:17:32.119 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:17:32.193 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-21T02:17:32.478 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T02:17:32.548 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T02:17:32.566 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.567 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T02:17:32.567 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T02:17:32.567 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T02:17:32.567 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T02:17:32.567 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T02:17:32.567 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T02:17:32.568 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T02:17:32.628 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.629 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T02:17:32.629 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -qi grub2-tools 2024-09-21T02:17:32.637 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.638 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T02:17:32.638 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T02:17:32.638 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T02:17:32.638 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T02:17:32.639 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-21T02:17:32.639 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T02:17:32.639 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T02:17:32.677 INFO:teuthology.orchestra.run.smithi066.stdout:Name : grub2-tools 2024-09-21T02:17:32.677 INFO:teuthology.orchestra.run.smithi066.stdout:Epoch : 1 2024-09-21T02:17:32.677 INFO:teuthology.orchestra.run.smithi066.stdout:Version : 2.06 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Release : 82.el9 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: x86_64 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Group : Unspecified 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Size : 8274670 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:License : GPLv3+ 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Packager : builder@centos.org 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Vendor : CentOS 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T02:17:32.678 INFO:teuthology.orchestra.run.smithi066.stdout:Summary : Support tools for GRUB. 2024-09-21T02:17:32.679 INFO:teuthology.orchestra.run.smithi066.stdout:Description : 2024-09-21T02:17:32.679 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:17:32.679 INFO:teuthology.orchestra.run.smithi066.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T02:17:32.679 INFO:teuthology.orchestra.run.smithi066.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T02:17:32.679 INFO:teuthology.orchestra.run.smithi066.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T02:17:32.679 INFO:teuthology.orchestra.run.smithi066.stdout:hardware devices. 2024-09-21T02:17:32.679 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:17:32.679 INFO:teuthology.orchestra.run.smithi066.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T02:17:32.680 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T02:17:32.680 INFO:teuthology.task.kernel:Updating grub on smithi066 to boot 5.14.0-508.el9.x86_64 2024-09-21T02:17:32.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T02:17:32.696 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:32.696 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T02:17:32.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-09-21T02:17:32.728 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-09-21T02:17:32.728 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 82.el9 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8274670 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-09-21T02:17:32.729 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:17:32.730 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T02:17:32.731 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T02:17:32.731 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-508.el9.x86_64 2024-09-21T02:17:32.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T02:17:33.286 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2024-09-21T02:17:33.359 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-09-21T02:17:34.589 INFO:teuthology.orchestra.run.smithi066.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T02:17:34.629 INFO:teuthology.orchestra.run.smithi066.stderr:done 2024-09-21T02:17:34.631 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-09-21T02:17:34.642 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T02:17:34.646 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.6Mi7c7CAi7 2024-09-21T02:17:34.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6Mi7c7CAi7 2024-09-21T02:17:34.679 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-09-21T02:17:34.681 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-21T02:17:34.696 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.wBkuC8CFOv 2024-09-21T02:17:34.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wBkuC8CFOv 2024-09-21T02:17:34.713 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.6Mi7c7CAi7 2024-09-21T02:17:34.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.wBkuC8CFOv 2024-09-21T02:17:34.851 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.6Mi7c7CAi7 is 6KB 2024-09-21T02:17:34.899 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.6Mi7c7CAi7 2024-09-21T02:17:34.915 DEBUG:teuthology.orchestra.run.smithi066:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T02:17:34.919 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.wBkuC8CFOv is 6KB 2024-09-21T02:17:34.939 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.wBkuC8CFOv 2024-09-21T02:17:34.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T02:17:34.986 INFO:teuthology.orchestra.run.smithi066.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T02:17:34.986 INFO:teuthology.orchestra.run.smithi066.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T02:17:34.986 INFO:teuthology.orchestra.run.smithi066.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T02:17:34.986 INFO:teuthology.orchestra.run.smithi066.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T02:17:34.987 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T02:17:35.019 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T02:17:35.019 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T02:17:35.019 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T02:17:35.019 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T02:17:35.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T02:17:35.143 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2024-09-21T02:17:35.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-09-21T02:17:38.317 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:38.359 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:38.728 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:38.728 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T02:17:38.728 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T02:17:38.728 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T02:17:39.791 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T02:17:39.791 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:17:39.791 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-09-21T02:17:39.791 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:39.791 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T02:17:39.792 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T02:17:39.792 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T02:17:39.792 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:17:39.792 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-09-21T02:17:40.258 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T02:17:40.365 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:40.366 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T02:17:40.366 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T02:17:40.366 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T02:17:40.367 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T02:17:40.367 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T02:17:40.367 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T02:17:40.367 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T02:17:40.427 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T02:17:40.427 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T02:17:40.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-09-21T02:17:40.458 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-09-21T02:17:40.458 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-09-21T02:17:40.458 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-09-21T02:17:40.458 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 82.el9 2024-09-21T02:17:40.458 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8274670 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-09-21T02:17:40.459 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-09-21T02:17:40.460 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:17:40.460 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T02:17:40.460 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T02:17:40.460 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T02:17:40.460 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-09-21T02:17:40.460 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:17:40.460 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T02:17:40.461 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T02:17:40.461 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-508.el9.x86_64 2024-09-21T02:17:40.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T02:17:41.236 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-09-21T02:17:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T02:17:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-09-21T02:17:42.799 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-09-21T02:17:42.814 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.Pr3OTWtkCQ 2024-09-21T02:17:42.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Pr3OTWtkCQ 2024-09-21T02:17:42.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.Pr3OTWtkCQ 2024-09-21T02:17:43.045 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.Pr3OTWtkCQ is 6KB 2024-09-21T02:17:43.093 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.Pr3OTWtkCQ 2024-09-21T02:17:43.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T02:17:43.177 INFO:teuthology.orchestra.run.smithi029.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T02:17:43.177 INFO:teuthology.orchestra.run.smithi029.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T02:17:43.177 INFO:teuthology.orchestra.run.smithi029.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T02:17:43.177 INFO:teuthology.orchestra.run.smithi029.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T02:17:43.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T02:17:43.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-09-21T02:18:05.155 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T02:18:05.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-21T02:18:05.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:18:05.170 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-21T02:18:05.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-21T02:18:05.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:18:13.345 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T02:18:13.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T02:18:13.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:18:23.603 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-09-21T02:18:23.603 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-21T02:18:31.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-09-21T02:18:32.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-21T02:18:32.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:18:32.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-21T02:18:32.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:18:35.663 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-09-21T02:18:35.663 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-21T02:18:40.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T02:18:40.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:18:47.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-21T02:18:47.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:18:47.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-21T02:18:47.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:18:50.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-09-21T02:19:05.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-21T02:19:05.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:19:19.502 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-21T02:19:20.096 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-21T02:19:20.096 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-508.el9.x86_64"... 2024-09-21T02:19:20.097 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-21T02:19:20.136 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-508.el9.x86_64 2024-09-21T02:19:20.136 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T02:19:20.137 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T02:19:20.137 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T02:19:21.138 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-21T02:19:21.138 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T02:19:21.405 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-09-21T02:19:21.427 DEBUG:teuthology.parallel:result is None 2024-09-21T02:19:40.797 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T02:19:52.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T02:19:52.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:19:53.203 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-09-21T02:19:53.593 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T02:19:53.593 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T02:19:53.593 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-09-21T02:19:53.610 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-508.el9.x86_64 2024-09-21T02:19:53.610 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T02:19:53.611 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T02:19:53.611 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T02:19:54.612 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T02:19:54.612 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T02:19:54.712 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-09-21T02:19:54.734 DEBUG:teuthology.parallel:result is None 2024-09-21T02:20:05.747 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T02:20:23.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-21T02:20:23.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:20:24.047 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-09-21T02:20:24.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-21T02:20:24.410 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T02:20:24.411 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-09-21T02:20:24.427 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-508.el9.x86_64 2024-09-21T02:20:24.428 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T02:20:24.428 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T02:20:24.428 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T02:20:25.429 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T02:20:25.429 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T02:20:25.494 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2024-09-21T02:20:25.517 DEBUG:teuthology.parallel:result is None 2024-09-21T02:20:25.517 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T02:20:25.531 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T02:20:25.531 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T02:20:25.534 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T02:20:25.536 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T02:20:25.556 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T02:20:25.562 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T02:20:25.574 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T02:20:25.574 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T02:20:25.594 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T02:20:25.596 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T02:20:25.628 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T02:20:25.634 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T02:20:25.634 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T02:20:25.662 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T02:20:25.664 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T02:20:25.693 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T02:20:25.698 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T02:20:25.707 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T02:20:25.707 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T02:20:25.711 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T02:20:25.719 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T02:20:25.721 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T02:20:25.728 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T02:20:25.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T02:20:25.751 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T02:20:25.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T02:20:25.784 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T02:20:25.792 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T02:20:25.792 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T02:20:25.820 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T02:20:25.826 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T02:20:25.842 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T02:20:25.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T02:20:25.972 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T02:20:26.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T02:20:26.099 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:20:26.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T02:20:26.167 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T02:20:26.214 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T02:20:26.270 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T02:20:26.353 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T02:20:26.400 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:20:26.400 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T02:20:26.469 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T02:20:26.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T02:20:26.569 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T02:20:26.653 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T02:20:26.696 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:20:26.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T02:20:26.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-09-21T02:20:26.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2024-09-21T02:20:26.767 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-09-21T02:20:26.805 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T02:20:26.813 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T02:20:26.850 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T02:20:27.299 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T02:20:27.306 INFO:teuthology.task.internal:Starting timer... 2024-09-21T02:20:27.307 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T02:20:27.318 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T02:20:27.326 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T02:20:27.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-09-21T02:20:27.400 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-09-21T02:20:27.402 DEBUG:teuthology.orchestra.run.smithi066:> sudo service auditd rotate 2024-09-21T02:20:27.447 INFO:teuthology.orchestra.run.smithi066.stdout:Rotating logs: 2024-09-21T02:20:27.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-09-21T02:20:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-09-21T02:20:27.506 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T02:20:27.506 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-09-21T02:20:27.529 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-09-21T02:20:27.529 DEBUG:teuthology.orchestra.run.smithi066:> /usr/sbin/getenforce 2024-09-21T02:20:27.551 INFO:teuthology.orchestra.run.smithi066.stdout:Permissive 2024-09-21T02:20:27.551 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-09-21T02:20:27.570 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-09-21T02:20:27.570 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi066.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-09-21T02:20:27.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T02:20:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:20:27.601 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T02:20:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:20:27.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T02:20:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:20:27.658 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T02:20:27.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-09-21T02:20:27.695 DEBUG:teuthology.orchestra.run.smithi066:> sudo /usr/sbin/setenforce permissive 2024-09-21T02:20:27.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-09-21T02:20:27.770 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T02:20:27.785 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-21T02:20:27.787 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-21T02:20:27.892 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T02:20:27.902 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T02:20:27.903 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi066.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T02:24:19.912 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi066.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-09-21T02:24:19.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T02:24:19.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:24:20.005 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-09-21T02:24:20.087 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T02:24:20.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-21T02:24:20.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:24:20.162 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-09-21T02:24:20.239 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-21T02:24:20.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-21T02:24:20.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T02:24:20.314 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-21T02:24:20.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-21T02:24:20.400 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T02:24:20.412 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T02:24:20.413 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T02:24:20.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T02:24:20.415 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T02:24:20.416 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T02:24:20.418 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T02:24:20.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T02:24:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T02:24:20.448 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T02:24:20.463 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T02:24:20.464 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T02:24:20.471 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T02:24:20.487 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T02:24:20.491 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: ntpd: command not found 2024-09-21T02:24:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-09-21T02:24:20.502 INFO:teuthology.orchestra.run.smithi066.stdout:506 Cannot talk to daemon 2024-09-21T02:24:20.504 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-09-21T02:24:20.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-09-21T02:24:20.518 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T02:24:20.520 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T02:24:20.523 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-09-21T02:24:20.532 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T02:24:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T02:24:20.538 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T02:24:20.552 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T02:24:20.591 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-09-21T02:24:20.593 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-09-21T02:24:20.595 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T02:24:20.595 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-09-21T02:24:20.595 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.596 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.596 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.596 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.597 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T02:24:20.597 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-09-21T02:24:20.597 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.598 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.598 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.598 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.614 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-09-21T02:24:20.617 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T02:24:20.617 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-09-21T02:24:20.617 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.618 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.618 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.618 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:24:20.618 INFO:teuthology.run_tasks:Running task install... 2024-09-21T02:24:20.626 DEBUG:teuthology.task.install:project ceph 2024-09-21T02:24:20.626 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}} 2024-09-21T02:24:20.627 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T02:24:20.627 INFO:teuthology.task.install:Using flavor: default 2024-09-21T02:24:20.632 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T02:24:20.632 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-21T02:24:20.632 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T02:24:20.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:24:20.634 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T02:24:20.634 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:24:20.635 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T02:24:20.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:24:20.781 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T02:24:20.782 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T02:24:20.788 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T02:24:20.788 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T02:24:20.804 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T02:24:20.804 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T02:24:20.891 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T02:24:20.891 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:24:20.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T02:24:20.896 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T02:24:20.897 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:24:20.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T02:24:20.909 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T02:24:20.909 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:24:20.909 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T02:24:20.919 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T02:24:20.919 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T02:24:20.922 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T02:24:20.923 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T02:24:20.936 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T02:24:20.936 DEBUG:teuthology.orchestra.run.smithi066:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T02:24:20.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T02:24:20.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T02:24:21.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T02:24:21.063 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T02:24:21.064 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T02:24:21.078 DEBUG:teuthology.orchestra.run.smithi066:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T02:24:21.127 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-09-21T02:24:21.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-09-21T02:24:21.131 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-09-21T02:24:21.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-09-21T02:24:21.143 INFO:teuthology.orchestra.run.smithi066.stdout:check_obsoletes = 1 2024-09-21T02:24:21.144 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum clean all 2024-09-21T02:24:21.364 INFO:teuthology.orchestra.run.smithi190.stdout:45 files removed 2024-09-21T02:24:21.368 INFO:teuthology.orchestra.run.smithi029.stdout:45 files removed 2024-09-21T02:24:21.374 INFO:teuthology.orchestra.run.smithi066.stdout:45 files removed 2024-09-21T02:24:21.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T02:24:21.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T02:24:21.401 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T02:24:21.951 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 307 kB/s | 84 kB 00:00 2024-09-21T02:24:21.968 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 309 kB/s | 84 kB 00:00 2024-09-21T02:24:21.971 INFO:teuthology.orchestra.run.smithi066.stdout:ceph packages for x86_64 313 kB/s | 84 kB 00:00 2024-09-21T02:24:22.252 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-21T02:24:22.263 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-21T02:24:22.272 INFO:teuthology.orchestra.run.smithi066.stdout:ceph noarch packages 95 kB/s | 24 kB 00:00 2024-09-21T02:24:22.527 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-21T02:24:22.547 INFO:teuthology.orchestra.run.smithi066.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-21T02:24:22.548 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-21T02:24:24.149 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.3 MB 00:01 2024-09-21T02:24:24.323 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.3 MB 00:01 2024-09-21T02:24:24.369 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.3 MB 00:01 2024-09-21T02:24:26.119 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-21T02:24:26.181 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-21T02:24:30.473 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-21T02:24:31.806 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-21T02:24:32.058 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-21T02:24:33.199 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 105 kB/s | 19 kB 00:00 2024-09-21T02:24:33.230 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-21T02:24:34.095 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T02:24:38.855 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-21T02:24:39.575 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-09-21T02:24:41.209 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T02:24:41.209 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T02:24:41.221 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T02:24:41.222 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T02:24:41.223 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T02:24:41.224 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T02:24:41.225 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T02:24:41.228 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T02:24:41.229 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T02:24:41.295 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T02:24:41.323 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T02:24:41.324 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T02:24:41.325 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T02:24:41.326 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T02:24:41.327 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T02:24:41.328 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T02:24:41.329 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T02:24:41.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T02:24:41.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T02:24:41.332 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T02:24:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout:Install 149 Packages 2024-09-21T02:24:41.334 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 8 Packages 2024-09-21T02:24:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:24:41.335 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 228 M 2024-09-21T02:24:41.335 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-09-21T02:24:41.777 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T02:24:41.778 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T02:24:41.789 INFO:teuthology.orchestra.run.smithi190.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T02:24:41.789 INFO:teuthology.orchestra.run.smithi190.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T02:24:41.790 INFO:teuthology.orchestra.run.smithi190.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T02:24:41.791 INFO:teuthology.orchestra.run.smithi190.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T02:24:41.793 INFO:teuthology.orchestra.run.smithi190.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T02:24:41.795 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T02:24:41.796 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T02:24:41.858 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-21T02:24:41.879 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T02:24:41.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T02:24:41.881 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T02:24:41.882 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-21T02:24:41.883 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T02:24:41.884 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T02:24:41.885 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T02:24:41.886 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T02:24:41.887 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T02:24:41.888 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T02:24:41.889 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T02:24:41.890 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T02:24:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout:Install 149 Packages 2024-09-21T02:24:41.892 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 8 Packages 2024-09-21T02:24:41.893 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:24:41.893 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 228 M 2024-09-21T02:24:41.893 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-21T02:24:42.108 INFO:teuthology.orchestra.run.smithi029.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 52 kB/s | 6.6 kB 00:00 2024-09-21T02:24:42.301 INFO:teuthology.orchestra.run.smithi029.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 4.4 MB/s | 873 kB 00:00 2024-09-21T02:24:42.401 INFO:teuthology.orchestra.run.smithi029.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 1.4 MB/s | 148 kB 00:00 2024-09-21T02:24:42.535 INFO:teuthology.orchestra.run.smithi029.stdout:(4/157): ceph-base-19.3.0-5120.g18432218.el9.x8 10 MB/s | 5.5 MB 00:00 2024-09-21T02:24:42.627 INFO:teuthology.orchestra.run.smithi029.stdout:(5/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 10 MB/s | 2.3 MB 00:00 2024-09-21T02:24:42.635 INFO:teuthology.orchestra.run.smithi190.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 49 kB/s | 6.6 kB 00:00 2024-09-21T02:24:42.828 INFO:teuthology.orchestra.run.smithi029.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 5.2 MB/s | 1.5 MB 00:00 2024-09-21T02:24:43.036 INFO:teuthology.orchestra.run.smithi190.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 2.1 MB/s | 873 kB 00:00 2024-09-21T02:24:43.094 INFO:teuthology.orchestra.run.smithi190.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 2.5 MB/s | 148 kB 00:00 2024-09-21T02:24:43.164 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:17 2024-09-21T02:24:43.172 INFO:teuthology.orchestra.run.smithi029.stdout:(7/157): ceph-common-19.3.0-5120.g18432218.el9. 17 MB/s | 20 MB 00:01 2024-09-21T02:24:43.297 INFO:teuthology.orchestra.run.smithi029.stdout:(8/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 7.4 MB/s | 5.0 MB 00:00 2024-09-21T02:24:43.339 INFO:teuthology.orchestra.run.smithi029.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 605 kB/s | 25 kB 00:00 2024-09-21T02:24:43.603 INFO:teuthology.orchestra.run.smithi190.stdout:(4/157): ceph-base-19.3.0-5120.g18432218.el9.x8 5.0 MB/s | 5.5 MB 00:01 2024-09-21T02:24:43.645 INFO:teuthology.orchestra.run.smithi190.stdout:(5/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 4.2 MB/s | 2.3 MB 00:00 2024-09-21T02:24:43.715 INFO:teuthology.orchestra.run.smithi029.stdout:(10/157): ceph-radosgw-19.3.0-5120.g18432218.el 19 MB/s | 10 MB 00:00 2024-09-21T02:24:43.770 INFO:teuthology.orchestra.run.smithi190.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 9.1 MB/s | 1.5 MB 00:00 2024-09-21T02:24:43.791 INFO:teuthology.orchestra.run.smithi029.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 2.9 MB/s | 226 kB 00:00 2024-09-21T02:24:43.866 INFO:teuthology.orchestra.run.smithi029.stdout:(12/157): cephfs-shell-19.3.0-5120.g18432218.el 359 kB/s | 27 kB 00:00 2024-09-21T02:24:43.950 INFO:teuthology.orchestra.run.smithi029.stdout:(13/157): libcephfs-devel-19.3.0-5120.g18432218 381 kB/s | 32 kB 00:00 2024-09-21T02:24:44.075 INFO:teuthology.orchestra.run.smithi029.stdout:(14/157): libcephfs2-19.3.0-5120.g18432218.el9. 6.0 MB/s | 766 kB 00:00 2024-09-21T02:24:44.168 INFO:teuthology.orchestra.run.smithi029.stdout:(15/157): libcephsqlite-19.3.0-5120.g18432218.e 1.7 MB/s | 158 kB 00:00 2024-09-21T02:24:44.268 INFO:teuthology.orchestra.run.smithi029.stdout:(16/157): librados-devel-19.3.0-5120.g18432218. 1.2 MB/s | 127 kB 00:00 2024-09-21T02:24:44.386 INFO:teuthology.orchestra.run.smithi029.stdout:(17/157): libradosstriper1-19.3.0-5120.g1843221 4.2 MB/s | 507 kB 00:00 2024-09-21T02:24:44.545 INFO:teuthology.orchestra.run.smithi190.stdout:(7/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 5.5 MB/s | 5.0 MB 00:00 2024-09-21T02:24:44.869 INFO:teuthology.orchestra.run.smithi029.stdout:(18/157): librgw2-19.3.0-5120.g18432218.el9.x86 11 MB/s | 5.3 MB 00:00 2024-09-21T02:24:44.953 INFO:teuthology.orchestra.run.smithi029.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 539 kB/s | 45 kB 00:00 2024-09-21T02:24:45.021 INFO:teuthology.orchestra.run.smithi029.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 2.2 MB/s | 148 kB 00:00 2024-09-21T02:24:45.112 INFO:teuthology.orchestra.run.smithi190.stdout:(8/157): ceph-common-19.3.0-5120.g18432218.el9. 7.6 MB/s | 20 MB 00:02 2024-09-21T02:24:45.162 INFO:teuthology.orchestra.run.smithi190.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 507 kB/s | 25 kB 00:00 2024-09-21T02:24:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:(21/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 7.1 MB/s | 18 MB 00:02 2024-09-21T02:24:45.390 INFO:teuthology.orchestra.run.smithi029.stdout:(22/157): python3-cephfs-19.3.0-5120.g18432218. 437 kB/s | 161 kB 00:00 2024-09-21T02:24:45.474 INFO:teuthology.orchestra.run.smithi029.stdout:(23/157): python3-rbd-19.3.0-5120.g18432218.el9 3.6 MB/s | 307 kB 00:00 2024-09-21T02:24:45.508 INFO:teuthology.orchestra.run.smithi029.stdout:(24/157): python3-rados-19.3.0-5120.g18432218.e 2.1 MB/s | 321 kB 00:00 2024-09-21T02:24:45.567 INFO:teuthology.orchestra.run.smithi029.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.1 MB/s | 99 kB 00:00 2024-09-21T02:24:45.592 INFO:teuthology.orchestra.run.smithi029.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.0 MB/s | 86 kB 00:00 2024-09-21T02:24:45.660 INFO:teuthology.orchestra.run.smithi029.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 2.5 MB/s | 170 kB 00:00 2024-09-21T02:24:45.688 INFO:teuthology.orchestra.run.smithi190.stdout:(10/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 9.3 MB/s | 18 MB 00:01 2024-09-21T02:24:45.727 INFO:teuthology.orchestra.run.smithi029.stdout:(28/157): ceph-grafana-dashboards-19.3.0-5120.g 512 kB/s | 34 kB 00:00 2024-09-21T02:24:45.754 INFO:teuthology.orchestra.run.smithi190.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 3.3 MB/s | 226 kB 00:00 2024-09-21T02:24:45.813 INFO:teuthology.orchestra.run.smithi190.stdout:(12/157): cephfs-shell-19.3.0-5120.g18432218.el 464 kB/s | 27 kB 00:00 2024-09-21T02:24:45.872 INFO:teuthology.orchestra.run.smithi190.stdout:(13/157): libcephfs-devel-19.3.0-5120.g18432218 547 kB/s | 32 kB 00:00 2024-09-21T02:24:46.005 INFO:teuthology.orchestra.run.smithi190.stdout:(14/157): libcephfs2-19.3.0-5120.g18432218.el9. 5.6 MB/s | 766 kB 00:00 2024-09-21T02:24:46.114 INFO:teuthology.orchestra.run.smithi190.stdout:(15/157): ceph-radosgw-19.3.0-5120.g18432218.el 6.6 MB/s | 10 MB 00:01 2024-09-21T02:24:46.139 INFO:teuthology.orchestra.run.smithi190.stdout:(16/157): libcephsqlite-19.3.0-5120.g18432218.e 1.2 MB/s | 158 kB 00:00 2024-09-21T02:24:46.164 INFO:teuthology.orchestra.run.smithi190.stdout:(17/157): librados-devel-19.3.0-5120.g18432218. 2.5 MB/s | 127 kB 00:00 2024-09-21T02:24:46.190 INFO:teuthology.orchestra.run.smithi190.stdout:(18/157): libradosstriper1-19.3.0-5120.g1843221 9.9 MB/s | 507 kB 00:00 2024-09-21T02:24:46.240 INFO:teuthology.orchestra.run.smithi190.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 904 kB/s | 45 kB 00:00 2024-09-21T02:24:46.315 INFO:teuthology.orchestra.run.smithi190.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 1.9 MB/s | 148 kB 00:00 2024-09-21T02:24:46.378 INFO:teuthology.orchestra.run.smithi029.stdout:(29/157): ceph-test-19.3.0-5120.g18432218.el9.x 13 MB/s | 40 MB 00:03 2024-09-21T02:24:46.399 INFO:teuthology.orchestra.run.smithi190.stdout:(21/157): python3-cephfs-19.3.0-5120.g18432218. 1.9 MB/s | 161 kB 00:00 2024-09-21T02:24:46.403 INFO:teuthology.orchestra.run.smithi029.stdout:(30/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 229 kB/s | 154 kB 00:00 2024-09-21T02:24:46.479 INFO:teuthology.orchestra.run.smithi029.stdout:(31/157): rbd-mirror-19.3.0-5120.g18432218.el9. 3.4 MB/s | 3.1 MB 00:00 2024-09-21T02:24:46.563 INFO:teuthology.orchestra.run.smithi029.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 3.4 MB/s | 292 kB 00:00 2024-09-21T02:24:46.630 INFO:teuthology.orchestra.run.smithi029.stdout:(33/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 741 kB/s | 49 kB 00:00 2024-09-21T02:24:46.688 INFO:teuthology.orchestra.run.smithi029.stdout:(34/157): ceph-prometheus-alerts-19.3.0-5120.g1 281 kB/s | 16 kB 00:00 2024-09-21T02:24:46.765 INFO:teuthology.orchestra.run.smithi029.stdout:(35/157): ceph-volume-19.3.0-5120.g18432218.el9 3.7 MB/s | 285 kB 00:00 2024-09-21T02:24:46.783 INFO:teuthology.orchestra.run.smithi190.stdout:(22/157): ceph-test-19.3.0-5120.g18432218.el9.x 25 MB/s | 40 MB 00:01 2024-09-21T02:24:46.808 INFO:teuthology.orchestra.run.smithi190.stdout:(23/157): python3-rados-19.3.0-5120.g18432218.e 786 kB/s | 321 kB 00:00 2024-09-21T02:24:46.850 INFO:teuthology.orchestra.run.smithi190.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 4.5 MB/s | 307 kB 00:00 2024-09-21T02:24:46.875 INFO:teuthology.orchestra.run.smithi190.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.5 MB/s | 99 kB 00:00 2024-09-21T02:24:46.891 INFO:teuthology.orchestra.run.smithi029.stdout:(36/157): cephadm-19.3.0-5120.g18432218.el9.noa 6.1 MB/s | 784 kB 00:00 2024-09-21T02:24:46.950 INFO:teuthology.orchestra.run.smithi029.stdout:(37/157): cephfs-top-19.3.0-5120.g18432218.el9. 315 kB/s | 18 kB 00:00 2024-09-21T02:24:46.950 INFO:teuthology.orchestra.run.smithi190.stdout:(26/157): librgw2-19.3.0-5120.g18432218.el9.x86 6.7 MB/s | 5.3 MB 00:00 2024-09-21T02:24:46.975 INFO:teuthology.orchestra.run.smithi190.stdout:(27/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 690 kB/s | 86 kB 00:00 2024-09-21T02:24:47.001 INFO:teuthology.orchestra.run.smithi190.stdout:(28/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-21T02:24:47.026 INFO:teuthology.orchestra.run.smithi190.stdout:(29/157): ceph-grafana-dashboards-19.3.0-5120.g 684 kB/s | 34 kB 00:00 2024-09-21T02:24:47.051 INFO:teuthology.orchestra.run.smithi190.stdout:(30/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 3.0 MB/s | 154 kB 00:00 2024-09-21T02:24:47.127 INFO:teuthology.orchestra.run.smithi190.stdout:(31/157): rbd-mirror-19.3.0-5120.g18432218.el9. 12 MB/s | 3.1 MB 00:00 2024-09-21T02:24:47.186 INFO:teuthology.orchestra.run.smithi190.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 4.9 MB/s | 292 kB 00:00 2024-09-21T02:24:47.217 INFO:teuthology.orchestra.run.smithi029.stdout:(38/157): ceph-mgr-dashboard-19.3.0-5120.g18432 17 MB/s | 14 MB 00:00 2024-09-21T02:24:47.277 INFO:teuthology.orchestra.run.smithi190.stdout:(33/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 540 kB/s | 49 kB 00:00 2024-09-21T02:24:47.336 INFO:teuthology.orchestra.run.smithi190.stdout:(34/157): ceph-prometheus-alerts-19.3.0-5120.g1 282 kB/s | 16 kB 00:00 2024-09-21T02:24:47.368 INFO:teuthology.orchestra.run.smithi029.stdout:(39/157): ceph-mgr-diskprediction-local-19.3.0- 7.6 MB/s | 7.4 MB 00:00 2024-09-21T02:24:47.393 INFO:teuthology.orchestra.run.smithi029.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 180 kB/s | 80 kB 00:00 2024-09-21T02:24:47.486 INFO:teuthology.orchestra.run.smithi029.stdout:(41/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-09-21T02:24:47.486 INFO:teuthology.orchestra.run.smithi190.stdout:(35/157): ceph-mgr-dashboard-19.3.0-5120.g18432 31 MB/s | 14 MB 00:00 2024-09-21T02:24:47.511 INFO:teuthology.orchestra.run.smithi190.stdout:(36/157): ceph-volume-19.3.0-5120.g18432218.el9 1.6 MB/s | 285 kB 00:00 2024-09-21T02:24:47.519 INFO:teuthology.orchestra.run.smithi029.stdout:(42/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-21T02:24:47.545 INFO:teuthology.orchestra.run.smithi190.stdout:(37/157): cephadm-19.3.0-5120.g18432218.el9.noa 13 MB/s | 784 kB 00:00 2024-09-21T02:24:47.561 INFO:teuthology.orchestra.run.smithi029.stdout:(43/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-21T02:24:47.571 INFO:teuthology.orchestra.run.smithi190.stdout:(38/157): cephfs-top-19.3.0-5120.g18432218.el9. 314 kB/s | 18 kB 00:00 2024-09-21T02:24:47.596 INFO:teuthology.orchestra.run.smithi029.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-21T02:24:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 2.2 MB/s | 135 kB 00:00 2024-09-21T02:24:47.655 INFO:teuthology.orchestra.run.smithi029.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-21T02:24:47.689 INFO:teuthology.orchestra.run.smithi029.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-09-21T02:24:47.720 INFO:teuthology.orchestra.run.smithi190.stdout:(39/157): fuse-2.9.9-16.el9.x86_64.rpm 456 kB/s | 80 kB 00:00 2024-09-21T02:24:47.739 INFO:teuthology.orchestra.run.smithi029.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 4.8 MB/s | 1.2 MB 00:00 2024-09-21T02:24:47.804 INFO:teuthology.orchestra.run.smithi190.stdout:(40/157): ceph-mgr-diskprediction-local-19.3.0- 9.8 MB/s | 7.4 MB 00:00 2024-09-21T02:24:47.846 INFO:teuthology.orchestra.run.smithi190.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-21T02:24:47.890 INFO:teuthology.orchestra.run.smithi029.stdout:(49/157): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 689 kB 00:00 2024-09-21T02:24:47.905 INFO:teuthology.orchestra.run.smithi190.stdout:(42/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-21T02:24:47.915 INFO:teuthology.orchestra.run.smithi029.stdout:(50/157): boost-program-options-1.75.0-8.el9.x8 608 kB/s | 107 kB 00:00 2024-09-21T02:24:47.943 INFO:teuthology.orchestra.run.smithi029.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 414 kB/s | 22 kB 00:00 2024-09-21T02:24:47.963 INFO:teuthology.orchestra.run.smithi190.stdout:(43/157): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-21T02:24:47.993 INFO:teuthology.orchestra.run.smithi029.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 3.0 MB/s | 943 kB 00:00 2024-09-21T02:24:48.019 INFO:teuthology.orchestra.run.smithi029.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 432 kB/s | 33 kB 00:00 2024-09-21T02:24:48.030 INFO:teuthology.orchestra.run.smithi190.stdout:(44/157): python3-cryptography-36.0.1-4.el9.x86 6.7 MB/s | 1.2 MB 00:00 2024-09-21T02:24:48.052 INFO:teuthology.orchestra.run.smithi029.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-09-21T02:24:48.055 INFO:teuthology.orchestra.run.smithi190.stdout:(45/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-21T02:24:48.078 INFO:teuthology.orchestra.run.smithi029.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 309 kB/s | 18 kB 00:00 2024-09-21T02:24:48.081 INFO:teuthology.orchestra.run.smithi190.stdout:(46/157): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-09-21T02:24:48.104 INFO:teuthology.orchestra.run.smithi029.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 219 kB/s | 11 kB 00:00 2024-09-21T02:24:48.106 INFO:teuthology.orchestra.run.smithi190.stdout:(47/157): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-09-21T02:24:48.129 INFO:teuthology.orchestra.run.smithi029.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 3.3 MB/s | 171 kB 00:00 2024-09-21T02:24:48.140 INFO:teuthology.orchestra.run.smithi190.stdout:(48/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-21T02:24:48.163 INFO:teuthology.orchestra.run.smithi029.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 160 kB 00:00 2024-09-21T02:24:48.189 INFO:teuthology.orchestra.run.smithi029.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 765 kB/s | 45 kB 00:00 2024-09-21T02:24:48.257 INFO:teuthology.orchestra.run.smithi190.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 918 kB/s | 107 kB 00:00 2024-09-21T02:24:48.264 INFO:teuthology.orchestra.run.smithi029.stdout:(60/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-09-21T02:24:48.299 INFO:teuthology.orchestra.run.smithi190.stdout:(50/157): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 689 kB 00:00 2024-09-21T02:24:48.307 INFO:teuthology.orchestra.run.smithi029.stdout:(61/157): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-21T02:24:48.324 INFO:teuthology.orchestra.run.smithi190.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 324 kB/s | 22 kB 00:00 2024-09-21T02:24:48.332 INFO:teuthology.orchestra.run.smithi029.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 363 kB/s | 24 kB 00:00 2024-09-21T02:24:48.358 INFO:teuthology.orchestra.run.smithi190.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 943 kB 00:00 2024-09-21T02:24:48.374 INFO:teuthology.orchestra.run.smithi029.stdout:(63/157): libtool-2.4.6-46.el9.x86_64.rpm 3.1 MB/s | 581 kB 00:00 2024-09-21T02:24:48.383 INFO:teuthology.orchestra.run.smithi190.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 558 kB/s | 33 kB 00:00 2024-09-21T02:24:48.391 INFO:teuthology.orchestra.run.smithi029.stdout:(64/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 561 kB/s | 47 kB 00:00 2024-09-21T02:24:48.408 INFO:teuthology.orchestra.run.smithi190.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-09-21T02:24:48.425 INFO:teuthology.orchestra.run.smithi029.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-21T02:24:48.433 INFO:teuthology.orchestra.run.smithi190.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-09-21T02:24:48.444 INFO:teuthology.orchestra.run.smithi029.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 735 kB/s | 38 kB 00:00 2024-09-21T02:24:48.459 INFO:teuthology.orchestra.run.smithi190.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-21T02:24:48.478 INFO:teuthology.orchestra.run.smithi029.stdout:(67/157): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-09-21T02:24:48.484 INFO:teuthology.orchestra.run.smithi190.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 3.3 MB/s | 171 kB 00:00 2024-09-21T02:24:48.543 INFO:teuthology.orchestra.run.smithi190.stdout:(58/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-09-21T02:24:48.568 INFO:teuthology.orchestra.run.smithi190.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 542 kB/s | 45 kB 00:00 2024-09-21T02:24:48.590 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.5 MB 00:01 2024-09-21T02:24:48.593 INFO:teuthology.orchestra.run.smithi190.stdout:(60/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-21T02:24:48.612 INFO:teuthology.orchestra.run.smithi029.stdout:(68/157): openblas-openmp-0.3.26-2.el9.x86_64.r 27 MB/s | 4.9 MB 00:00 2024-09-21T02:24:48.635 INFO:teuthology.orchestra.run.smithi190.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 8.4 MB/s | 581 kB 00:00 2024-09-21T02:24:48.660 INFO:teuthology.orchestra.run.smithi190.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 366 kB/s | 24 kB 00:00 2024-09-21T02:24:48.671 INFO:teuthology.orchestra.run.smithi029.stdout:(69/157): postgresql-13.16-1.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-09-21T02:24:48.686 INFO:teuthology.orchestra.run.smithi190.stdout:(63/157): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-21T02:24:48.711 INFO:teuthology.orchestra.run.smithi190.stdout:(64/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-21T02:24:48.713 INFO:teuthology.orchestra.run.smithi029.stdout:(70/157): postgresql-contrib-13.16-1.el9.x86_64 8.8 MB/s | 910 kB 00:00 2024-09-21T02:24:48.736 INFO:teuthology.orchestra.run.smithi190.stdout:(65/157): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-09-21T02:24:48.738 INFO:teuthology.orchestra.run.smithi029.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-21T02:24:48.761 INFO:teuthology.orchestra.run.smithi190.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-21T02:24:48.787 INFO:teuthology.orchestra.run.smithi190.stdout:(67/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-21T02:24:48.831 INFO:teuthology.orchestra.run.smithi029.stdout:(72/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-21T02:24:48.871 INFO:teuthology.orchestra.run.smithi190.stdout:(68/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-21T02:24:48.873 INFO:teuthology.orchestra.run.smithi029.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 7.5 MB/s | 1.0 MB 00:00 2024-09-21T02:24:48.899 INFO:teuthology.orchestra.run.smithi029.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-21T02:24:48.938 INFO:teuthology.orchestra.run.smithi190.stdout:(69/157): openblas-openmp-0.3.26-2.el9.x86_64.r 25 MB/s | 4.9 MB 00:00 2024-09-21T02:24:48.949 INFO:teuthology.orchestra.run.smithi029.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-21T02:24:48.971 INFO:teuthology.orchestra.run.smithi190.stdout:(70/157): postgresql-contrib-13.16-1.el9.x86_64 8.9 MB/s | 910 kB 00:00 2024-09-21T02:24:48.988 INFO:teuthology.orchestra.run.smithi190.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 2.7 MB/s | 137 kB 00:00 2024-09-21T02:24:49.030 INFO:teuthology.orchestra.run.smithi190.stdout:(72/157): postgresql-13.16-1.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-09-21T02:24:49.063 INFO:teuthology.orchestra.run.smithi190.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-09-21T02:24:49.089 INFO:teuthology.orchestra.run.smithi190.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-09-21T02:24:49.108 INFO:teuthology.orchestra.run.smithi029.stdout:(76/157): postgresql-server-13.16-1.el9.x86_64. 15 MB/s | 5.8 MB 00:00 2024-09-21T02:24:49.131 INFO:teuthology.orchestra.run.smithi190.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-09-21T02:24:49.142 INFO:teuthology.orchestra.run.smithi029.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-21T02:24:49.176 INFO:teuthology.orchestra.run.smithi029.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-09-21T02:24:49.193 INFO:teuthology.orchestra.run.smithi029.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 506 kB/s | 26 kB 00:00 2024-09-21T02:24:49.220 INFO:teuthology.orchestra.run.smithi029.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 435 kB/s | 19 kB 00:00 2024-09-21T02:24:49.248 INFO:teuthology.orchestra.run.smithi190.stdout:(76/157): postgresql-server-13.16-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-09-21T02:24:49.254 INFO:teuthology.orchestra.run.smithi029.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 172 kB 00:00 2024-09-21T02:24:49.288 INFO:teuthology.orchestra.run.smithi029.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 516 kB/s | 35 kB 00:00 2024-09-21T02:24:49.406 INFO:teuthology.orchestra.run.smithi190.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 890 kB/s | 245 kB 00:00 2024-09-21T02:24:49.413 INFO:teuthology.orchestra.run.smithi029.stdout:(83/157): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-21T02:24:49.432 INFO:teuthology.orchestra.run.smithi190.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-09-21T02:24:49.447 INFO:teuthology.orchestra.run.smithi029.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-09-21T02:24:49.448 INFO:teuthology.orchestra.run.smithi190.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 610 kB/s | 26 kB 00:00 2024-09-21T02:24:49.465 INFO:teuthology.orchestra.run.smithi190.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 571 kB/s | 19 kB 00:00 2024-09-21T02:24:49.481 INFO:teuthology.orchestra.run.smithi029.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 222 kB 00:00 2024-09-21T02:24:49.491 INFO:teuthology.orchestra.run.smithi190.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-21T02:24:49.507 INFO:teuthology.orchestra.run.smithi029.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-09-21T02:24:49.508 INFO:teuthology.orchestra.run.smithi190.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 831 kB/s | 35 kB 00:00 2024-09-21T02:24:49.532 INFO:teuthology.orchestra.run.smithi029.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 832 kB/s | 42 kB 00:00 2024-09-21T02:24:49.566 INFO:teuthology.orchestra.run.smithi029.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-21T02:24:49.583 INFO:teuthology.orchestra.run.smithi190.stdout:(83/157): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-21T02:24:49.608 INFO:teuthology.orchestra.run.smithi190.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-09-21T02:24:49.609 INFO:teuthology.orchestra.run.smithi029.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 159 kB 00:00 2024-09-21T02:24:49.634 INFO:teuthology.orchestra.run.smithi190.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-21T02:24:49.642 INFO:teuthology.orchestra.run.smithi029.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-09-21T02:24:49.659 INFO:teuthology.orchestra.run.smithi190.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-21T02:24:49.668 INFO:teuthology.orchestra.run.smithi029.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 862 kB/s | 51 kB 00:00 2024-09-21T02:24:49.684 INFO:teuthology.orchestra.run.smithi190.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-09-21T02:24:49.693 INFO:teuthology.orchestra.run.smithi029.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-21T02:24:49.718 INFO:teuthology.orchestra.run.smithi190.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-21T02:24:49.743 INFO:teuthology.orchestra.run.smithi190.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-21T02:24:49.777 INFO:teuthology.orchestra.run.smithi190.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-09-21T02:24:49.802 INFO:teuthology.orchestra.run.smithi190.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-09-21T02:24:49.811 INFO:teuthology.orchestra.run.smithi029.stdout:(93/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.2 MB/s | 5.1 MB 00:00 2024-09-21T02:24:49.827 INFO:teuthology.orchestra.run.smithi190.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-21T02:24:49.836 INFO:teuthology.orchestra.run.smithi029.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 292 kB/s | 42 kB 00:00 2024-09-21T02:24:49.864 INFO:teuthology.orchestra.run.smithi029.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 846 kB/s | 44 kB 00:00 2024-09-21T02:24:49.897 INFO:teuthology.orchestra.run.smithi029.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 3.6 MB/s | 221 kB 00:00 2024-09-21T02:24:49.911 INFO:teuthology.orchestra.run.smithi190.stdout:(93/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-09-21T02:24:49.914 INFO:teuthology.orchestra.run.smithi029.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 710 kB/s | 36 kB 00:00 2024-09-21T02:24:49.936 INFO:teuthology.orchestra.run.smithi190.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-09-21T02:24:49.940 INFO:teuthology.orchestra.run.smithi029.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-21T02:24:49.961 INFO:teuthology.orchestra.run.smithi190.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 886 kB/s | 44 kB 00:00 2024-09-21T02:24:49.974 INFO:teuthology.orchestra.run.smithi029.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 5.0 MB/s | 304 kB 00:00 2024-09-21T02:24:49.987 INFO:teuthology.orchestra.run.smithi190.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-21T02:24:49.991 INFO:teuthology.orchestra.run.smithi029.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-21T02:24:50.012 INFO:teuthology.orchestra.run.smithi190.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-09-21T02:24:50.025 INFO:teuthology.orchestra.run.smithi029.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.8 MB/s | 96 kB 00:00 2024-09-21T02:24:50.037 INFO:teuthology.orchestra.run.smithi190.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-09-21T02:24:50.050 INFO:teuthology.orchestra.run.smithi029.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-21T02:24:50.071 INFO:teuthology.orchestra.run.smithi190.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-09-21T02:24:50.076 INFO:teuthology.orchestra.run.smithi029.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 884 kB/s | 45 kB 00:00 2024-09-21T02:24:50.096 INFO:teuthology.orchestra.run.smithi190.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 989 kB/s | 58 kB 00:00 2024-09-21T02:24:50.122 INFO:teuthology.orchestra.run.smithi190.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-21T02:24:50.147 INFO:teuthology.orchestra.run.smithi190.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T02:24:50.172 INFO:teuthology.orchestra.run.smithi190.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 897 kB/s | 45 kB 00:00 2024-09-21T02:24:50.185 INFO:teuthology.orchestra.run.smithi029.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-21T02:24:50.247 INFO:teuthology.orchestra.run.smithi190.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-09-21T02:24:50.249 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 19 kB 00:00 2024-09-21T02:24:50.306 INFO:teuthology.orchestra.run.smithi190.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-21T02:24:50.339 INFO:teuthology.orchestra.run.smithi190.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-21T02:24:50.356 INFO:teuthology.orchestra.run.smithi190.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-21T02:24:50.477 INFO:teuthology.orchestra.run.smithi029.stdout:(105/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-21T02:24:50.515 INFO:teuthology.orchestra.run.smithi190.stdout:(108/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-09-21T02:24:50.527 INFO:teuthology.orchestra.run.smithi029.stdout:(106/157): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-09-21T02:24:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:(109/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-09-21T02:24:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:(110/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-09-21T02:24:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:(111/157): liboath-2.6.7-2.el9.x86_64.rpm 714 kB/s | 49 kB 00:00 2024-09-21T02:24:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:(112/157): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-09-21T02:24:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:(113/157): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-21T02:24:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:(114/157): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-21T02:24:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:(115/157): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-09-21T02:24:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:(116/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(117/157): python3-cachetools-4.2.4-1.el9.noarc 567 kB/s | 32 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(118/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 326 kB/s | 43 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(119/157): python3-certifi-2023.05.07-4.el9.noa 187 kB/s | 14 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(120/157): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(121/157): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(122/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(123/157): python3-google-auth-2.34.0-1.el9.noa 2.6 MB/s | 220 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(124/157): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(125/157): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-09-21T02:24:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-09-21T02:24:51.153 INFO:teuthology.orchestra.run.smithi190.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-09-21T02:24:51.155 INFO:teuthology.orchestra.run.smithi029.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 850 kB/s | 19 kB 00:00 2024-09-21T02:24:51.155 INFO:teuthology.orchestra.run.smithi029.stdout:(108/157): gperftools-libs-2.9.1-3.el9.x86_64.r 2.6 MB/s | 308 kB 00:00 2024-09-21T02:24:51.155 INFO:teuthology.orchestra.run.smithi029.stdout:(109/157): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-09-21T02:24:51.155 INFO:teuthology.orchestra.run.smithi029.stdout:(110/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 343 kB/s | 26 kB 00:00 2024-09-21T02:24:51.155 INFO:teuthology.orchestra.run.smithi029.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 1.9 MB/s | 67 kB 00:00 2024-09-21T02:24:51.155 INFO:teuthology.orchestra.run.smithi029.stdout:(112/157): liboath-2.6.7-2.el9.x86_64.rpm 515 kB/s | 49 kB 00:00 2024-09-21T02:24:51.155 INFO:teuthology.orchestra.run.smithi029.stdout:(113/157): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-21T02:24:51.156 INFO:teuthology.orchestra.run.smithi029.stdout:(114/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-21T02:24:51.156 INFO:teuthology.orchestra.run.smithi029.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-09-21T02:24:51.156 INFO:teuthology.orchestra.run.smithi029.stdout:(116/157): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-09-21T02:24:51.156 INFO:teuthology.orchestra.run.smithi029.stdout:(117/157): python3-cachetools-4.2.4-1.el9.noarc 544 kB/s | 32 kB 00:00 2024-09-21T02:24:51.156 INFO:teuthology.orchestra.run.smithi029.stdout:(118/157): python3-certifi-2023.05.07-4.el9.noa 418 kB/s | 14 kB 00:00 2024-09-21T02:24:51.156 INFO:teuthology.orchestra.run.smithi029.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-21T02:24:51.157 INFO:teuthology.orchestra.run.smithi029.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-09-21T02:24:51.157 INFO:teuthology.orchestra.run.smithi029.stdout:(121/157): libarrow-9.0.0-11.el9.x86_64.rpm 7.6 MB/s | 4.4 MB 00:00 2024-09-21T02:24:51.157 INFO:teuthology.orchestra.run.smithi190.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-09-21T02:24:51.171 INFO:teuthology.orchestra.run.smithi029.stdout:(122/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-09-21T02:24:51.179 INFO:teuthology.orchestra.run.smithi190.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-09-21T02:24:51.205 INFO:teuthology.orchestra.run.smithi029.stdout:(123/157): python3-google-auth-2.34.0-1.el9.noa 1.3 MB/s | 220 kB 00:00 2024-09-21T02:24:51.213 INFO:teuthology.orchestra.run.smithi190.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-09-21T02:24:51.230 INFO:teuthology.orchestra.run.smithi029.stdout:(124/157): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-09-21T02:24:51.255 INFO:teuthology.orchestra.run.smithi190.stdout:(131/157): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-09-21T02:24:51.256 INFO:teuthology.orchestra.run.smithi029.stdout:(125/157): python3-jaraco-8.2.1-3.el9.noarch.rp 211 kB/s | 11 kB 00:00 2024-09-21T02:24:51.272 INFO:teuthology.orchestra.run.smithi190.stdout:(132/157): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-09-21T02:24:51.283 INFO:teuthology.orchestra.run.smithi029.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 338 kB/s | 18 kB 00:00 2024-09-21T02:24:51.286 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-21T02:24:51.297 INFO:teuthology.orchestra.run.smithi190.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-09-21T02:24:51.309 INFO:teuthology.orchestra.run.smithi029.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 444 kB/s | 23 kB 00:00 2024-09-21T02:24:51.322 INFO:teuthology.orchestra.run.smithi190.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 665 kB/s | 16 kB 00:00 2024-09-21T02:24:51.334 INFO:teuthology.orchestra.run.smithi029.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-09-21T02:24:51.348 INFO:teuthology.orchestra.run.smithi190.stdout:(135/157): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-21T02:24:51.359 INFO:teuthology.orchestra.run.smithi029.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 389 kB/s | 20 kB 00:00 2024-09-21T02:24:51.373 INFO:teuthology.orchestra.run.smithi190.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-21T02:24:51.385 INFO:teuthology.orchestra.run.smithi029.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T02:24:51.398 INFO:teuthology.orchestra.run.smithi190.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-09-21T02:24:51.419 INFO:teuthology.orchestra.run.smithi029.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-21T02:24:51.424 INFO:teuthology.orchestra.run.smithi190.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-09-21T02:24:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-09-21T02:24:51.474 INFO:teuthology.orchestra.run.smithi190.stdout:(139/157): python3-grpcio-1.46.7-10.el9.x86_64. 4.1 MB/s | 2.0 MB 00:00 2024-09-21T02:24:51.499 INFO:teuthology.orchestra.run.smithi190.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 786 kB/s | 59 kB 00:00 2024-09-21T02:24:51.520 INFO:teuthology.orchestra.run.smithi029.stdout:(133/157): python3-grpcio-1.46.7-10.el9.x86_64. 5.3 MB/s | 2.0 MB 00:00 2024-09-21T02:24:51.524 INFO:teuthology.orchestra.run.smithi190.stdout:(141/157): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-09-21T02:24:51.546 INFO:teuthology.orchestra.run.smithi029.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 655 kB/s | 16 kB 00:00 2024-09-21T02:24:51.550 INFO:teuthology.orchestra.run.smithi190.stdout:(142/157): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-09-21T02:24:51.575 INFO:teuthology.orchestra.run.smithi190.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 988 kB/s | 74 kB 00:00 2024-09-21T02:24:51.596 INFO:teuthology.orchestra.run.smithi029.stdout:(135/157): python3-kubernetes-26.1.0-2.el9.noar 3.9 MB/s | 1.0 MB 00:00 2024-09-21T02:24:51.600 INFO:teuthology.orchestra.run.smithi190.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-21T02:24:51.626 INFO:teuthology.orchestra.run.smithi190.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 804 kB/s | 20 kB 00:00 2024-09-21T02:24:51.638 INFO:teuthology.orchestra.run.smithi029.stdout:(136/157): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-09-21T02:24:51.651 INFO:teuthology.orchestra.run.smithi190.stdout:(146/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-21T02:24:51.664 INFO:teuthology.orchestra.run.smithi029.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 419 kB/s | 28 kB 00:00 2024-09-21T02:24:51.676 INFO:teuthology.orchestra.run.smithi190.stdout:(147/157): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-09-21T02:24:51.690 INFO:teuthology.orchestra.run.smithi029.stdout:(138/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 626 kB/s | 90 kB 00:00 2024-09-21T02:24:51.715 INFO:teuthology.orchestra.run.smithi029.stdout:(139/157): python3-repoze-lru-0.7-16.el9.noarch 402 kB/s | 31 kB 00:00 2024-09-21T02:24:51.718 INFO:teuthology.orchestra.run.smithi190.stdout:(148/157): re2-20211101-20.el9.x86_64.rpm 2.0 MB/s | 191 kB 00:00 2024-09-21T02:24:51.749 INFO:teuthology.orchestra.run.smithi029.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-21T02:24:51.769 INFO:teuthology.orchestra.run.smithi190.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-21T02:24:51.776 INFO:teuthology.orchestra.run.smithi029.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 592 kB/s | 36 kB 00:00 2024-09-21T02:24:51.810 INFO:teuthology.orchestra.run.smithi029.stdout:(142/157): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-09-21T02:24:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 867 kB/s | 74 kB 00:00 2024-09-21T02:24:51.869 INFO:teuthology.orchestra.run.smithi029.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 2.6 MB/s | 90 kB 00:00 2024-09-21T02:24:51.903 INFO:teuthology.orchestra.run.smithi029.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-09-21T02:24:51.929 INFO:teuthology.orchestra.run.smithi029.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 338 kB/s | 20 kB 00:00 2024-09-21T02:24:51.968 INFO:teuthology.orchestra.run.smithi029.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 2.9 MB/s | 191 kB 00:00 2024-09-21T02:24:51.977 INFO:teuthology.orchestra.run.smithi190.stdout:(150/157): librbd1-19.3.0-5120.g18432218.el9.x8 12 MB/s | 3.2 MB 00:00 2024-09-21T02:24:52.010 INFO:teuthology.orchestra.run.smithi029.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-09-21T02:24:52.036 INFO:teuthology.orchestra.run.smithi190.stdout:(151/157): librados2-19.3.0-5120.g18432218.el9. 9.4 MB/s | 3.4 MB 00:00 2024-09-21T02:24:52.069 INFO:teuthology.orchestra.run.smithi029.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-21T02:24:52.078 INFO:teuthology.orchestra.run.smithi190.stdout:(152/157): NetworkManager-1.51.0-1.el9.x86_64.r 7.5 MB/s | 2.3 MB 00:00 2024-09-21T02:24:52.136 INFO:teuthology.orchestra.run.smithi190.stdout:(153/157): NetworkManager-tui-1.51.0-1.el9.x86_ 4.2 MB/s | 249 kB 00:00 2024-09-21T02:24:52.162 INFO:teuthology.orchestra.run.smithi190.stdout:(154/157): NetworkManager-team-1.51.0-1.el9.x86 326 kB/s | 41 kB 00:00 2024-09-21T02:24:52.237 INFO:teuthology.orchestra.run.smithi190.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 14 MB/s | 1.4 MB 00:00 2024-09-21T02:24:52.279 INFO:teuthology.orchestra.run.smithi190.stdout:(156/157): NetworkManager-libnm-1.51.0-1.el9.x8 6.0 MB/s | 1.8 MB 00:00 2024-09-21T02:24:52.303 INFO:teuthology.orchestra.run.smithi029.stdout:(150/157): librados2-19.3.0-5120.g18432218.el9. 10 MB/s | 3.4 MB 00:00 2024-09-21T02:24:52.370 INFO:teuthology.orchestra.run.smithi029.stdout:(151/157): NetworkManager-1.51.0-1.el9.x86_64.r 7.9 MB/s | 2.3 MB 00:00 2024-09-21T02:24:52.387 INFO:teuthology.orchestra.run.smithi190.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-09-21T02:24:52.394 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-21T02:24:52.394 INFO:teuthology.orchestra.run.smithi190.stdout:Total 22 MB/s | 228 MB 00:10 2024-09-21T02:24:52.412 INFO:teuthology.orchestra.run.smithi029.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 983 kB/s | 41 kB 00:00 2024-09-21T02:24:52.505 INFO:teuthology.orchestra.run.smithi029.stdout:(153/157): librbd1-19.3.0-5120.g18432218.el9.x8 6.4 MB/s | 3.2 MB 00:00 2024-09-21T02:24:52.547 INFO:teuthology.orchestra.run.smithi029.stdout:(154/157): NetworkManager-tui-1.51.0-1.el9.x86_ 1.8 MB/s | 249 kB 00:00 2024-09-21T02:24:52.606 INFO:teuthology.orchestra.run.smithi029.stdout:(155/157): NetworkManager-libnm-1.51.0-1.el9.x8 6.0 MB/s | 1.8 MB 00:00 2024-09-21T02:24:52.715 INFO:teuthology.orchestra.run.smithi029.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-09-21T02:24:52.782 INFO:teuthology.orchestra.run.smithi029.stdout:(157/157): openssl-3.2.2-6.el9.x86_64.rpm 4.9 MB/s | 1.4 MB 00:00 2024-09-21T02:24:52.788 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-09-21T02:24:52.789 INFO:teuthology.orchestra.run.smithi029.stdout:Total 20 MB/s | 228 MB 00:11 2024-09-21T02:24:53.618 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-21T02:24:53.773 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-21T02:24:53.773 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-21T02:24:54.072 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-09-21T02:24:54.246 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-09-21T02:24:54.246 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-09-21T02:24:56.054 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-21T02:24:56.055 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-21T02:24:56.711 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-09-21T02:24:56.711 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-09-21T02:24:57.017 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-09-21T02:24:59.295 INFO:teuthology.orchestra.run.smithi066.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T02:24:59.296 INFO:teuthology.orchestra.run.smithi066.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T02:24:59.308 INFO:teuthology.orchestra.run.smithi066.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T02:24:59.309 INFO:teuthology.orchestra.run.smithi066.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T02:24:59.310 INFO:teuthology.orchestra.run.smithi066.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T02:24:59.311 INFO:teuthology.orchestra.run.smithi066.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T02:24:59.312 INFO:teuthology.orchestra.run.smithi066.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T02:24:59.315 INFO:teuthology.orchestra.run.smithi066.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T02:24:59.316 INFO:teuthology.orchestra.run.smithi066.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T02:24:59.383 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-21T02:24:59.403 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================= 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================= 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T02:24:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T02:24:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T02:24:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrading: 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T02:24:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T02:24:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T02:24:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T02:24:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T02:24:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T02:24:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T02:24:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T02:24:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T02:24:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout:Installing weak dependencies: 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================= 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout:Install 149 Packages 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrade 8 Packages 2024-09-21T02:24:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:24:59.417 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 228 M 2024-09-21T02:24:59.417 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-21T02:24:59.719 INFO:teuthology.orchestra.run.smithi066.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 52 kB/s | 6.6 kB 00:00 2024-09-21T02:24:59.978 INFO:teuthology.orchestra.run.smithi066.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 3.3 MB/s | 873 kB 00:00 2024-09-21T02:25:00.078 INFO:teuthology.orchestra.run.smithi066.stdout:(3/157): ceph-base-19.3.0-5120.g18432218.el9.x8 11 MB/s | 5.5 MB 00:00 2024-09-21T02:25:00.103 INFO:teuthology.orchestra.run.smithi066.stdout:(4/157): ceph-immutable-object-cache-19.3.0-512 1.2 MB/s | 148 kB 00:00 2024-09-21T02:25:00.179 INFO:teuthology.orchestra.run.smithi066.stdout:(5/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 23 MB/s | 2.3 MB 00:00 2024-09-21T02:25:00.354 INFO:teuthology.orchestra.run.smithi066.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 6.1 MB/s | 1.5 MB 00:00 2024-09-21T02:25:00.438 INFO:teuthology.orchestra.run.smithi066.stdout:(7/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 19 MB/s | 5.0 MB 00:00 2024-09-21T02:25:00.680 INFO:teuthology.orchestra.run.smithi066.stdout:(8/157): ceph-common-19.3.0-5120.g18432218.el9. 18 MB/s | 20 MB 00:01 2024-09-21T02:25:00.721 INFO:teuthology.orchestra.run.smithi066.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 610 kB/s | 25 kB 00:00 2024-09-21T02:25:01.047 INFO:teuthology.orchestra.run.smithi066.stdout:(10/157): ceph-radosgw-19.3.0-5120.g18432218.el 17 MB/s | 10 MB 00:00 2024-09-21T02:25:01.114 INFO:teuthology.orchestra.run.smithi066.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 3.3 MB/s | 226 kB 00:00 2024-09-21T02:25:01.197 INFO:teuthology.orchestra.run.smithi066.stdout:(12/157): cephfs-shell-19.3.0-5120.g18432218.el 324 kB/s | 27 kB 00:00 2024-09-21T02:25:01.272 INFO:teuthology.orchestra.run.smithi066.stdout:(13/157): libcephfs-devel-19.3.0-5120.g18432218 425 kB/s | 32 kB 00:00 2024-09-21T02:25:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:(14/157): libcephfs2-19.3.0-5120.g18432218.el9. 7.5 MB/s | 766 kB 00:00 2024-09-21T02:25:01.565 INFO:teuthology.orchestra.run.smithi066.stdout:(15/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 15 MB/s | 18 MB 00:01 2024-09-21T02:25:01.590 INFO:teuthology.orchestra.run.smithi066.stdout:(16/157): libcephsqlite-19.3.0-5120.g18432218.e 730 kB/s | 158 kB 00:00 2024-09-21T02:25:01.615 INFO:teuthology.orchestra.run.smithi066.stdout:(17/157): librados-devel-19.3.0-5120.g18432218. 2.5 MB/s | 127 kB 00:00 2024-09-21T02:25:01.649 INFO:teuthology.orchestra.run.smithi066.stdout:(18/157): libradosstriper1-19.3.0-5120.g1843221 8.5 MB/s | 507 kB 00:00 2024-09-21T02:25:01.716 INFO:teuthology.orchestra.run.smithi066.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 679 kB/s | 45 kB 00:00 2024-09-21T02:25:01.808 INFO:teuthology.orchestra.run.smithi066.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 1.6 MB/s | 148 kB 00:00 2024-09-21T02:25:01.891 INFO:teuthology.orchestra.run.smithi066.stdout:(21/157): librgw2-19.3.0-5120.g18432218.el9.x86 19 MB/s | 5.3 MB 00:00 2024-09-21T02:25:01.917 INFO:teuthology.orchestra.run.smithi066.stdout:(22/157): python3-cephfs-19.3.0-5120.g18432218. 1.5 MB/s | 161 kB 00:00 2024-09-21T02:25:01.942 INFO:teuthology.orchestra.run.smithi066.stdout:(23/157): python3-rados-19.3.0-5120.g18432218.e 6.2 MB/s | 321 kB 00:00 2024-09-21T02:25:01.976 INFO:teuthology.orchestra.run.smithi066.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 5.1 MB/s | 307 kB 00:00 2024-09-21T02:25:02.009 INFO:teuthology.orchestra.run.smithi066.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.5 MB/s | 99 kB 00:00 2024-09-21T02:25:02.034 INFO:teuthology.orchestra.run.smithi066.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.4 MB/s | 86 kB 00:00 2024-09-21T02:25:02.168 INFO:teuthology.orchestra.run.smithi066.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 1.2 MB/s | 170 kB 00:00 2024-09-21T02:25:02.235 INFO:teuthology.orchestra.run.smithi066.stdout:(28/157): rbd-mirror-19.3.0-5120.g18432218.el9. 14 MB/s | 3.1 MB 00:00 2024-09-21T02:25:02.261 INFO:teuthology.orchestra.run.smithi066.stdout:(29/157): ceph-grafana-dashboards-19.3.0-5120.g 372 kB/s | 34 kB 00:00 2024-09-21T02:25:02.286 INFO:teuthology.orchestra.run.smithi066.stdout:(30/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 3.0 MB/s | 154 kB 00:00 2024-09-21T02:25:02.386 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-21T02:25:02.530 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T02:25:02.620 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T02:25:02.792 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T02:25:02.858 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T02:25:02.861 INFO:teuthology.orchestra.run.smithi066.stdout:(31/157): ceph-test-19.3.0-5120.g18432218.el9.x 19 MB/s | 40 MB 00:02 2024-09-21T02:25:02.884 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T02:25:02.912 INFO:teuthology.orchestra.run.smithi066.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 5.7 MB/s | 292 kB 00:00 2024-09-21T02:25:02.948 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T02:25:03.012 INFO:teuthology.orchestra.run.smithi066.stdout:(33/157): ceph-mgr-diskprediction-local-19.3.0- 10 MB/s | 7.4 MB 00:00 2024-09-21T02:25:03.037 INFO:teuthology.orchestra.run.smithi066.stdout:(34/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 394 kB/s | 49 kB 00:00 2024-09-21T02:25:03.062 INFO:teuthology.orchestra.run.smithi066.stdout:(35/157): ceph-prometheus-alerts-19.3.0-5120.g1 330 kB/s | 16 kB 00:00 2024-09-21T02:25:03.088 INFO:teuthology.orchestra.run.smithi066.stdout:(36/157): ceph-volume-19.3.0-5120.g18432218.el9 5.6 MB/s | 285 kB 00:00 2024-09-21T02:25:03.121 INFO:teuthology.orchestra.run.smithi066.stdout:(37/157): cephadm-19.3.0-5120.g18432218.el9.noa 13 MB/s | 784 kB 00:00 2024-09-21T02:25:03.146 INFO:teuthology.orchestra.run.smithi066.stdout:(38/157): cephfs-top-19.3.0-5120.g18432218.el9. 315 kB/s | 18 kB 00:00 2024-09-21T02:25:03.301 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T02:25:03.305 INFO:teuthology.orchestra.run.smithi066.stdout:(39/157): ceph-mgr-dashboard-19.3.0-5120.g18432 14 MB/s | 14 MB 00:01 2024-09-21T02:25:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-09-21T02:25:03.397 INFO:teuthology.orchestra.run.smithi066.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 290 kB/s | 80 kB 00:00 2024-09-21T02:25:03.419 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:25:03.456 INFO:teuthology.orchestra.run.smithi066.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-21T02:25:03.481 INFO:teuthology.orchestra.run.smithi066.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-09-21T02:25:03.515 INFO:teuthology.orchestra.run.smithi066.stdout:(43/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.1 MB/s | 801 kB 00:00 2024-09-21T02:25:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T02:25:03.538 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:25:03.540 INFO:teuthology.orchestra.run.smithi066.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-21T02:25:03.565 INFO:teuthology.orchestra.run.smithi066.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-21T02:25:03.572 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:25:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T02:25:03.590 INFO:teuthology.orchestra.run.smithi066.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-21T02:25:03.615 INFO:teuthology.orchestra.run.smithi066.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-09-21T02:25:03.649 INFO:teuthology.orchestra.run.smithi066.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 6.3 MB/s | 1.2 MB 00:00 2024-09-21T02:25:03.651 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T02:25:03.662 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:25:03.702 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T02:25:03.732 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T02:25:03.790 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T02:25:03.858 INFO:teuthology.orchestra.run.smithi066.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 513 kB/s | 107 kB 00:00 2024-09-21T02:25:03.891 INFO:teuthology.orchestra.run.smithi066.stdout:(50/157): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-09-21T02:25:03.917 INFO:teuthology.orchestra.run.smithi066.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 370 kB/s | 22 kB 00:00 2024-09-21T02:25:03.918 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T02:25:03.951 INFO:teuthology.orchestra.run.smithi066.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-09-21T02:25:03.974 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T02:25:03.976 INFO:teuthology.orchestra.run.smithi066.stdout:(53/157): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 313 kB 00:00 2024-09-21T02:25:04.001 INFO:teuthology.orchestra.run.smithi066.stdout:(54/157): flexiblas-3.0.4-8.el9.x86_64.rpm 391 kB/s | 33 kB 00:00 2024-09-21T02:25:04.002 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:25:04.053 INFO:teuthology.orchestra.run.smithi066.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 242 kB/s | 18 kB 00:00 2024-09-21T02:25:04.077 INFO:teuthology.orchestra.run.smithi066.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-09-21T02:25:04.108 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:25:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T02:25:04.144 INFO:teuthology.orchestra.run.smithi066.stdout:(57/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-09-21T02:25:04.145 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T02:25:04.169 INFO:teuthology.orchestra.run.smithi066.stdout:(58/157): libnbd-1.20.2-2.el9.x86_64.rpm 1.4 MB/s | 171 kB 00:00 2024-09-21T02:25:04.192 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T02:25:04.198 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T02:25:04.219 INFO:teuthology.orchestra.run.smithi066.stdout:(59/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-21T02:25:04.221 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T02:25:04.243 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T02:25:04.253 INFO:teuthology.orchestra.run.smithi066.stdout:(60/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 416 kB/s | 45 kB 00:00 2024-09-21T02:25:04.286 INFO:teuthology.orchestra.run.smithi066.stdout:(61/157): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-21T02:25:04.312 INFO:teuthology.orchestra.run.smithi066.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 417 kB/s | 24 kB 00:00 2024-09-21T02:25:04.346 INFO:teuthology.orchestra.run.smithi066.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 807 kB/s | 47 kB 00:00 2024-09-21T02:25:04.371 INFO:teuthology.orchestra.run.smithi066.stdout:(64/157): libtool-2.4.6-46.el9.x86_64.rpm 3.8 MB/s | 581 kB 00:00 2024-09-21T02:25:04.421 INFO:teuthology.orchestra.run.smithi066.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-21T02:25:04.423 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T02:25:04.446 INFO:teuthology.orchestra.run.smithi066.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 511 kB/s | 38 kB 00:00 2024-09-21T02:25:04.471 INFO:teuthology.orchestra.run.smithi066.stdout:(67/157): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-21T02:25:04.594 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T02:25:04.597 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T02:25:04.630 INFO:teuthology.orchestra.run.smithi066.stdout:(68/157): openblas-openmp-0.3.26-2.el9.x86_64.r 24 MB/s | 4.9 MB 00:00 2024-09-21T02:25:04.672 INFO:teuthology.orchestra.run.smithi066.stdout:(69/157): postgresql-13.16-1.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-09-21T02:25:04.728 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T02:25:04.739 INFO:teuthology.orchestra.run.smithi066.stdout:(70/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-09-21T02:25:04.764 INFO:teuthology.orchestra.run.smithi066.stdout:(71/157): postgresql-contrib-13.16-1.el9.x86_64 6.6 MB/s | 910 kB 00:00 2024-09-21T02:25:04.789 INFO:teuthology.orchestra.run.smithi066.stdout:(72/157): postgresql-private-libs-13.16-1.el9.x 1.1 MB/s | 137 kB 00:00 2024-09-21T02:25:04.798 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:25:04.833 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:04.865 INFO:teuthology.orchestra.run.smithi066.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-09-21T02:25:04.890 INFO:teuthology.orchestra.run.smithi066.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 870 kB/s | 87 kB 00:00 2024-09-21T02:25:04.914 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:04.937 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:25:04.977 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:25:04.990 INFO:teuthology.orchestra.run.smithi066.stdout:(75/157): postgresql-server-13.16-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-09-21T02:25:05.016 INFO:teuthology.orchestra.run.smithi066.stdout:(76/157): python3-colorama-0.4.6-3.el9.noarch.r 489 kB/s | 61 kB 00:00 2024-09-21T02:25:05.069 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:25:05.074 INFO:teuthology.orchestra.run.smithi066.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-21T02:25:05.099 INFO:teuthology.orchestra.run.smithi066.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-09-21T02:25:05.108 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T02:25:05.142 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:05.155 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T02:25:05.183 INFO:teuthology.orchestra.run.smithi066.stdout:(79/157): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-09-21T02:25:05.208 INFO:teuthology.orchestra.run.smithi066.stdout:(80/157): python3-jsonpatch-1.21-16.el9.noarch. 191 kB/s | 26 kB 00:00 2024-09-21T02:25:05.234 INFO:teuthology.orchestra.run.smithi066.stdout:(81/157): python3-jsonpointer-2.0-4.el9.noarch. 143 kB/s | 19 kB 00:00 2024-09-21T02:25:05.255 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T02:25:05.259 INFO:teuthology.orchestra.run.smithi066.stdout:(82/157): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-21T02:25:05.293 INFO:teuthology.orchestra.run.smithi066.stdout:(83/157): python3-markupsafe-1.1.1-12.el9.x86_6 595 kB/s | 35 kB 00:00 2024-09-21T02:25:05.297 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T02:25:05.336 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T02:25:05.343 INFO:teuthology.orchestra.run.smithi066.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-09-21T02:25:05.358 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T02:25:05.368 INFO:teuthology.orchestra.run.smithi066.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-09-21T02:25:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T02:25:05.417 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T02:25:05.435 INFO:teuthology.orchestra.run.smithi066.stdout:(86/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-09-21T02:25:05.455 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:25:05.460 INFO:teuthology.orchestra.run.smithi066.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 457 kB/s | 42 kB 00:00 2024-09-21T02:25:05.461 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T02:25:05.486 INFO:teuthology.orchestra.run.smithi066.stdout:(88/157): python3-packaging-20.9-5.el9.noarch.r 541 kB/s | 77 kB 00:00 2024-09-21T02:25:05.494 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T02:25:05.528 INFO:teuthology.orchestra.run.smithi066.stdout:(89/157): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 269 kB 00:00 2024-09-21T02:25:05.537 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T02:25:05.553 INFO:teuthology.orchestra.run.smithi066.stdout:(90/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-09-21T02:25:05.563 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:25:05.570 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T02:25:05.578 INFO:teuthology.orchestra.run.smithi066.stdout:(91/157): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-09-21T02:25:05.597 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T02:25:05.599 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T02:25:05.603 INFO:teuthology.orchestra.run.smithi066.stdout:(92/157): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-21T02:25:05.629 INFO:teuthology.orchestra.run.smithi066.stdout:(93/157): python3-requests-oauthlib-1.3.0-12.el 711 kB/s | 54 kB 00:00 2024-09-21T02:25:05.662 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T02:25:05.666 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T02:25:05.679 INFO:teuthology.orchestra.run.smithi066.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-21T02:25:05.701 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T02:25:05.704 INFO:teuthology.orchestra.run.smithi066.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 585 kB/s | 44 kB 00:00 2024-09-21T02:25:05.726 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T02:25:05.728 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T02:25:05.755 INFO:teuthology.orchestra.run.smithi066.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-21T02:25:05.759 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T02:25:05.797 INFO:teuthology.orchestra.run.smithi066.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 390 kB/s | 36 kB 00:00 2024-09-21T02:25:05.803 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T02:25:05.838 INFO:teuthology.orchestra.run.smithi066.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 568 kB/s | 47 kB 00:00 2024-09-21T02:25:05.880 INFO:teuthology.orchestra.run.smithi066.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 3.5 MB/s | 304 kB 00:00 2024-09-21T02:25:05.883 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T02:25:05.921 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T02:25:05.922 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T02:25:05.965 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T02:25:05.993 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T02:25:06.056 INFO:teuthology.orchestra.run.smithi066.stdout:(100/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-21T02:25:06.063 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T02:25:06.081 INFO:teuthology.orchestra.run.smithi066.stdout:(101/157): uuid-1.6.2-55.el9.x86_64.rpm 238 kB/s | 58 kB 00:00 2024-09-21T02:25:06.106 INFO:teuthology.orchestra.run.smithi066.stdout:(102/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 424 kB/s | 96 kB 00:00 2024-09-21T02:25:06.108 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T02:25:06.131 INFO:teuthology.orchestra.run.smithi066.stdout:(103/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 845 kB/s | 64 kB 00:00 2024-09-21T02:25:06.160 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T02:25:06.165 INFO:teuthology.orchestra.run.smithi066.stdout:(104/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 534 kB/s | 45 kB 00:00 2024-09-21T02:25:06.182 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T02:25:06.215 INFO:teuthology.orchestra.run.smithi066.stdout:(105/157): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-09-21T02:25:06.282 INFO:teuthology.orchestra.run.smithi066.stdout:(106/157): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-09-21T02:25:06.307 INFO:teuthology.orchestra.run.smithi066.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-21T02:25:06.324 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T02:25:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T02:25:06.341 INFO:teuthology.orchestra.run.smithi066.stdout:(108/157): gperftools-libs-2.9.1-3.el9.x86_64.r 2.4 MB/s | 308 kB 00:00 2024-09-21T02:25:06.375 INFO:teuthology.orchestra.run.smithi066.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 779 kB/s | 26 kB 00:00 2024-09-21T02:25:06.386 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T02:25:06.409 INFO:teuthology.orchestra.run.smithi066.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-21T02:25:06.450 INFO:teuthology.orchestra.run.smithi066.stdout:(111/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.6 MB/s | 863 kB 00:00 2024-09-21T02:25:06.476 INFO:teuthology.orchestra.run.smithi066.stdout:(112/157): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-21T02:25:06.483 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:06.568 INFO:teuthology.orchestra.run.smithi066.stdout:(113/157): libarrow-9.0.0-11.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-09-21T02:25:06.593 INFO:teuthology.orchestra.run.smithi066.stdout:(114/157): luarocks-3.9.2-5.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-09-21T02:25:06.620 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:06.626 INFO:teuthology.orchestra.run.smithi066.stdout:(115/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.7 MB/s | 839 kB 00:00 2024-09-21T02:25:06.651 INFO:teuthology.orchestra.run.smithi066.stdout:(116/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 740 kB/s | 43 kB 00:00 2024-09-21T02:25:06.677 INFO:teuthology.orchestra.run.smithi066.stdout:(117/157): python3-cachetools-4.2.4-1.el9.noarc 636 kB/s | 32 kB 00:00 2024-09-21T02:25:06.702 INFO:teuthology.orchestra.run.smithi066.stdout:(118/157): python3-certifi-2023.05.07-4.el9.noa 279 kB/s | 14 kB 00:00 2024-09-21T02:25:06.736 INFO:teuthology.orchestra.run.smithi066.stdout:(119/157): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-09-21T02:25:06.762 INFO:teuthology.orchestra.run.smithi066.stdout:(120/157): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-09-21T02:25:06.786 INFO:teuthology.orchestra.run.smithi066.stdout:(121/157): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-09-21T02:25:06.812 INFO:teuthology.orchestra.run.smithi066.stdout:(122/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-09-21T02:25:06.837 INFO:teuthology.orchestra.run.smithi066.stdout:(123/157): python3-google-auth-2.34.0-1.el9.noa 2.8 MB/s | 220 kB 00:00 2024-09-21T02:25:06.863 INFO:teuthology.orchestra.run.smithi066.stdout:(124/157): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-21T02:25:06.880 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:06.888 INFO:teuthology.orchestra.run.smithi066.stdout:(125/157): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-09-21T02:25:06.905 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T02:25:06.913 INFO:teuthology.orchestra.run.smithi066.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-09-21T02:25:06.939 INFO:teuthology.orchestra.run.smithi066.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-09-21T02:25:06.964 INFO:teuthology.orchestra.run.smithi066.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-09-21T02:25:06.989 INFO:teuthology.orchestra.run.smithi066.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-21T02:25:06.989 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T02:25:07.014 INFO:teuthology.orchestra.run.smithi066.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T02:25:07.021 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T02:25:07.027 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T02:25:07.039 INFO:teuthology.orchestra.run.smithi066.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-21T02:25:07.059 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T02:25:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T02:25:07.073 INFO:teuthology.orchestra.run.smithi066.stdout:(132/157): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-09-21T02:25:07.098 INFO:teuthology.orchestra.run.smithi066.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 984 kB/s | 58 kB 00:00 2024-09-21T02:25:07.103 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T02:25:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T02:25:07.124 INFO:teuthology.orchestra.run.smithi066.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 655 kB/s | 16 kB 00:00 2024-09-21T02:25:07.149 INFO:teuthology.orchestra.run.smithi066.stdout:(135/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-21T02:25:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T02:25:07.174 INFO:teuthology.orchestra.run.smithi066.stdout:(136/157): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-21T02:25:07.197 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T02:25:07.257 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T02:25:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T02:25:07.308 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T02:25:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T02:25:07.366 INFO:teuthology.orchestra.run.smithi066.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 130 kB/s | 28 kB 00:00 2024-09-21T02:25:07.386 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T02:25:07.392 INFO:teuthology.orchestra.run.smithi066.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 142 kB/s | 31 kB 00:00 2024-09-21T02:25:07.400 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T02:25:07.417 INFO:teuthology.orchestra.run.smithi066.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T02:25:07.437 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T02:25:07.442 INFO:teuthology.orchestra.run.smithi066.stdout:(140/157): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-09-21T02:25:07.459 INFO:teuthology.orchestra.run.smithi066.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 856 kB/s | 36 kB 00:00 2024-09-21T02:25:07.476 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T02:25:07.501 INFO:teuthology.orchestra.run.smithi066.stdout:(142/157): python3-grpcio-1.46.7-10.el9.x86_64. 2.9 MB/s | 2.0 MB 00:00 2024-09-21T02:25:07.517 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T02:25:07.527 INFO:teuthology.orchestra.run.smithi066.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 889 kB/s | 74 kB 00:00 2024-09-21T02:25:07.533 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T02:25:07.552 INFO:teuthology.orchestra.run.smithi066.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-21T02:25:07.577 INFO:teuthology.orchestra.run.smithi066.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-09-21T02:25:07.588 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T02:25:07.604 INFO:teuthology.orchestra.run.smithi066.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 382 kB/s | 20 kB 00:00 2024-09-21T02:25:07.616 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T02:25:07.630 INFO:teuthology.orchestra.run.smithi066.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-21T02:25:07.634 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T02:25:07.663 INFO:teuthology.orchestra.run.smithi066.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-21T02:25:07.679 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T02:25:07.713 INFO:teuthology.orchestra.run.smithi066.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T02:25:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T02:25:07.798 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T02:25:07.807 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T02:25:07.852 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T02:25:07.888 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T02:25:07.908 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T02:25:07.939 INFO:teuthology.orchestra.run.smithi066.stdout:(150/157): NetworkManager-1.51.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-09-21T02:25:07.970 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T02:25:08.005 INFO:teuthology.orchestra.run.smithi066.stdout:(151/157): NetworkManager-libnm-1.51.0-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-09-21T02:25:08.030 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T02:25:08.032 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T02:25:08.048 INFO:teuthology.orchestra.run.smithi066.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 980 kB/s | 41 kB 00:00 2024-09-21T02:25:08.066 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T02:25:08.089 INFO:teuthology.orchestra.run.smithi066.stdout:(153/157): NetworkManager-tui-1.51.0-1.el9.x86_ 5.9 MB/s | 249 kB 00:00 2024-09-21T02:25:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T02:25:08.117 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T02:25:08.147 INFO:teuthology.orchestra.run.smithi066.stdout:(154/157): openssl-3.2.2-6.el9.x86_64.rpm 23 MB/s | 1.4 MB 00:00 2024-09-21T02:25:08.198 INFO:teuthology.orchestra.run.smithi066.stdout:(155/157): librados2-19.3.0-5120.g18432218.el9. 6.0 MB/s | 3.4 MB 00:00 2024-09-21T02:25:08.228 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T02:25:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T02:25:08.265 INFO:teuthology.orchestra.run.smithi066.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 21 MB/s | 2.4 MB 00:00 2024-09-21T02:25:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T02:25:08.517 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T02:25:08.561 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T02:25:08.602 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T02:25:08.713 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T02:25:08.762 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T02:25:08.802 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T02:25:08.835 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T02:25:08.860 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T02:25:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T02:25:08.957 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T02:25:08.980 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T02:25:08.998 INFO:teuthology.orchestra.run.smithi066.stdout:(157/157): librbd1-19.3.0-5120.g18432218.el9.x8 2.4 MB/s | 3.2 MB 00:01 2024-09-21T02:25:09.005 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-21T02:25:09.005 INFO:teuthology.orchestra.run.smithi066.stdout:Total 24 MB/s | 228 MB 00:09 2024-09-21T02:25:09.015 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T02:25:09.085 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T02:25:09.134 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T02:25:09.162 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T02:25:09.196 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T02:25:09.235 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T02:25:09.267 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T02:25:09.297 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T02:25:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T02:25:09.331 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T02:25:09.365 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T02:25:09.410 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T02:25:09.445 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T02:25:09.530 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T02:25:09.574 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T02:25:09.606 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T02:25:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T02:25:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T02:25:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T02:25:09.774 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T02:25:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T02:25:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T02:25:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T02:25:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T02:25:10.273 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T02:25:10.284 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-21T02:25:10.312 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T02:25:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T02:25:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T02:25:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T02:25:10.455 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-21T02:25:10.455 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-21T02:25:10.496 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T02:25:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T02:25:10.538 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T02:25:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T02:25:10.572 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T02:25:10.673 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T02:25:10.708 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T02:25:10.756 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T02:25:10.774 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T02:25:10.806 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T02:25:10.817 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T02:25:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T02:25:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T02:25:10.927 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T02:25:10.970 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T02:25:11.019 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T02:25:11.061 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T02:25:11.108 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T02:25:11.127 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T02:25:11.139 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T02:25:11.216 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T02:25:11.745 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T02:25:11.872 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T02:25:11.989 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T02:25:12.042 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T02:25:12.092 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T02:25:12.129 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T02:25:12.165 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T02:25:12.206 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T02:25:12.254 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T02:25:12.293 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T02:25:12.327 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T02:25:12.362 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T02:25:12.413 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T02:25:12.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T02:25:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T02:25:12.623 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T02:25:12.670 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T02:25:12.779 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T02:25:12.796 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-21T02:25:12.797 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-21T02:25:12.841 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T02:25:13.264 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T02:25:13.337 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T02:25:13.365 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T02:25:13.398 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T02:25:13.617 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T02:25:13.681 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T02:25:13.722 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T02:25:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T02:25:13.821 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T02:25:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T02:25:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T02:25:13.937 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T02:25:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T02:25:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T02:25:14.037 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T02:25:14.052 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T02:25:14.082 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T02:25:14.274 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T02:25:14.305 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T02:25:14.395 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T02:25:14.565 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T02:25:14.598 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T02:25:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T02:25:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T02:25:15.088 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T02:25:15.225 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T02:25:16.555 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T02:25:16.684 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T02:25:16.720 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T02:25:16.763 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T02:25:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:17.116 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T02:25:17.176 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T02:25:17.305 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T02:25:17.343 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T02:25:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T02:25:17.635 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:17.676 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T02:25:17.769 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T02:25:17.769 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T02:25:17.769 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T02:25:17.770 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:17.811 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T02:25:17.932 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T02:25:17.967 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T02:25:18.055 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T02:25:18.095 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T02:25:19.394 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-21T02:25:19.527 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T02:25:19.646 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T02:25:19.829 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T02:25:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T02:25:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T02:25:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T02:25:20.373 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T02:25:20.437 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:25:20.521 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:20.566 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:20.567 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:25:20.597 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:25:20.690 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:25:20.718 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T02:25:20.749 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T02:25:20.928 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T02:25:20.983 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T02:25:21.010 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:25:21.118 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:25:21.144 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T02:25:21.167 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:21.216 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T02:25:21.222 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T02:25:21.245 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T02:25:21.269 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T02:25:21.327 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T02:25:21.327 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T02:25:21.327 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T02:25:21.327 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:21.368 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T02:25:21.441 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T02:25:21.596 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T02:25:21.718 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T02:25:21.826 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:21.895 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:22.129 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T02:25:22.245 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T02:25:22.290 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T02:25:22.326 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T02:25:22.410 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T02:25:22.461 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T02:25:22.501 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T02:25:22.544 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T02:25:22.577 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T02:25:22.604 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T02:25:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T02:25:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T02:25:22.774 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T02:25:22.818 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T02:25:22.901 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T02:25:22.944 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T02:25:22.989 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T02:25:23.017 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T02:25:23.092 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T02:25:23.140 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T02:25:23.218 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T02:25:23.370 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T02:25:23.435 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T02:25:23.995 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T02:25:24.082 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T02:25:24.125 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T02:25:24.168 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T02:25:24.206 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T02:25:24.250 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T02:25:24.410 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T02:25:24.468 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T02:25:24.563 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T02:25:24.643 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T02:25:24.723 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T02:25:24.761 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T02:25:24.797 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T02:25:24.845 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T02:25:24.880 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T02:25:24.923 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T02:25:24.966 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T02:25:25.109 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T02:25:25.317 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T02:25:25.359 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T02:25:25.466 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T02:25:25.500 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T02:25:25.548 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T02:25:25.581 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T02:25:25.606 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T02:25:25.697 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T02:25:25.825 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T02:25:25.867 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T02:25:25.901 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T02:25:25.938 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T02:25:25.977 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T02:25:26.009 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T02:25:26.038 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T02:25:26.073 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T02:25:26.114 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T02:25:26.156 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T02:25:26.190 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T02:25:26.280 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T02:25:26.330 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T02:25:26.368 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T02:25:26.476 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T02:25:26.528 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T02:25:27.306 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T02:25:27.358 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T02:25:27.392 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T02:25:27.502 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T02:25:27.547 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T02:25:27.601 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T02:25:27.690 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T02:25:27.727 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T02:25:27.918 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T02:25:27.949 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T02:25:28.030 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T02:25:28.723 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T02:25:28.834 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T02:25:30.175 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T02:25:30.175 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-09-21T02:25:30.175 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-09-21T02:25:30.176 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-09-21T02:25:30.176 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-09-21T02:25:30.176 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-09-21T02:25:30.176 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-09-21T02:25:30.176 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-09-21T02:25:30.176 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:30.180 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T02:25:30.359 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T02:25:30.375 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T02:25:30.375 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:30.375 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T02:25:30.375 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:30.378 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T02:25:30.415 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T02:25:30.456 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T02:25:30.809 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T02:25:30.902 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T02:25:30.994 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T02:25:31.023 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T02:25:31.060 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T02:25:31.586 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T02:25:31.621 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T02:25:31.675 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T02:25:31.714 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T02:25:31.970 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T02:25:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T02:25:31.997 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:31.997 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T02:25:31.997 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T02:25:31.997 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T02:25:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:32.221 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T02:25:32.244 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T02:25:32.244 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:32.244 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T02:25:32.244 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T02:25:32.244 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T02:25:32.245 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:32.713 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T02:25:32.737 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T02:25:32.737 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:32.737 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T02:25:32.737 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T02:25:32.737 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T02:25:32.737 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:32.852 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T02:25:32.909 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T02:25:32.964 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T02:25:33.158 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T02:25:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T02:25:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T02:25:33.372 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T02:25:33.520 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T02:25:33.564 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T02:25:33.597 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T02:25:33.624 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T02:25:34.034 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T02:25:34.053 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:34.064 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T02:25:34.085 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:34.097 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T02:25:34.162 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T02:25:34.187 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T02:25:34.696 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T02:25:34.739 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T02:25:34.833 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T02:25:34.833 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T02:25:34.833 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T02:25:34.833 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:34.873 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T02:25:37.412 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T02:25:37.477 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-09-21T02:25:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:37.988 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T02:25:38.005 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T02:25:38.005 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:38.005 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T02:25:38.005 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:38.449 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T02:25:38.485 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T02:25:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T02:25:38.762 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T02:25:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T02:25:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T02:25:38.812 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:38.812 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T02:25:38.812 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T02:25:38.812 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T02:25:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T02:25:38.883 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T02:25:38.999 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T02:25:39.663 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T02:25:39.691 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T02:25:39.692 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:39.692 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T02:25:39.692 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T02:25:39.692 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T02:25:39.692 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:39.931 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T02:25:39.956 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T02:25:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T02:25:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T02:25:39.956 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T02:25:39.957 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:40.025 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T02:25:40.057 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T02:25:40.081 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T02:25:40.081 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:40.081 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T02:25:40.081 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T02:25:40.081 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T02:25:40.081 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:40.125 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T02:25:40.148 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T02:25:40.148 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:40.148 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T02:25:40.148 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:40.200 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T02:25:40.223 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T02:25:40.223 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:40.223 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T02:25:40.224 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T02:25:40.224 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T02:25:40.224 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:40.449 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T02:25:40.474 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T02:25:40.474 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:40.474 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T02:25:40.474 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T02:25:40.474 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T02:25:40.475 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:40.551 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T02:25:40.575 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T02:25:40.575 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:40.575 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T02:25:40.575 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T02:25:40.575 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T02:25:40.575 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T02:25:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T02:25:40.720 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T02:25:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T02:25:40.990 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T02:25:41.070 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T02:25:41.177 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T02:25:41.339 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T02:25:41.386 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T02:25:41.419 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T02:25:41.455 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T02:25:41.937 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T02:25:41.975 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T02:25:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T02:25:42.093 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T02:25:42.125 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T02:25:44.598 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T02:25:44.647 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T02:25:44.687 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T02:25:44.719 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T02:25:44.747 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T02:25:44.954 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T02:25:44.994 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T02:25:45.024 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T02:25:45.053 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T02:25:45.100 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T02:25:45.495 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T02:25:45.564 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T02:25:45.696 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:25:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:25:45.759 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:25:46.580 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T02:25:46.586 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T02:25:46.615 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T02:25:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T02:25:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T02:25:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T02:25:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T02:25:46.839 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T02:25:46.865 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T02:25:46.891 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T02:25:46.891 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T02:25:46.917 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T02:25:46.952 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T02:25:46.975 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T02:25:46.975 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T02:25:46.979 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T02:25:46.979 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:46.979 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T02:25:46.979 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T02:25:46.979 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T02:25:46.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:47.030 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T02:25:47.042 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T02:25:47.067 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T02:25:47.092 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T02:25:47.097 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:25:47.098 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:25:47.154 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /sys 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /proc 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /mnt 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /var/tmp 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /home 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /root 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /tmp 2024-09-21T02:25:47.185 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:47.209 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T02:25:47.277 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:25:47.277 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T02:25:47.353 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T02:25:47.385 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T02:25:47.402 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T02:25:47.402 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:47.402 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T02:25:47.402 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:47.420 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:25:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T02:25:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T02:25:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T02:25:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T02:25:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T02:25:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T02:25:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:48.337 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T02:25:48.361 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T02:25:48.361 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:48.361 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T02:25:48.361 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:48.412 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T02:25:48.438 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T02:25:48.438 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:48.438 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T02:25:48.438 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T02:25:48.438 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T02:25:48.438 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:48.782 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T02:25:48.808 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T02:25:48.808 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:48.808 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T02:25:48.808 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T02:25:48.808 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T02:25:48.808 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:25:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T02:25:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T02:25:49.073 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:49.073 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T02:25:49.073 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T02:25:49.073 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T02:25:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:49.320 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T02:25:49.345 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T02:25:49.345 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:49.345 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T02:25:49.345 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T02:25:49.345 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T02:25:49.346 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:49.846 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T02:25:49.871 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T02:25:49.871 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:49.871 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T02:25:49.871 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T02:25:49.871 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T02:25:49.871 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:49.985 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T02:25:50.045 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T02:25:50.109 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T02:25:50.303 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T02:25:50.358 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T02:25:50.438 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T02:25:50.527 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T02:25:50.679 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T02:25:50.724 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T02:25:50.756 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T02:25:50.783 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T02:25:51.254 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T02:25:51.282 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T02:25:51.315 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T02:25:51.383 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T02:25:51.413 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T02:25:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T02:25:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T02:25:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T02:25:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T02:25:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T02:25:53.456 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T02:25:53.513 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T02:25:53.545 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T02:25:53.576 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T02:25:53.639 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T02:25:53.649 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:25:53.649 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T02:25:53.649 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T02:25:53.649 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T02:25:53.649 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T02:25:53.649 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T02:25:53.649 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T02:25:53.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T02:25:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T02:25:53.662 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T02:25:53.662 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T02:25:53.662 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T02:25:53.663 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T02:25:53.664 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T02:25:53.665 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T02:25:53.666 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T02:25:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T02:25:53.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T02:25:53.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T02:25:53.670 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T02:25:53.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T02:25:53.672 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T02:25:54.279 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:25:54.330 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:25:54.352 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:25:54.788 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T02:25:54.845 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T02:25:55.267 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T02:25:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T02:25:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T02:25:55.533 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T02:25:55.573 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T02:25:55.607 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T02:25:55.642 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T02:25:55.642 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T02:25:55.735 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T02:25:55.735 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T02:25:55.821 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T02:25:55.855 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T02:25:55.858 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T02:25:55.861 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:25:55.861 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:25:55.886 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T02:25:56.007 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T02:25:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:25:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T02:25:56.151 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T02:25:56.186 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T02:25:56.222 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T02:25:56.228 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:25:56.247 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T02:25:56.248 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:56.248 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T02:25:56.248 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T02:25:56.248 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T02:25:56.248 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:56.297 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T02:25:56.309 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.322 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.323 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.324 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T02:25:56.325 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T02:25:56.326 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T02:25:56.327 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T02:25:56.328 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T02:25:56.329 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T02:25:56.330 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T02:25:56.331 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T02:25:56.331 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T02:25:56.331 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T02:25:56.331 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T02:25:56.331 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T02:25:56.331 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T02:25:56.331 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T02:25:56.331 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T02:25:56.332 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T02:25:56.333 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T02:25:56.334 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T02:25:56.334 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T02:25:56.334 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:25:56.334 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-21T02:25:56.412 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T02:25:56.473 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T02:25:56.719 DEBUG:teuthology.parallel:result is None 2024-09-21T02:25:57.491 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T02:25:57.525 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T02:25:57.550 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T02:25:57.550 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:57.550 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T02:25:57.550 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T02:25:57.550 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T02:25:57.550 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:57.592 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T02:25:57.618 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T02:25:57.618 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:57.618 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T02:25:57.618 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:57.668 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T02:25:57.693 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T02:25:57.693 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:57.693 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T02:25:57.693 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T02:25:57.693 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T02:25:57.693 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:25:58.035 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T02:25:58.060 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T02:25:58.060 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T02:25:58.061 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T02:25:58.061 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T02:25:58.061 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T02:25:58.061 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:26:02.315 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T02:26:02.356 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T02:26:02.397 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T02:26:02.428 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T02:26:02.456 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T02:26:02.676 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T02:26:02.727 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T02:26:02.757 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T02:26:02.787 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T02:26:02.833 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T02:26:03.527 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:26:03.572 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:26:03.594 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T02:26:04.501 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T02:26:04.602 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T02:26:04.651 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T02:26:04.758 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T02:26:04.788 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T02:26:04.813 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T02:26:04.847 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T02:26:04.848 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T02:26:04.940 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T02:26:04.940 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T02:26:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T02:26:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T02:26:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:26:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:26:05.242 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T02:26:05.243 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T02:26:05.335 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T02:26:05.411 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:26:07.763 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T02:26:07.764 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T02:26:07.765 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T02:26:07.765 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T02:26:07.765 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T02:26:07.777 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T02:26:07.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T02:26:07.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T02:26:07.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T02:26:07.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T02:26:07.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T02:26:07.783 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T02:26:07.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T02:26:07.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T02:26:07.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T02:26:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:26:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:26:10.446 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T02:26:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T02:26:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T02:26:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T02:26:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T02:26:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T02:26:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T02:26:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T02:26:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T02:26:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T02:26:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T02:26:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T02:26:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T02:26:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T02:26:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T02:26:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T02:26:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:26:10.458 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-09-21T02:26:11.031 DEBUG:teuthology.parallel:result is None 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T02:26:11.644 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T02:26:11.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T02:26:11.646 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T02:26:11.646 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T02:26:11.646 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T02:26:11.646 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T02:26:11.657 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T02:26:11.657 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T02:26:11.657 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T02:26:11.657 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T02:26:11.657 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T02:26:11.657 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T02:26:11.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T02:26:11.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T02:26:11.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T02:26:11.661 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T02:26:11.662 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T02:26:11.663 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T02:26:11.664 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T02:26:11.665 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T02:26:11.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T02:26:14.360 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T02:26:14.360 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:26:14.360 INFO:teuthology.orchestra.run.smithi066.stdout:Upgraded: 2024-09-21T02:26:14.360 INFO:teuthology.orchestra.run.smithi066.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T02:26:14.360 INFO:teuthology.orchestra.run.smithi066.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T02:26:14.360 INFO:teuthology.orchestra.run.smithi066.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T02:26:14.360 INFO:teuthology.orchestra.run.smithi066.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.361 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.362 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T02:26:14.363 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T02:26:14.364 INFO:teuthology.orchestra.run.smithi066.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T02:26:14.365 INFO:teuthology.orchestra.run.smithi066.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T02:26:14.366 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T02:26:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T02:26:14.368 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T02:26:14.369 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T02:26:14.370 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.371 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:26:14.372 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-21T02:26:14.823 DEBUG:teuthology.parallel:result is None 2024-09-21T02:26:14.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:26:14.973 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T02:26:15.001 INFO:teuthology.orchestra.run.smithi029.stdout:19.3.0-5120.g18432218.el9 2024-09-21T02:26:15.001 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T02:26:15.001 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T02:26:15.003 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:26:15.151 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T02:26:15.175 INFO:teuthology.orchestra.run.smithi066.stdout:19.3.0-5120.g18432218.el9 2024-09-21T02:26:15.175 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T02:26:15.175 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T02:26:15.177 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:26:15.316 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T02:26:15.343 INFO:teuthology.orchestra.run.smithi190.stdout:19.3.0-5120.g18432218.el9 2024-09-21T02:26:15.343 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T02:26:15.343 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T02:26:15.345 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T02:26:15.345 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:26:15.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T02:26:15.379 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:26:15.379 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T02:26:15.412 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:26:15.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T02:26:15.444 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T02:26:15.444 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:26:15.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T02:26:15.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T02:26:15.545 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:26:15.545 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T02:26:15.575 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T02:26:15.644 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:26:15.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T02:26:15.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T02:26:15.743 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T02:26:15.743 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:26:15.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T02:26:15.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T02:26:15.844 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:26:15.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T02:26:15.874 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T02:26:15.943 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:26:15.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T02:26:15.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T02:26:16.033 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T02:26:16.033 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:26:16.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T02:26:16.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T02:26:16.134 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:26:16.134 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T02:26:16.165 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T02:26:16.232 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:26:16.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T02:26:16.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T02:26:16.322 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T02:26:16.437 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'mds': {'mds_export_ephemeral_random_max': 0.1}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'subvols': {'create': 2}} 2024-09-21T02:26:16.437 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:26:16.437 INFO:tasks.cephadm:Cluster fsid is e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:26:16.437 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T02:26:16.438 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.29', 'mon.b': '172.21.15.66', 'mon.c': '172.21.15.190'} 2024-09-21T02:26:16.438 INFO:tasks.cephadm:First mon is mon.a on smithi029 2024-09-21T02:26:16.438 INFO:tasks.cephadm:First mgr is x 2024-09-21T02:26:16.438 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T02:26:16.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-09-21T02:26:16.470 DEBUG:teuthology.orchestra.run.smithi066:> sudo hostname $(hostname -s) 2024-09-21T02:26:16.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-09-21T02:26:16.526 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T02:26:16.526 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:26:16.672 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.4.75+confusa13', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 00:04:17.484514', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.2.22+braggi22', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 22:54:16.499090', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-21T02:26:16.814 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20240920.212106-debug, sha1 1843221895adccfb3b692060251dd6a0183bc410 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:26:16.815 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T02:26:16.815 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T02:26:16.816 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:17.352 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 02:26 /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:17.352 DEBUG:teuthology.orchestra.run.smithi066:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:17.876 INFO:teuthology.orchestra.run.smithi066.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 02:26 /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:17.876 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:18.399 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 02:26 /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:18.400 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:18.420 DEBUG:teuthology.orchestra.run.smithi066:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:18.440 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T02:26:18.473 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 on all hosts... 2024-09-21T02:26:18.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T02:26:18.475 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T02:26:18.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T02:26:20.001 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T02:26:20.042 INFO:teuthology.orchestra.run.smithi066.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T02:26:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T02:27:03.603 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:27:03.603 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T02:27:03.603 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T02:27:03.603 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-09-21T02:27:03.603 INFO:teuthology.orchestra.run.smithi029.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T02:27:03.603 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T02:27:03.603 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:27:20.579 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-21T02:27:20.580 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T02:27:20.580 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T02:27:20.580 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2024-09-21T02:27:20.580 INFO:teuthology.orchestra.run.smithi190.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T02:27:20.580 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-21T02:27:20.580 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-21T02:27:21.067 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-09-21T02:27:21.067 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T02:27:21.068 INFO:teuthology.orchestra.run.smithi066.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T02:27:21.068 INFO:teuthology.orchestra.run.smithi066.stdout: "repo_digests": [ 2024-09-21T02:27:21.068 INFO:teuthology.orchestra.run.smithi066.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T02:27:21.068 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-09-21T02:27:21.068 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-09-21T02:27:21.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-09-21T02:27:21.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph 2024-09-21T02:27:21.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-09-21T02:27:21.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-09-21T02:27:21.215 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 777 /etc/ceph 2024-09-21T02:27:21.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-09-21T02:27:21.264 INFO:tasks.cephadm:Writing seed config... 2024-09-21T02:27:21.265 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-09-21T02:27:21.265 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-21T02:27:21.265 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T02:27:21.265 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T02:27:21.265 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T02:27:21.265 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-21T02:27:21.266 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T02:27:21.267 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-09-21T02:27:21.267 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:27:21.267 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T02:27:21.331 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e1049baa-77c0-11ef-baf5-efdc52797490 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [mds] mds_export_ephemeral_random_max = 0.1 2024-09-21T02:27:21.331 DEBUG:teuthology.orchestra.run.smithi029:mon.a> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.a.service 2024-09-21T02:27:21.334 DEBUG:teuthology.orchestra.run.smithi029:mgr.x> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x.service 2024-09-21T02:27:21.377 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T02:27:21.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 -v bootstrap --fsid e1049baa-77c0-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:27:21.593 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-09-21T02:27:21.594 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', '-v', 'bootstrap', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-09-21T02:27:21.594 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T02:27:21.594 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-09-21T02:27:21.723 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 5.2.2 2024-09-21T02:27:21.723 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-09-21T02:27:21.723 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-09-21T02:27:21.731 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T02:27:21.731 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T02:27:21.738 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T02:27:21.738 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-09-21T02:27:21.745 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-09-21T02:27:21.752 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-09-21T02:27:21.752 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-09-21T02:27:21.752 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-09-21T02:27:21.777 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 5.2.2 2024-09-21T02:27:21.777 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T02:27:21.777 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-09-21T02:27:21.777 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-09-21T02:27:21.784 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T02:27:21.784 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T02:27:21.791 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T02:27:21.791 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-09-21T02:27:21.797 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-09-21T02:27:21.804 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-09-21T02:27:21.804 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-09-21T02:27:21.805 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-09-21T02:27:21.805 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:21.805 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 140517673425264 on /run/cephadm/e1049baa-77c0-11ef-baf5-efdc52797490.lock 2024-09-21T02:27:21.805 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 140517673425264 acquired on /run/cephadm/e1049baa-77c0-11ef-baf5-efdc52797490.lock 2024-09-21T02:27:21.805 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-09-21T02:27:21.805 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-09-21T02:27:21.806 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-09-21T02:27:21.809 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-09-21T02:27:21.809 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-09-21T02:27:21.811 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T02:27:21.811 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-21T02:27:21.812 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1747sec hoplimit 64 pref medium 2024-09-21T02:27:21.814 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T02:27:21.814 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T02:27:21.814 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T02:27:21.814 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-21T02:27:21.814 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-09-21T02:27:21.814 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T02:27:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-09-21T02:27:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-09-21T02:27:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T02:27:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T02:27:21.816 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T02:27:25.927 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T02:27:25.927 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T02:27:25.927 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T02:27:25.927 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:8425bc7408d91b180e7fdc62f23e7ae6603757f4f36e9969a4eaf9fd9416b764 2024-09-21T02:27:25.927 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T02:27:25.927 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T02:27:25.927 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T02:27:39.499 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T02:27:39.499 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T02:27:39.499 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T02:27:40.518 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-09-21T02:27:40.519 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-09-21T02:27:41.474 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQAcL+5mN7QoMxAArQQw4pfrnK5C4bk8C9d1Gw== 2024-09-21T02:27:42.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQAdL+5mWW6MMRAA9Z6VEJzVQ5u22xxlUdccKg== 2024-09-21T02:27:43.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQAeL+5mQFxjMRAAzM2uyfNq+IcWuBzq7QNh6w== 2024-09-21T02:27:43.384 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-09-21T02:27:45.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T02:27:45.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T02:27:45.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:45.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T02:27:45.502 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T02:27:45.502 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-09-21T02:27:45.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:45.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T02:27:45.502 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:27:45.502 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-09-21T02:27:46.440 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.a on 2024-09-21T02:27:46.672 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T02:27:46.821 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T02:27:47.015 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e1049baa-77c0-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-e1049baa-77c0-11ef-baf5-efdc52797490.target. 2024-09-21T02:27:47.015 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e1049baa-77c0-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-e1049baa-77c0-11ef-baf5-efdc52797490.target. 2024-09-21T02:27:47.245 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.a 2024-09-21T02:27:47.245 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-21T02:27:47.418 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e1049baa-77c0-11ef-baf5-efdc52797490.target.wants/ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-e1049baa-77c0-11ef-baf5-efdc52797490@.service. 2024-09-21T02:27:47.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:47 smithi029 podman[30616]: 2024-09-21 02:27:47.876188203 +0000 UTC m=+0.332102192 container start 4c3c13e563f13b15a3604d9c1dd3d9b9b4c7a76614ad2e740eceeb2d8dc501dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T02:27:47.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:47 smithi029 ceph-mon[30635]: mkfs e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:47.980 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T02:27:47.980 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-09-21T02:27:47.987 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T02:27:47.987 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-09-21T02:27:47.987 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-09-21T02:27:47.988 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T02:27:47.988 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-09-21T02:27:47.988 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-09-21T02:27:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:47 smithi029 ceph-mon[30635]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T02:27:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:47 smithi029 bash[30616]: 4c3c13e563f13b15a3604d9c1dd3d9b9b4c7a76614ad2e740eceeb2d8dc501dd 2024-09-21T02:27:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:47 smithi029 systemd[1]: Started Ceph mon.a for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: monmap epoch 1 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: last_changed 2024-09-21T02:27:43.754319+0000 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: created 2024-09-21T02:27:43.754319+0000 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: min_mon_release 19 (squid) 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: election_strategy: 1 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: fsmap 2024-09-21T02:27:49.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T02:27:49.109 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: mgrmap e1: no daemons active 2024-09-21T02:27:49.109 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:48 smithi029 ceph-mon[30635]: from='client.? 172.21.15.29:0/1990787038' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-21T02:27:49.110 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T02:27:49.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:49.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T02:27:49.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:27:49.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-09-21T02:27:49.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.594872s) 2024-09-21T02:27:49.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T02:27:49.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T02:27:49.112 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:27:49.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-09-21T02:27:49.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T02:27:49.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T02:27:49.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T02:27:49.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T02:27:49.113 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:27:49.113 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-09-21T02:27:49.113 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T02:27:50.205 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:49 smithi029 ceph-mon[30635]: from='client.? 172.21.15.29:0/420841217' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T02:27:50.205 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:49 smithi029 ceph-mon[30635]: from='client.? 172.21.15.29:0/420841217' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T02:27:50.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:27:50.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T02:27:50.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-21T02:27:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-21T02:27:50.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T02:27:50.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T02:27:50.211 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-09-21T02:27:51.312 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:50 smithi029 ceph-mon[30635]: from='client.? 172.21.15.29:0/297955381' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:27:51.315 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-09-21T02:27:51.617 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:51 smithi029 systemd[1]: Stopping Ceph mon.a for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:27:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:51 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a[30631]: 2024-09-21T02:27:51.615+0000 7f8b3ba00640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:27:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:51 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a[30631]: 2024-09-21T02:27:51.615+0000 7f8b3ba00640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T02:27:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:51 smithi029 podman[30879]: 2024-09-21 02:27:51.708115097 +0000 UTC m=+0.217160662 container died 4c3c13e563f13b15a3604d9c1dd3d9b9b4c7a76614ad2e740eceeb2d8dc501dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T02:27:52.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 podman[30879]: 2024-09-21 02:27:52.059110573 +0000 UTC m=+0.568156127 container cleanup 4c3c13e563f13b15a3604d9c1dd3d9b9b4c7a76614ad2e740eceeb2d8dc501dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:27:52.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 bash[30879]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a 2024-09-21T02:27:52.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 podman[30891]: 2024-09-21 02:27:52.320758683 +0000 UTC m=+0.611527196 container remove 4c3c13e563f13b15a3604d9c1dd3d9b9b4c7a76614ad2e740eceeb2d8dc501dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T02:27:52.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T02:27:52.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 systemd[1]: Stopped Ceph mon.a for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:27:52.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 systemd[1]: Starting Ceph mon.a for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:27:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 podman[30987]: 2024-09-21 02:27:52.561261023 +0000 UTC m=+0.016193218 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:27:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 podman[30987]: 2024-09-21 02:27:52.664012112 +0000 UTC m=+0.118944297 container create d52d85c4a57f56fdc11a6559214e6a2e6d2b58e7892a5a010198d42608a0d6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T02:27:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 podman[30987]: 2024-09-21 02:27:52.859924665 +0000 UTC m=+0.314856860 container init d52d85c4a57f56fdc11a6559214e6a2e6d2b58e7892a5a010198d42608a0d6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 podman[30987]: 2024-09-21 02:27:52.863303945 +0000 UTC m=+0.318236141 container start d52d85c4a57f56fdc11a6559214e6a2e6d2b58e7892a5a010198d42608a0d6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: pidfile_write: ignore empty --pid-file 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: load: jerasure load: lrc 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: RocksDB version: 7.9.2 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Git sha 0 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: DB SUMMARY 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: DB Session ID: FH3GN55MUBZ8VLK9EASX 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: CURRENT file: CURRENT 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T02:27:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T02:27:52.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94312 ; 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.error_if_exists: 0 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.create_if_missing: 0 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.paranoid_checks: 1 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.env: 0x562634db2d60 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.info_log: 0x56263a25d360 2024-09-21T02:27:52.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.statistics: (nil) 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.use_fsync: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_log_file_size: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.allow_fallocate: 1 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.use_direct_reads: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T02:27:52.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.db_log_dir: 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.wal_dir: 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.write_buffer_manager: 0x56263a261d60 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T02:27:52.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.unordered_write: 0 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.row_cache: None 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.wal_filter: None 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.two_write_queues: 0 2024-09-21T02:27:52.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.wal_compression: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.atomic_flush: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.log_readahead_size: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T02:27:52.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_background_jobs: 2 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_background_compactions: -1 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_subcompactions: 1 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_open_files: -1 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T02:27:52.971 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_background_flushes: -1 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Compression algorithms supported: 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: kZSTD supported: 0 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: kXpressCompression supported: 0 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: kBZip2Compression supported: 0 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: kLZ4Compression supported: 1 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: kZlibCompression supported: 1 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: kSnappyCompression supported: 1 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T02:27:52.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.merge_operator: 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_filter: None 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56263a25cfa0) 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_type: 0 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_index_type: 0 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_shortening: 1 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T02:27:52.973 INFO:journalctl@ceph.mon.a.smithi029.stdout: checksum: 4 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: no_block_cache: 0 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache: 0x562636221610 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_options: 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: capacity : 536870912 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_shard_bits : 4 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: strict_capacity_limit : 0 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_compressed: (nil) 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: persistent_cache: (nil) 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size: 4096 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size_deviation: 10 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_restart_interval: 16 2024-09-21T02:27:52.974 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_block_restart_interval: 1 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: metadata_block_size: 4096 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: partition_filters: 0 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: use_delta_encoding: 1 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: filter_policy: bloomfilter 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: whole_key_filtering: 1 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: verify_compression: 0 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: format_version: 5 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_index_compression: 1 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_align: 0 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: max_auto_readahead_size: 262144 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: prepopulate_block_cache: 0 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: initial_auto_readahead_size: 8192 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T02:27:52.975 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression: NoCompression 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.num_levels: 7 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T02:27:52.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T02:27:52.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T02:27:52.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T02:27:52.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.inplace_update_support: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.bloom_locality: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.max_successive_merges: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.ttl: 2592000 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T02:27:52.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.enable_blob_files: false 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.min_blob_size: 0 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bbc2ce9b-38e2-43d0-977f-fff8e4fd2b5e 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726885672902096, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T02:27:52.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726885672903273, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90616, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 281, "table_properties": {"data_size": 88657, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11470, "raw_average_key_size": 47, "raw_value_size": 82129, "raw_average_value_size": 342, "num_data_blocks": 12, "num_entries": 240, "num_filter_entries": 240, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726885672, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bbc2ce9b-38e2-43d0-977f-fff8e4fd2b5e", "db_session_id": "FH3GN55MUBZ8VLK9EASX", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726885672903379, "job": 1, "event": "recovery_finished"} 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56263a280a80 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: DB pointer 0x56263b026000 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** DB Stats ** 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T02:27:52.982 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: L0 2/0 90.35 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: Sum 2/0 90.35 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 120.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-21T02:27:52.983 INFO:journalctl@ceph.mon.a.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative compaction: 0.00 GB write, 24.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval compaction: 0.00 GB write, 24.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache BinnedLRUCache@0x562636221610#2 capacity: 512.00 MB usage: 1.31 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-21T02:27:52.984 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a@-1(???) e1 preinit fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a@-1(???).mds e1 new map 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a@-1(???).mds e1 print_map 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout: e1 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout: btime 2024-09-21T02:27:47:924894+0000 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout: legacy client fscid: -1 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout: No filesystems configured 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T02:27:52.985 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: monmap epoch 1 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: last_changed 2024-09-21T02:27:43.754319+0000 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: created 2024-09-21T02:27:43.754319+0000 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: min_mon_release 19 (squid) 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: election_strategy: 1 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: fsmap 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T02:27:52.986 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 ceph-mon[31006]: mgrmap e1: no daemons active 2024-09-21T02:27:52.990 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T02:27:53.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 bash[30987]: d52d85c4a57f56fdc11a6559214e6a2e6d2b58e7892a5a010198d42608a0d6b1 2024-09-21T02:27:53.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:52 smithi029 systemd[1]: Started Ceph mon.a for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:27:54.046 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T02:27:54.076 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:27:54.076 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-09-21T02:27:54.076 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T02:27:54.077 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T02:27:54.344 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x 2024-09-21T02:27:54.344 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x.service: Unit ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x.service not loaded. 2024-09-21T02:27:54.497 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e1049baa-77c0-11ef-baf5-efdc52797490.target.wants/ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x.service → /etc/systemd/system/ceph-e1049baa-77c0-11ef-baf5-efdc52797490@.service. 2024-09-21T02:27:54.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:54 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/882622904' entity='client.admin' 2024-09-21T02:27:54.731 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:27:54 smithi029 systemd[1]: Starting Ceph mgr.x for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:27:55.054 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:27:54 smithi029 podman[31221]: 2024-09-21 02:27:54.633070544 +0000 UTC m=+0.016423416 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:27:55.054 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:27:54 smithi029 podman[31221]: 2024-09-21 02:27:54.739015185 +0000 UTC m=+0.122368056 container create d8db2ab5cded05173c066343f091af730c1d9486c1a551d46029653b4377abe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T02:27:55.054 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:27:54 smithi029 podman[31221]: 2024-09-21 02:27:54.96961443 +0000 UTC m=+0.352967301 container init d8db2ab5cded05173c066343f091af730c1d9486c1a551d46029653b4377abe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:27:55.054 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:27:54 smithi029 podman[31221]: 2024-09-21 02:27:54.972506127 +0000 UTC m=+0.355859001 container start d8db2ab5cded05173c066343f091af730c1d9486c1a551d46029653b4377abe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T02:27:55.054 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:27:55 smithi029 bash[31221]: d8db2ab5cded05173c066343f091af730c1d9486c1a551d46029653b4377abe4 2024-09-21T02:27:55.071 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T02:27:55.071 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-09-21T02:27:55.079 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T02:27:55.079 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-09-21T02:27:55.080 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-09-21T02:27:55.080 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T02:27:55.091 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T02:27:55.091 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-09-21T02:27:55.100 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T02:27:55.100 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-09-21T02:27:55.100 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-09-21T02:27:55.100 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T02:27:55.100 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-09-21T02:27:55.100 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-09-21T02:27:55.344 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:27:55 smithi029 systemd[1]: Started Ceph mgr.x for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:27:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:55 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2864228829' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T02:27:56.317 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:27:56.317 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-09-21T02:27:56.317 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:27:56.318 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:56.319 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T02:27:47:924894+0000", 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T02:27:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T02:27:47.925706+0000", 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T02:27:56.321 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:56.322 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T02:27:56.322 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-09-21T02:27:56.322 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-09-21T02:27:59.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:27:58 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3743721714' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T02:27:59.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:27:59.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-09-21T02:27:59.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:27:59.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T02:27:59.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T02:27:59.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:27:59.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T02:27:59.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T02:27:59.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:59.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T02:27:59.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T02:27:47:924894+0000", 2024-09-21T02:27:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:59.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T02:27:59.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:27:59.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T02:27:47.925706+0000", 2024-09-21T02:27:59.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T02:27:59.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:27:59.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T02:27:59.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-09-21T02:27:59.507 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (2/15)... 2024-09-21T02:28:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: Activating manager daemon x 2024-09-21T02:28:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: mgrmap e2: x(active, starting, since 0.00376062s) 2024-09-21T02:28:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T02:28:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T02:28:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T02:28:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:28:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T02:28:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: Manager daemon x is now available 2024-09-21T02:28:00.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T02:28:00.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T02:28:00.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' 2024-09-21T02:28:00.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' 2024-09-21T02:28:00.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:00 smithi029 ceph-mon[31006]: from='mgr.14100 172.21.15.29:0/1933534005' entity='mgr.x' 2024-09-21T02:28:02.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:02 smithi029 ceph-mon[31006]: mgrmap e3: x(active, since 1.00909s) 2024-09-21T02:28:02.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:02 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2922782947' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T02:28:02.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:28:02.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T02:28:02.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T02:28:02.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T02:28:02.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T02:28:02.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T02:28:02.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T02:28:02.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:28:02.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T02:28:02.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T02:28:02.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T02:27:47:924894+0000", 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T02:28:02.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T02:27:47.925706+0000", 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-09-21T02:28:02.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T02:28:02.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-09-21T02:28:02.890 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-09-21T02:28:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:03 smithi029 ceph-mon[31006]: mgrmap e4: x(active, since 2s) 2024-09-21T02:28:04.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:28:04.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T02:28:04.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:28:04.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T02:28:04.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-09-21T02:28:04.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T02:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-09-21T02:28:04.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:04 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/544773003' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T02:28:05.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:05 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2738782883' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-21T02:28:05.632 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:05 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: ignoring --setuser ceph since I am not root 2024-09-21T02:28:05.632 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:05 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: ignoring --setgroup ceph since I am not root 2024-09-21T02:28:06.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:06 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2738782883' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T02:28:06.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:06 smithi029 ceph-mon[31006]: mgrmap e5: x(active, since 5s) 2024-09-21T02:28:07.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-09-21T02:28:07.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T02:28:07.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T02:28:07.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-21T02:28:07.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T02:28:07.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-09-21T02:28:07.376 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-09-21T02:28:07.376 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-09-21T02:28:07.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:07 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2660244420' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: Active manager daemon x restarted 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: Activating manager daemon x 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: mgrmap e6: x(active, starting, since 0.0035734s) 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T02:28:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:10 smithi029 ceph-mon[31006]: Manager daemon x is now available 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: Found migration_current of "None". Setting to last migration. 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T02:28:12.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T02:28:12.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-mon[31006]: mgrmap e7: x(active, since 1.00631s) 2024-09-21T02:28:12.265 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-09-21T02:28:12.265 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T02:28:12.266 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T02:28:12.266 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-09-21T02:28:12.266 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-09-21T02:28:12.266 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying orchestrator module is enabled... 2024-09-21T02:28:12.541 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:28:12.271+0000 7f393800f640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:02:28:12] ENGINE Error in HTTPServer.serve 2024-09-21T02:28:12.541 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: Traceback (most recent call last): 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: self._connections.run(self.expiration_interval) 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: self._run(expiration_interval) 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: s = self.context.wrap_socket( 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: return self.sslsocket_class._create( 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T02:28:12.542 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: self.do_handshake() 2024-09-21T02:28:12.543 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T02:28:12.543 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: self._sslobj.do_handshake() 2024-09-21T02:28:12.543 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T02:28:12.543 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:28:13.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T02:28:13.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T02:28:13.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:12] ENGINE Bus STARTING 2024-09-21T02:28:13.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:12] ENGINE Serving on https://172.21.15.29:7150 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:12] ENGINE Error in HTTPServer.serve 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: Traceback (most recent call last): 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._run(expiration_interval) 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: s = self.context.wrap_socket( 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T02:28:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: return self.sslsocket_class._create( 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout: self.do_handshake() 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._sslobj.do_handshake() 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:12] ENGINE Serving on http://172.21.15.29:8765 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:12] ENGINE Bus STARTED 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:28:13.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:13 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1176446280' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-21T02:28:13.758 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-21T02:28:13.758 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-09-21T02:28:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:14 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1176446280' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-21T02:28:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:14 smithi029 ceph-mon[31006]: mgrmap e8: x(active, since 2s) 2024-09-21T02:28:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:15 smithi029 ceph-mon[31006]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:15 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:15 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:28:16.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T02:28:16.304 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-09-21T02:28:16.560 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:16 smithi029 ceph-mon[31006]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: Generating public/private rsa key pair. 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: Your identification has been saved in /tmp/tmpd6e3vjnv/key 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: Your public key has been saved in /tmp/tmpd6e3vjnv/key.pub 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: The key fingerprint is: 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: SHA256:bv6BkOyzQLFoZrRB/Oxl/puslXHUNi85o5qBf6c9kvY ceph-e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: The key's randomart image is: 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: +---[RSA 3072]----+ 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | .. | 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | .. . | 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | oo. . + | 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | . +o+o. . . + | 2024-09-21T02:28:17.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | *.o++ S . = . | 2024-09-21T02:28:17.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | + ....+ = . + | 2024-09-21T02:28:17.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | . +.B o. | 2024-09-21T02:28:17.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | . O.==oo | 2024-09-21T02:28:17.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: | o.O=o=E. | 2024-09-21T02:28:17.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: +----[SHA256]-----+ 2024-09-21T02:28:18.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:18 smithi029 ceph-mon[31006]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:18.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:18 smithi029 ceph-mon[31006]: Generating ssh key... 2024-09-21T02:28:18.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:18 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:18.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:18 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:19.076 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGJxmgxKknCxdO1u/gGJvpVG2hzXPkhwvyzd02ilt7LKor2DkRN+p95XDQkJL3LoIrqL/C7YGOQZvG+fOzhFC22d/9CEI5hd7cT39EboVOnTmHUWgOBT+IW90dsFyKDS7h32toiQ76q0dO/PJUmA/++ZugmO0a7IdCJoTOvFuhy+6ylAavOjyGTLyVcJ2TZ8yFTQ4SBTRIe11xsqamTBY1E4M+MJ937oy4lGi7mv+ddYr50CUAGsPNu2lZIIet8g46n1vUxvu0SBCCCCavWUSDMylwzCZawOSPa54FKX6WPFMHQDM+qyXwB++w6NIddyhH6ePbpiJa+Gq9apPZUte1zI4hcTaCBmiMcI1QYCE0i/2C/q4Xos2DGID6V51V9l2AcqeV3DxraSl+RV6KgYyut9EWlPUtDeYOWpBmnTTnrao5qRheSCR8oi1JRI9D8qVG6wWtaW9dRBoAn0UvpJ/+BQlOWuNcRiqGdRZoUELjhsbTcRghbtpCN16XL5QG+sU= ceph-e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:28:19.076 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T02:28:19.076 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T02:28:19.083 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-09-21T02:28:19.208 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:19 smithi029 ceph-mon[31006]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:20.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:20 smithi029 ceph-mon[31006]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:21.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:21 smithi029 ceph-mon[31006]: Deploying cephadm binary to smithi029 2024-09-21T02:28:24.490 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-09-21T02:28:24.491 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mon service... 2024-09-21T02:28:24.812 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:24 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:24.812 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:24 smithi029 ceph-mon[31006]: Added host smithi029 2024-09-21T02:28:24.812 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:24 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:28:25.838 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T02:28:25.838 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mgr service... 2024-09-21T02:28:26.480 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:26 smithi029 ceph-mon[31006]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:26.481 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:26 smithi029 ceph-mon[31006]: Saving service mon spec with placement count:5 2024-09-21T02:28:26.481 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:26 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:27 smithi029 ceph-mon[31006]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:27 smithi029 ceph-mon[31006]: Saving service mgr spec with placement count:2 2024-09-21T02:28:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:27 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:27 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:27.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T02:28:29.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:28 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:29.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:28 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2447513079' entity='client.admin' 2024-09-21T02:28:30.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:30 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:31.135 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-09-21T02:28:31.539 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:31 smithi029 ceph-mon[31006]: from='mgr.14118 172.21.15.29:0/2286337466' entity='mgr.x' 2024-09-21T02:28:31.539 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:31 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4242568885' entity='client.admin' 2024-09-21T02:28:31.539 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:31 smithi029 ceph-mon[31006]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:28:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:32 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4271753154' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-21T02:28:32.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:32 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: ignoring --setuser ceph since I am not root 2024-09-21T02:28:32.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:32 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: ignoring --setgroup ceph since I am not root 2024-09-21T02:28:33.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4271753154' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T02:28:33.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:33 smithi029 ceph-mon[31006]: mgrmap e9: x(active, since 21s) 2024-09-21T02:28:34.792 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:34 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1867732924' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T02:28:35.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-09-21T02:28:35.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T02:28:35.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T02:28:35.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-21T02:28:35.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T02:28:35.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-09-21T02:28:35.157 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-09-21T02:28:35.157 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: Active manager daemon x restarted 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: Activating manager daemon x 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: mgrmap e10: x(active, starting, since 0.00403167s) 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T02:28:38.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:37 smithi029 ceph-mon[31006]: Manager daemon x is now available 2024-09-21T02:28:39.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:28:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T02:28:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T02:28:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:38 smithi029 ceph-mon[31006]: mgrmap e11: x(active, since 1.00714s) 2024-09-21T02:28:39.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-09-21T02:28:39.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T02:28:39.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T02:28:39.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-09-21T02:28:39.834 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-09-21T02:28:39.834 INFO:teuthology.orchestra.run.smithi029.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-21T02:28:40.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:39 smithi029 ceph-mon[31006]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T02:28:40.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:39 smithi029 ceph-mon[31006]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T02:28:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-mon[31006]: mgrmap e12: x(active, since 2s) 2024-09-21T02:28:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:41.382 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:28:40.889+0000 7fab6901b640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:02:28:40] ENGINE Error in HTTPServer.serve 2024-09-21T02:28:41.382 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: Traceback (most recent call last): 2024-09-21T02:28:41.382 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: self._connections.run(self.expiration_interval) 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: self._run(expiration_interval) 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: s = self.context.wrap_socket( 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T02:28:41.383 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: return self.sslsocket_class._create( 2024-09-21T02:28:41.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T02:28:41.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: self.do_handshake() 2024-09-21T02:28:41.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T02:28:41.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: self._sslobj.do_handshake() 2024-09-21T02:28:41.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T02:28:41.384 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:28:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:28:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:42 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:40] ENGINE Bus STARTING 2024-09-21T02:28:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:42 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:40] ENGINE Serving on https://172.21.15.29:7150 2024-09-21T02:28:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:42 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:40] ENGINE Error in HTTPServer.serve 2024-09-21T02:28:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout: Traceback (most recent call last): 2024-09-21T02:28:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T02:28:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-09-21T02:28:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T02:28:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._run(expiration_interval) 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: s = self.context.wrap_socket( 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: return self.sslsocket_class._create( 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: self.do_handshake() 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T02:28:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._sslobj.do_handshake() 2024-09-21T02:28:42.384 INFO:journalctl@ceph.mon.a.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T02:28:42.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:42 smithi029 ceph-mon[31006]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:42.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:42 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:40] ENGINE Serving on http://172.21.15.29:8765 2024-09-21T02:28:42.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:42 smithi029 ceph-mon[31006]: [21/Sep/2024:02:28:40] ENGINE Bus STARTED 2024-09-21T02:28:42.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:43.160 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIFEDCCAvigAwIBAgIUXnOkq1fucKjCMszxu0TxG4JbbaMwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMTAyMjg0MVoXDTM0\nMDkyMjAyMjg0MVowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjI5MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAwoO57A06tTKSDLSNLqcq2ISagZnecdJlhkYi\nsyDQlcJnFgiAo5qpF//WE1l2rkdHkDqq3n1PFsWK20xeu5/2HEYTLy5S0w0a7Zkk\nPgIEOKZueVFRvfWed4U4VO04yX+8ehztule7MvQZaqdn3m1uTNIIxmmkKFILJFog\nDI5onM6lRpVYytLlPGRG9dKP/5yjWJy0aetx9bHevzIxjm6OA0gKiKvV05+msmwy\nj6gvq+v0PqoCwGM/UUNkGZfbjZf63xbAloxb2SaByxJ8Ep2UfTvF2KF1MMkOZn96\n+gTxnv25G449WAnNh9bzIFjgmLGfkliA+09KP0cdrwnI0g9c2PDbfEn47sVIgu6A\nuJfJWOlCJ+wRMaIdRVlGN5GgC2BwOEymHNLftJYEbKO/Pdw1YcGoyF/VXsyaSx6L\nJWxY7iVNWkvgRSmebY5HvwMqZN+sZiycafx2l/oJlYN05eVpNTbWcK1MIfnYjprY\nVBznYborFZXYj0jgEDwQI+6hyeXO+ZDWfBwQHUXVWKWfjVAj1OUkkNH4M59wpuUh\nvGNdxL6JvqbWwOTt3aYlvH3paLKGicE4sKUO0N8gWq5Cx2O+I8HHWP7qPMlvWItR\n7wD+5zCEFR4QcPblLmvZ8k4mFF1ve9GmWATDAueZxmwaIGNvJM0hPkq1PqCI0zwC\n7+mB3bMCAwEAAaNUMFIwQgYDVR0RBDswOYIec21pdGhpMDI5LmZyb250LnNlcGlh\nLmNlcGguY29tghFkYXNoYm9hcmRfc2VydmVyc4cErBUPHTAMBgNVHRMBAf8EAjAA\nMA0GCSqGSIb3DQEBCwUAA4ICAQBGiEmXQE4CUahLHcNpG2JfCP9ys0j9LjSL4dTW\nIsQbEspRFal6JhpdtsGF9IDfPPkdzOM6vGl4iDtcpH1kKHQMcyyS6fZf2b1rgVeQ\n662AJMTCnIJF1kKoClSTsGCamuZ5YRSYHImGbzXRWDYwur0yO1/ZgGWuxb4W6KcB\nchr72WmbC3RENpzkbHUatYbcecx+TSpBGH+KE1UlbSu2oe4+do6xH0b2wejjYN08\n4eT0xaDyAAr1316TWbmlKPvxe9HENZVhW2SccNFJScGC63wiKujRFbnMgX2oR9k6\n+5LUX65gA5RJbF40kQ9fP52WCNy9ON6qdUI+fUty9C+bkSS5Jj07MGLtDKwXUYyW\ngK9uNUmvYWIKHc24YYH4t1gPaH3wRgBaT8yCNst+2lbU6Dm1HJpL6co7AbPwpiJ7\nh979Z9KN+eDK5PP7JXU6YyfGjMrjcbK1KbPzvXvWxFWjXpisnsrtfCrpNB3LMk97\nEXYZwoJTe5ooWWBslVMlXNNNWirP+TswNShD1aamwyzJcjno2PFbqd8o00/eCKq1\noKCaPm/DSVnjgm2CGFYXfRhSd4tjPfqVm2cWjLRi7ncSCKhGz9DyxuRTQhLWF6LV\nXaHF+LdOiKoAiv6jSfFvsMUpvd+degWCMSsIXsXOb3ZT09m4znL8WfZzlU2Hw4BA\nI8F8FQ==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEAwoO57A06tTKSDLSNLqcq2ISagZnecdJlhkYisyDQlcJnFgiA\no5qpF//WE1l2rkdHkDqq3n1PFsWK20xeu5/2HEYTLy5S0w0a7ZkkPgIEOKZueVFR\nvfWed4U4VO04yX+8ehztule7MvQZaqdn3m1uTNIIxmmkKFILJFogDI5onM6lRpVY\nytLlPGRG9dKP/5yjWJy0aetx9bHevzIxjm6OA0gKiKvV05+msmwyj6gvq+v0PqoC\nwGM/UUNkGZfbjZf63xbAloxb2SaByxJ8Ep2UfTvF2KF1MMkOZn96+gTxnv25G449\nWAnNh9bzIFjgmLGfkliA+09KP0cdrwnI0g9c2PDbfEn47sVIgu6AuJfJWOlCJ+wR\nMaIdRVlGN5GgC2BwOEymHNLftJYEbKO/Pdw1YcGoyF/VXsyaSx6LJWxY7iVNWkvg\nRSmebY5HvwMqZN+sZiycafx2l/oJlYN05eVpNTbWcK1MIfnYjprYVBznYborFZXY\nj0jgEDwQI+6hyeXO+ZDWfBwQHUXVWKWfjVAj1OUkkNH4M59wpuUhvGNdxL6JvqbW\nwOTt3aYlvH3paLKGicE4sKUO0N8gWq5Cx2O+I8HHWP7qPMlvWItR7wD+5zCEFR4Q\ncPblLmvZ8k4mFF1ve9GmWATDAueZxmwaIGNvJM0hPkq1PqCI0zwC7+mB3bMCAwEA\nAQKCAgAh31d+lDUvfZ1ySfS19ZVf/AN94yIZ9xYH4mnDgMjh82cJLrTcCKA+liYc\n23zvz2jWZGkB8pdVs1PBaZSrnqj4dnl7w1jX+NXCH2NwyuAaT104kiEmb8HHNXNA\nUskLScu0Bhqd0w49ysUsTfeoA61sAj1V8+qKvPLEw08GcLwl5lRgrLw4Ia6arCmx\nQBgkXo4RbcR7F22SaAWOjPqVP4pKZv6ymx3Syt/3/SB25MZcbWvsh6WQBBhwGciy\nLkAk8ZzVwXwLG2kZMPv0NYVWpotHnaOP3Fnwg2qXDccIB9ldovhbpJLpJFfeul8/\nOErwPCo6DWB0ovOefk9cflsyvUshDGOG+SE23ar8lrRrAmmK3PzAsrHqFhoDcA/W\n73HhweUZ9WHJd5TfPid48XN8BTIBYNhwDx65l6C9I2e0gHGXsrf7Stm57YHPZRXf\nmcDq6poE2F9kDRTyVcGwEuLJzw0o6S/EgF6WEMqPhPB/ux83NpQ8X/hw3nfbfE2C\nvbfHOf9aB6GUi0LS0+R/ZwbrhwNY33zbDIy10XsC7PSV8UIDG/vQfYbMrXfTSaZZ\nhndDZiH606R+bH9OBYWmtzxFC66/gucgzevgFZK9eUbh7gsp2zGbkrEh5FjARwBs\nEpjo6bMZY+BGcnF/YYIpvvYkagosymNNA7FnX+RpQDJQqUXv7QKCAQEA4S2n6FD7\nLr67BtetoYMxZebPC+5DXxgqVCfpuqicUtLdNOlT9zUVx7ZqOvMXkdIdMR0ReVz3\nDAvbR8qlm0Y0RSv7FnOSsVYaSDodr1gp2r46l+SwfrKROR3bNtnD/X617neXGp4o\nhn1YwVFTNGtuBVmC4AjPuALYY4No5FAQSoEXcKTkUTmXzlS3BtuiUmBUP/biHIcY\nyLxgnzHtZ73tOagQbesHRziXQYA7vROvDE5rZizdMFMePMXubzwpSun83aaJXqPl\n+zXrnc8OVU+7NqUeI0cXMhudf3unGx7N/RL8W/B3h2qiwOk/5y/lsfT0VA4FE9g2\nxiJjy3mUGzOJ5QKCAQEA3SOZK32rDAyVrSig7pFQ90fLPnKF0fZ09SQRrAOkq2mc\nCyuWroER4Q17nrO7xTYLT7vjahS/lhBJgAj9KSvBqdprshOGiE31MrYTYLUtiZeK\nckTMW1g2BgVK0vOb+2MlZydfWgP6eVe5+5P5859lw50HOeo+WdteW3XWl6M1d6uD\nD65/wq/kaijcfcADfO5Q0ExHO/JenH+HYalxbKhvpl+iXu+OaCn4JwkBXmBYDU5c\nQLRRrbFmus+ZklkgLbeUvCnKTCcGRwiI7uTdfK1HT+WQm0eVESQhdI6WUnd0b+f2\nyPNA6v3pwd63r7SkoiSyAKNZLe0fsPOSRfuBNWBvtwKCAQEAvfzmgXXqKIuWg3Rw\nNCjcAe1W3ese/toFxNGUPABNPPha2WzQgnBQuxtmz+wRA+Cy7qBnAO32JzzNu+n4\nBmWabo5oII5Mw8ExFaWrLCmHzt7X0xaJ76w8KWPOdJ4R5g0F9XkDjdrJuIeD3j/+\noujNgEgAy4PDtq734iaa1p/SzvnSTCFj3kBonBcQAgJIF5bdHsbwCj/WInAGWesn\nMnOl9xKdIhC+KFJQFFzcinBULl7v8NqDmWVXytD5lvK8qE96IY2Es8w3m9fnv1NW\n9ZX7EGUKY5t0r7trDIfI+3HnPNLLB8uBHklTns5uM2JPO/HAQDdAA3lW5Z9YcfJX\niwfWHQKCAQA2noNHfauQt08EzOMq6ZOA8CegDfNFj8UFg1omDWRPvi7/h02ZXmHH\nH2hoI2Z+aEHU+MhLvEIx02YEDLRSxe74f+1Xt4DkC7VDSDtzCGzt7hMQ9DMxYJE/\nXXFEt0zQg4WKe+wONKtZ6zQRklQLO1SMQy20vUmXMhqVJn7RR0t+ektNxUDjGkSl\nf+At4w45fjdRq3VwtgVuTAVpNsvAFb417qsRHcQjsDINmijPYaZ96tvYU3FVIFsK\nSxpYTlEnWY+QBhZkRsNDKjWAI+Ec2UOi1UWpRL3p1O26dzljtwN4XGfU7FgdcoeZ\nDeR6rnh3X+7jP3VYZZgNQavgQyV/xIjzAoIBAFjbFjaB/I6T+kjq7vD6oF8iQDbp\nXX2S9ejBKdl09EDxNvMYvsN7wxZ7eZ2lPdW/2nY7CSaSOWlIS9OeCC2/Kplzp57C\nOflURsBvH2K9D82bcX/+vSi36wNdI6r3cixzSe0m8VTyHfCRbp4UOubPnNv9WIyD\n5TZMDeVTohYYdpYWb7CCmFaQvmgBOC+xBITQrl8SB9ZxGVzn56eDmgjHPSYNaHtr\nc/QRzxX16TWvw8fThGjE+ieq/+9YRi431gz++fVdwAEkr6cj2pBR9n6V47WMsqYP\nL8mqXo/XvRmWD+OuwYJNFP0PhwWnk6xjFfMqXYyhT0QHlZgeHh0VjgVyKwI=\n-----END RSA PRIVATE KEY-----\n"} 2024-09-21T02:28:44.459 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-21T02:28:45.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:44 smithi029 ceph-mon[31006]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:45.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:45.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-21T02:28:45.734 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-09-21T02:28:46.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:46 smithi029 ceph-mon[31006]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:47.287 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:47 smithi029 ceph-mon[31006]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:47.287 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:47.290 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$cmxBXbiy0P6sjV8Ny438PeYbhvfCy6Zp/VAJOmGrvPcQ86Ib9vnWe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726885726, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T02:28:47.290 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-09-21T02:28:48.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:48 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2648855' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-21T02:28:48.546 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T02:28:48.556 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T02:28:48.556 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-09-21T02:28:48.563 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T02:28:48.563 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-09-21T02:28:48.563 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-09-21T02:28:48.563 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T02:28:48.565 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-09-21T02:28:48.565 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:48.565 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-09-21T02:28:48.566 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-09-21T02:28:48.566 INFO:teuthology.orchestra.run.smithi029.stdout: Password: 01i6bzev79 2024-09-21T02:28:48.566 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:48.566 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config directory 2024-09-21T02:28:49.961 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T02:28:49.961 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T02:28:49.961 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:49.961 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:28:49.961 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:28:49.962 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-09-21T02:28:50.008 INFO:tasks.cephadm:Fetching config... 2024-09-21T02:28:50.009 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:28:50.009 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T02:28:50.026 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T02:28:50.026 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:28:50.026 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T02:28:50.083 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T02:28:50.083 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:28:50.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T02:28:50.150 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T02:28:50.150 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:28:50.151 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T02:28:50.207 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T02:28:50.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGJxmgxKknCxdO1u/gGJvpVG2hzXPkhwvyzd02ilt7LKor2DkRN+p95XDQkJL3LoIrqL/C7YGOQZvG+fOzhFC22d/9CEI5hd7cT39EboVOnTmHUWgOBT+IW90dsFyKDS7h32toiQ76q0dO/PJUmA/++ZugmO0a7IdCJoTOvFuhy+6ylAavOjyGTLyVcJ2TZ8yFTQ4SBTRIe11xsqamTBY1E4M+MJ937oy4lGi7mv+ddYr50CUAGsPNu2lZIIet8g46n1vUxvu0SBCCCCavWUSDMylwzCZawOSPa54FKX6WPFMHQDM+qyXwB++w6NIddyhH6ePbpiJa+Gq9apPZUte1zI4hcTaCBmiMcI1QYCE0i/2C/q4Xos2DGID6V51V9l2AcqeV3DxraSl+RV6KgYyut9EWlPUtDeYOWpBmnTTnrao5qRheSCR8oi1JRI9D8qVG6wWtaW9dRBoAn0UvpJ/+BQlOWuNcRiqGdRZoUELjhsbTcRghbtpCN16XL5QG+sU= ceph-e1049baa-77c0-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T02:28:50.281 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGJxmgxKknCxdO1u/gGJvpVG2hzXPkhwvyzd02ilt7LKor2DkRN+p95XDQkJL3LoIrqL/C7YGOQZvG+fOzhFC22d/9CEI5hd7cT39EboVOnTmHUWgOBT+IW90dsFyKDS7h32toiQ76q0dO/PJUmA/++ZugmO0a7IdCJoTOvFuhy+6ylAavOjyGTLyVcJ2TZ8yFTQ4SBTRIe11xsqamTBY1E4M+MJ937oy4lGi7mv+ddYr50CUAGsPNu2lZIIet8g46n1vUxvu0SBCCCCavWUSDMylwzCZawOSPa54FKX6WPFMHQDM+qyXwB++w6NIddyhH6ePbpiJa+Gq9apPZUte1zI4hcTaCBmiMcI1QYCE0i/2C/q4Xos2DGID6V51V9l2AcqeV3DxraSl+RV6KgYyut9EWlPUtDeYOWpBmnTTnrao5qRheSCR8oi1JRI9D8qVG6wWtaW9dRBoAn0UvpJ/+BQlOWuNcRiqGdRZoUELjhsbTcRghbtpCN16XL5QG+sU= ceph-e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:28:50.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGJxmgxKknCxdO1u/gGJvpVG2hzXPkhwvyzd02ilt7LKor2DkRN+p95XDQkJL3LoIrqL/C7YGOQZvG+fOzhFC22d/9CEI5hd7cT39EboVOnTmHUWgOBT+IW90dsFyKDS7h32toiQ76q0dO/PJUmA/++ZugmO0a7IdCJoTOvFuhy+6ylAavOjyGTLyVcJ2TZ8yFTQ4SBTRIe11xsqamTBY1E4M+MJ937oy4lGi7mv+ddYr50CUAGsPNu2lZIIet8g46n1vUxvu0SBCCCCavWUSDMylwzCZawOSPa54FKX6WPFMHQDM+qyXwB++w6NIddyhH6ePbpiJa+Gq9apPZUte1zI4hcTaCBmiMcI1QYCE0i/2C/q4Xos2DGID6V51V9l2AcqeV3DxraSl+RV6KgYyut9EWlPUtDeYOWpBmnTTnrao5qRheSCR8oi1JRI9D8qVG6wWtaW9dRBoAn0UvpJ/+BQlOWuNcRiqGdRZoUELjhsbTcRghbtpCN16XL5QG+sU= ceph-e1049baa-77c0-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T02:28:50.355 INFO:teuthology.orchestra.run.smithi066.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGJxmgxKknCxdO1u/gGJvpVG2hzXPkhwvyzd02ilt7LKor2DkRN+p95XDQkJL3LoIrqL/C7YGOQZvG+fOzhFC22d/9CEI5hd7cT39EboVOnTmHUWgOBT+IW90dsFyKDS7h32toiQ76q0dO/PJUmA/++ZugmO0a7IdCJoTOvFuhy+6ylAavOjyGTLyVcJ2TZ8yFTQ4SBTRIe11xsqamTBY1E4M+MJ937oy4lGi7mv+ddYr50CUAGsPNu2lZIIet8g46n1vUxvu0SBCCCCavWUSDMylwzCZawOSPa54FKX6WPFMHQDM+qyXwB++w6NIddyhH6ePbpiJa+Gq9apPZUte1zI4hcTaCBmiMcI1QYCE0i/2C/q4Xos2DGID6V51V9l2AcqeV3DxraSl+RV6KgYyut9EWlPUtDeYOWpBmnTTnrao5qRheSCR8oi1JRI9D8qVG6wWtaW9dRBoAn0UvpJ/+BQlOWuNcRiqGdRZoUELjhsbTcRghbtpCN16XL5QG+sU= ceph-e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:28:50.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGJxmgxKknCxdO1u/gGJvpVG2hzXPkhwvyzd02ilt7LKor2DkRN+p95XDQkJL3LoIrqL/C7YGOQZvG+fOzhFC22d/9CEI5hd7cT39EboVOnTmHUWgOBT+IW90dsFyKDS7h32toiQ76q0dO/PJUmA/++ZugmO0a7IdCJoTOvFuhy+6ylAavOjyGTLyVcJ2TZ8yFTQ4SBTRIe11xsqamTBY1E4M+MJ937oy4lGi7mv+ddYr50CUAGsPNu2lZIIet8g46n1vUxvu0SBCCCCavWUSDMylwzCZawOSPa54FKX6WPFMHQDM+qyXwB++w6NIddyhH6ePbpiJa+Gq9apPZUte1zI4hcTaCBmiMcI1QYCE0i/2C/q4Xos2DGID6V51V9l2AcqeV3DxraSl+RV6KgYyut9EWlPUtDeYOWpBmnTTnrao5qRheSCR8oi1JRI9D8qVG6wWtaW9dRBoAn0UvpJ/+BQlOWuNcRiqGdRZoUELjhsbTcRghbtpCN16XL5QG+sU= ceph-e1049baa-77c0-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T02:28:50.443 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGJxmgxKknCxdO1u/gGJvpVG2hzXPkhwvyzd02ilt7LKor2DkRN+p95XDQkJL3LoIrqL/C7YGOQZvG+fOzhFC22d/9CEI5hd7cT39EboVOnTmHUWgOBT+IW90dsFyKDS7h32toiQ76q0dO/PJUmA/++ZugmO0a7IdCJoTOvFuhy+6ylAavOjyGTLyVcJ2TZ8yFTQ4SBTRIe11xsqamTBY1E4M+MJ937oy4lGi7mv+ddYr50CUAGsPNu2lZIIet8g46n1vUxvu0SBCCCCavWUSDMylwzCZawOSPa54FKX6WPFMHQDM+qyXwB++w6NIddyhH6ePbpiJa+Gq9apPZUte1zI4hcTaCBmiMcI1QYCE0i/2C/q4Xos2DGID6V51V9l2AcqeV3DxraSl+RV6KgYyut9EWlPUtDeYOWpBmnTTnrao5qRheSCR8oi1JRI9D8qVG6wWtaW9dRBoAn0UvpJ/+BQlOWuNcRiqGdRZoUELjhsbTcRghbtpCN16XL5QG+sU= ceph-e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:28:50.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T02:28:50.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:50 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/216252937' entity='client.admin' 2024-09-21T02:28:50.711 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:28:51.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:51 smithi029 ceph-mon[31006]: mgrmap e13: x(active, since 12s) 2024-09-21T02:28:52.172 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T02:28:52.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T02:28:52.698 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:52 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1688361245' entity='client.admin' 2024-09-21T02:28:53.203 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:28:55.137 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi066 2024-09-21T02:28:55.137 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:28:55.137 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/etc/ceph/ceph.conf 2024-09-21T02:28:55.157 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:28:55.157 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T02:28:55.216 INFO:tasks.cephadm:Adding host smithi066 to orchestrator... 2024-09-21T02:28:55.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch host add smithi066 2024-09-21T02:28:55.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:54 smithi029 ceph-mon[31006]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:55.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:55.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:55.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:55.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"} : dispatch 2024-09-21T02:28:55.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:28:55.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:28:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:28:56.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: Updating smithi029:/etc/ceph/ceph.conf 2024-09-21T02:28:56.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: Updating smithi029:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:28:56.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-09-21T02:28:56.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: Updating smithi029:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T02:28:56.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:56.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:56.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:56.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:28:56.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:28:56.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:28:56.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:56 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:28:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:57 smithi029 ceph-mon[31006]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:58.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:58 smithi029 ceph-mon[31006]: Deploying cephadm binary to smithi066 2024-09-21T02:28:59.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:28:59 smithi029 ceph-mon[31006]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:28:59.944 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi066' with addr '172.21.15.66' 2024-09-21T02:29:00.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T02:29:00.930 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:01.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:00 smithi029 ceph-mon[31006]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:01.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:01.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:00 smithi029 ceph-mon[31006]: Added host smithi066 2024-09-21T02:29:01.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:01.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:01.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:01.559 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:29:01.559 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.66", "hostname": "smithi066", "labels": [], "status": ""}] 2024-09-21T02:29:02.217 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-09-21T02:29:02.218 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:29:02.218 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-09-21T02:29:02.236 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:29:02.236 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T02:29:02.295 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-09-21T02:29:02.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch host add smithi190 2024-09-21T02:29:02.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:02 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:02.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:02 smithi029 ceph-mon[31006]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:29:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:03.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:03 smithi029 ceph-mon[31006]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:04.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:04 smithi029 ceph-mon[31006]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:04.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:04.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:04.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:04.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:04.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi066", "name": "osd_memory_target"} : dispatch 2024-09-21T02:29:04.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:04.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: Updating smithi066:/etc/ceph/ceph.conf 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: Updating smithi066:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: Deploying cephadm binary to smithi190 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: Updating smithi066:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:05 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi190' with addr '172.21.15.190' 2024-09-21T02:29:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:07 smithi029 ceph-mon[31006]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:07.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T02:29:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:29:08.780 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.66", "hostname": "smithi066", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-09-21T02:29:08.780 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:08 smithi029 ceph-mon[31006]: Added host smithi190 2024-09-21T02:29:08.780 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:08 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:08.780 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:08 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:09.424 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T02:29:09.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-21T02:29:09.679 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:09.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:09 smithi029 ceph-mon[31006]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-09-21T02:29:10.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:10 smithi029 ceph-mon[31006]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:29:10.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:10 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2558849411' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T02:29:11.263 INFO:tasks.cephadm:Adding mon.a on smithi029 2024-09-21T02:29:11.263 INFO:tasks.cephadm:Adding mon.b on smithi066 2024-09-21T02:29:11.263 INFO:tasks.cephadm:Adding mon.c on smithi190 2024-09-21T02:29:11.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi029:172.21.15.29=a;smithi066:172.21.15.66=b;smithi190:172.21.15.190=c' 2024-09-21T02:29:11.506 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2558849411' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T02:29:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T02:29:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"} : dispatch 2024-09-21T02:29:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:12.187 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2024-09-21T02:29:12.744 DEBUG:teuthology.orchestra.run.smithi066:mon.b> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.b.service 2024-09-21T02:29:12.747 DEBUG:teuthology.orchestra.run.smithi190:mon.c> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.c.service 2024-09-21T02:29:12.749 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T02:29:12.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T02:29:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: Updating smithi190:/etc/ceph/ceph.conf 2024-09-21T02:29:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: Updating smithi190:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-09-21T02:29:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: Updating smithi190:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T02:29:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:12.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:12.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:12.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:13.712 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:13 smithi029 ceph-mon[31006]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:13 smithi029 ceph-mon[31006]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi066:172.21.15.66=b;smithi190:172.21.15.190=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:13 smithi029 ceph-mon[31006]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi066:172.21.15.66=b;smithi190:172.21.15.190=c;count:3 2024-09-21T02:29:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:13 smithi029 ceph-mon[31006]: Deploying daemon mon.c on smithi190 2024-09-21T02:29:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:29:15.332 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","modified":"2024-09-21T02:27:43.754319Z","created":"2024-09-21T02:27:43.754319Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T02:29:15.332 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-09-21T02:29:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:15 smithi029 ceph-mon[31006]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:15 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/2713197811' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T02:29:16.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:16 smithi190 systemd[1]: Started Ceph mon.c for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:29:17.157 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T02:29:17.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T02:29:17.412 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:29:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: mon.a calling monitor election 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: mon.c calling monitor election 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T02:29:21.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: monmap epoch 2 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: last_changed 2024-09-21T02:29:16.565623+0000 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: created 2024-09-21T02:27:43.754319+0000 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: min_mon_release 19 (squid) 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: election_strategy: 1 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.c 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: fsmap 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: mgrmap e13: x(active, since 43s) 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: overall HEALTH_OK 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:21.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:21.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:29:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: mon.a calling monitor election 2024-09-21T02:29:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: mon.c calling monitor election 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: monmap epoch 2 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: last_changed 2024-09-21T02:29:16.565623+0000 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: created 2024-09-21T02:27:43.754319+0000 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: min_mon_release 19 (squid) 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: election_strategy: 1 2024-09-21T02:29:22.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.c 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: fsmap 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: mgrmap e13: x(active, since 43s) 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: overall HEALTH_OK 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:22.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:22.548 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:29:22.548 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":2,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","modified":"2024-09-21T02:29:16.565623Z","created":"2024-09-21T02:27:43.754319Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T02:29:22.548 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 2 2024-09-21T02:29:22.850 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:22 smithi190 ceph-mon[33871]: Deploying daemon mon.b on smithi066 2024-09-21T02:29:22.850 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:22 smithi190 ceph-mon[33871]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:22.851 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:22 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/1342928126' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T02:29:22.851 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:22 smithi029 ceph-mon[31006]: Deploying daemon mon.b on smithi066 2024-09-21T02:29:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:22 smithi029 ceph-mon[31006]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:22 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/1342928126' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T02:29:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:24.146 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T02:29:24.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T02:29:24.397 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:29:25.044 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:29:25.044 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":2,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","modified":"2024-09-21T02:29:16.565623Z","created":"2024-09-21T02:27:43.754319Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T02:29:25.044 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 2 2024-09-21T02:29:25.044 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:24 smithi190 ceph-mon[33871]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:25.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:24 smithi029 ceph-mon[31006]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:25 smithi066 systemd[1]: Started Ceph mon.b for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:29:26.603 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T02:29:26.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T02:29:26.853 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: mon.a calling monitor election 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: mon.c calling monitor election 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: mon.b calling monitor election 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: monmap epoch 3 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: last_changed 2024-09-21T02:29:25.622113+0000 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: created 2024-09-21T02:27:43.754319+0000 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: min_mon_release 19 (squid) 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: election_strategy: 1 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-09-21T02:29:31.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.c 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: 2: [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] mon.b 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: fsmap 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: mgrmap e13: x(active, since 52s) 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: overall HEALTH_OK 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: mon.a calling monitor election 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: mon.c calling monitor election 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: mon.b calling monitor election 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: monmap epoch 3 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: last_changed 2024-09-21T02:29:25.622113+0000 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: created 2024-09-21T02:27:43.754319+0000 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: min_mon_release 19 (squid) 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: election_strategy: 1 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.c 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: 2: [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] mon.b 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: fsmap 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T02:29:31.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: mgrmap e13: x(active, since 52s) 2024-09-21T02:29:31.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: overall HEALTH_OK 2024-09-21T02:29:31.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:29:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: mon.a calling monitor election 2024-09-21T02:29:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:29:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: mon.c calling monitor election 2024-09-21T02:29:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: mon.b calling monitor election 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: monmap epoch 3 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: last_changed 2024-09-21T02:29:25.622113+0000 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: created 2024-09-21T02:27:43.754319+0000 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: min_mon_release 19 (squid) 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: election_strategy: 1 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.c 2024-09-21T02:29:31.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: 2: [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] mon.b 2024-09-21T02:29:31.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: fsmap 2024-09-21T02:29:31.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T02:29:31.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: mgrmap e13: x(active, since 52s) 2024-09-21T02:29:31.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: overall HEALTH_OK 2024-09-21T02:29:31.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:31.182 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:29:31.182 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":3,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","modified":"2024-09-21T02:29:25.622113Z","created":"2024-09-21T02:27:43.754319Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:3300","nonce":0},{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T02:29:31.182 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 3 2024-09-21T02:29:31.957 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:31 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:31.957 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:31 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/3866894725' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T02:29:31.958 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:31 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:31 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:31 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/3866894725' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T02:29:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:31 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:32.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:31 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:32.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:31 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/3866894725' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T02:29:32.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:31 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:29:32.335 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T02:29:32.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T02:29:32.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:29:32.622+0000 7fab9ca50640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T02:29:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: Updating smithi029:/etc/ceph/ceph.conf 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: Updating smithi066:/etc/ceph/ceph.conf 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: Updating smithi190:/etc/ceph/ceph.conf 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: Updating smithi190:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: Updating smithi066:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: Updating smithi029:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:32.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:32 smithi029 ceph-mon[31006]: Reconfiguring daemon mon.a on smithi029 2024-09-21T02:29:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: Updating smithi029:/etc/ceph/ceph.conf 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: Updating smithi066:/etc/ceph/ceph.conf 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: Updating smithi190:/etc/ceph/ceph.conf 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: Updating smithi190:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: Updating smithi066:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: Updating smithi029:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:33.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:32 smithi190 ceph-mon[33871]: Reconfiguring daemon mon.a on smithi029 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: Updating smithi029:/etc/ceph/ceph.conf 2024-09-21T02:29:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: Updating smithi066:/etc/ceph/ceph.conf 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: Updating smithi190:/etc/ceph/ceph.conf 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: Updating smithi190:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: Updating smithi066:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: Updating smithi029:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:33.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T02:29:33.262 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:33.262 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:33.262 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:33.262 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:32 smithi066 ceph-mon[33436]: Reconfiguring daemon mon.a on smithi029 2024-09-21T02:29:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:34.365 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.365 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.365 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.365 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:34.365 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:34.365 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:34.387 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:29:34.387 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-09-21T02:29:34.387 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:29:34.387 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-21T02:29:34.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:34.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:34.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:34.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:34.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:34.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:34.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:35.029 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T02:29:35.029 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:29:35.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T02:29:35.060 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:29:35.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T02:29:35.128 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:29:35.170 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T02:29:35.202 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:29:35.202 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T02:29:35.270 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:29:35.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T02:29:35.297 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:29:35.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T02:29:35.361 INFO:tasks.cephadm:Adding mgr.x on smithi029 2024-09-21T02:29:35.361 INFO:tasks.cephadm:Adding mgr.y on smithi066 2024-09-21T02:29:35.361 INFO:tasks.cephadm:Adding mgr.z on smithi190 2024-09-21T02:29:35.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch apply mgr '3;smithi029=x;smithi066=y;smithi190=z' 2024-09-21T02:29:35.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: Reconfiguring mon.b (monmap changed)... 2024-09-21T02:29:35.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: Reconfiguring daemon mon.b on smithi066 2024-09-21T02:29:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2183059531' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:35.548 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: Reconfiguring mon.b (monmap changed)... 2024-09-21T02:29:35.548 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: Reconfiguring daemon mon.b on smithi066 2024-09-21T02:29:35.548 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:35.548 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2183059531' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:35.548 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:35.548 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:35.548 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:35.549 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:35.549 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:35.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: Reconfiguring mon.b (monmap changed)... 2024-09-21T02:29:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: Reconfiguring daemon mon.b on smithi066 2024-09-21T02:29:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2183059531' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T02:29:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T02:29:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:36.476 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:29:36.596 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:36 smithi190 ceph-mon[33871]: Reconfiguring mon.c (monmap changed)... 2024-09-21T02:29:36.596 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:36 smithi190 ceph-mon[33871]: Reconfiguring daemon mon.c on smithi190 2024-09-21T02:29:36.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:36 smithi029 ceph-mon[31006]: Reconfiguring mon.c (monmap changed)... 2024-09-21T02:29:36.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:36 smithi029 ceph-mon[31006]: Reconfiguring daemon mon.c on smithi190 2024-09-21T02:29:36.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:36 smithi066 ceph-mon[33436]: Reconfiguring mon.c (monmap changed)... 2024-09-21T02:29:36.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:36 smithi066 ceph-mon[33436]: Reconfiguring daemon mon.c on smithi190 2024-09-21T02:29:37.120 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mgr update... 2024-09-21T02:29:37.679 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:37.679 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.679 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:37.680 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.707 DEBUG:teuthology.orchestra.run.smithi066:mgr.y> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.y.service 2024-09-21T02:29:37.710 DEBUG:teuthology.orchestra.run.smithi190:mgr.z> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.z.service 2024-09-21T02:29:37.712 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-09-21T02:29:37.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set client client_mount_timeout 600 2024-09-21T02:29:37.714 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-09-21T02:29:37.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set client debug_client 20 2024-09-21T02:29:37.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:37.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:37.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.738 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:37.738 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.738 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.738 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:37.738 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:37.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:37.740 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-09-21T02:29:37.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set client debug_ms 1 2024-09-21T02:29:37.783 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-09-21T02:29:37.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set client rados_mon_op_timeout 900 2024-09-21T02:29:37.827 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-09-21T02:29:37.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set client rados_osd_op_timeout 900 2024-09-21T02:29:37.870 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-09-21T02:29:37.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds 20 2024-09-21T02:29:37.914 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-09-21T02:29:37.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds_balancer 20 2024-09-21T02:29:37.957 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-09-21T02:29:37.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds debug_ms 2 2024-09-21T02:29:38.000 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-09-21T02:29:38.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-09-21T02:29:38.058 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-09-21T02:29:38.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_merge_size 5 2024-09-21T02:29:38.058 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:38.087 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-09-21T02:29:38.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_split_bits 3 2024-09-21T02:29:38.130 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-09-21T02:29:38.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_split_size 100 2024-09-21T02:29:38.179 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-09-21T02:29:38.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_frag True 2024-09-21T02:29:38.222 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-09-21T02:29:38.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_scatterstat True 2024-09-21T02:29:38.268 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-09-21T02:29:38.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds mds_op_complaint_time 180 2024-09-21T02:29:38.312 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-09-21T02:29:38.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds mds_verify_scatter True 2024-09-21T02:29:38.364 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-09-21T02:29:38.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds osd_op_complaint_time 180 2024-09-21T02:29:38.419 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-09-21T02:29:38.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds rados_mon_op_timeout 900 2024-09-21T02:29:38.476 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-09-21T02:29:38.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mds rados_osd_op_timeout 900 2024-09-21T02:29:38.516 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:38 smithi029 ceph-mon[31006]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi066=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:38.516 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:38 smithi029 ceph-mon[31006]: Saving service mgr spec with placement smithi029=x;smithi066=y;smithi190=z;count:3 2024-09-21T02:29:38.516 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:38 smithi029 ceph-mon[31006]: Deploying daemon mgr.z on smithi190 2024-09-21T02:29:38.518 INFO:tasks.cephadm:setting config [mgr] client_mount_timeout = 30 2024-09-21T02:29:38.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mgr client_mount_timeout 30 2024-09-21T02:29:38.562 INFO:tasks.cephadm:setting config [mgr] debug_client = 20 2024-09-21T02:29:38.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mgr debug_client 20 2024-09-21T02:29:38.610 INFO:tasks.cephadm:setting config [mgr] debug_mgr = 20 2024-09-21T02:29:38.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mgr debug_mgr 20 2024-09-21T02:29:38.664 INFO:tasks.cephadm:setting config [mgr] debug_ms = 1 2024-09-21T02:29:38.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mgr debug_ms 1 2024-09-21T02:29:38.708 INFO:tasks.cephadm:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-21T02:29:38.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mgr mon_warn_on_pool_no_app False 2024-09-21T02:29:38.753 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-09-21T02:29:38.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set mon mon_op_complaint_time 120 2024-09-21T02:29:38.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:38 smithi066 ceph-mon[33436]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi066=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:38.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:38 smithi066 ceph-mon[33436]: Saving service mgr spec with placement smithi029=x;smithi066=y;smithi190=z;count:3 2024-09-21T02:29:38.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:38 smithi066 ceph-mon[33436]: Deploying daemon mgr.z on smithi190 2024-09-21T02:29:38.801 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:38 smithi190 ceph-mon[33871]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi066=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:38.801 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:38 smithi190 ceph-mon[33871]: Saving service mgr spec with placement smithi029=x;smithi066=y;smithi190=z;count:3 2024-09-21T02:29:38.801 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:38 smithi190 ceph-mon[33871]: Deploying daemon mgr.z on smithi190 2024-09-21T02:29:38.810 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-09-21T02:29:38.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config set osd osd_op_complaint_time 180 2024-09-21T02:29:38.856 DEBUG:tasks.cephadm:set 26 configs 2024-09-21T02:29:38.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi029') 2024-09-21T02:29:39.101 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:39.516 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:39 smithi190 ceph-mon[33871]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:39.516 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:39 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2510507228' entity='client.admin' 2024-09-21T02:29:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:39 smithi066 ceph-mon[33436]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:39.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:39 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2510507228' entity='client.admin' 2024-09-21T02:29:39.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:39 smithi029 ceph-mon[31006]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:39.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:39 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2510507228' entity='client.admin' 2024-09-21T02:29:39.873 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Sep 21 02:29:39 smithi190 systemd[1]: Started Ceph mgr.z for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:29:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 -- 172.21.15.29:0/2641365530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf3803fb20 msgr2=0x7fdf38042030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 --2- 172.21.15.29:0/2641365530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf3803fb20 0x7fdf38042030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf48005d00 tx=0x7fdf4800b410 comp rx=0 tx=0).stop 2024-09-21T02:29:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 -- 172.21.15.29:0/2641365530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c106900 msgr2=0x7fdf4c19a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 --2- 172.21.15.29:0/2641365530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c106900 0x7fdf4c19a300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c005940 tx=0x7fdf3c0042f0 comp rx=0 tx=0).stop 2024-09-21T02:29:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 -- 172.21.15.29:0/2641365530 shutdown_connections 2024-09-21T02:29:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 --2- 172.21.15.29:0/2641365530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf3803fb20 0x7fdf38042030 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 --2- 172.21.15.29:0/2641365530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf4c1094a0 0x7fdf4c19aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 --2- 172.21.15.29:0/2641365530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c106900 0x7fdf4c19a300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 --2- 172.21.15.29:0/2641365530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf4c06c660 0x7fdf4c199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 -- 172.21.15.29:0/2641365530 >> 172.21.15.29:0/2641365530 conn(0x7fdf4c0fdfe0 msgr2=0x7fdf4c104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 -- 172.21.15.29:0/2641365530 shutdown_connections 2024-09-21T02:29:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:40.538+0000 7fdf53816640 1 -- 172.21.15.29:0/2641365530 wait complete. 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: Deploying daemon mgr.y on smithi066 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1756778516' entity='client.admin' 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:41.081 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:40 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2641365530' entity='client.admin' 2024-09-21T02:29:41.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:41.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T02:29:41.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:41.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:41.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: Deploying daemon mgr.y on smithi066 2024-09-21T02:29:41.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1756778516' entity='client.admin' 2024-09-21T02:29:41.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:41.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:40 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2641365530' entity='client.admin' 2024-09-21T02:29:41.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:41.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:41.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T02:29:41.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:41.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:41.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: Deploying daemon mgr.y on smithi066 2024-09-21T02:29:41.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1756778516' entity='client.admin' 2024-09-21T02:29:41.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:41.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:40 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2641365530' entity='client.admin' 2024-09-21T02:29:41.548 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi029') 2024-09-21T02:29:41.549 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi029') 2024-09-21T02:29:42.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 -- 172.21.15.29:0/631036888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc06b900 msgr2=0x7f64cc106410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 --2- 172.21.15.29:0/631036888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc06b900 0x7f64cc106410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f64c8009900 tx=0x7f64c802a410 comp rx=0 tx=0).stop 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 -- 172.21.15.29:0/631036888 shutdown_connections 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 --2- 172.21.15.29:0/631036888 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64cc109490 0x7f64cc10b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 --2- 172.21.15.29:0/631036888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc106950 0x7f64cc108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 --2- 172.21.15.29:0/631036888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc06b900 0x7f64cc106410 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 -- 172.21.15.29:0/631036888 >> 172.21.15.29:0/631036888 conn(0x7f64cc0fdfb0 msgr2=0x7f64cc100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 -- 172.21.15.29:0/631036888 shutdown_connections 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 -- 172.21.15.29:0/631036888 wait complete. 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.002+0000 7f64d45ed640 1 Processor -- start 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d45ed640 1 -- start start 2024-09-21T02:29:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d45ed640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64cc06b900 0x7f64cc199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d45ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc106950 0x7f64cc19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d45ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc109490 0x7f64cc19af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d45ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc10eff0 con 0x7f64cc106950 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d45ed640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc10ee70 con 0x7f64cc06b900 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d45ed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc10f170 con 0x7f64cc109490 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc106950 0x7f64cc19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc106950 0x7f64cc19a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53176/0 (socket says 172.21.15.29:53176) 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 -- 172.21.15.29:0/3260176267 learned_addr learned my addr 172.21.15.29:0/3260176267 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d2b8a640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc109490 0x7f64cc19af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d2389640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64cc06b900 0x7f64cc199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 -- 172.21.15.29:0/3260176267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc109490 msgr2=0x7f64cc19af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc109490 0x7f64cc19af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 -- 172.21.15.29:0/3260176267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64cc06b900 msgr2=0x7f64cc199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64cc06b900 0x7f64cc199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 -- 172.21.15.29:0/3260176267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64cc19b760 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.003+0000 7f64d1b88640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc106950 0x7f64cc19a3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00e9f0 tx=0x7f64bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.004+0000 7f64c37fe640 1 -- 172.21.15.29:0/3260176267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc00cd80 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.004+0000 7f64c37fe640 1 -- 172.21.15.29:0/3260176267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f64bc005470 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.004+0000 7f64c37fe640 1 -- 172.21.15.29:0/3260176267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc004e60 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.004+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64cc19ba50 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.004+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64cc1a2150 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.005+0000 7f64c37fe640 1 -- 172.21.15.29:0/3260176267 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 51898+0+0 (secure 0 0 0) 0x7f64bc021440 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.005+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64cc10f7b0 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.005+0000 7f64c37fe640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64b803fb20 0x7f64b8042030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.005+0000 7f64c37fe640 1 -- 172.21.15.29:0/3260176267 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f64bc0549e0 con 0x7f64cc106950 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.005+0000 7f64d2389640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64b803fb20 0x7f64b8042030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.006+0000 7f64d2389640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64b803fb20 0x7f64b8042030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64c8005ab0 tx=0x7f64c8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:42.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.010+0000 7f64c37fe640 1 -- 172.21.15.29:0/3260176267 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f64bc005020 con 0x7f64cc106950 2024-09-21T02:29:42.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.206+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f64cc02e850 con 0x7f64cc106950 2024-09-21T02:29:42.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.209+0000 7f64c37fe640 1 -- 172.21.15.29:0/3260176267 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v13)=0 v13) ==== 121+0+0 (secure 0 0 0) 0x7f64bc021730 con 0x7f64cc106950 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.211+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64b803fb20 msgr2=0x7f64b8042030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.211+0000 7f64d45ed640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64b803fb20 0x7f64b8042030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64c8005ab0 tx=0x7f64c8005a20 comp rx=0 tx=0).stop 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc106950 msgr2=0x7f64cc19a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc106950 0x7f64cc19a3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00e9f0 tx=0x7f64bc00edf0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 shutdown_connections 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64b803fb20 0x7f64b8042030 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc109490 0x7f64cc19af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64cc106950 0x7f64cc19a3a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 --2- 172.21.15.29:0/3260176267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64cc06b900 0x7f64cc199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 >> 172.21.15.29:0/3260176267 conn(0x7f64cc0fdfb0 msgr2=0x7f64cc104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 shutdown_connections 2024-09-21T02:29:42.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:42.212+0000 7f64d45ed640 1 -- 172.21.15.29:0/3260176267 wait complete. 2024-09-21T02:29:42.259 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Sep 21 02:29:41 smithi066 systemd[1]: Started Ceph mgr.y for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:29:43.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:43 smithi029 ceph-mon[31006]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:43.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:43.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:43 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3260176267' entity='client.admin' 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 -- 172.21.15.29:0/2994930323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 msgr2=0x7f34a410bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2994930323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 0x7f34a410bbb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f349c009ca0 tx=0x7f349c02a950 comp rx=0 tx=0).stop 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 -- 172.21.15.29:0/2994930323 shutdown_connections 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2994930323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 0x7f34a410bbb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2994930323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a4104870 0x7f34a4104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2994930323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34a41037e0 0x7f34a4103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 -- 172.21.15.29:0/2994930323 >> 172.21.15.29:0/2994930323 conn(0x7f34a40ff150 msgr2=0x7f34a4101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 -- 172.21.15.29:0/2994930323 shutdown_connections 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.229+0000 7f34ab62e640 1 -- 172.21.15.29:0/2994930323 wait complete. 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34ab62e640 1 Processor -- start 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34ab62e640 1 -- start start 2024-09-21T02:29:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34ab62e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a41037e0 0x7f34a4076e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34ab62e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34a4104870 0x7f34a4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34ab62e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 0x7f34a4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34ab62e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34a410f040 con 0x7f34a4105210 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34ab62e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f34a410eec0 con 0x7f34a4104870 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34ab62e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34a410f1c0 con 0x7f34a41037e0 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34a9bcb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 0x7f34a4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34a93ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a41037e0 0x7f34a4076e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34a9bcb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 0x7f34a4075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53198/0 (socket says 172.21.15.29:53198) 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34a93ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a41037e0 0x7f34a4076e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51196/0 (socket says 172.21.15.29:51196) 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34a9bcb640 1 -- 172.21.15.29:0/2327046403 learned_addr learned my addr 172.21.15.29:0/2327046403 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.230+0000 7f34a8bc9640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34a4104870 0x7f34a4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34a93ca640 1 -- 172.21.15.29:0/2327046403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34a4104870 msgr2=0x7f34a4075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34a93ca640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34a4104870 0x7f34a4075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34a93ca640 1 -- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 msgr2=0x7f34a4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34a93ca640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 0x7f34a4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34a93ca640 1 -- 172.21.15.29:0/2327046403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a4076350 con 0x7f34a41037e0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34a9bcb640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 0x7f34a4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34a8bc9640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34a4104870 0x7f34a4075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34a93ca640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a41037e0 0x7f34a4076e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f34a000e9f0 tx=0x7f34a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f349a7fc640 1 -- 172.21.15.29:0/2327046403 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a000ccc0 con 0x7f34a41037e0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a41a3d80 con 0x7f34a41037e0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f349a7fc640 1 -- 172.21.15.29:0/2327046403 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f34a0010880 con 0x7f34a41037e0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f349a7fc640 1 -- 172.21.15.29:0/2327046403 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a0018680 con 0x7f34a41037e0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.231+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a41a4180 con 0x7f34a41037e0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.232+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a4103c50 con 0x7f34a41037e0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.233+0000 7f349a7fc640 1 -- 172.21.15.29:0/2327046403 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 13) ==== 51898+0+0 (secure 0 0 0) 0x7f34a0010430 con 0x7f34a41037e0 2024-09-21T02:29:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.233+0000 7f349a7fc640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f349003fb20 0x7f3490042030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:43.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.233+0000 7f349a7fc640 1 -- 172.21.15.29:0/2327046403 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f34a0054c60 con 0x7f34a41037e0 2024-09-21T02:29:43.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.233+0000 7f34a8bc9640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f349003fb20 0x7f3490042030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:43.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.234+0000 7f34a8bc9640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f349003fb20 0x7f3490042030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3494009fc0 tx=0x7f3494005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:43.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.242+0000 7f349a7fc640 1 -- 172.21.15.29:0/2327046403 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f34a0010d30 con 0x7f34a41037e0 2024-09-21T02:29:43.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:43 smithi190 ceph-mon[33871]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:43.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:43 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3260176267' entity='client.admin' 2024-09-21T02:29:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.435+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f34a410a9c0 con 0x7f34a41037e0 2024-09-21T02:29:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.442+0000 7f349a7fc640 1 -- 172.21.15.29:0/2327046403 <== mon.1 v2:172.21.15.190:3300/0 7 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f34a001cd30 con 0x7f34a41037e0 2024-09-21T02:29:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.442+0000 7f349a7fc640 1 -- 172.21.15.29:0/2327046403 <== mon.1 v2:172.21.15.190:3300/0 8 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f34a00535b0 con 0x7f34a41037e0 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f349003fb20 msgr2=0x7f3490042030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f349003fb20 0x7f3490042030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3494009fc0 tx=0x7f3494005f90 comp rx=0 tx=0).stop 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a41037e0 msgr2=0x7f34a4076e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a41037e0 0x7f34a4076e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f34a000e9f0 tx=0x7f34a000eeb0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 shutdown_connections 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f349003fb20 0x7f3490042030 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a4105210 0x7f34a4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34a4104870 0x7f34a4075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 --2- 172.21.15.29:0/2327046403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a41037e0 0x7f34a4076e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:43.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 >> 172.21.15.29:0/2327046403 conn(0x7f34a40ff150 msgr2=0x7f34a4107b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:43.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 shutdown_connections 2024-09-21T02:29:43.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:43.445+0000 7f34ab62e640 1 -- 172.21.15.29:0/2327046403 wait complete. 2024-09-21T02:29:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:43 smithi066 ceph-mon[33436]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:43.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:43.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:43 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3260176267' entity='client.admin' 2024-09-21T02:29:43.748 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi029') 2024-09-21T02:29:44.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: Reconfiguring daemon mgr.x on smithi029 2024-09-21T02:29:44.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:44 smithi190 ceph-mon[33871]: from='client.? ' entity='client.admin' 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: Reconfiguring daemon mgr.x on smithi029 2024-09-21T02:29:44.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:44 smithi029 ceph-mon[31006]: from='client.? ' entity='client.admin' 2024-09-21T02:29:44.491 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi029') 2024-09-21T02:29:44.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: Reconfiguring daemon mgr.x on smithi029 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:44.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:44 smithi066 ceph-mon[33436]: from='client.? ' entity='client.admin' 2024-09-21T02:29:45.223 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:45 smithi029 ceph-mon[31006]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:45 smithi029 ceph-mon[31006]: Standby manager daemon z started 2024-09-21T02:29:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:45 smithi029 ceph-mon[31006]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T02:29:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:45 smithi029 ceph-mon[31006]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T02:29:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:45 smithi029 ceph-mon[31006]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T02:29:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:45 smithi029 ceph-mon[31006]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T02:29:45.689 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.689 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.689 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.693 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.697 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.699 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.704 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:45.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:45 smithi066 ceph-mon[33436]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:45.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:45 smithi066 ceph-mon[33436]: Standby manager daemon z started 2024-09-21T02:29:45.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:45 smithi066 ceph-mon[33436]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T02:29:45.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:45 smithi066 ceph-mon[33436]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T02:29:45.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:45 smithi066 ceph-mon[33436]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T02:29:45.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:45 smithi066 ceph-mon[33436]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T02:29:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:45 smithi190 ceph-mon[33871]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:45 smithi190 ceph-mon[33871]: Standby manager daemon z started 2024-09-21T02:29:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:45 smithi190 ceph-mon[33871]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T02:29:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:45 smithi190 ceph-mon[33871]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T02:29:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:45 smithi190 ceph-mon[33871]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T02:29:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:45 smithi190 ceph-mon[33871]: from='mgr.? 172.21.15.190:0/3185175330' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T02:29:46.740 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:46 smithi029 ceph-mon[31006]: mgrmap e14: x(active, since 67s), standbys: z 2024-09-21T02:29:46.740 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T02:29:46.740 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:46.740 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:46.740 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:46.740 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:46.740 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:46.740 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:46.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:46 smithi066 ceph-mon[33436]: mgrmap e14: x(active, since 67s), standbys: z 2024-09-21T02:29:46.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T02:29:46.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:46.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:46.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:46.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:46.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:46.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:46.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:46 smithi190 ceph-mon[33871]: mgrmap e14: x(active, since 67s), standbys: z 2024-09-21T02:29:46.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T02:29:46.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:46.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:46.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:46.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:46.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:46.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:47.541 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:47.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:47.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: Standby manager daemon y started 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2292205541' entity='client.admin' 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:47.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: Standby manager daemon y started 2024-09-21T02:29:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T02:29:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T02:29:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T02:29:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T02:29:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2292205541' entity='client.admin' 2024-09-21T02:29:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: Standby manager daemon y started 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='mgr.? 172.21.15.66:0/556825277' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2292205541' entity='client.admin' 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:47.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:47.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.878+0000 7f232bdd2640 1 -- 172.21.15.29:0/3355593445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232410b690 msgr2=0x7f232410dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.878+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3355593445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232410b690 0x7f232410dba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f231c009ca0 tx=0x7f231c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:29:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.878+0000 7f232bdd2640 1 -- 172.21.15.29:0/3355593445 shutdown_connections 2024-09-21T02:29:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.878+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3355593445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232410b690 0x7f232410dba0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.879+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3355593445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2324108ce0 0x7f232410b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.879+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3355593445 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f232406c490 0x7f23241087a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.879+0000 7f232bdd2640 1 -- 172.21.15.29:0/3355593445 >> 172.21.15.29:0/3355593445 conn(0x7f23240fdfe0 msgr2=0x7f2324100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.879+0000 7f232bdd2640 1 -- 172.21.15.29:0/3355593445 shutdown_connections 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.879+0000 7f232bdd2640 1 -- 172.21.15.29:0/3355593445 wait complete. 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.879+0000 7f232bdd2640 1 Processor -- start 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.879+0000 7f232bdd2640 1 -- start start 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.879+0000 7f232bdd2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232406c490 0x7f232419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232bdd2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2324108ce0 0x7f232419c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232bdd2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232410b690 0x7f232419d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232bdd2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2324111340 con 0x7f232406c490 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232bdd2640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f23241111c0 con 0x7f2324108ce0 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232bdd2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f23241114c0 con 0x7f232410b690 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232406c490 0x7f232419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232406c490 0x7f232419bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36766/0 (socket says 172.21.15.29:36766) 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 -- 172.21.15.29:0/3233035184 learned_addr learned my addr 172.21.15.29:0/3233035184 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232a36f640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232410b690 0x7f232419d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232936d640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2324108ce0 0x7f232419c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 -- 172.21.15.29:0/3233035184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232410b690 msgr2=0x7f232419d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232410b690 0x7f232419d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 -- 172.21.15.29:0/3233035184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2324108ce0 msgr2=0x7f232419c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2324108ce0 0x7f232419c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 -- 172.21.15.29:0/3233035184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232419d9d0 con 0x7f232406c490 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232a36f640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232410b690 0x7f232419d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232936d640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2324108ce0 0x7f232419c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f2329b6e640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232406c490 0x7f232419bfa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f232000e9f0 tx=0x7f232000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f231affd640 1 -- 172.21.15.29:0/3233035184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232000cd80 con 0x7f232406c490 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f231affd640 1 -- 172.21.15.29:0/3233035184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f23200108a0 con 0x7f232406c490 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f231affd640 1 -- 172.21.15.29:0/3233035184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320018750 con 0x7f232406c490 2024-09-21T02:29:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232419dcc0 con 0x7f232406c490 2024-09-21T02:29:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.880+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23241a43c0 con 0x7f232406c490 2024-09-21T02:29:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.881+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23240ffb00 con 0x7f232406c490 2024-09-21T02:29:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.883+0000 7f231affd640 1 -- 172.21.15.29:0/3233035184 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f2320010430 con 0x7f232406c490 2024-09-21T02:29:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.884+0000 7f231affd640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23100b6500 0x7f23100b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.884+0000 7f232936d640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23100b6500 0x7f23100b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.884+0000 7f231affd640 1 -- 172.21.15.29:0/3233035184 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f23200e5ed0 con 0x7f232406c490 2024-09-21T02:29:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.884+0000 7f232936d640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23100b6500 0x7f23100b8a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2314005d00 tx=0x7f2314009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.887+0000 7f231affd640 1 -- 172.21.15.29:0/3233035184 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f23200ae430 con 0x7f232406c490 2024-09-21T02:29:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.076+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f232402e640 con 0x7f232406c490 2024-09-21T02:29:48.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.080+0000 7f231affd640 1 -- 172.21.15.29:0/3233035184 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v16)=0 v16) ==== 119+0+0 (secure 0 0 0) 0x7f23200b2810 con 0x7f232406c490 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23100b6500 msgr2=0x7f23100b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23100b6500 0x7f23100b8a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2314005d00 tx=0x7f2314009550 comp rx=0 tx=0).stop 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232406c490 msgr2=0x7f232419bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232406c490 0x7f232419bfa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f232000e9f0 tx=0x7f232000eeb0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 shutdown_connections 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23100b6500 0x7f23100b8a10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232410b690 0x7f232419d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2324108ce0 0x7f232419c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 --2- 172.21.15.29:0/3233035184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f232406c490 0x7f232419bfa0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.082+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 >> 172.21.15.29:0/3233035184 conn(0x7f23240fdfe0 msgr2=0x7f2324109bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:48.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.083+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 shutdown_connections 2024-09-21T02:29:48.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.083+0000 7f232bdd2640 1 -- 172.21.15.29:0/3233035184 wait complete. 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.718+0000 7f246791e640 1 -- 172.21.15.29:0/1617541225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2460102640 msgr2=0x7f2460102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.718+0000 7f246791e640 1 --2- 172.21.15.29:0/1617541225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2460102640 0x7f2460102ab0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f245c009900 tx=0x7f245c02a480 comp rx=0 tx=0).stop 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 -- 172.21.15.29:0/1617541225 shutdown_connections 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 --2- 172.21.15.29:0/1617541225 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2460104070 0x7f246010aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 --2- 172.21.15.29:0/1617541225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24601036d0 0x7f2460103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 --2- 172.21.15.29:0/1617541225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2460102640 0x7f2460102ab0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 -- 172.21.15.29:0/1617541225 >> 172.21.15.29:0/1617541225 conn(0x7f24600fdfb0 msgr2=0x7f2460100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 -- 172.21.15.29:0/1617541225 shutdown_connections 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 -- 172.21.15.29:0/1617541225 wait complete. 2024-09-21T02:29:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 Processor -- start 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 -- start start 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.719+0000 7f246791e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2460102640 0x7f2460199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f246791e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24601036d0 0x7f246019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f246791e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2460104070 0x7f246019b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f246791e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f246010e020 con 0x7f24601036d0 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f246791e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f246010dea0 con 0x7f2460104070 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f246791e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f246010e1a0 con 0x7f2460102640 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24601036d0 0x7f246019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24601036d0 0x7f246019a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36800/0 (socket says 172.21.15.29:36800) 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 -- 172.21.15.29:0/1107830252 learned_addr learned my addr 172.21.15.29:0/1107830252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f24656ba640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2460102640 0x7f2460199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2465ebb640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2460104070 0x7f246019b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:48.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 -- 172.21.15.29:0/1107830252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2460102640 msgr2=0x7f2460199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2460102640 0x7f2460199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 -- 172.21.15.29:0/1107830252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2460104070 msgr2=0x7f246019b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2460104070 0x7f246019b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 -- 172.21.15.29:0/1107830252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246019b840 con 0x7f24601036d0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2465ebb640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2460104070 0x7f246019b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f2464eb9640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24601036d0 0x7f246019a480 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f245000d950 tx=0x7f245000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f24567fc640 1 -- 172.21.15.29:0/1107830252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2450017070 con 0x7f24601036d0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f24567fc640 1 -- 172.21.15.29:0/1107830252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f2450009c30 con 0x7f24601036d0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f24567fc640 1 -- 172.21.15.29:0/1107830252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245000f620 con 0x7f24601036d0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.720+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246019bb30 con 0x7f24601036d0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.721+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24601a2230 con 0x7f24601036d0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.722+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24601a2460 con 0x7f24601036d0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.723+0000 7f24567fc640 1 -- 172.21.15.29:0/1107830252 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f2450010460 con 0x7f24601036d0 2024-09-21T02:29:48.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.724+0000 7f24567fc640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f244c0b6480 0x7f244c0b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:48.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.724+0000 7f24656ba640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f244c0b6480 0x7f244c0b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:48.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.724+0000 7f24567fc640 1 -- 172.21.15.29:0/1107830252 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f24500e5340 con 0x7f24601036d0 2024-09-21T02:29:48.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.724+0000 7f24656ba640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f244c0b6480 0x7f244c0b8990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f245c006010 tx=0x7f245c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:48.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.728+0000 7f24567fc640 1 -- 172.21.15.29:0/1107830252 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f24500ad8a0 con 0x7f24601036d0 2024-09-21T02:29:48.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.618+0000 7fe595f4d640 1 -- 172.21.15.29:0/3631925581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 msgr2=0x7fe590108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3631925581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 0x7fe590108d70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe578009900 tx=0x7fe57802a410 comp rx=0 tx=0).stop 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 -- 172.21.15.29:0/3631925581 shutdown_connections 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3631925581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5901094b0 0x7fe59010b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3631925581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 0x7fe590108d70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3631925581 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe59006c6a0 0x7fe5901063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 -- 172.21.15.29:0/3631925581 >> 172.21.15.29:0/3631925581 conn(0x7fe5900fdff0 msgr2=0x7fe5901004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 -- 172.21.15.29:0/3631925581 shutdown_connections 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 -- 172.21.15.29:0/3631925581 wait complete. 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 Processor -- start 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 -- start start 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe59006c6a0 0x7fe590199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.619+0000 7fe595f4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 0x7fe59019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:48.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe595f4d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5901094b0 0x7fe59019b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe595f4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe59010f130 con 0x7fe590106910 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe595f4d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe59010efb0 con 0x7fe59006c6a0 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe595f4d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe59010f2b0 con 0x7fe5901094b0 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 0x7fe59019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 0x7fe59019a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36784/0 (socket says 172.21.15.29:36784) 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 -- 172.21.15.29:0/3287047992 learned_addr learned my addr 172.21.15.29:0/3287047992 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe58f7fe640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe59006c6a0 0x7fe590199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe58ffff640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5901094b0 0x7fe59019b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 -- 172.21.15.29:0/3287047992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5901094b0 msgr2=0x7fe59019b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5901094b0 0x7fe59019b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 -- 172.21.15.29:0/3287047992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe59006c6a0 msgr2=0x7fe590199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe59006c6a0 0x7fe590199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 -- 172.21.15.29:0/3287047992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59019b850 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe58f7fe640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe59006c6a0 0x7fe590199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe587fff640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 0x7fe59019a490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe578006010 tx=0x7fe578003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe58d7fa640 1 -- 172.21.15.29:0/3287047992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57803f070 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe59019bae0 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe58d7fa640 1 -- 172.21.15.29:0/3287047992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fe578031cf0 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe58d7fa640 1 -- 172.21.15.29:0/3287047992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe578035030 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.620+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5901a2070 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.621+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5900ffb10 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.624+0000 7fe58d7fa640 1 -- 172.21.15.29:0/3287047992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7fe578042070 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.625+0000 7fe58d7fa640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe57c0b6480 0x7fe57c0b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.625+0000 7fe58d7fa640 1 -- 172.21.15.29:0/3287047992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe578106840 con 0x7fe590106910 2024-09-21T02:29:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.625+0000 7fe58f7fe640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe57c0b6480 0x7fe57c0b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.626+0000 7fe58f7fe640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe57c0b6480 0x7fe57c0b8990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe5800058c0 tx=0x7fe580005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.629+0000 7fe58d7fa640 1 -- 172.21.15.29:0/3287047992 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe5780ceda0 con 0x7fe590106910 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.821+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fe59002e640 con 0x7fe590106910 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.826+0000 7fe58d7fa640 1 -- 172.21.15.29:0/3287047992 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v17)=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7fe5780d3180 con 0x7fe590106910 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe57c0b6480 msgr2=0x7fe57c0b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe57c0b6480 0x7fe57c0b8990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe5800058c0 tx=0x7fe580005ec0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 msgr2=0x7fe59019a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 0x7fe59019a490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe578006010 tx=0x7fe578003d50 comp rx=0 tx=0).stop 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 shutdown_connections 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe57c0b6480 0x7fe57c0b8990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5901094b0 0x7fe59019b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe590106910 0x7fe59019a490 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 --2- 172.21.15.29:0/3287047992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe59006c6a0 0x7fe590199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 >> 172.21.15.29:0/3287047992 conn(0x7fe5900fdff0 msgr2=0x7fe590104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:48.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 shutdown_connections 2024-09-21T02:29:48.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.828+0000 7fe595f4d640 1 -- 172.21.15.29:0/3287047992 wait complete. 2024-09-21T02:29:48.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.928+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f24601a2630 con 0x7f24601036d0 2024-09-21T02:29:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.932+0000 7f24567fc640 1 -- 172.21.15.29:0/1107830252 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) ==== 116+0+0 (secure 0 0 0) 0x7f2450009de0 con 0x7f24601036d0 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f244c0b6480 msgr2=0x7f244c0b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f244c0b6480 0x7f244c0b8990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f245c006010 tx=0x7f245c000bc0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24601036d0 msgr2=0x7f246019a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24601036d0 0x7f246019a480 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f245000d950 tx=0x7f245000de10 comp rx=0 tx=0).stop 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 shutdown_connections 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f244c0b6480 0x7f244c0b8990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2460104070 0x7f246019b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24601036d0 0x7f246019a480 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 --2- 172.21.15.29:0/1107830252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2460102640 0x7f2460199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.934+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 >> 172.21.15.29:0/1107830252 conn(0x7f24600fdfb0 msgr2=0x7f24600ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.935+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 shutdown_connections 2024-09-21T02:29:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:48.935+0000 7f246791e640 1 -- 172.21.15.29:0/1107830252 wait complete. 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: mgrmap e15: x(active, since 70s), standbys: z, y 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3233035184' entity='client.admin' 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:48 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3287047992' entity='client.admin' 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: mgrmap e15: x(active, since 70s), standbys: z, y 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3233035184' entity='client.admin' 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:48 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3287047992' entity='client.admin' 2024-09-21T02:29:49.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: mgrmap e15: x(active, since 70s), standbys: z, y 2024-09-21T02:29:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T02:29:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3233035184' entity='client.admin' 2024-09-21T02:29:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:29:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:29:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:29:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:29:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:48 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3287047992' entity='client.admin' 2024-09-21T02:29:49.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.360+0000 7f8431d4e640 1 -- 172.21.15.29:0/3953167055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 msgr2=0x7f842c102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.360+0000 7f8431d4e640 1 --2- 172.21.15.29:0/3953167055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 0x7f842c102270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f841c009ca0 tx=0x7f841c02a950 comp rx=0 tx=0).stop 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.360+0000 7f8431d4e640 1 -- 172.21.15.29:0/3953167055 shutdown_connections 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.360+0000 7f8431d4e640 1 --2- 172.21.15.29:0/3953167055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f842c103830 0x7f842c10a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.361+0000 7f8431d4e640 1 --2- 172.21.15.29:0/3953167055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f842c102e90 0x7f842c1032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.361+0000 7f8431d4e640 1 --2- 172.21.15.29:0/3953167055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 0x7f842c102270 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.361+0000 7f8431d4e640 1 -- 172.21.15.29:0/3953167055 >> 172.21.15.29:0/3953167055 conn(0x7f842c0fd770 msgr2=0x7f842c0ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.361+0000 7f8431d4e640 1 -- 172.21.15.29:0/3953167055 shutdown_connections 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.361+0000 7f8431d4e640 1 -- 172.21.15.29:0/3953167055 wait complete. 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.361+0000 7f8431d4e640 1 Processor -- start 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.361+0000 7f8431d4e640 1 -- start start 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8431d4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 0x7f842c199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8431d4e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f842c102e90 0x7f842c19a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8431d4e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f842c103830 0x7f842c19b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8431d4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f842c10cf90 con 0x7f842c101e00 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8431d4e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f842c10ce10 con 0x7f842c103830 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8431d4e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f842c10d110 con 0x7f842c102e90 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 0x7f842c199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 0x7f842c199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36808/0 (socket says 172.21.15.29:36808) 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 -- 172.21.15.29:0/2661211266 learned_addr learned my addr 172.21.15.29:0/2661211266 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842affd640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f842c102e90 0x7f842c19a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842bfff640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f842c103830 0x7f842c19b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 -- 172.21.15.29:0/2661211266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f842c102e90 msgr2=0x7f842c19a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f842c102e90 0x7f842c19a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 -- 172.21.15.29:0/2661211266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f842c103830 msgr2=0x7f842c19b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f842c103830 0x7f842c19b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 -- 172.21.15.29:0/2661211266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842c19b890 con 0x7f842c101e00 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f842b7fe640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 0x7f842c199e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f841c004af0 tx=0x7f841c02fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8428ff9640 1 -- 172.21.15.29:0/2661211266 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c034040 con 0x7f842c101e00 2024-09-21T02:29:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8428ff9640 1 -- 172.21.15.29:0/2661211266 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f841c003de0 con 0x7f842c101e00 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.362+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842c19bb20 con 0x7f842c101e00 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.363+0000 7f8428ff9640 1 -- 172.21.15.29:0/2661211266 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c0047c0 con 0x7f842c101e00 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.363+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f842c1a2280 con 0x7f842c101e00 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.363+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f842c102270 con 0x7f842c101e00 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.365+0000 7f8428ff9640 1 -- 172.21.15.29:0/2661211266 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f841c037020 con 0x7f842c101e00 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.366+0000 7f8428ff9640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f84180b6500 0x7f84180b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.366+0000 7f842affd640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f84180b6500 0x7f84180b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.366+0000 7f8428ff9640 1 -- 172.21.15.29:0/2661211266 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f841c1067f0 con 0x7f842c101e00 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.367+0000 7f842affd640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f84180b6500 0x7f84180b8a10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8414009890 tx=0x7f8414008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.369+0000 7f8428ff9640 1 -- 172.21.15.29:0/2661211266 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f841c0cee70 con 0x7f842c101e00 2024-09-21T02:29:49.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.561+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7f842c1a24f0 con 0x7f842c101e00 2024-09-21T02:29:49.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.561+0000 7f8428ff9640 1 -- 172.21.15.29:0/2661211266 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v18)=0 v18) ==== 108+0+0 (secure 0 0 0) 0x7f841c0d3250 con 0x7f842c101e00 2024-09-21T02:29:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f84180b6500 msgr2=0x7f84180b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f84180b6500 0x7f84180b8a10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8414009890 tx=0x7f8414008040 comp rx=0 tx=0).stop 2024-09-21T02:29:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 msgr2=0x7f842c199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 0x7f842c199e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f841c004af0 tx=0x7f841c02fad0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 shutdown_connections 2024-09-21T02:29:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f84180b6500 0x7f84180b8a10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f842c103830 0x7f842c19b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f842c102e90 0x7f842c19a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 --2- 172.21.15.29:0/2661211266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f842c101e00 0x7f842c199e60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 >> 172.21.15.29:0/2661211266 conn(0x7f842c0fd770 msgr2=0x7f842c1083c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:49.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 shutdown_connections 2024-09-21T02:29:49.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.564+0000 7f8431d4e640 1 -- 172.21.15.29:0/2661211266 wait complete. 2024-09-21T02:29:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.590+0000 7f1c00f2d640 1 -- 172.21.15.29:0/3244264600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc102670 msgr2=0x7f1bfc102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.590+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/3244264600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc102670 0x7f1bfc102ae0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009900 tx=0x7f1bec02a480 comp rx=0 tx=0).stop 2024-09-21T02:29:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.590+0000 7f1c00f2d640 1 -- 172.21.15.29:0/3244264600 shutdown_connections 2024-09-21T02:29:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.590+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/3244264600 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bfc1040a0 0x7f1bfc10aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.590+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/3244264600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bfc103700 0x7f1bfc103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.590+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/3244264600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc102670 0x7f1bfc102ae0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.590+0000 7f1c00f2d640 1 -- 172.21.15.29:0/3244264600 >> 172.21.15.29:0/3244264600 conn(0x7f1bfc0fdfe0 msgr2=0x7f1bfc100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.591+0000 7f1c00f2d640 1 -- 172.21.15.29:0/3244264600 shutdown_connections 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.591+0000 7f1c00f2d640 1 -- 172.21.15.29:0/3244264600 wait complete. 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.591+0000 7f1c00f2d640 1 Processor -- start 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1c00f2d640 1 -- start start 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1c00f2d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bfc102670 0x7f1bfc198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1c00f2d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bfc103700 0x7f1bfc199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1c00f2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc1040a0 0x7f1bfc199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1c00f2d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc10e000 con 0x7f1bfc1040a0 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1c00f2d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc10de80 con 0x7f1bfc102670 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1c00f2d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc10e180 con 0x7f1bfc103700 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1bfad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc1040a0 0x7f1bfc199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1bfad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc1040a0 0x7f1bfc199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36836/0 (socket says 172.21.15.29:36836) 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1bfad9d640 1 -- 172.21.15.29:0/1388832552 learned_addr learned my addr 172.21.15.29:0/1388832552 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.592+0000 7f1bfa59c640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bfc102670 0x7f1bfc198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bfa59c640 1 -- 172.21.15.29:0/1388832552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bfc103700 msgr2=0x7f1bfc199380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bfa59c640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bfc103700 0x7f1bfc199380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bfa59c640 1 -- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc1040a0 msgr2=0x7f1bfc199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bfa59c640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc1040a0 0x7f1bfc199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bfa59c640 1 -- 172.21.15.29:0/1388832552 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bfc19a740 con 0x7f1bfc102670 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bfad9d640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc1040a0 0x7f1bfc199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bfa59c640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bfc102670 0x7f1bfc198d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1bec005940 tx=0x7f1bec003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bf37fe640 1 -- 172.21.15.29:0/1388832552 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec03f070 con 0x7f1bfc102670 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bfc19a9d0 con 0x7f1bfc102670 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.593+0000 7f1bf37fe640 1 -- 172.21.15.29:0/1388832552 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f1bec031af0 con 0x7f1bfc102670 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.594+0000 7f1bf37fe640 1 -- 172.21.15.29:0/1388832552 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec035030 con 0x7f1bfc102670 2024-09-21T02:29:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.594+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bfc1a1130 con 0x7f1bfc102670 2024-09-21T02:29:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.595+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bfc102ae0 con 0x7f1bfc102670 2024-09-21T02:29:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.596+0000 7f1bf37fe640 1 -- 172.21.15.29:0/1388832552 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f1bec042070 con 0x7f1bfc102670 2024-09-21T02:29:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.598+0000 7f1bf37fe640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1be80b6500 0x7f1be80b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.599+0000 7f1bf3fff640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1be80b6500 0x7f1be80b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.599+0000 7f1bf37fe640 1 -- 172.21.15.29:0/1388832552 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1bec1060b0 con 0x7f1bfc102670 2024-09-21T02:29:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.599+0000 7f1bf3fff640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1be80b6500 0x7f1be80b8a10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009a60 tx=0x7f1be4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.602+0000 7f1bf37fe640 1 -- 172.21.15.29:0/1388832552 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1bec0ce570 con 0x7f1bfc102670 2024-09-21T02:29:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.793+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f1bfc109850 con 0x7f1bfc102670 2024-09-21T02:29:49.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.799+0000 7f1bf37fe640 1 -- 172.21.15.29:0/1388832552 <== mon.2 v2:172.21.15.66:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v19)=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f1bec0d2950 con 0x7f1bfc102670 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1be80b6500 msgr2=0x7f1be80b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1be80b6500 0x7f1be80b8a10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009a60 tx=0x7f1be4005f90 comp rx=0 tx=0).stop 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bfc102670 msgr2=0x7f1bfc198d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bfc102670 0x7f1bfc198d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1bec005940 tx=0x7f1bec003c10 comp rx=0 tx=0).stop 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 shutdown_connections 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1be80b6500 0x7f1be80b8a10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bfc1040a0 0x7f1bfc199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bfc103700 0x7f1bfc199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 --2- 172.21.15.29:0/1388832552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bfc102670 0x7f1bfc198d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:49.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 >> 172.21.15.29:0/1388832552 conn(0x7f1bfc0fdfe0 msgr2=0x7f1bfc106a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:49.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 shutdown_connections 2024-09-21T02:29:49.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:49.802+0000 7f1c00f2d640 1 -- 172.21.15.29:0/1388832552 wait complete. 2024-09-21T02:29:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:49 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1107830252' entity='client.admin' 2024-09-21T02:29:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:49 smithi029 ceph-mon[31006]: from='client.? ' entity='client.admin' 2024-09-21T02:29:50.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:49 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1107830252' entity='client.admin' 2024-09-21T02:29:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:49 smithi066 ceph-mon[33436]: from='client.? ' entity='client.admin' 2024-09-21T02:29:50.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:49 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1107830252' entity='client.admin' 2024-09-21T02:29:50.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:49 smithi190 ceph-mon[33871]: from='client.? ' entity='client.admin' 2024-09-21T02:29:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 -- 172.21.15.29:0/2239213220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec41068e0 msgr2=0x7fcec4108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 --2- 172.21.15.29:0/2239213220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec41068e0 0x7fcec4108d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fceac009ca0 tx=0x7fceac02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 -- 172.21.15.29:0/2239213220 shutdown_connections 2024-09-21T02:29:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 --2- 172.21.15.29:0/2239213220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec4109480 0x7fcec410b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 --2- 172.21.15.29:0/2239213220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec41068e0 0x7fcec4108d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 --2- 172.21.15.29:0/2239213220 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcec406c660 0x7fcec41063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 -- 172.21.15.29:0/2239213220 >> 172.21.15.29:0/2239213220 conn(0x7fcec40fdfe0 msgr2=0x7fcec4100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 -- 172.21.15.29:0/2239213220 shutdown_connections 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.234+0000 7fcecaea3640 1 -- 172.21.15.29:0/2239213220 wait complete. 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcecaea3640 1 Processor -- start 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcecaea3640 1 -- start start 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcecaea3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec406c660 0x7fcec4199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcecaea3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcec41068e0 0x7fcec419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcecaea3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec4109480 0x7fcec419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcecaea3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410f0e0 con 0x7fcec4109480 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcecaea3640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410ef60 con 0x7fcec41068e0 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcecaea3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410f260 con 0x7fcec406c660 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec9440640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec4109480 0x7fcec419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec9440640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec4109480 0x7fcec419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36862/0 (socket says 172.21.15.29:36862) 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec9440640 1 -- 172.21.15.29:0/1865165857 learned_addr learned my addr 172.21.15.29:0/1865165857 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec8c3f640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec406c660 0x7fcec4199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec3fff640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcec41068e0 0x7fcec419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec9440640 1 -- 172.21.15.29:0/1865165857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec406c660 msgr2=0x7fcec4199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec9440640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec406c660 0x7fcec4199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec9440640 1 -- 172.21.15.29:0/1865165857 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcec41068e0 msgr2=0x7fcec419a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec9440640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcec41068e0 0x7fcec419a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.235+0000 7fcec9440640 1 -- 172.21.15.29:0/1865165857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec419b8f0 con 0x7fcec4109480 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.236+0000 7fcec9440640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec4109480 0x7fcec419b0d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00d950 tx=0x7fcebc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.236+0000 7fcec1ffb640 1 -- 172.21.15.29:0/1865165857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc017030 con 0x7fcec4109480 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.236+0000 7fcec1ffb640 1 -- 172.21.15.29:0/1865165857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fcebc004020 con 0x7fcec4109480 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.236+0000 7fcec1ffb640 1 -- 172.21.15.29:0/1865165857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc0039c0 con 0x7fcec4109480 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.236+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec419bbe0 con 0x7fcec4109480 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.236+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41a22e0 con 0x7fcec4109480 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.237+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec40ffb00 con 0x7fcec4109480 2024-09-21T02:29:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.238+0000 7fcec1ffb640 1 -- 172.21.15.29:0/1865165857 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7fcebc01a070 con 0x7fcec4109480 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.239+0000 7fcec1ffb640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fceb00b6500 0x7fceb00b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.239+0000 7fcec8c3f640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fceb00b6500 0x7fceb00b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.239+0000 7fcec1ffb640 1 -- 172.21.15.29:0/1865165857 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcebc0e4ce0 con 0x7fcec4109480 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.240+0000 7fcec8c3f640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fceb00b6500 0x7fceb00b8a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fceb4005d00 tx=0x7fceb400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.244+0000 7fcec1ffb640 1 -- 172.21.15.29:0/1865165857 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcebc021070 con 0x7fcec4109480 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.438+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fcec402e640 con 0x7fcec4109480 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.439+0000 7fcec1ffb640 1 -- 172.21.15.29:0/1865165857 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v19)=0 v19) ==== 106+0+0 (secure 0 0 0) 0x7fcebc0ad240 con 0x7fcec4109480 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fceb00b6500 msgr2=0x7fceb00b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fceb00b6500 0x7fceb00b8a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fceb4005d00 tx=0x7fceb400b410 comp rx=0 tx=0).stop 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec4109480 msgr2=0x7fcec419b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec4109480 0x7fcec419b0d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00d950 tx=0x7fcebc00de10 comp rx=0 tx=0).stop 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 shutdown_connections 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fceb00b6500 0x7fceb00b8a10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcec4109480 0x7fcec419b0d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcec41068e0 0x7fcec419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 --2- 172.21.15.29:0/1865165857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec406c660 0x7fcec4199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 >> 172.21.15.29:0/1865165857 conn(0x7fcec40fdfe0 msgr2=0x7fcec4104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:50.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 shutdown_connections 2024-09-21T02:29:50.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.441+0000 7fcecaea3640 1 -- 172.21.15.29:0/1865165857 wait complete. 2024-09-21T02:29:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 -- 172.21.15.29:0/964708921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44105180 msgr2=0x7f8c4410bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/964708921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44105180 0x7f8c4410bb20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009ca0 tx=0x7f8c3c02a950 comp rx=0 tx=0).stop 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 -- 172.21.15.29:0/964708921 shutdown_connections 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/964708921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44105180 0x7f8c4410bb20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/964708921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c441047e0 0x7f8c44104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/964708921 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c44103750 0x7f8c44103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 -- 172.21.15.29:0/964708921 >> 172.21.15.29:0/964708921 conn(0x7f8c440ff0c0 msgr2=0x7f8c44101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 -- 172.21.15.29:0/964708921 shutdown_connections 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 -- 172.21.15.29:0/964708921 wait complete. 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 Processor -- start 2024-09-21T02:29:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.953+0000 7f8c49dbe640 1 -- start start 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c49dbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44103750 0x7f8c44076e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c49dbe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c441047e0 0x7f8c44075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c49dbe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c44105180 0x7f8c44075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c49dbe640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4410f1c0 con 0x7f8c44103750 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c49dbe640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4410f040 con 0x7f8c44105180 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c49dbe640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4410f340 con 0x7f8c441047e0 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44103750 0x7f8c44076e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44103750 0x7f8c44076e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36874/0 (socket says 172.21.15.29:36874) 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c437fe640 1 -- 172.21.15.29:0/1166074878 learned_addr learned my addr 172.21.15.29:0/1166074878 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c43fff640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c44105180 0x7f8c44075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c437fe640 1 -- 172.21.15.29:0/1166074878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c441047e0 msgr2=0x7f8c44075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c437fe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c441047e0 0x7f8c44075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c437fe640 1 -- 172.21.15.29:0/1166074878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c44105180 msgr2=0x7f8c44075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c437fe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c44105180 0x7f8c44075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c437fe640 1 -- 172.21.15.29:0/1166074878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c44076350 con 0x7f8c44103750 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.954+0000 7f8c43fff640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c44105180 0x7f8c44075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.955+0000 7f8c437fe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44103750 0x7f8c44076e40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400d950 tx=0x7f8c3400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.955+0000 7f8c417fa640 1 -- 172.21.15.29:0/1166074878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c34017070 con 0x7f8c44103750 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.955+0000 7f8c417fa640 1 -- 172.21.15.29:0/1166074878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f8c34009c30 con 0x7f8c44103750 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.955+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c441a3eb0 con 0x7f8c44103750 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.955+0000 7f8c417fa640 1 -- 172.21.15.29:0/1166074878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400f620 con 0x7f8c44103750 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.955+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c441a4250 con 0x7f8c44103750 2024-09-21T02:29:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.956+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c44103bc0 con 0x7f8c44103750 2024-09-21T02:29:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.958+0000 7f8c417fa640 1 -- 172.21.15.29:0/1166074878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f8c34010460 con 0x7f8c44103750 2024-09-21T02:29:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.959+0000 7f8c417fa640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c300b6480 0x7f8c300b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.959+0000 7f8c3affd640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c300b6480 0x7f8c300b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.959+0000 7f8c417fa640 1 -- 172.21.15.29:0/1166074878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8c340e5300 con 0x7f8c44103750 2024-09-21T02:29:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.959+0000 7f8c3affd640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c300b6480 0x7f8c300b8990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c009710 tx=0x7f8c2c006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:50.962+0000 7f8c417fa640 1 -- 172.21.15.29:0/1166074878 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8c340ad860 con 0x7f8c44103750 2024-09-21T02:29:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:50 smithi029 ceph-mon[31006]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:51.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.156+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f8c4410a930 con 0x7f8c44103750 2024-09-21T02:29:51.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.160+0000 7f8c417fa640 1 -- 172.21.15.29:0/1166074878 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v20)=0 v20) ==== 108+0+0 (secure 0 0 0) 0x7f8c34009de0 con 0x7f8c44103750 2024-09-21T02:29:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.162+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c300b6480 msgr2=0x7f8c300b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c300b6480 0x7f8c300b8990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c009710 tx=0x7f8c2c006c90 comp rx=0 tx=0).stop 2024-09-21T02:29:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44103750 msgr2=0x7f8c44076e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44103750 0x7f8c44076e40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400d950 tx=0x7f8c3400de10 comp rx=0 tx=0).stop 2024-09-21T02:29:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 shutdown_connections 2024-09-21T02:29:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c300b6480 0x7f8c300b8990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c44105180 0x7f8c44075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c441047e0 0x7f8c44075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 --2- 172.21.15.29:0/1166074878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c44103750 0x7f8c44076e40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 >> 172.21.15.29:0/1166074878 conn(0x7f8c440ff0c0 msgr2=0x7f8c44107b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 shutdown_connections 2024-09-21T02:29:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.163+0000 7f8c49dbe640 1 -- 172.21.15.29:0/1166074878 wait complete. 2024-09-21T02:29:51.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:50 smithi066 ceph-mon[33436]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:51.275 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:50 smithi190 ceph-mon[33871]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:52.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:52 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1166074878' entity='client.admin' 2024-09-21T02:29:52.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:52 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4163632565' entity='client.admin' 2024-09-21T02:29:52.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:52 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1166074878' entity='client.admin' 2024-09-21T02:29:52.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:52 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4163632565' entity='client.admin' 2024-09-21T02:29:52.545 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:52 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1166074878' entity='client.admin' 2024-09-21T02:29:52.546 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:52 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4163632565' entity='client.admin' 2024-09-21T02:29:53.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:53 smithi029 ceph-mon[31006]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:53.499 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:53 smithi190 ceph-mon[33871]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:53.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:53 smithi066 ceph-mon[33436]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:54.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 -- 172.21.15.29:0/4114743521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 msgr2=0x7fbb80101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:54.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 --2- 172.21.15.29:0/4114743521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 0x7fbb80101f10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009900 tx=0x7fbb7002a410 comp rx=0 tx=0).stop 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 -- 172.21.15.29:0/4114743521 shutdown_connections 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 --2- 172.21.15.29:0/4114743521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb80109500 0x7fbb8010b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 --2- 172.21.15.29:0/4114743521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80106ba0 0x7fbb80108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 --2- 172.21.15.29:0/4114743521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 0x7fbb80101f10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 -- 172.21.15.29:0/4114743521 >> 172.21.15.29:0/4114743521 conn(0x7fbb800fd770 msgr2=0x7fbb800ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 -- 172.21.15.29:0/4114743521 shutdown_connections 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 -- 172.21.15.29:0/4114743521 wait complete. 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.065+0000 7fbb85360640 1 Processor -- start 2024-09-21T02:29:54.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb85360640 1 -- start start 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb85360640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 0x7fbb801041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb85360640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb80106ba0 0x7fbb80102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb85360640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80109500 0x7fbb80102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb85360640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8010f230 con 0x7fbb80101b00 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb85360640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8010f0b0 con 0x7fbb80106ba0 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb85360640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8010f3b0 con 0x7fbb80109500 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 0x7fbb801041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 0x7fbb801041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36932/0 (socket says 172.21.15.29:36932) 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7effd640 1 -- 172.21.15.29:0/814062198 learned_addr learned my addr 172.21.15.29:0/814062198 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:54.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7f7fe640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80109500 0x7fbb80102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7e7fc640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb80106ba0 0x7fbb80102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7effd640 1 -- 172.21.15.29:0/814062198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80109500 msgr2=0x7fbb80102fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7effd640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80109500 0x7fbb80102fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7effd640 1 -- 172.21.15.29:0/814062198 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb80106ba0 msgr2=0x7fbb80102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7effd640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb80106ba0 0x7fbb80102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.066+0000 7fbb7effd640 1 -- 172.21.15.29:0/814062198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb801036e0 con 0x7fbb80101b00 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.067+0000 7fbb7effd640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 0x7fbb801041d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009230 tx=0x7fbb70005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.067+0000 7fbb67fff640 1 -- 172.21.15.29:0/814062198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7003f030 con 0x7fbb80101b00 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.067+0000 7fbb67fff640 1 -- 172.21.15.29:0/814062198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb700043d0 con 0x7fbb80101b00 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.067+0000 7fbb67fff640 1 -- 172.21.15.29:0/814062198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70035070 con 0x7fbb80101b00 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.067+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb801a3f50 con 0x7fbb80101b00 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.067+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb801a42f0 con 0x7fbb80101b00 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.068+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb800ff290 con 0x7fbb80101b00 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.070+0000 7fbb67fff640 1 -- 172.21.15.29:0/814062198 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7fbb70042030 con 0x7fbb80101b00 2024-09-21T02:29:54.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.071+0000 7fbb67fff640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbb680b6480 0x7fbb680b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:54.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.071+0000 7fbb7e7fc640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbb680b6480 0x7fbb680b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:54.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.071+0000 7fbb67fff640 1 -- 172.21.15.29:0/814062198 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbb70105a60 con 0x7fbb80101b00 2024-09-21T02:29:54.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.071+0000 7fbb7e7fc640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbb680b6480 0x7fbb680b8990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009bd0 tx=0x7fbb74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:54.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.075+0000 7fbb67fff640 1 -- 172.21.15.29:0/814062198 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbb700d23a0 con 0x7fbb80101b00 2024-09-21T02:29:54.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.275+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7fbb8002e640 con 0x7fbb80101b00 2024-09-21T02:29:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.279+0000 7fbb67fff640 1 -- 172.21.15.29:0/814062198 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v24)=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7fbb700d72e0 con 0x7fbb80101b00 2024-09-21T02:29:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbb680b6480 msgr2=0x7fbb680b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbb680b6480 0x7fbb680b8990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009bd0 tx=0x7fbb74005f90 comp rx=0 tx=0).stop 2024-09-21T02:29:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 msgr2=0x7fbb801041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 0x7fbb801041d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009230 tx=0x7fbb70005e80 comp rx=0 tx=0).stop 2024-09-21T02:29:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 shutdown_connections 2024-09-21T02:29:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbb680b6480 0x7fbb680b8990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80109500 0x7fbb80102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb80106ba0 0x7fbb80102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 --2- 172.21.15.29:0/814062198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb80101b00 0x7fbb801041d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:54.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 >> 172.21.15.29:0/814062198 conn(0x7fbb800fd770 msgr2=0x7fbb80107a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:54.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 shutdown_connections 2024-09-21T02:29:54.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.282+0000 7fbb85360640 1 -- 172.21.15.29:0/814062198 wait complete. 2024-09-21T02:29:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:55 smithi066 ceph-mon[33436]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:55 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1175512591' entity='client.admin' 2024-09-21T02:29:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:55 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1273463919' entity='client.admin' 2024-09-21T02:29:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:55 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/814062198' entity='client.admin' 2024-09-21T02:29:55.296 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:55 smithi029 ceph-mon[31006]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:55.296 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:55 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1175512591' entity='client.admin' 2024-09-21T02:29:55.296 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:55 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1273463919' entity='client.admin' 2024-09-21T02:29:55.296 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:55 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/814062198' entity='client.admin' 2024-09-21T02:29:55.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:55 smithi190 ceph-mon[33871]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:55.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:55 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1175512591' entity='client.admin' 2024-09-21T02:29:55.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:55 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1273463919' entity='client.admin' 2024-09-21T02:29:55.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:55 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/814062198' entity='client.admin' 2024-09-21T02:29:56.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 -- 172.21.15.29:0/463725396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 msgr2=0x7f2e1c10bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 --2- 172.21.15.29:0/463725396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 0x7f2e1c10bb20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009900 tx=0x7f2e1402a480 comp rx=0 tx=0).stop 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 -- 172.21.15.29:0/463725396 shutdown_connections 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 --2- 172.21.15.29:0/463725396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 0x7f2e1c10bb20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 --2- 172.21.15.29:0/463725396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e1c1047e0 0x7f2e1c104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 --2- 172.21.15.29:0/463725396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103750 0x7f2e1c103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 -- 172.21.15.29:0/463725396 >> 172.21.15.29:0/463725396 conn(0x7f2e1c0ff0c0 msgr2=0x7f2e1c101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 -- 172.21.15.29:0/463725396 shutdown_connections 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 -- 172.21.15.29:0/463725396 wait complete. 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 Processor -- start 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.122+0000 7f2e24037640 1 -- start start 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e24037640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e1c103750 0x7f2e1c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e24037640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c1047e0 0x7f2e1c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e24037640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 0x7f2e1c19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:56.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e24037640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c10f250 con 0x7f2e1c105180 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e24037640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c10f0d0 con 0x7f2e1c103750 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e24037640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c10f3d0 con 0x7f2e1c1047e0 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 0x7f2e1c19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 0x7f2e1c19b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36076/0 (socket says 172.21.15.29:36076) 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 -- 172.21.15.29:0/3599450738 learned_addr learned my addr 172.21.15.29:0/3599450738 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e215d2640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c1047e0 0x7f2e1c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e21dd3640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e1c103750 0x7f2e1c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 -- 172.21.15.29:0/3599450738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c1047e0 msgr2=0x7f2e1c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c1047e0 0x7f2e1c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 -- 172.21.15.29:0/3599450738 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e1c103750 msgr2=0x7f2e1c199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e1c103750 0x7f2e1c199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 -- 172.21.15.29:0/3599450738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c19b870 con 0x7f2e1c105180 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.123+0000 7f2e225d4640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 0x7f2e1c19b0e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2e14005ab0 tx=0x7f2e140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:56.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.124+0000 7f2e1affd640 1 -- 172.21.15.29:0/3599450738 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1403f030 con 0x7f2e1c105180 2024-09-21T02:29:56.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.124+0000 7f2e1affd640 1 -- 172.21.15.29:0/3599450738 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e14003c90 con 0x7f2e1c105180 2024-09-21T02:29:56.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.124+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1c19bb00 con 0x7f2e1c105180 2024-09-21T02:29:56.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.124+0000 7f2e1affd640 1 -- 172.21.15.29:0/3599450738 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14035070 con 0x7f2e1c105180 2024-09-21T02:29:56.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.124+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e1c1a2120 con 0x7f2e1c105180 2024-09-21T02:29:56.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.125+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e1c10a930 con 0x7f2e1c105180 2024-09-21T02:29:56.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.126+0000 7f2e1affd640 1 -- 172.21.15.29:0/3599450738 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f2e14042030 con 0x7f2e1c105180 2024-09-21T02:29:56.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.127+0000 7f2e1affd640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2e040b6500 0x7f2e040b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:56.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.127+0000 7f2e21dd3640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2e040b6500 0x7f2e040b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:56.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.127+0000 7f2e1affd640 1 -- 172.21.15.29:0/3599450738 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2e14105ec0 con 0x7f2e1c105180 2024-09-21T02:29:56.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.128+0000 7f2e21dd3640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2e040b6500 0x7f2e040b8a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c005fd0 tx=0x7f2e0c009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:56.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.132+0000 7f2e1affd640 1 -- 172.21.15.29:0/3599450738 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2e140ce420 con 0x7f2e1c105180 2024-09-21T02:29:56.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.321+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f2e1c02e640 con 0x7f2e1c105180 2024-09-21T02:29:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.325+0000 7f2e1affd640 1 -- 172.21.15.29:0/3599450738 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v26)=0 v26) ==== 106+0+0 (secure 0 0 0) 0x7f2e140d2800 con 0x7f2e1c105180 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.327+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2e040b6500 msgr2=0x7f2e040b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.327+0000 7f2e24037640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2e040b6500 0x7f2e040b8a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c005fd0 tx=0x7f2e0c009680 comp rx=0 tx=0).stop 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 msgr2=0x7f2e1c19b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 0x7f2e1c19b0e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2e14005ab0 tx=0x7f2e140042f0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 shutdown_connections 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2e040b6500 0x7f2e040b8a10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e1c105180 0x7f2e1c19b0e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c1047e0 0x7f2e1c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 --2- 172.21.15.29:0/3599450738 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e1c103750 0x7f2e1c199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 >> 172.21.15.29:0/3599450738 conn(0x7f2e1c0ff0c0 msgr2=0x7f2e1c107b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:56.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 shutdown_connections 2024-09-21T02:29:56.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:56.328+0000 7f2e24037640 1 -- 172.21.15.29:0/3599450738 wait complete. 2024-09-21T02:29:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 -- 172.21.15.29:0/3517564642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 msgr2=0x7ff0d0101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 --2- 172.21.15.29:0/3517564642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 0x7ff0d0101f70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009900 tx=0x7ff0cc02a410 comp rx=0 tx=0).stop 2024-09-21T02:29:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 -- 172.21.15.29:0/3517564642 shutdown_connections 2024-09-21T02:29:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 --2- 172.21.15.29:0/3517564642 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff0d01094b0 0x7ff0d010b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 --2- 172.21.15.29:0/3517564642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0d0102590 0x7ff0d0108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 --2- 172.21.15.29:0/3517564642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 0x7ff0d0101f70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 -- 172.21.15.29:0/3517564642 >> 172.21.15.29:0/3517564642 conn(0x7ff0d00fd770 msgr2=0x7ff0d00ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 -- 172.21.15.29:0/3517564642 shutdown_connections 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.939+0000 7ff0d7851640 1 -- 172.21.15.29:0/3517564642 wait complete. 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.940+0000 7ff0d7851640 1 Processor -- start 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.940+0000 7ff0d7851640 1 -- start start 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d7851640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 0x7ff0d0199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d7851640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0d0102590 0x7ff0d019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d7851640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff0d01094b0 0x7ff0d019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d7851640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d010f240 con 0x7ff0d0101b60 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d7851640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d010f0c0 con 0x7ff0d01094b0 2024-09-21T02:29:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d7851640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d010f3c0 con 0x7ff0d0102590 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 0x7ff0d0199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 0x7ff0d0199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36920/0 (socket says 172.21.15.29:36920) 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 -- 172.21.15.29:0/1273463919 learned_addr learned my addr 172.21.15.29:0/1273463919 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d4dec640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0d0102590 0x7ff0d019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d5dee640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff0d01094b0 0x7ff0d019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 -- 172.21.15.29:0/1273463919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0d0102590 msgr2=0x7ff0d019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0d0102590 0x7ff0d019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 -- 172.21.15.29:0/1273463919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff0d01094b0 msgr2=0x7ff0d019b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff0d01094b0 0x7ff0d019b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 -- 172.21.15.29:0/1273463919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d019b870 con 0x7ff0d0101b60 2024-09-21T02:29:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.941+0000 7ff0d55ed640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 0x7ff0d0199ed0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009230 tx=0x7ff0cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.942+0000 7ff0c67fc640 1 -- 172.21.15.29:0/1273463919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc03f030 con 0x7ff0d0101b60 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.942+0000 7ff0c67fc640 1 -- 172.21.15.29:0/1273463919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff0cc003c90 con 0x7ff0d0101b60 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.942+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d019bb00 con 0x7ff0d0101b60 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.942+0000 7ff0c67fc640 1 -- 172.21.15.29:0/1273463919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc035070 con 0x7ff0d0101b60 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.942+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d01a2120 con 0x7ff0d0101b60 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.943+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d00ff290 con 0x7ff0d0101b60 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.944+0000 7ff0c67fc640 1 -- 172.21.15.29:0/1273463919 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7ff0cc042030 con 0x7ff0d0101b60 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.945+0000 7ff0c67fc640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff0bc0b6480 0x7ff0bc0b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.945+0000 7ff0d4dec640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff0bc0b6480 0x7ff0bc0b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.946+0000 7ff0c67fc640 1 -- 172.21.15.29:0/1273463919 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff0cc106080 con 0x7ff0d0101b60 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.946+0000 7ff0d4dec640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff0bc0b6480 0x7ff0bc0b8990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0005d00 tx=0x7ff0c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.949+0000 7ff0c67fc640 1 -- 172.21.15.29:0/1273463919 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff0cc0ce5c0 con 0x7ff0d0101b60 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.151+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7ff0d002e640 con 0x7ff0d0101b60 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.155+0000 7ff0c67fc640 1 -- 172.21.15.29:0/1273463919 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v23)=0 v23) ==== 119+0+0 (secure 0 0 0) 0x7ff0cc0d29a0 con 0x7ff0d0101b60 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff0bc0b6480 msgr2=0x7ff0bc0b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff0bc0b6480 0x7ff0bc0b8990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0005d00 tx=0x7ff0c000b410 comp rx=0 tx=0).stop 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 msgr2=0x7ff0d0199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 0x7ff0d0199ed0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009230 tx=0x7ff0cc0042f0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 shutdown_connections 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff0bc0b6480 0x7ff0bc0b8990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff0d01094b0 0x7ff0d019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0d0102590 0x7ff0d019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 --2- 172.21.15.29:0/1273463919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d0101b60 0x7ff0d0199ed0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 >> 172.21.15.29:0/1273463919 conn(0x7ff0d00fd770 msgr2=0x7ff0d01077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:56.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 shutdown_connections 2024-09-21T02:29:56.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.158+0000 7ff0d7851640 1 -- 172.21.15.29:0/1273463919 wait complete. 2024-09-21T02:29:56.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:56 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2094244294' entity='client.admin' 2024-09-21T02:29:56.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:56 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3599450738' entity='client.admin' 2024-09-21T02:29:56.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:56 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2094244294' entity='client.admin' 2024-09-21T02:29:56.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:56 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3599450738' entity='client.admin' 2024-09-21T02:29:56.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:56 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2094244294' entity='client.admin' 2024-09-21T02:29:56.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:56 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3599450738' entity='client.admin' 2024-09-21T02:29:57.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 -- 172.21.15.29:0/2960801840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134105180 msgr2=0x7f213410bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:57.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 --2- 172.21.15.29:0/2960801840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134105180 0x7f213410bb20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f212c009ca0 tx=0x7f212c02a950 comp rx=0 tx=0).stop 2024-09-21T02:29:57.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 -- 172.21.15.29:0/2960801840 shutdown_connections 2024-09-21T02:29:57.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 --2- 172.21.15.29:0/2960801840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134105180 0x7f213410bb20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 --2- 172.21.15.29:0/2960801840 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f21341047e0 0x7f2134104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 --2- 172.21.15.29:0/2960801840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2134103750 0x7f2134103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 -- 172.21.15.29:0/2960801840 >> 172.21.15.29:0/2960801840 conn(0x7f21340ff0c0 msgr2=0x7f2134101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:57.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 -- 172.21.15.29:0/2960801840 shutdown_connections 2024-09-21T02:29:57.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.231+0000 7f213a430640 1 -- 172.21.15.29:0/2960801840 wait complete. 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.232+0000 7f213a430640 1 Processor -- start 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.232+0000 7f213a430640 1 -- start start 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.232+0000 7f213a430640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134103750 0x7f2134197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.232+0000 7f213a430640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21341047e0 0x7f2134198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.232+0000 7f213a430640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2134105180 0x7f2134198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.232+0000 7f213a430640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f213410f250 con 0x7f2134103750 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.232+0000 7f213a430640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f213410f0d0 con 0x7f2134105180 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.232+0000 7f213a430640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f213410f3d0 con 0x7f21341047e0 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134103750 0x7f2134197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134103750 0x7f2134197cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36106/0 (socket says 172.21.15.29:36106) 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 -- 172.21.15.29:0/3930236337 learned_addr learned my addr 172.21.15.29:0/3930236337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f21389cd640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2134105180 0x7f2134198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f21337fe640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21341047e0 0x7f2134198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 -- 172.21.15.29:0/3930236337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21341047e0 msgr2=0x7f2134198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21341047e0 0x7f2134198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 -- 172.21.15.29:0/3930236337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2134105180 msgr2=0x7f2134198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2134105180 0x7f2134198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 -- 172.21.15.29:0/3930236337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2134199720 con 0x7f2134103750 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f21337fe640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21341047e0 0x7f2134198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f21389cd640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2134105180 0x7f2134198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:29:57.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f2133fff640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134103750 0x7f2134197cf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f212400e9f0 tx=0x7f212400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:57.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.233+0000 7f21317fa640 1 -- 172.21.15.29:0/3930236337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212400cd80 con 0x7f2134103750 2024-09-21T02:29:57.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.234+0000 7f21317fa640 1 -- 172.21.15.29:0/3930236337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21240054b0 con 0x7f2134103750 2024-09-21T02:29:57.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.234+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2134199a10 con 0x7f2134103750 2024-09-21T02:29:57.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.234+0000 7f21317fa640 1 -- 172.21.15.29:0/3930236337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2124010430 con 0x7f2134103750 2024-09-21T02:29:57.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.234+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f213419c0f0 con 0x7f2134103750 2024-09-21T02:29:57.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.235+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2134199c70 con 0x7f2134103750 2024-09-21T02:29:57.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.237+0000 7f21317fa640 1 -- 172.21.15.29:0/3930236337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f21240046c0 con 0x7f2134103750 2024-09-21T02:29:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.238+0000 7f21317fa640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f211c0b6480 0x7f211c0b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.238+0000 7f21337fe640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f211c0b6480 0x7f211c0b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.238+0000 7f21317fa640 1 -- 172.21.15.29:0/3930236337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f21240e5c20 con 0x7f2134103750 2024-09-21T02:29:57.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.239+0000 7f21337fe640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f211c0b6480 0x7f211c0b8990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2128009fc0 tx=0x7f2128005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:57.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.241+0000 7f21317fa640 1 -- 172.21.15.29:0/3930236337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f21240ae0e0 con 0x7f2134103750 2024-09-21T02:29:57.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.434+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f213419cbd0 con 0x7f2134103750 2024-09-21T02:29:57.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.438+0000 7f21317fa640 1 -- 172.21.15.29:0/3930236337 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v27)=0 v27) ==== 120+0+0 (secure 0 0 0) 0x7f21240b24c0 con 0x7f2134103750 2024-09-21T02:29:57.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f211c0b6480 msgr2=0x7f211c0b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:57.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f211c0b6480 0x7f211c0b8990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2128009fc0 tx=0x7f2128005f90 comp rx=0 tx=0).stop 2024-09-21T02:29:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134103750 msgr2=0x7f2134197cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134103750 0x7f2134197cf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f212400e9f0 tx=0x7f212400edf0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 shutdown_connections 2024-09-21T02:29:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f211c0b6480 0x7f211c0b8990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2134105180 0x7f2134198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21341047e0 0x7f2134198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 --2- 172.21.15.29:0/3930236337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2134103750 0x7f2134197cf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 >> 172.21.15.29:0/3930236337 conn(0x7f21340ff0c0 msgr2=0x7f2134105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 shutdown_connections 2024-09-21T02:29:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:57.440+0000 7f213a430640 1 -- 172.21.15.29:0/3930236337 wait complete. 2024-09-21T02:29:57.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:57 smithi029 ceph-mon[31006]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:29:57.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:57 smithi066 ceph-mon[33436]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:57.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:57 smithi190 ceph-mon[33871]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.226+0000 7f083cd15640 1 -- 172.21.15.29:0/1794229352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08381036d0 msgr2=0x7f0838103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.226+0000 7f083cd15640 1 --2- 172.21.15.29:0/1794229352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08381036d0 0x7f0838103ae0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f082c009900 tx=0x7f082c02a480 comp rx=0 tx=0).stop 2024-09-21T02:29:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.227+0000 7f083cd15640 1 -- 172.21.15.29:0/1794229352 shutdown_connections 2024-09-21T02:29:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.227+0000 7f083cd15640 1 --2- 172.21.15.29:0/1794229352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0838104070 0x7f083810aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.227+0000 7f083cd15640 1 --2- 172.21.15.29:0/1794229352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08381036d0 0x7f0838103ae0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.227+0000 7f083cd15640 1 --2- 172.21.15.29:0/1794229352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0838102640 0x7f0838102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.227+0000 7f083cd15640 1 -- 172.21.15.29:0/1794229352 >> 172.21.15.29:0/1794229352 conn(0x7f08380fdfb0 msgr2=0x7f0838100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.227+0000 7f083cd15640 1 -- 172.21.15.29:0/1794229352 shutdown_connections 2024-09-21T02:29:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.227+0000 7f083cd15640 1 -- 172.21.15.29:0/1794229352 wait complete. 2024-09-21T02:29:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.227+0000 7f083cd15640 1 Processor -- start 2024-09-21T02:29:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f083cd15640 1 -- start start 2024-09-21T02:29:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f083cd15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0838102640 0x7f0838199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f083cd15640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08381036d0 0x7f083819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f083cd15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0838104070 0x7f083819b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f083cd15640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f083810e170 con 0x7f0838104070 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f083cd15640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f083810dff0 con 0x7f08381036d0 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f083cd15640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f083810e2f0 con 0x7f0838102640 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0836d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0838104070 0x7f083819b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0836d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0838104070 0x7f083819b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36134/0 (socket says 172.21.15.29:36134) 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0836d9d640 1 -- 172.21.15.29:0/1482032626 learned_addr learned my addr 172.21.15.29:0/1482032626 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f083659c640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0838102640 0x7f0838199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0835d9b640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08381036d0 0x7f083819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0836d9d640 1 -- 172.21.15.29:0/1482032626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0838102640 msgr2=0x7f0838199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0836d9d640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0838102640 0x7f0838199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0836d9d640 1 -- 172.21.15.29:0/1482032626 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08381036d0 msgr2=0x7f083819a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0836d9d640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08381036d0 0x7f083819a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.228+0000 7f0836d9d640 1 -- 172.21.15.29:0/1482032626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083819b940 con 0x7f0838104070 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.229+0000 7f0836d9d640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0838104070 0x7f083819b120 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f083000eaa0 tx=0x7f083000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.229+0000 7f08277fe640 1 -- 172.21.15.29:0/1482032626 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083000cdb0 con 0x7f0838104070 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.229+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083819bc30 con 0x7f0838104070 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.229+0000 7f08277fe640 1 -- 172.21.15.29:0/1482032626 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0830005020 con 0x7f0838104070 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.229+0000 7f08277fe640 1 -- 172.21.15.29:0/1482032626 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08300116d0 con 0x7f0838104070 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.229+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08381a2330 con 0x7f0838104070 2024-09-21T02:29:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.230+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08381a2500 con 0x7f0838104070 2024-09-21T02:29:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.232+0000 7f08277fe640 1 -- 172.21.15.29:0/1482032626 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f08300051d0 con 0x7f0838104070 2024-09-21T02:29:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.232+0000 7f08277fe640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f08200b6480 0x7f08200b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.232+0000 7f083659c640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f08200b6480 0x7f08200b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.232+0000 7f08277fe640 1 -- 172.21.15.29:0/1482032626 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f08300e5bd0 con 0x7f0838104070 2024-09-21T02:29:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.233+0000 7f083659c640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f08200b6480 0x7f08200b8990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f082800bbe0 tx=0x7f082800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.235+0000 7f08277fe640 1 -- 172.21.15.29:0/1482032626 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0830016030 con 0x7f0838104070 2024-09-21T02:29:58.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.426+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f08381a2880 con 0x7f0838104070 2024-09-21T02:29:58.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.430+0000 7f08277fe640 1 -- 172.21.15.29:0/1482032626 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v28)=0 v28) ==== 127+0+0 (secure 0 0 0) 0x7f08300ae130 con 0x7f0838104070 2024-09-21T02:29:58.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f08200b6480 msgr2=0x7f08200b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f08200b6480 0x7f08200b8990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f082800bbe0 tx=0x7f082800b390 comp rx=0 tx=0).stop 2024-09-21T02:29:58.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0838104070 msgr2=0x7f083819b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0838104070 0x7f083819b120 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f083000eaa0 tx=0x7f083000ef60 comp rx=0 tx=0).stop 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 shutdown_connections 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f08200b6480 0x7f08200b8990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0838104070 0x7f083819b120 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08381036d0 0x7f083819a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 --2- 172.21.15.29:0/1482032626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0838102640 0x7f0838199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 >> 172.21.15.29:0/1482032626 conn(0x7f08380fdfb0 msgr2=0x7f08380ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 shutdown_connections 2024-09-21T02:29:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:58.432+0000 7f083cd15640 1 -- 172.21.15.29:0/1482032626 wait complete. 2024-09-21T02:29:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 -- 172.21.15.29:0/545756261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c077060 msgr2=0x7f830c075730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 --2- 172.21.15.29:0/545756261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c077060 0x7f830c075730 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8304009900 tx=0x7f830402a410 comp rx=0 tx=0).stop 2024-09-21T02:29:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 -- 172.21.15.29:0/545756261 shutdown_connections 2024-09-21T02:29:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 --2- 172.21.15.29:0/545756261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f830c10b770 0x7f830c10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 --2- 172.21.15.29:0/545756261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c075c70 0x7f830c076080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 --2- 172.21.15.29:0/545756261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c077060 0x7f830c075730 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 -- 172.21.15.29:0/545756261 >> 172.21.15.29:0/545756261 conn(0x7f830c0ff0e0 msgr2=0x7f830c101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 -- 172.21.15.29:0/545756261 shutdown_connections 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.800+0000 7f8314615640 1 -- 172.21.15.29:0/545756261 wait complete. 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8314615640 1 Processor -- start 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8314615640 1 -- start start 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8314615640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c075c70 0x7f830c19c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8314615640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c077060 0x7f830c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8314615640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f830c10b770 0x7f830c19d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8314615640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f830c111460 con 0x7f830c075c70 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8314615640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f830c1112e0 con 0x7f830c10b770 2024-09-21T02:29:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8314615640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f830c1115e0 con 0x7f830c077060 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f83123b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c075c70 0x7f830c19c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f83123b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c075c70 0x7f830c19c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36896/0 (socket says 172.21.15.29:36896) 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f83123b1640 1 -- 172.21.15.29:0/1175512591 learned_addr learned my addr 172.21.15.29:0/1175512591 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8312bb2640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f830c10b770 0x7f830c19d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f8311bb0640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c077060 0x7f830c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.801+0000 7f83123b1640 1 -- 172.21.15.29:0/1175512591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c077060 msgr2=0x7f830c19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f83123b1640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c077060 0x7f830c19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f83123b1640 1 -- 172.21.15.29:0/1175512591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f830c10b770 msgr2=0x7f830c19d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f83123b1640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f830c10b770 0x7f830c19d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f83123b1640 1 -- 172.21.15.29:0/1175512591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f830c19db40 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f83123b1640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c075c70 0x7f830c19c110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8304005760 tx=0x7f8304003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f830b7fe640 1 -- 172.21.15.29:0/1175512591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830403f070 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f830b7fe640 1 -- 172.21.15.29:0/1175512591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8304031ba0 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830c19ddd0 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f830b7fe640 1 -- 172.21.15.29:0/1175512591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304035030 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.802+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f830c1a4530 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.803+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f830c10a580 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.805+0000 7f830b7fe640 1 -- 172.21.15.29:0/1175512591 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f8304042070 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.805+0000 7f830b7fe640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82f40b6500 0x7f82f40b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.805+0000 7f8311bb0640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82f40b6500 0x7f82f40b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.806+0000 7f830b7fe640 1 -- 172.21.15.29:0/1175512591 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8304106610 con 0x7f830c075c70 2024-09-21T02:29:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.806+0000 7f8311bb0640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82f40b6500 0x7f82f40b8a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f830000bbe0 tx=0x7f830000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:53.809+0000 7f830b7fe640 1 -- 172.21.15.29:0/1175512591 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f83040d2f50 con 0x7f830c075c70 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.002+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f830c02e640 con 0x7f830c075c70 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.008+0000 7f830b7fe640 1 -- 172.21.15.29:0/1175512591 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v22)=0 v22) ==== 118+0+0 (secure 0 0 0) 0x7f83040d7e90 con 0x7f830c075c70 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.010+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82f40b6500 msgr2=0x7f82f40b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82f40b6500 0x7f82f40b8a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f830000bbe0 tx=0x7f830000b390 comp rx=0 tx=0).stop 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c075c70 msgr2=0x7f830c19c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c075c70 0x7f830c19c110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8304005760 tx=0x7f8304003c10 comp rx=0 tx=0).stop 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 shutdown_connections 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82f40b6500 0x7f82f40b8a10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f830c10b770 0x7f830c19d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c077060 0x7f830c19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 --2- 172.21.15.29:0/1175512591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f830c075c70 0x7f830c19c110 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 >> 172.21.15.29:0/1175512591 conn(0x7f830c0ff0e0 msgr2=0x7f830c107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 shutdown_connections 2024-09-21T02:29:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:54.011+0000 7f8314615640 1 -- 172.21.15.29:0/1175512591 wait complete. 2024-09-21T02:29:58.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:58 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3930236337' entity='client.admin' 2024-09-21T02:29:58.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:58 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1482032626' entity='client.admin' 2024-09-21T02:29:58.773 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:58 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3930236337' entity='client.admin' 2024-09-21T02:29:58.773 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:58 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1482032626' entity='client.admin' 2024-09-21T02:29:58.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:58 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3930236337' entity='client.admin' 2024-09-21T02:29:58.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:58 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1482032626' entity='client.admin' 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3507058624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8104870 msgr2=0x7f45f8104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3507058624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8104870 0x7f45f8104c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f45e8009900 tx=0x7f45e802a480 comp rx=0 tx=0).stop 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3507058624 shutdown_connections 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3507058624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45f8105210 0x7f45f810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3507058624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8104870 0x7f45f8104c80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3507058624 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45f81037e0 0x7f45f8103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3507058624 >> 172.21.15.29:0/3507058624 conn(0x7f45f80ff150 msgr2=0x7f45f8101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3507058624 shutdown_connections 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.295+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3507058624 wait complete. 2024-09-21T02:29:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45ffcfa640 1 Processor -- start 2024-09-21T02:29:59.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45ffcfa640 1 -- start start 2024-09-21T02:29:59.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45ffcfa640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45f81037e0 0x7f45f8076e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:59.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45ffcfa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45f8104870 0x7f45f8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:59.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45ffcfa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8105210 0x7f45f8075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:59.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45ffcfa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f45f810f250 con 0x7f45f8105210 2024-09-21T02:29:59.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45ffcfa640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f45f810f0d0 con 0x7f45f81037e0 2024-09-21T02:29:59.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45ffcfa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f45f810f3d0 con 0x7f45f8104870 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45fe297640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8105210 0x7f45f8075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45fe297640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8105210 0x7f45f8075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36142/0 (socket says 172.21.15.29:36142) 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.296+0000 7f45fe297640 1 -- 172.21.15.29:0/3123954630 learned_addr learned my addr 172.21.15.29:0/3123954630 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45fd295640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45f8104870 0x7f45f8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45fda96640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45f81037e0 0x7f45f8076e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45fe297640 1 -- 172.21.15.29:0/3123954630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45f8104870 msgr2=0x7f45f8075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45fe297640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45f8104870 0x7f45f8075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45fe297640 1 -- 172.21.15.29:0/3123954630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45f81037e0 msgr2=0x7f45f8076e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45fe297640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45f81037e0 0x7f45f8076e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45fe297640 1 -- 172.21.15.29:0/3123954630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f80762c0 con 0x7f45f8105210 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45fe297640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8105210 0x7f45f8075c50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45f000eaa0 tx=0x7f45f000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45eeffd640 1 -- 172.21.15.29:0/3123954630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f000cdb0 con 0x7f45f8105210 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45eeffd640 1 -- 172.21.15.29:0/3123954630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45f0005020 con 0x7f45f8105210 2024-09-21T02:29:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f81a3fa0 con 0x7f45f8105210 2024-09-21T02:29:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45eeffd640 1 -- 172.21.15.29:0/3123954630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f00116d0 con 0x7f45f8105210 2024-09-21T02:29:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.297+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f81a4340 con 0x7f45f8105210 2024-09-21T02:29:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.299+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f8103c50 con 0x7f45f8105210 2024-09-21T02:29:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.301+0000 7f45eeffd640 1 -- 172.21.15.29:0/3123954630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f45f00051d0 con 0x7f45f8105210 2024-09-21T02:29:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.302+0000 7f45eeffd640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f45e40b6500 0x7f45e40b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.302+0000 7f45fda96640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f45e40b6500 0x7f45e40b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.302+0000 7f45eeffd640 1 -- 172.21.15.29:0/3123954630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f45f00e5b80 con 0x7f45f8105210 2024-09-21T02:29:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.303+0000 7f45fda96640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f45e40b6500 0x7f45e40b8a10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45f4005e10 tx=0x7f45f4005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.306+0000 7f45eeffd640 1 -- 172.21.15.29:0/3123954630 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f45f0016030 con 0x7f45f8105210 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.168+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/1306888341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc1040a0 msgr2=0x7fd4dc10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.168+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/1306888341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc1040a0 0x7fd4dc10aa40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009ca0 tx=0x7fd4d402a950 comp rx=0 tx=0).stop 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/1306888341 shutdown_connections 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/1306888341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc1040a0 0x7fd4dc10aa40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/1306888341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4dc103700 0x7fd4dc103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/1306888341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4dc102670 0x7fd4dc102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/1306888341 >> 172.21.15.29:0/1306888341 conn(0x7fd4dc0fdfe0 msgr2=0x7fd4dc100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/1306888341 shutdown_connections 2024-09-21T02:29:59.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/1306888341 wait complete. 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 Processor -- start 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.169+0000 7fd4e1f0a640 1 -- start start 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4e1f0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc102670 0x7fd4dc06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4e1f0a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4dc103700 0x7fd4dc19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4e1f0a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4dc1040a0 0x7fd4dc1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4e1f0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4dc10e150 con 0x7fd4dc102670 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4e1f0a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd4dc10dfd0 con 0x7fd4dc103700 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4e1f0a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd4dc10e2d0 con 0x7fd4dc1040a0 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc102670 0x7fd4dc06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc102670 0x7fd4dc06c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36060/0 (socket says 172.21.15.29:36060) 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 -- 172.21.15.29:0/2094244294 learned_addr learned my addr 172.21.15.29:0/2094244294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4daffd640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4dc103700 0x7fd4dc19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4dbfff640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4dc1040a0 0x7fd4dc1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:59.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 -- 172.21.15.29:0/2094244294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4dc1040a0 msgr2=0x7fd4dc1a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4dc1040a0 0x7fd4dc1a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 -- 172.21.15.29:0/2094244294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4dc103700 msgr2=0x7fd4dc19ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4dc103700 0x7fd4dc19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 -- 172.21.15.29:0/2094244294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4dc1a1950 con 0x7fd4dc102670 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4daffd640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4dc103700 0x7fd4dc19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4dbfff640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4dc1040a0 0x7fd4dc1a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.170+0000 7fd4db7fe640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc102670 0x7fd4dc06c3a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00d950 tx=0x7fd4cc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.171+0000 7fd4d8ff9640 1 -- 172.21.15.29:0/2094244294 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc017070 con 0x7fd4dc102670 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.171+0000 7fd4d8ff9640 1 -- 172.21.15.29:0/2094244294 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4cc009c30 con 0x7fd4dc102670 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.171+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4dc1a1c40 con 0x7fd4dc102670 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.171+0000 7fd4d8ff9640 1 -- 172.21.15.29:0/2094244294 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00f620 con 0x7fd4dc102670 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.171+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4dc1a2180 con 0x7fd4dc102670 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.172+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4dc109830 con 0x7fd4dc102670 2024-09-21T02:29:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.174+0000 7fd4d8ff9640 1 -- 172.21.15.29:0/2094244294 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7fd4cc010460 con 0x7fd4dc102670 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.174+0000 7fd4d8ff9640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd4c40b6480 0x7fd4c40b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.174+0000 7fd4daffd640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd4c40b6480 0x7fd4c40b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.174+0000 7fd4d8ff9640 1 -- 172.21.15.29:0/2094244294 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd4cc0e5310 con 0x7fd4dc102670 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.175+0000 7fd4daffd640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd4c40b6480 0x7fd4c40b8990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0005d00 tx=0x7fd4d0009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.177+0000 7fd4d8ff9640 1 -- 172.21.15.29:0/2094244294 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd4cc0ad870 con 0x7fd4dc102670 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.369+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7fd4dc02e800 con 0x7fd4dc102670 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.373+0000 7fd4d8ff9640 1 -- 172.21.15.29:0/2094244294 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v25)=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7fd4cc009de0 con 0x7fd4dc102670 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd4c40b6480 msgr2=0x7fd4c40b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd4c40b6480 0x7fd4c40b8990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0005d00 tx=0x7fd4d0009550 comp rx=0 tx=0).stop 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc102670 msgr2=0x7fd4dc06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc102670 0x7fd4dc06c3a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00d950 tx=0x7fd4cc00de10 comp rx=0 tx=0).stop 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 shutdown_connections 2024-09-21T02:29:59.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd4c40b6480 0x7fd4c40b8990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4dc1040a0 0x7fd4dc1a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4dc103700 0x7fd4dc19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 --2- 172.21.15.29:0/2094244294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc102670 0x7fd4dc06c3a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 >> 172.21.15.29:0/2094244294 conn(0x7fd4dc0fdfe0 msgr2=0x7fd4dc106a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:59.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 shutdown_connections 2024-09-21T02:29:59.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:55.376+0000 7fd4e1f0a640 1 -- 172.21.15.29:0/2094244294 wait complete. 2024-09-21T02:29:59.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:29:59 smithi029 ceph-mon[31006]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:59.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.498+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f45f810a9c0 con 0x7f45f8105210 2024-09-21T02:29:59.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.502+0000 7f45eeffd640 1 -- 172.21.15.29:0/3123954630 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v29)=0 v29) ==== 111+0+0 (secure 0 0 0) 0x7f45f00ae0c0 con 0x7f45f8105210 2024-09-21T02:29:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.505+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f45e40b6500 msgr2=0x7f45e40b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.505+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f45e40b6500 0x7f45e40b8a10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45f4005e10 tx=0x7f45f4005da0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.505+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8105210 msgr2=0x7f45f8075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:29:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.505+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8105210 0x7f45f8075c50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45f000eaa0 tx=0x7f45f000ef60 comp rx=0 tx=0).stop 2024-09-21T02:29:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.506+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 shutdown_connections 2024-09-21T02:29:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.506+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f45e40b6500 0x7f45e40b8a10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.506+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45f8105210 0x7f45f8075c50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.506+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45f8104870 0x7f45f8075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.506+0000 7f45ffcfa640 1 --2- 172.21.15.29:0/3123954630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45f81037e0 0x7f45f8076e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:29:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.506+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 >> 172.21.15.29:0/3123954630 conn(0x7f45f80ff150 msgr2=0x7f45f8107b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:29:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.506+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 shutdown_connections 2024-09-21T02:29:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:59.506+0000 7f45ffcfa640 1 -- 172.21.15.29:0/3123954630 wait complete. 2024-09-21T02:29:59.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:29:59 smithi066 ceph-mon[33436]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:29:59.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:29:59 smithi190 ceph-mon[33871]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:00.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:00 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3123954630' entity='client.admin' 2024-09-21T02:30:00.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:00 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:00.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:00 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:00.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:00 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:00.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:00 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:00.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:00 smithi066 ceph-mon[33436]: overall HEALTH_OK 2024-09-21T02:30:00.835 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:00 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3123954630' entity='client.admin' 2024-09-21T02:30:00.835 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:00.835 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:00.836 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:00.836 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:00.836 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:00 smithi029 ceph-mon[31006]: overall HEALTH_OK 2024-09-21T02:30:00.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:00 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3123954630' entity='client.admin' 2024-09-21T02:30:00.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:00 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:00.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:00 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:00.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:00 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:00.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:00 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:00.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:00 smithi190 ceph-mon[33871]: overall HEALTH_OK 2024-09-21T02:30:01.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:01 smithi029 ceph-mon[31006]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:01.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:01 smithi190 ceph-mon[33871]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 -- 172.21.15.29:0/22176957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c077060 msgr2=0x7f799c075730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 --2- 172.21.15.29:0/22176957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c077060 0x7f799c075730 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7994009900 tx=0x7f799402a410 comp rx=0 tx=0).stop 2024-09-21T02:30:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 -- 172.21.15.29:0/22176957 shutdown_connections 2024-09-21T02:30:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 --2- 172.21.15.29:0/22176957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f799c10b7a0 0x7f799c10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 --2- 172.21.15.29:0/22176957 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f799c075c70 0x7f799c076080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 --2- 172.21.15.29:0/22176957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c077060 0x7f799c075730 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 -- 172.21.15.29:0/22176957 >> 172.21.15.29:0/22176957 conn(0x7f799c0ff0f0 msgr2=0x7f799c1015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 -- 172.21.15.29:0/22176957 shutdown_connections 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.493+0000 7f79a36af640 1 -- 172.21.15.29:0/22176957 wait complete. 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a36af640 1 Processor -- start 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a36af640 1 -- start start 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a36af640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f799c075c70 0x7f799c19c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a36af640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f799c077060 0x7f799c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a36af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c10b7a0 0x7f799c19d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a36af640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f799c111300 con 0x7f799c10b7a0 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a36af640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f799c111180 con 0x7f799c077060 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a36af640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f799c111480 con 0x7f799c075c70 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a1c4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c10b7a0 0x7f799c19d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a0c4a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f799c077060 0x7f799c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a1c4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c10b7a0 0x7f799c19d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36884/0 (socket says 172.21.15.29:36884) 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a1c4c640 1 -- 172.21.15.29:0/4163632565 learned_addr learned my addr 172.21.15.29:0/4163632565 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.494+0000 7f79a144b640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f799c075c70 0x7f799c19c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a1c4c640 1 -- 172.21.15.29:0/4163632565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f799c075c70 msgr2=0x7f799c19c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a1c4c640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f799c075c70 0x7f799c19c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a1c4c640 1 -- 172.21.15.29:0/4163632565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f799c077060 msgr2=0x7f799c19c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a1c4c640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f799c077060 0x7f799c19c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a1c4c640 1 -- 172.21.15.29:0/4163632565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c19da40 con 0x7f799c10b7a0 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a0c4a640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f799c077060 0x7f799c19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a1c4c640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c10b7a0 0x7f799c19d220 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f798c00eaa0 tx=0x7f798c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f799a7fc640 1 -- 172.21.15.29:0/4163632565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c00cdb0 con 0x7f799c10b7a0 2024-09-21T02:30:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799c19dd30 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f799a7fc640 1 -- 172.21.15.29:0/4163632565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f798c005020 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f799a7fc640 1 -- 172.21.15.29:0/4163632565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c0116d0 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.495+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f799c1a4430 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.496+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799c10a5b0 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.498+0000 7f799a7fc640 1 -- 172.21.15.29:0/4163632565 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f798c004a40 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.498+0000 7f799a7fc640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79840b6480 0x7f79840b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.499+0000 7f79a144b640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79840b6480 0x7f79840b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.499+0000 7f799a7fc640 1 -- 172.21.15.29:0/4163632565 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f798c0e5770 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.499+0000 7f79a144b640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79840b6480 0x7f79840b8990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7994009230 tx=0x7f79940058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.503+0000 7f799a7fc640 1 -- 172.21.15.29:0/4163632565 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f798c016030 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.693+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f799c02e640 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.697+0000 7f799a7fc640 1 -- 172.21.15.29:0/4163632565 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v21)=0 v21) ==== 123+0+0 (secure 0 0 0) 0x7f798c0b20b0 con 0x7f799c10b7a0 2024-09-21T02:30:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.698+0000 7f799a7fc640 1 -- 172.21.15.29:0/4163632565 <== mon.0 v2:172.21.15.29:3300/0 8 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f798c0ad190 con 0x7f799c10b7a0 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79840b6480 msgr2=0x7f79840b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79840b6480 0x7f79840b8990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7994009230 tx=0x7f79940058d0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c10b7a0 msgr2=0x7f799c19d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c10b7a0 0x7f799c19d220 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f798c00eaa0 tx=0x7f798c00ef60 comp rx=0 tx=0).stop 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 shutdown_connections 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79840b6480 0x7f79840b8990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f799c10b7a0 0x7f799c19d220 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f799c077060 0x7f799c19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 --2- 172.21.15.29:0/4163632565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f799c075c70 0x7f799c19c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 >> 172.21.15.29:0/4163632565 conn(0x7f799c0ff0f0 msgr2=0x7f799c107780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.699+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 shutdown_connections 2024-09-21T02:30:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:51.700+0000 7f79a36af640 1 -- 172.21.15.29:0/4163632565 wait complete. 2024-09-21T02:30:02.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:01 smithi066 ceph-mon[33436]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:02.609 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi029') 2024-09-21T02:30:02.609 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi029') 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 -- 172.21.15.29:0/3137861526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe60102640 msgr2=0x7efe60102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 --2- 172.21.15.29:0/3137861526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe60102640 0x7efe60102ab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009900 tx=0x7efe5c02a480 comp rx=0 tx=0).stop 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 -- 172.21.15.29:0/3137861526 shutdown_connections 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 --2- 172.21.15.29:0/3137861526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe60104070 0x7efe6010aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 --2- 172.21.15.29:0/3137861526 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efe601036d0 0x7efe60103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 --2- 172.21.15.29:0/3137861526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe60102640 0x7efe60102ab0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 -- 172.21.15.29:0/3137861526 >> 172.21.15.29:0/3137861526 conn(0x7efe600fdfb0 msgr2=0x7efe60100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 -- 172.21.15.29:0/3137861526 shutdown_connections 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.177+0000 7efe67e29640 1 -- 172.21.15.29:0/3137861526 wait complete. 2024-09-21T02:30:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe67e29640 1 Processor -- start 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe67e29640 1 -- start start 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe67e29640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efe60102640 0x7efe60199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe67e29640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe601036d0 0x7efe6019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe67e29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe60104070 0x7efe6019b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe67e29640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe6010dff0 con 0x7efe601036d0 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe67e29640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7efe6010de70 con 0x7efe60102640 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe67e29640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efe6010e170 con 0x7efe60104070 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe653c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe601036d0 0x7efe6019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe653c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe601036d0 0x7efe6019a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36746/0 (socket says 172.21.15.29:36746) 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe653c4640 1 -- 172.21.15.29:0/2292205541 learned_addr learned my addr 172.21.15.29:0/2292205541 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe65bc5640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efe60102640 0x7efe60199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe663c6640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe60104070 0x7efe6019b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:03.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe653c4640 1 -- 172.21.15.29:0/2292205541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe60104070 msgr2=0x7efe6019b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe653c4640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe60104070 0x7efe6019b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe653c4640 1 -- 172.21.15.29:0/2292205541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efe60102640 msgr2=0x7efe60199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe653c4640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efe60102640 0x7efe60199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe653c4640 1 -- 172.21.15.29:0/2292205541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6019b820 con 0x7efe601036d0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.178+0000 7efe663c6640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe60104070 0x7efe6019b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.179+0000 7efe653c4640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe601036d0 0x7efe6019a460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efe5400e9f0 tx=0x7efe5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.179+0000 7efe52ffd640 1 -- 172.21.15.29:0/2292205541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5400cd80 con 0x7efe601036d0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.179+0000 7efe52ffd640 1 -- 172.21.15.29:0/2292205541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7efe540108a0 con 0x7efe601036d0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.179+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6019bb10 con 0x7efe601036d0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.179+0000 7efe52ffd640 1 -- 172.21.15.29:0/2292205541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54018750 con 0x7efe601036d0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.179+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe601a2210 con 0x7efe601036d0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.180+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe601a23e0 con 0x7efe601036d0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.181+0000 7efe52ffd640 1 -- 172.21.15.29:0/2292205541 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) ==== 103177+0+0 (secure 0 0 0) 0x7efe54010430 con 0x7efe601036d0 2024-09-21T02:30:03.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.181+0000 7efe52ffd640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efe4c07afd0 0x7efe4c07d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.182+0000 7efe52ffd640 1 -- 172.21.15.29:0/2292205541 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efe5409d0a0 con 0x7efe601036d0 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.182+0000 7efe65bc5640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efe4c07afd0 0x7efe4c07d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.182+0000 7efe65bc5640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efe4c07afd0 0x7efe4c07d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009230 tx=0x7efe5c0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.186+0000 7efe52ffd640 1 -- 172.21.15.29:0/2292205541 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7efe54069ba0 con 0x7efe601036d0 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.383+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7efe601a2690 con 0x7efe601036d0 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.387+0000 7efe52ffd640 1 -- 172.21.15.29:0/2292205541 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v15)=0 v15) ==== 125+0+0 (secure 0 0 0) 0x7efe5406ea50 con 0x7efe601036d0 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efe4c07afd0 msgr2=0x7efe4c07d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efe4c07afd0 0x7efe4c07d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009230 tx=0x7efe5c0058d0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe601036d0 msgr2=0x7efe6019a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe601036d0 0x7efe6019a460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efe5400e9f0 tx=0x7efe5400eeb0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 shutdown_connections 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efe4c07afd0 0x7efe4c07d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe60104070 0x7efe6019b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe601036d0 0x7efe6019a460 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 --2- 172.21.15.29:0/2292205541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efe60102640 0x7efe60199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:03.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 >> 172.21.15.29:0/2292205541 conn(0x7efe600fdfb0 msgr2=0x7efe600ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:03.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 shutdown_connections 2024-09-21T02:30:03.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:29:47.390+0000 7efe67e29640 1 -- 172.21.15.29:0/2292205541 wait complete. 2024-09-21T02:30:03.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:03 smithi066 ceph-mon[33436]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:03.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:03 smithi190 ceph-mon[33871]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:03 smithi029 ceph-mon[31006]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:30:05.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:05 smithi029 ceph-mon[31006]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:05 smithi190 ceph-mon[33871]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:06.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:05 smithi066 ceph-mon[33436]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:06.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 -- 172.21.15.29:0/1840808415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc41047e0 msgr2=0x7fdcc4104bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:06.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 --2- 172.21.15.29:0/1840808415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc41047e0 0x7fdcc4104bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009900 tx=0x7fdcb402a480 comp rx=0 tx=0).stop 2024-09-21T02:30:06.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 -- 172.21.15.29:0/1840808415 shutdown_connections 2024-09-21T02:30:06.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 --2- 172.21.15.29:0/1840808415 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdcc4105180 0x7fdcc410bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 --2- 172.21.15.29:0/1840808415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc41047e0 0x7fdcc4104bf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 --2- 172.21.15.29:0/1840808415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc4103750 0x7fdcc4103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 -- 172.21.15.29:0/1840808415 >> 172.21.15.29:0/1840808415 conn(0x7fdcc40ff0c0 msgr2=0x7fdcc4101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 -- 172.21.15.29:0/1840808415 shutdown_connections 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.413+0000 7fdccbcba640 1 -- 172.21.15.29:0/1840808415 wait complete. 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdccbcba640 1 Processor -- start 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdccbcba640 1 -- start start 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdccbcba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc4103750 0x7fdcc4199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdccbcba640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdcc41047e0 0x7fdcc419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdccbcba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc4105180 0x7fdcc419b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdccbcba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc410f250 con 0x7fdcc4105180 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdccbcba640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc410f0d0 con 0x7fdcc41047e0 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdccbcba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc410f3d0 con 0x7fdcc4103750 2024-09-21T02:30:06.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcca257640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc4105180 0x7fdcc419b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcca257640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc4105180 0x7fdcc419b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40974/0 (socket says 172.21.15.29:40974) 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcca257640 1 -- 172.21.15.29:0/2108338250 learned_addr learned my addr 172.21.15.29:0/2108338250 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcc9a56640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc4103750 0x7fdcc4199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcc9255640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdcc41047e0 0x7fdcc419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcca257640 1 -- 172.21.15.29:0/2108338250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc4103750 msgr2=0x7fdcc4199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcca257640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc4103750 0x7fdcc4199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcca257640 1 -- 172.21.15.29:0/2108338250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdcc41047e0 msgr2=0x7fdcc419a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcca257640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdcc41047e0 0x7fdcc419a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcca257640 1 -- 172.21.15.29:0/2108338250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc419b930 con 0x7fdcc4105180 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.414+0000 7fdcc9255640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdcc41047e0 0x7fdcc419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.415+0000 7fdcca257640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc4105180 0x7fdcc419b110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00eaa0 tx=0x7fdcbc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.415+0000 7fdcbaffd640 1 -- 172.21.15.29:0/2108338250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc00cdb0 con 0x7fdcc4105180 2024-09-21T02:30:06.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.415+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc419bc20 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.415+0000 7fdcbaffd640 1 -- 172.21.15.29:0/2108338250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdcbc005020 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.415+0000 7fdcbaffd640 1 -- 172.21.15.29:0/2108338250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc0116d0 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.415+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc41a2320 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.416+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc410a930 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.418+0000 7fdcbaffd640 1 -- 172.21.15.29:0/2108338250 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7fdcbc0051d0 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.418+0000 7fdcbaffd640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdcb00b6480 0x7fdcb00b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.419+0000 7fdcc9a56640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdcb00b6480 0x7fdcb00b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.419+0000 7fdcbaffd640 1 -- 172.21.15.29:0/2108338250 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdcbc0e5bf0 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.419+0000 7fdcc9a56640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdcb00b6480 0x7fdcb00b8990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009bd0 tx=0x7fdcc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.421+0000 7fdcbaffd640 1 -- 172.21.15.29:0/2108338250 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdcbc016030 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.616+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7fdcc402e640 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.620+0000 7fdcbaffd640 1 -- 172.21.15.29:0/2108338250 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v30)=0 v30) ==== 116+0+0 (secure 0 0 0) 0x7fdcbc0ae150 con 0x7fdcc4105180 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.622+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdcb00b6480 msgr2=0x7fdcb00b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:06.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.622+0000 7fdccbcba640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdcb00b6480 0x7fdcb00b8990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009bd0 tx=0x7fdcc0005f90 comp rx=0 tx=0).stop 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.622+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc4105180 msgr2=0x7fdcc419b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.622+0000 7fdccbcba640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc4105180 0x7fdcc419b110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00eaa0 tx=0x7fdcbc00ef60 comp rx=0 tx=0).stop 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.623+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 shutdown_connections 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.623+0000 7fdccbcba640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdcb00b6480 0x7fdcb00b8990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.623+0000 7fdccbcba640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcc4105180 0x7fdcc419b110 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.623+0000 7fdccbcba640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdcc41047e0 0x7fdcc419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.623+0000 7fdccbcba640 1 --2- 172.21.15.29:0/2108338250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc4103750 0x7fdcc4199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.623+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 >> 172.21.15.29:0/2108338250 conn(0x7fdcc40ff0c0 msgr2=0x7fdcc4107b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.623+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 shutdown_connections 2024-09-21T02:30:06.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:06.623+0000 7fdccbcba640 1 -- 172.21.15.29:0/2108338250 wait complete. 2024-09-21T02:30:07.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.096+0000 7fbbcde72640 1 -- 172.21.15.29:0/586935937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc81036f0 msgr2=0x7fbbc8103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.096+0000 7fbbcde72640 1 --2- 172.21.15.29:0/586935937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc81036f0 0x7fbbc8103b00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009900 tx=0x7fbbb002a480 comp rx=0 tx=0).stop 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.096+0000 7fbbcde72640 1 -- 172.21.15.29:0/586935937 shutdown_connections 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.096+0000 7fbbcde72640 1 --2- 172.21.15.29:0/586935937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc8104090 0x7fbbc810aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.096+0000 7fbbcde72640 1 --2- 172.21.15.29:0/586935937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc81036f0 0x7fbbc8103b00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.096+0000 7fbbcde72640 1 --2- 172.21.15.29:0/586935937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc8102660 0x7fbbc8102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.096+0000 7fbbcde72640 1 -- 172.21.15.29:0/586935937 >> 172.21.15.29:0/586935937 conn(0x7fbbc80fdfd0 msgr2=0x7fbbc8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.097+0000 7fbbcde72640 1 -- 172.21.15.29:0/586935937 shutdown_connections 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.097+0000 7fbbcde72640 1 -- 172.21.15.29:0/586935937 wait complete. 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.097+0000 7fbbcde72640 1 Processor -- start 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.097+0000 7fbbcde72640 1 -- start start 2024-09-21T02:30:07.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbcde72640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc8102660 0x7fbbc8199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbcde72640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc81036f0 0x7fbbc819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbcde72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8104090 0x7fbbc819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbcde72640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc810e160 con 0x7fbbc8104090 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbcde72640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc810dfe0 con 0x7fbbc8102660 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbcde72640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc810e2e0 con 0x7fbbc81036f0 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8104090 0x7fbbc819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8104090 0x7fbbc819b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40996/0 (socket says 172.21.15.29:40996) 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 -- 172.21.15.29:0/2638560562 learned_addr learned my addr 172.21.15.29:0/2638560562 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc81036f0 0x7fbbc819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc77fe640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc8102660 0x7fbbc8199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 -- 172.21.15.29:0/2638560562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc81036f0 msgr2=0x7fbbc819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc81036f0 0x7fbbc819a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 -- 172.21.15.29:0/2638560562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc8102660 msgr2=0x7fbbc8199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc8102660 0x7fbbc8199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 -- 172.21.15.29:0/2638560562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc819b9f0 con 0x7fbbc8104090 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc7fff640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8104090 0x7fbbc819b1d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000d950 tx=0x7fbbc000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc4ff9640 1 -- 172.21.15.29:0/2638560562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0017030 con 0x7fbbc8104090 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.098+0000 7fbbc4ff9640 1 -- 172.21.15.29:0/2638560562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbbc0004020 con 0x7fbbc8104090 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.099+0000 7fbbc4ff9640 1 -- 172.21.15.29:0/2638560562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0003ab0 con 0x7fbbc8104090 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.099+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc819bce0 con 0x7fbbc8104090 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.099+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc81a23e0 con 0x7fbbc8104090 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.100+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc81a25b0 con 0x7fbbc8104090 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.102+0000 7fbbc4ff9640 1 -- 172.21.15.29:0/2638560562 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7fbbc001a070 con 0x7fbbc8104090 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.103+0000 7fbbc4ff9640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6500 0x7fbbb40b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.103+0000 7fbbc77fe640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6500 0x7fbbb40b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:07.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.103+0000 7fbbc4ff9640 1 -- 172.21.15.29:0/2638560562 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbbc00e5040 con 0x7fbbc8104090 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.104+0000 7fbbc77fe640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6500 0x7fbbb40b8a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8005fd0 tx=0x7fbbb8009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.106+0000 7fbbc4ff9640 1 -- 172.21.15.29:0/2638560562 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbbc0021070 con 0x7fbbc8104090 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.298+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7fbbc81a2780 con 0x7fbbc8104090 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.302+0000 7fbbc4ff9640 1 -- 172.21.15.29:0/2638560562 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v31)=0 v31) ==== 120+0+0 (secure 0 0 0) 0x7fbbc00ad600 con 0x7fbbc8104090 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6500 msgr2=0x7fbbb40b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6500 0x7fbbb40b8a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8005fd0 tx=0x7fbbb8009680 comp rx=0 tx=0).stop 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8104090 msgr2=0x7fbbc819b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8104090 0x7fbbc819b1d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000d950 tx=0x7fbbc000de10 comp rx=0 tx=0).stop 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 shutdown_connections 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6500 0x7fbbb40b8a10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8104090 0x7fbbc819b1d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc81036f0 0x7fbbc819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 --2- 172.21.15.29:0/2638560562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc8102660 0x7fbbc8199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:07.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.304+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 >> 172.21.15.29:0/2638560562 conn(0x7fbbc80fdfd0 msgr2=0x7fbbc80ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:07.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.305+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 shutdown_connections 2024-09-21T02:30:07.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:07.305+0000 7fbbcde72640 1 -- 172.21.15.29:0/2638560562 wait complete. 2024-09-21T02:30:07.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:07 smithi029 ceph-mon[31006]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:07.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:07 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2108338250' entity='client.admin' 2024-09-21T02:30:07.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:07 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2638560562' entity='client.admin' 2024-09-21T02:30:07.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:07 smithi190 ceph-mon[33871]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:07 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2108338250' entity='client.admin' 2024-09-21T02:30:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:07 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2638560562' entity='client.admin' 2024-09-21T02:30:08.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:07 smithi066 ceph-mon[33436]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:07 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2108338250' entity='client.admin' 2024-09-21T02:30:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:07 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2638560562' entity='client.admin' 2024-09-21T02:30:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 -- 172.21.15.29:0/1704508859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8105180 msgr2=0x7f7ed810bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 --2- 172.21.15.29:0/1704508859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8105180 0x7f7ed810bb20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009ca0 tx=0x7f7ed002a950 comp rx=0 tx=0).stop 2024-09-21T02:30:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 -- 172.21.15.29:0/1704508859 shutdown_connections 2024-09-21T02:30:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 --2- 172.21.15.29:0/1704508859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8105180 0x7f7ed810bb20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 --2- 172.21.15.29:0/1704508859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ed81047e0 0x7f7ed8104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 --2- 172.21.15.29:0/1704508859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed8103750 0x7f7ed8103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 -- 172.21.15.29:0/1704508859 >> 172.21.15.29:0/1704508859 conn(0x7f7ed80ff0c0 msgr2=0x7f7ed8101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 -- 172.21.15.29:0/1704508859 shutdown_connections 2024-09-21T02:30:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.013+0000 7f7edcc52640 1 -- 172.21.15.29:0/1704508859 wait complete. 2024-09-21T02:30:09.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7edcc52640 1 Processor -- start 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7edcc52640 1 -- start start 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7edcc52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8103750 0x7f7ed8199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7edcc52640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed81047e0 0x7f7ed819a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7edcc52640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ed8105180 0x7f7ed819b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7edcc52640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810f230 con 0x7f7ed8103750 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7edcc52640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810f0b0 con 0x7f7ed8105180 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7edcc52640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810f3b0 con 0x7f7ed81047e0 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7ed659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8103750 0x7f7ed8199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7ed659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8103750 0x7f7ed8199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41024/0 (socket says 172.21.15.29:41024) 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7ed659c640 1 -- 172.21.15.29:0/2266350159 learned_addr learned my addr 172.21.15.29:0/2266350159 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7ed6d9d640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ed8105180 0x7f7ed819b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.014+0000 7f7ed5d9b640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed81047e0 0x7f7ed819a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ed659c640 1 -- 172.21.15.29:0/2266350159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed81047e0 msgr2=0x7f7ed819a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ed659c640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed81047e0 0x7f7ed819a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ed659c640 1 -- 172.21.15.29:0/2266350159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ed8105180 msgr2=0x7f7ed819b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ed659c640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ed8105180 0x7f7ed819b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ed659c640 1 -- 172.21.15.29:0/2266350159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed819b8e0 con 0x7f7ed8103750 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ed659c640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8103750 0x7f7ed8199eb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800d950 tx=0x7f7ec800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ec77fe640 1 -- 172.21.15.29:0/2266350159 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8017070 con 0x7f7ed8103750 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed819bbd0 con 0x7f7ed8103750 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ec77fe640 1 -- 172.21.15.29:0/2266350159 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ec8009c30 con 0x7f7ed8103750 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7ec77fe640 1 -- 172.21.15.29:0/2266350159 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec800f620 con 0x7f7ed8103750 2024-09-21T02:30:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.015+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed81a22d0 con 0x7f7ed8103750 2024-09-21T02:30:09.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.016+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed810a930 con 0x7f7ed8103750 2024-09-21T02:30:09.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.019+0000 7f7ec77fe640 1 -- 172.21.15.29:0/2266350159 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f7ec8010460 con 0x7f7ed8103750 2024-09-21T02:30:09.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.020+0000 7f7ec77fe640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7ec00b6480 0x7f7ec00b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:09.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.020+0000 7f7ed5d9b640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7ec00b6480 0x7f7ec00b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:09.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.020+0000 7f7ec77fe640 1 -- 172.21.15.29:0/2266350159 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7ec80e5210 con 0x7f7ed8103750 2024-09-21T02:30:09.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.020+0000 7f7ed5d9b640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7ec00b6480 0x7f7ec00b8990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00bbe0 tx=0x7f7ecc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:09.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.023+0000 7f7ec77fe640 1 -- 172.21.15.29:0/2266350159 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7ec8009de0 con 0x7f7ed8103750 2024-09-21T02:30:09.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.212+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f7ed802e640 con 0x7f7ed8103750 2024-09-21T02:30:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.216+0000 7f7ec77fe640 1 -- 172.21.15.29:0/2266350159 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v32)=0 v32) ==== 117+0+0 (secure 0 0 0) 0x7f7ec80b1ab0 con 0x7f7ed8103750 2024-09-21T02:30:09.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7ec00b6480 msgr2=0x7f7ec00b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7ec00b6480 0x7f7ec00b8990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00bbe0 tx=0x7f7ecc00b390 comp rx=0 tx=0).stop 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8103750 msgr2=0x7f7ed8199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8103750 0x7f7ed8199eb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800d950 tx=0x7f7ec800de10 comp rx=0 tx=0).stop 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 shutdown_connections 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7ec00b6480 0x7f7ec00b8990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ed8105180 0x7f7ed819b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed81047e0 0x7f7ed819a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 --2- 172.21.15.29:0/2266350159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ed8103750 0x7f7ed8199eb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 >> 172.21.15.29:0/2266350159 conn(0x7f7ed80ff0c0 msgr2=0x7f7ed8107b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 shutdown_connections 2024-09-21T02:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.218+0000 7f7edcc52640 1 -- 172.21.15.29:0/2266350159 wait complete. 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 -- 172.21.15.29:0/468748634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c102650 msgr2=0x7fb62c102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 --2- 172.21.15.29:0/468748634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c102650 0x7fb62c102ac0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb628009900 tx=0x7fb62802a480 comp rx=0 tx=0).stop 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 -- 172.21.15.29:0/468748634 shutdown_connections 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 --2- 172.21.15.29:0/468748634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c104080 0x7fb62c10aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 --2- 172.21.15.29:0/468748634 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c1036e0 0x7fb62c103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 --2- 172.21.15.29:0/468748634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c102650 0x7fb62c102ac0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 -- 172.21.15.29:0/468748634 >> 172.21.15.29:0/468748634 conn(0x7fb62c0fdfa0 msgr2=0x7fb62c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 -- 172.21.15.29:0/468748634 shutdown_connections 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.230+0000 7fb63434f640 1 -- 172.21.15.29:0/468748634 wait complete. 2024-09-21T02:30:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb63434f640 1 Processor -- start 2024-09-21T02:30:09.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb63434f640 1 -- start start 2024-09-21T02:30:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb63434f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c102650 0x7fb62c06c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb63434f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1036e0 0x7fb62c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb63434f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c104080 0x7fb62c1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb63434f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c10e190 con 0x7fb62c104080 2024-09-21T02:30:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb63434f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c10e010 con 0x7fb62c102650 2024-09-21T02:30:09.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb63434f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c10e310 con 0x7fb62c1036e0 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.231+0000 7fb6328ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c104080 0x7fb62c1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6328ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c104080 0x7fb62c1a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41032/0 (socket says 172.21.15.29:41032) 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6328ec640 1 -- 172.21.15.29:0/2337363671 learned_addr learned my addr 172.21.15.29:0/2337363671 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6320eb640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c102650 0x7fb62c06c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6318ea640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1036e0 0x7fb62c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6328ec640 1 -- 172.21.15.29:0/2337363671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1036e0 msgr2=0x7fb62c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6328ec640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1036e0 0x7fb62c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6328ec640 1 -- 172.21.15.29:0/2337363671 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c102650 msgr2=0x7fb62c06c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6328ec640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c102650 0x7fb62c06c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6328ec640 1 -- 172.21.15.29:0/2337363671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c198a40 con 0x7fb62c104080 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6318ea640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1036e0 0x7fb62c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6328ec640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c104080 0x7fb62c1a11c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb62400eb10 tx=0x7fb62400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6237fe640 1 -- 172.21.15.29:0/2337363671 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb624004b20 con 0x7fb62c104080 2024-09-21T02:30:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb6237fe640 1 -- 172.21.15.29:0/2337363671 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb624004cc0 con 0x7fb62c104080 2024-09-21T02:30:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.232+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62c198d30 con 0x7fb62c104080 2024-09-21T02:30:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.233+0000 7fb6237fe640 1 -- 172.21.15.29:0/2337363671 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb624010690 con 0x7fb62c104080 2024-09-21T02:30:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.233+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb62c1992c0 con 0x7fb62c104080 2024-09-21T02:30:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.234+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb62c19a750 con 0x7fb62c104080 2024-09-21T02:30:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.235+0000 7fb6237fe640 1 -- 172.21.15.29:0/2337363671 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7fb624005000 con 0x7fb62c104080 2024-09-21T02:30:09.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.236+0000 7fb6237fe640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6180b6500 0x7fb6180b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:09.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.237+0000 7fb6320eb640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6180b6500 0x7fb6180b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:09.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.237+0000 7fb6237fe640 1 -- 172.21.15.29:0/2337363671 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb6240e5db0 con 0x7fb62c104080 2024-09-21T02:30:09.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.237+0000 7fb6320eb640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6180b6500 0x7fb6180b8a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb628005760 tx=0x7fb628031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.240+0000 7fb6237fe640 1 -- 172.21.15.29:0/2337363671 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb624056710 con 0x7fb62c104080 2024-09-21T02:30:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.432+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7fb62c19a9a0 con 0x7fb62c104080 2024-09-21T02:30:09.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.437+0000 7fb6237fe640 1 -- 172.21.15.29:0/2337363671 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v33)=0 v33) ==== 122+0+0 (secure 0 0 0) 0x7fb6240ae310 con 0x7fb62c104080 2024-09-21T02:30:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6180b6500 msgr2=0x7fb6180b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6180b6500 0x7fb6180b8a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb628005760 tx=0x7fb628031400 comp rx=0 tx=0).stop 2024-09-21T02:30:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c104080 msgr2=0x7fb62c1a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c104080 0x7fb62c1a11c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb62400eb10 tx=0x7fb62400c2d0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 shutdown_connections 2024-09-21T02:30:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6180b6500 0x7fb6180b8a10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c104080 0x7fb62c1a11c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1036e0 0x7fb62c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 --2- 172.21.15.29:0/2337363671 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c102650 0x7fb62c06c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:09.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 >> 172.21.15.29:0/2337363671 conn(0x7fb62c0fdfa0 msgr2=0x7fb62c106a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:09.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 shutdown_connections 2024-09-21T02:30:09.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:09.439+0000 7fb63434f640 1 -- 172.21.15.29:0/2337363671 wait complete. 2024-09-21T02:30:09.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:09 smithi190 ceph-mon[33871]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:09 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2266350159' entity='client.admin' 2024-09-21T02:30:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:09 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2337363671' entity='client.admin' 2024-09-21T02:30:09.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:09 smithi029 ceph-mon[31006]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:09 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2266350159' entity='client.admin' 2024-09-21T02:30:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:09 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2337363671' entity='client.admin' 2024-09-21T02:30:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:09 smithi066 ceph-mon[33436]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:09 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2266350159' entity='client.admin' 2024-09-21T02:30:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:09 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2337363671' entity='client.admin' 2024-09-21T02:30:10.107 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi029') 2024-09-21T02:30:10.107 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi029') 2024-09-21T02:30:10.108 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi029') 2024-09-21T02:30:10.108 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi029') 2024-09-21T02:30:10.562 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi029') 2024-09-21T02:30:10.562 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi029') 2024-09-21T02:30:10.562 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.041 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.041 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.041 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.041 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.042 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.042 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.042 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.042 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.042 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.042 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.042 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.042 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', 'e1049baa-77c0-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi029') 2024-09-21T02:30:11.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config dump 2024-09-21T02:30:11.305 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:30:11.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:11 smithi029 ceph-mon[31006]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 -- 172.21.15.29:0/2804826155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 msgr2=0x7f4cec10b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/2804826155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 0x7f4cec10b9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009ca0 tx=0x7f4ce402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 -- 172.21.15.29:0/2804826155 shutdown_connections 2024-09-21T02:30:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/2804826155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 0x7f4cec10b9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/2804826155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cec1025d0 0x7f4cec108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/2804826155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cec101ba0 0x7f4cec101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 -- 172.21.15.29:0/2804826155 >> 172.21.15.29:0/2804826155 conn(0x7f4cec0fd770 msgr2=0x7f4cec0ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 -- 172.21.15.29:0/2804826155 shutdown_connections 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.780+0000 7f4cf11eb640 1 -- 172.21.15.29:0/2804826155 wait complete. 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cf11eb640 1 Processor -- start 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cf11eb640 1 -- start start 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cf11eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cec101ba0 0x7f4cec199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cf11eb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cec1025d0 0x7f4cec19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cf11eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 0x7f4cec19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cf11eb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4cec10f2d0 con 0x7f4cec1094b0 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cf11eb640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4cec10f150 con 0x7f4cec1025d0 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cf11eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4cec10f450 con 0x7f4cec101ba0 2024-09-21T02:30:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ceb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 0x7f4cec19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ceb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 0x7f4cec19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41050/0 (socket says 172.21.15.29:41050) 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4cead9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cec101ba0 0x7f4cec199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ceb59e640 1 -- 172.21.15.29:0/463322549 learned_addr learned my addr 172.21.15.29:0/463322549 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ce3fff640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cec1025d0 0x7f4cec19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ceb59e640 1 -- 172.21.15.29:0/463322549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cec101ba0 msgr2=0x7f4cec199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ceb59e640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cec101ba0 0x7f4cec199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ceb59e640 1 -- 172.21.15.29:0/463322549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cec1025d0 msgr2=0x7f4cec19a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ceb59e640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cec1025d0 0x7f4cec19a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ceb59e640 1 -- 172.21.15.29:0/463322549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cec19b8c0 con 0x7f4cec1094b0 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.781+0000 7f4ce3fff640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cec1025d0 0x7f4cec19a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.782+0000 7f4ceb59e640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 0x7f4cec19b0a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4031040 tx=0x7f4ce4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.782+0000 7f4ce8d99640 1 -- 172.21.15.29:0/463322549 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4004040 con 0x7f4cec1094b0 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.782+0000 7f4ce8d99640 1 -- 172.21.15.29:0/463322549 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ce4004670 con 0x7f4cec1094b0 2024-09-21T02:30:11.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.782+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cec19bb50 con 0x7f4cec1094b0 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.782+0000 7f4ce8d99640 1 -- 172.21.15.29:0/463322549 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4030d50 con 0x7f4cec1094b0 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.782+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cec1a22b0 con 0x7f4cec1094b0 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.783+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cec101fb0 con 0x7f4cec1094b0 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.785+0000 7f4ce8d99640 1 -- 172.21.15.29:0/463322549 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f4ce40305c0 con 0x7f4cec1094b0 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.786+0000 7f4ce8d99640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cd80b6480 0x7f4cd80b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.786+0000 7f4cead9d640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cd80b6480 0x7f4cd80b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.786+0000 7f4ce8d99640 1 -- 172.21.15.29:0/463322549 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4ce4106c10 con 0x7f4cec1094b0 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.787+0000 7f4cead9d640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cd80b6480 0x7f4cd80b8990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc005fd0 tx=0x7f4cdc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.789+0000 7f4ce8d99640 1 -- 172.21.15.29:0/463322549 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4ce40cf0d0 con 0x7f4cec1094b0 2024-09-21T02:30:11.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:11 smithi190 ceph-mon[33871]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.981+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f4cec0ff290 con 0x7f4cec1094b0 2024-09-21T02:30:11.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.982+0000 7f4ce8d99640 1 -- 172.21.15.29:0/463322549 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v33) ==== 59+0+21112 (secure 0 0 0) 0x7f4cec0ff290 con 0x7f4cec1094b0 2024-09-21T02:30:11.983 INFO:teuthology.orchestra.run.smithi029.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-21T02:30:11.983 INFO:teuthology.orchestra.run.smithi029.stdout:global dev auth_debug true 2024-09-21T02:30:11.983 INFO:teuthology.orchestra.run.smithi029.stdout:global basic container_image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 * 2024-09-21T02:30:11.983 INFO:teuthology.orchestra.run.smithi029.stdout:global dev debug_asserts_on_shutdown true 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global basic log_to_file true 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global basic log_to_journald false 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global basic log_to_stderr false 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_allow_pool_delete true 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_clock_drift_allowed 1.000000 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_cluster_log_to_file true 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_max_pg_per_osd 10000 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_pg_warn_max_object_skew 0.000000 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_warn_on_crush_straw_calc_version_zero false 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_warn_on_legacy_crush_tunables false 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_warn_on_osd_down_out_interval_zero false 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global dev mon_warn_on_pool_pg_num_not_power_of_two false 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced mon_warn_on_too_few_osds false 2024-09-21T02:30:11.984 INFO:teuthology.orchestra.run.smithi029.stdout:global dev ms_die_on_bug true 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:global dev ms_die_on_old_message true 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced osd_pool_default_erasure_code_profile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:global advanced osd_pool_default_pg_autoscale_mode off 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced auth_allow_insecure_global_id_reclaim false 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced auth_mon_ticket_ttl 660.000000 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced auth_service_ticket_ttl 240.000000 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced debug_mon 20/20 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced debug_ms 1/1 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced debug_paxos 20/20 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_data_avail_warn 5 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_down_mkfs_grace 300 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_mgr_mkfs_grace 240 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_op_complaint_time 120 2024-09-21T02:30:11.985 INFO:teuthology.orchestra.run.smithi029.stdout:mon dev mon_osd_prime_pg_temp true 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_osd_reporter_subtree_level osd 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim false 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mon advanced public_network 172.21.0.0/20 * 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced cephsqlite_lock_renewal_timeout 900000 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced client_mount_timeout 30 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced debug_client 20/20 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced debug_mgr 20/20 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced debug_ms 1/1 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced mgr/cephadm/allow_ptrace true * 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced mgr/cephadm/container_init True * 2024-09-21T02:30:11.986 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced mgr/cephadm/migration_current 7 * 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced mgr/dashboard/ssl_server_port 8443 * 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced mgr/orchestrator/orchestrator cephadm 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:mgr advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev bdev_debug_aio true 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev bluestore_block_size 96636764160 * 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced bluestore_compression_mode aggressive 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev bluestore_fsck_on_mount true 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced debug_bluefs 20/20 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced debug_bluestore 20/20 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced debug_ms 1/1 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced debug_osd 20/20 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced debug_rocksdb 10/10 2024-09-21T02:30:11.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev osd_debug_misdirected_ops true 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev osd_debug_op_order true 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev osd_debug_pg_log_writeout true 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev osd_debug_shutdown true 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev osd_debug_verify_cached_snaps true 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev osd_debug_verify_missing_on_start true 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd dev osd_debug_verify_stray_on_activate true 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_deep_scrub_update_digest_min_age 30 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_failsafe_full_ratio 0.950000 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_max_omap_entries_per_request 10000 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_mclock_profile high_recovery_ops 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_memory_target_autotune true 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_objectstore bluestore * 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_op_queue debug_random * 2024-09-21T02:30:11.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_op_queue_cut_off debug_random * 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_recover_clone_overlap true 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_recovery_max_chunk 1048576 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_scrub_load_threshold 5.000000 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_scrub_max_interval 600.000000 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:osd advanced osd_shutdown_pgref_assert true 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced debug_mds 20/20 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced debug_mds_balancer 20/20 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced debug_ms 2/2 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced mds_bal_fragment_size_max 10000 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced mds_bal_merge_size 5 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced mds_bal_split_bits 3 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced mds_bal_split_size 100 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds dev mds_debug_frag true 2024-09-21T02:30:11.989 INFO:teuthology.orchestra.run.smithi029.stdout:mds dev mds_debug_scatterstat true 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced mds_export_ephemeral_random_max 0.100000 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:mds dev mds_verify_scatter true 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced rados_mon_op_timeout 900 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:mds advanced rados_osd_op_timeout 900 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:client advanced client_mount_timeout 600 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:client advanced debug_client 20/20 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:client advanced debug_ms 1/1 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:client advanced rados_mon_op_timeout 900 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:client advanced rados_osd_op_timeout 900 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:client.rgw advanced rgw_cache_enabled true 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:client.rgw advanced rgw_enable_ops_log true 2024-09-21T02:30:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:client.rgw advanced rgw_enable_usage_log true 2024-09-21T02:30:11.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cd80b6480 msgr2=0x7f4cd80b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:11.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cd80b6480 0x7f4cd80b8990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc005fd0 tx=0x7f4cdc00b390 comp rx=0 tx=0).stop 2024-09-21T02:30:11.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 msgr2=0x7f4cec19b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:11.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 0x7f4cec19b0a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4031040 tx=0x7f4ce4003ba0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 shutdown_connections 2024-09-21T02:30:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cd80b6480 0x7f4cd80b8990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec1094b0 0x7f4cec19b0a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cec1025d0 0x7f4cec19a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 --2- 172.21.15.29:0/463322549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cec101ba0 0x7f4cec199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 >> 172.21.15.29:0/463322549 conn(0x7f4cec0fd770 msgr2=0x7f4cec1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 shutdown_connections 2024-09-21T02:30:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:11.984+0000 7f4cf11eb640 1 -- 172.21.15.29:0/463322549 wait complete. 2024-09-21T02:30:12.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:11 smithi066 ceph-mon[33436]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:12.696 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:12 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/463322549' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T02:30:12.728 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T02:30:12.729 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:30:12.729 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T02:30:12.746 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T02:30:12.746 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 02:28:53.524254437 +0000 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 02:24:01.354220856 +0000 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 02:24:01.354220856 +0000 2024-09-21T02:30:12.803 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-09-21 02:24:01.354220856 +0000 2024-09-21T02:30:12.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T02:30:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-21T02:30:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-21T02:30:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000101471 s, 5.0 MB/s 2024-09-21T02:30:12.870 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T02:30:12.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:12 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/463322549' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T02:30:12.927 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 02:28:53.551254016 +0000 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 02:24:01.672215950 +0000 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 02:24:01.672215950 +0000 2024-09-21T02:30:12.985 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-09-21 02:24:01.672215950 +0000 2024-09-21T02:30:12.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T02:30:13.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:12 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/463322549' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T02:30:13.050 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-21T02:30:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-21T02:30:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000116808 s, 4.4 MB/s 2024-09-21T02:30:13.052 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T02:30:13.110 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-09-21T02:30:13.166 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T02:30:13.166 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:13.166 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T02:30:13.166 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:13.167 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:13.167 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 02:28:53.577253610 +0000 2024-09-21T02:30:13.167 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 02:24:01.999210904 +0000 2024-09-21T02:30:13.167 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 02:24:01.999210904 +0000 2024-09-21T02:30:13.167 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-09-21 02:24:01.999210904 +0000 2024-09-21T02:30:13.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T02:30:13.232 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-21T02:30:13.232 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-21T02:30:13.232 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000112827 s, 4.5 MB/s 2024-09-21T02:30:13.233 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T02:30:13.291 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 02:28:53.604253189 +0000 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 02:24:02.316206013 +0000 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 02:24:02.316206013 +0000 2024-09-21T02:30:13.347 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-09-21 02:24:02.316206013 +0000 2024-09-21T02:30:13.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T02:30:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-21T02:30:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-21T02:30:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000110774 s, 4.6 MB/s 2024-09-21T02:30:13.412 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T02:30:13.470 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-21T02:30:13.470 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T02:30:13.490 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T02:30:13.490 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 2024-09-21T02:30:13.550 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T02:30:13.550 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:13.550 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-21T02:30:13.550 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:13.550 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:13.551 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-09-21 02:29:41.384258308 +0000 2024-09-21T02:30:13.551 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-09-21 02:23:56.258219840 +0000 2024-09-21T02:30:13.551 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-09-21 02:23:56.258219840 +0000 2024-09-21T02:30:13.551 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-09-21 02:23:56.258219840 +0000 2024-09-21T02:30:13.551 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T02:30:13.622 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-09-21T02:30:13.622 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-09-21T02:30:13.622 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000301941 s, 1.7 MB/s 2024-09-21T02:30:13.623 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T02:30:13.683 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-09-21 02:29:41.384258308 +0000 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-09-21 02:23:56.595217099 +0000 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-09-21 02:23:56.595217099 +0000 2024-09-21T02:30:13.741 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-09-21 02:23:56.595217099 +0000 2024-09-21T02:30:13.742 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T02:30:13.805 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:13 smithi066 ceph-mon[33436]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:13.808 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-09-21T02:30:13.808 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-09-21T02:30:13.808 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000317745 s, 1.6 MB/s 2024-09-21T02:30:13.809 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T02:30:13.867 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 2024-09-21T02:30:13.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:13 smithi190 ceph-mon[33871]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:13.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:13 smithi029 ceph-mon[31006]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:13.924 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T02:30:13.924 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:13.924 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T02:30:13.924 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:13.924 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:13.924 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-09-21 02:29:41.385258300 +0000 2024-09-21T02:30:13.924 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-09-21 02:23:56.945214252 +0000 2024-09-21T02:30:13.924 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-09-21 02:23:56.945214252 +0000 2024-09-21T02:30:13.925 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-09-21 02:23:56.945214252 +0000 2024-09-21T02:30:13.925 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T02:30:13.990 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-09-21T02:30:13.990 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-09-21T02:30:13.990 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000275666 s, 1.9 MB/s 2024-09-21T02:30:13.992 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T02:30:14.049 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-09-21 02:29:41.385258300 +0000 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-09-21 02:23:57.290211446 +0000 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-09-21 02:23:57.290211446 +0000 2024-09-21T02:30:14.107 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-09-21 02:23:57.290211446 +0000 2024-09-21T02:30:14.107 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T02:30:14.172 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-09-21T02:30:14.172 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-09-21T02:30:14.172 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000170368 s, 3.0 MB/s 2024-09-21T02:30:14.173 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T02:30:14.230 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-21T02:30:14.230 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T02:30:14.250 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T02:30:14.250 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-21 02:29:39.157097303 +0000 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-21 02:23:52.125984691 +0000 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-21 02:23:52.125984691 +0000 2024-09-21T02:30:14.309 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-21 02:23:52.125984691 +0000 2024-09-21T02:30:14.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T02:30:14.379 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-21T02:30:14.379 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-21T02:30:14.379 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000313948 s, 1.6 MB/s 2024-09-21T02:30:14.380 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T02:30:14.440 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-09-21T02:30:14.498 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T02:30:14.499 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:14.499 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T02:30:14.499 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:14.499 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:14.499 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-21 02:29:39.158097276 +0000 2024-09-21T02:30:14.499 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-21 02:23:52.539973616 +0000 2024-09-21T02:30:14.499 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-21 02:23:52.539973616 +0000 2024-09-21T02:30:14.499 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-21 02:23:52.539973616 +0000 2024-09-21T02:30:14.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T02:30:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-21T02:30:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-21T02:30:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000220939 s, 2.3 MB/s 2024-09-21T02:30:14.566 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T02:30:14.624 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-09-21T02:30:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T02:30:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:14.681 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T02:30:14.681 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:14.681 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:14.681 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-21 02:29:39.158097276 +0000 2024-09-21T02:30:14.682 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-21 02:23:52.863964952 +0000 2024-09-21T02:30:14.682 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-21 02:23:52.863964952 +0000 2024-09-21T02:30:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-21 02:23:52.863964952 +0000 2024-09-21T02:30:14.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T02:30:14.747 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-21T02:30:14.747 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-21T02:30:14.747 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00021622 s, 2.4 MB/s 2024-09-21T02:30:14.748 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T02:30:14.806 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-09-21T02:30:14.862 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T02:30:14.862 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T02:30:14.862 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-21T02:30:14.862 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:30:14.862 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T02:30:14.862 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-21 02:29:39.158097276 +0000 2024-09-21T02:30:14.862 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-21 02:23:53.184956369 +0000 2024-09-21T02:30:14.863 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-21 02:23:53.184956369 +0000 2024-09-21T02:30:14.863 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-21 02:23:53.184956369 +0000 2024-09-21T02:30:14.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T02:30:14.926 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-21T02:30:14.926 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-21T02:30:14.926 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000228333 s, 2.2 MB/s 2024-09-21T02:30:14.927 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T02:30:14.984 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-09-21T02:30:14.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T02:30:15.246 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:30:15.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:15 smithi029 ceph-mon[31006]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:15.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:15 smithi190 ceph-mon[33871]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:16.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:15 smithi066 ceph-mon[33436]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:17.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:17 smithi190 ceph-mon[33871]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:17 smithi029 ceph-mon[31006]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:17.919 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:30:17.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-09-21T02:30:18.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:17 smithi066 ceph-mon[33436]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:18.207 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 -- 172.21.15.29:0/3972622272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11141040c0 msgr2=0x7f111410aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 --2- 172.21.15.29:0/3972622272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11141040c0 0x7f111410aa60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f110c009ca0 tx=0x7f110c02a950 comp rx=0 tx=0).stop 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 -- 172.21.15.29:0/3972622272 shutdown_connections 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 --2- 172.21.15.29:0/3972622272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11141040c0 0x7f111410aa60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 --2- 172.21.15.29:0/3972622272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1114103720 0x7f1114103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 --2- 172.21.15.29:0/3972622272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1114102690 0x7f1114102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 -- 172.21.15.29:0/3972622272 >> 172.21.15.29:0/3972622272 conn(0x7f11140fe000 msgr2=0x7f11141004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 -- 172.21.15.29:0/3972622272 shutdown_connections 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.751+0000 7f1119b44640 1 -- 172.21.15.29:0/3972622272 wait complete. 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.752+0000 7f1119b44640 1 Processor -- start 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.752+0000 7f1119b44640 1 -- start start 2024-09-21T02:30:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.752+0000 7f1119b44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1114102690 0x7f1114198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.752+0000 7f1119b44640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1114103720 0x7f11141994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.752+0000 7f1119b44640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11141040c0 0x7f111419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.752+0000 7f1119b44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f111410e240 con 0x7f1114102690 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.752+0000 7f1119b44640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f111410e0c0 con 0x7f11141040c0 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.752+0000 7f1119b44640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f111410e3c0 con 0x7f1114103720 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1114102690 0x7f1114198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1114102690 0x7f1114198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50730/0 (socket says 172.21.15.29:50730) 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 -- 172.21.15.29:0/1594051868 learned_addr learned my addr 172.21.15.29:0/1594051868 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f1112ffd640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1114103720 0x7f11141994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f1113fff640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11141040c0 0x7f111419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 -- 172.21.15.29:0/1594051868 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1114103720 msgr2=0x7f11141994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1114103720 0x7f11141994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 -- 172.21.15.29:0/1594051868 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11141040c0 msgr2=0x7f111419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11141040c0 0x7f111419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 -- 172.21.15.29:0/1594051868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111419a860 con 0x7f1114102690 2024-09-21T02:30:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f11137fe640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1114102690 0x7f1114198e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f110400c890 tx=0x7f110400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f1110ff9640 1 -- 172.21.15.29:0/1594051868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104004220 con 0x7f1114102690 2024-09-21T02:30:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f1110ff9640 1 -- 172.21.15.29:0/1594051868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1104007e40 con 0x7f1114102690 2024-09-21T02:30:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f1110ff9640 1 -- 172.21.15.29:0/1594051868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104004a00 con 0x7f1114102690 2024-09-21T02:30:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.753+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111419ab50 con 0x7f1114102690 2024-09-21T02:30:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.754+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f111406ba70 con 0x7f1114102690 2024-09-21T02:30:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.755+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1114102b00 con 0x7f1114102690 2024-09-21T02:30:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.756+0000 7f1110ff9640 1 -- 172.21.15.29:0/1594051868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f11040079b0 con 0x7f1114102690 2024-09-21T02:30:18.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.757+0000 7f1110ff9640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10fc0b6580 0x7f10fc0b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:18.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.757+0000 7f1112ffd640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10fc0b6580 0x7f10fc0b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:18.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.757+0000 7f1110ff9640 1 -- 172.21.15.29:0/1594051868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f11040e5810 con 0x7f1114102690 2024-09-21T02:30:18.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.758+0000 7f1112ffd640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10fc0b6580 0x7f10fc0b8a90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1108009890 tx=0x7f1108008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.760+0000 7f1110ff9640 1 -- 172.21.15.29:0/1594051868 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1104014030 con 0x7f1114102690 2024-09-21T02:30:18.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:18.959+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1114109490 con 0x7f10fc0b6580 2024-09-21T02:30:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:19 smithi029 ceph-mon[31006]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:30:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:30:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:19.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:19 smithi190 ceph-mon[33871]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:19.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:30:19.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:30:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:19 smithi066 ceph-mon[33436]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:30:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:30:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:20.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:20 smithi029 ceph-mon[31006]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:20.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:20 smithi190 ceph-mon[33871]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:21.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:20 smithi066 ceph-mon[33436]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:21 smithi190 ceph-mon[33871]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:21 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/77652477' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60"} : dispatch 2024-09-21T02:30:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:21 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/77652477' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60"}]': finished 2024-09-21T02:30:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:21 smithi190 ceph-mon[33871]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T02:30:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:21 smithi029 ceph-mon[31006]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:21 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/77652477' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60"} : dispatch 2024-09-21T02:30:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:21 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/77652477' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60"}]': finished 2024-09-21T02:30:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:21 smithi029 ceph-mon[31006]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T02:30:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:21 smithi066 ceph-mon[33436]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:21 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/77652477' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60"} : dispatch 2024-09-21T02:30:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:21 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/77652477' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60"}]': finished 2024-09-21T02:30:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:21 smithi066 ceph-mon[33436]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T02:30:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:21 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:22.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:22 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1725841088' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:30:22.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:22 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1725841088' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:30:22.910 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:22 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1725841088' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:30:23.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:23 smithi190 ceph-mon[33871]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:23 smithi029 ceph-mon[31006]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:24.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:23 smithi066 ceph-mon[33436]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:25.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:25 smithi190 ceph-mon[33871]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:25 smithi029 ceph-mon[31006]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:25 smithi066 ceph-mon[33436]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:27 smithi029 ceph-mon[31006]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:27 smithi190 ceph-mon[33871]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:27 smithi066 ceph-mon[33436]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:29.827 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:29 smithi029 ceph-mon[31006]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:29 smithi190 ceph-mon[33871]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:29 smithi066 ceph-mon[33436]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:30.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T02:30:30.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:30.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T02:30:30.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:31.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T02:30:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:31 smithi190 ceph-mon[33871]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:31 smithi190 ceph-mon[33871]: Deploying daemon osd.0 on smithi029 2024-09-21T02:30:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:31 smithi029 ceph-mon[31006]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:31 smithi029 ceph-mon[31006]: Deploying daemon osd.0 on smithi029 2024-09-21T02:30:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:31 smithi066 ceph-mon[33436]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:31 smithi066 ceph-mon[33436]: Deploying daemon osd.0 on smithi029 2024-09-21T02:30:33.832 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:33 smithi029 ceph-mon[31006]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:33 smithi190 ceph-mon[33871]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:34.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:33 smithi066 ceph-mon[33436]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:35 smithi190 ceph-mon[33871]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:35 smithi029 ceph-mon[31006]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:36.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:35 smithi066 ceph-mon[33436]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:37.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:37 smithi190 ceph-mon[33871]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:37 smithi029 ceph-mon[31006]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:38.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:37 smithi066 ceph-mon[33436]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:39.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:39 smithi190 ceph-mon[33871]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:39 smithi029 ceph-mon[31006]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:40.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:39 smithi066 ceph-mon[33436]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:41.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:41 smithi066 ceph-mon[33436]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:41.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:41.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:41.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:41 smithi029 ceph-mon[31006]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:41.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:41.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:41.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:41 smithi190 ceph-mon[33871]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:41.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:41.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.915+0000 7f1110ff9640 1 -- 172.21.15.29:0/1594051868 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1114109490 con 0x7f10fc0b6580 2024-09-21T02:30:41.916 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10fc0b6580 msgr2=0x7f10fc0b8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10fc0b6580 0x7f10fc0b8a90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1108009890 tx=0x7f1108008040 comp rx=0 tx=0).stop 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1114102690 msgr2=0x7f1114198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1114102690 0x7f1114198e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f110400c890 tx=0x7f110400cd50 comp rx=0 tx=0).stop 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 shutdown_connections 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10fc0b6580 0x7f10fc0b8a90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11141040c0 0x7f111419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1114103720 0x7f11141994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 --2- 172.21.15.29:0/1594051868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1114102690 0x7f1114198e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:41.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 >> 172.21.15.29:0/1594051868 conn(0x7f11140fe000 msgr2=0x7f1114106a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:41.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 shutdown_connections 2024-09-21T02:30:41.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:30:41.918+0000 7f1119b44640 1 -- 172.21.15.29:0/1594051868 wait complete. 2024-09-21T02:30:42.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:42.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:42.538 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:42.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:42.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:42.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:42.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:30:42.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:42.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:42.625 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.0.service 2024-09-21T02:30:42.630 INFO:tasks.cephadm:Deploying osd.1 on smithi066 with /dev/vg_nvme/lv_4... 2024-09-21T02:30:42.630 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T02:30:42.892 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:30:43.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:30:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0[46236]: 2024-09-21T02:30:42.837+0000 7fb677382640 -1 osd.0 0 log_to_monitors true 2024-09-21T02:30:43.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:43 smithi066 ceph-mon[33436]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:43.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:43 smithi066 ceph-mon[33436]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T02:30:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:43 smithi029 ceph-mon[31006]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:43 smithi029 ceph-mon[31006]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T02:30:43.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:43 smithi190 ceph-mon[33871]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:43.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:43 smithi190 ceph-mon[33871]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T02:30:44.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:44 smithi029 ceph-mon[31006]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T02:30:44.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:44 smithi029 ceph-mon[31006]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T02:30:44.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:44 smithi029 ceph-mon[31006]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:30:44.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:44.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:44 smithi029 ceph-mon[31006]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:30:44.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:44 smithi029 ceph-mon[31006]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T02:30:44.444 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:44 smithi066 ceph-mon[33436]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T02:30:44.444 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:44 smithi066 ceph-mon[33436]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T02:30:44.444 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:44 smithi066 ceph-mon[33436]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:30:44.444 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:44.444 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:44 smithi066 ceph-mon[33436]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:30:44.444 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:44 smithi066 ceph-mon[33436]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T02:30:44.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:44 smithi190 ceph-mon[33871]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T02:30:44.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:44 smithi190 ceph-mon[33871]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T02:30:44.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:44 smithi190 ceph-mon[33871]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:30:44.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:44.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:44 smithi190 ceph-mon[33871]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:30:44.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:44 smithi190 ceph-mon[33871]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T02:30:45.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:30:44 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0[46236]: 2024-09-21T02:30:44.691+0000 7fb673304640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T02:30:45.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:30:44 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0[46236]: 2024-09-21T02:30:44.694+0000 7fb66b0fa640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:30:45.227 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:30:45.252 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi066:vg_nvme/lv_4 2024-09-21T02:30:45.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:45 smithi066 ceph-mon[33436]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:45.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:45 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:45.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:45 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:45.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:45 smithi066 ceph-mon[33436]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' 2024-09-21T02:30:45.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:45 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:45.518 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:30:45.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:45 smithi190 ceph-mon[33871]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:45.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:45 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:45.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:45 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:45.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:45 smithi190 ceph-mon[33871]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' 2024-09-21T02:30:45.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:45 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:45 smithi029 ceph-mon[31006]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:45 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:45 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:45 smithi029 ceph-mon[31006]: from='osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118]' entity='osd.0' 2024-09-21T02:30:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:45 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:46.019 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 -- 172.21.15.66:0/4021653504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0105180 msgr2=0x7fc6c010bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 --2- 172.21.15.66:0/4021653504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0105180 0x7fc6c010bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009ca0 tx=0x7fc6b802a950 comp rx=0 tx=0).stop 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 -- 172.21.15.66:0/4021653504 shutdown_connections 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 --2- 172.21.15.66:0/4021653504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0105180 0x7fc6c010bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 --2- 172.21.15.66:0/4021653504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6c01047e0 0x7fc6c0104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 --2- 172.21.15.66:0/4021653504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6c0103750 0x7fc6c0103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 -- 172.21.15.66:0/4021653504 >> 172.21.15.66:0/4021653504 conn(0x7fc6c00ff0e0 msgr2=0x7fc6c0101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 -- 172.21.15.66:0/4021653504 shutdown_connections 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.015+0000 7fc6c813a640 1 -- 172.21.15.66:0/4021653504 wait complete. 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.016+0000 7fc6c813a640 1 Processor -- start 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.016+0000 7fc6c813a640 1 -- start start 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.016+0000 7fc6c813a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0103750 0x7fc6c0199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c813a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6c01047e0 0x7fc6c019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c813a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6c0105180 0x7fc6c019b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c813a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c010f250 con 0x7fc6c0105180 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c813a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c010f0d0 con 0x7fc6c0103750 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c813a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c010f3d0 con 0x7fc6c01047e0 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0103750 0x7fc6c0199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0103750 0x7fc6c0199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.66:52502/0 (socket says 172.21.15.66:52502) 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 -- 172.21.15.66:0/2866989050 learned_addr learned my addr 172.21.15.66:0/2866989050 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c56d5640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6c01047e0 0x7fc6c019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c66d7640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6c0105180 0x7fc6c019b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 -- 172.21.15.66:0/2866989050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6c01047e0 msgr2=0x7fc6c019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6c01047e0 0x7fc6c019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 -- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6c0105180 msgr2=0x7fc6c019b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6c0105180 0x7fc6c019b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 -- 172.21.15.66:0/2866989050 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c019b8e0 con 0x7fc6c0103750 2024-09-21T02:30:46.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.017+0000 7fc6c5ed6640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0103750 0x7fc6c0199eb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00e9f0 tx=0x7fc6bc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:46.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.018+0000 7fc6b6ffd640 1 -- 172.21.15.66:0/2866989050 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc00cd80 con 0x7fc6c0103750 2024-09-21T02:30:46.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.018+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c019bbd0 con 0x7fc6c0103750 2024-09-21T02:30:46.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.018+0000 7fc6b6ffd640 1 -- 172.21.15.66:0/2866989050 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6bc0108c0 con 0x7fc6c0103750 2024-09-21T02:30:46.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.018+0000 7fc6b6ffd640 1 -- 172.21.15.66:0/2866989050 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0187c0 con 0x7fc6c0103750 2024-09-21T02:30:46.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.018+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c01a22d0 con 0x7fc6c0103750 2024-09-21T02:30:46.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.019+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c0103bc0 con 0x7fc6c0103750 2024-09-21T02:30:46.023 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.020+0000 7fc6b6ffd640 1 -- 172.21.15.66:0/2866989050 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7fc6bc010430 con 0x7fc6c0103750 2024-09-21T02:30:46.023 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.021+0000 7fc6b6ffd640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc6ac0b6480 0x7fc6ac0b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:30:46.023 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.021+0000 7fc6b6ffd640 1 -- 172.21.15.66:0/2866989050 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7fc6bc0e5ca0 con 0x7fc6c0103750 2024-09-21T02:30:46.023 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.021+0000 7fc6c56d5640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc6ac0b6480 0x7fc6ac0b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:30:46.023 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.022+0000 7fc6c56d5640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc6ac0b6480 0x7fc6ac0b8990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0005fd0 tx=0x7fc6b000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:30:46.027 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.026+0000 7fc6b6ffd640 1 -- 172.21.15.66:0/2866989050 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc6bc0adfa0 con 0x7fc6c0103750 2024-09-21T02:30:46.224 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:30:46.222+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc6c0063800 con 0x7fc6ac0b6480 2024-09-21T02:30:46.871 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:30:46.871 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:30:46.871 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] boot 2024-09-21T02:30:46.871 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi029 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi029 to 17092M 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:30:46.872 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:30:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:30:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] boot 2024-09-21T02:30:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T02:30:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi029 2024-09-21T02:30:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi029 to 17092M 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:30:47.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: osd.0 [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] boot 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi029 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi029 to 17092M 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:30:47.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:47.961 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:47 smithi066 ceph-mon[33436]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T02:30:48.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:47 smithi190 ceph-mon[33871]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T02:30:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:47 smithi029 ceph-mon[31006]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T02:30:49.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:48 smithi066 ceph-mon[33436]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:48 smithi066 ceph-mon[33436]: from='client.? 172.21.15.66:0/572153188' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"} : dispatch 2024-09-21T02:30:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:48 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"} : dispatch 2024-09-21T02:30:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:48 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"}]': finished 2024-09-21T02:30:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:48 smithi066 ceph-mon[33436]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T02:30:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:30:49.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:48 smithi190 ceph-mon[33871]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:49.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:48 smithi190 ceph-mon[33871]: from='client.? 172.21.15.66:0/572153188' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"} : dispatch 2024-09-21T02:30:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:48 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"} : dispatch 2024-09-21T02:30:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:48 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"}]': finished 2024-09-21T02:30:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:48 smithi190 ceph-mon[33871]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T02:30:49.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:30:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:48 smithi029 ceph-mon[31006]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:48 smithi029 ceph-mon[31006]: from='client.? 172.21.15.66:0/572153188' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"} : dispatch 2024-09-21T02:30:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:48 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"} : dispatch 2024-09-21T02:30:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:48 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d7d39c-63c1-4b1c-905d-cbbad4c7e516"}]': finished 2024-09-21T02:30:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:48 smithi029 ceph-mon[31006]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T02:30:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:30:50.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:49 smithi066 ceph-mon[33436]: from='client.? 172.21.15.66:0/339180241' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:30:50.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:49 smithi190 ceph-mon[33871]: from='client.? 172.21.15.66:0/339180241' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:30:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:49 smithi029 ceph-mon[31006]: from='client.? 172.21.15.66:0/339180241' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:30:51.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:50 smithi066 ceph-mon[33436]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:51.025 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:50 smithi190 ceph-mon[33871]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:50 smithi029 ceph-mon[31006]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:53.247 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:52 smithi190 ceph-mon[33871]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:53.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:52 smithi066 ceph-mon[33436]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:53.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:52 smithi029 ceph-mon[31006]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:55.110 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:54 smithi066 ceph-mon[33436]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:55.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:54 smithi190 ceph-mon[33871]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:55.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:54 smithi029 ceph-mon[31006]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:57.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:56 smithi066 ceph-mon[33436]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:57.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:56 smithi190 ceph-mon[33871]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:56 smithi029 ceph-mon[31006]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:58.208 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T02:30:58.208 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:58.208 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:57 smithi066 ceph-mon[33436]: Deploying daemon osd.1 on smithi066 2024-09-21T02:30:58.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T02:30:58.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:58.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:57 smithi190 ceph-mon[33871]: Deploying daemon osd.1 on smithi066 2024-09-21T02:30:58.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T02:30:58.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:30:58.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:57 smithi029 ceph-mon[31006]: Deploying daemon osd.1 on smithi066 2024-09-21T02:30:59.047 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:30:58 smithi066 ceph-mon[33436]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:59.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:30:58 smithi190 ceph-mon[33871]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:30:59.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:30:58 smithi029 ceph-mon[31006]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:01.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:00 smithi066 ceph-mon[33436]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:01.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:00 smithi190 ceph-mon[33871]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:01.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:00 smithi029 ceph-mon[31006]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:03.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:02 smithi190 ceph-mon[33871]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:03.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:02 smithi029 ceph-mon[31006]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:03.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:02 smithi066 ceph-mon[33436]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:04.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:04.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:04.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:04.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:04.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:04.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:04.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:04.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:04.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:05.453 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:05 smithi066 ceph-mon[33436]: pgmap v73: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:05.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:05 smithi190 ceph-mon[33871]: pgmap v73: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:05.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:05 smithi029 ceph-mon[31006]: pgmap v73: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:06.785 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:06 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:06.785 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:06 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:06.785 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:06 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:06.785 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:06 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:06.785 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:06 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:06.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:06 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:06.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:06 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:06.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:06 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:06.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:06 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:06.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:06 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:06.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:06 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:06.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:06 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:06.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:06 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:06.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:06 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:06.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:06 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:07.398 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 1 on host 'smithi066' 2024-09-21T02:31:07.398 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.396+0000 7fc6b6ffd640 1 -- 172.21.15.66:0/2866989050 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc6c0063800 con 0x7fc6ac0b6480 2024-09-21T02:31:07.400 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.398+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc6ac0b6480 msgr2=0x7fc6ac0b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:07.400 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.398+0000 7fc6c813a640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc6ac0b6480 0x7fc6ac0b8990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0005fd0 tx=0x7fc6b000b390 comp rx=0 tx=0).stop 2024-09-21T02:31:07.400 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.398+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0103750 msgr2=0x7fc6c0199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:07.400 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.398+0000 7fc6c813a640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0103750 0x7fc6c0199eb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00e9f0 tx=0x7fc6bc00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:31:07.400 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.399+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 shutdown_connections 2024-09-21T02:31:07.400 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.399+0000 7fc6c813a640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc6ac0b6480 0x7fc6ac0b8990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:07.401 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.399+0000 7fc6c813a640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6c0105180 0x7fc6c019b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:07.401 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.399+0000 7fc6c813a640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6c01047e0 0x7fc6c019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:07.401 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.399+0000 7fc6c813a640 1 --2- 172.21.15.66:0/2866989050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c0103750 0x7fc6c0199eb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:07.401 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.399+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 >> 172.21.15.66:0/2866989050 conn(0x7fc6c00ff0e0 msgr2=0x7fc6c0107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:31:07.401 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.399+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 shutdown_connections 2024-09-21T02:31:07.401 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:31:07.399+0000 7fc6c813a640 1 -- 172.21.15.66:0/2866989050 wait complete. 2024-09-21T02:31:07.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:07 smithi066 ceph-mon[33436]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:07.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:07.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:07.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:07.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:07 smithi190 ceph-mon[33871]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:07 smithi029 ceph-mon[31006]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:08.013 DEBUG:teuthology.orchestra.run.smithi066:osd.1> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.1.service 2024-09-21T02:31:08.015 INFO:tasks.cephadm:Deploying osd.2 on smithi190 with /dev/vg_nvme/lv_4... 2024-09-21T02:31:08.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T02:31:08.266 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:31:09.260 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:09 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1[37383]: 2024-09-21T02:31:09.227+0000 7fc4ff8eb640 -1 osd.1 0 log_to_monitors true 2024-09-21T02:31:09.562 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:09 smithi190 ceph-mon[33871]: pgmap v75: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:09.562 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:09 smithi190 ceph-mon[33871]: from='osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T02:31:09.562 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:09 smithi190 ceph-mon[33871]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T02:31:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:09 smithi029 ceph-mon[31006]: pgmap v75: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:09 smithi029 ceph-mon[31006]: from='osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T02:31:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:09 smithi029 ceph-mon[31006]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T02:31:09.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:09 smithi066 ceph-mon[33436]: pgmap v75: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:09.985 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:09 smithi066 ceph-mon[33436]: from='osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T02:31:09.985 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:09 smithi066 ceph-mon[33436]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T02:31:10.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:10 smithi190 ceph-mon[33871]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T02:31:10.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:10 smithi190 ceph-mon[33871]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T02:31:10.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:10.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:10 smithi190 ceph-mon[33871]: from='osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:31:10.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:10 smithi190 ceph-mon[33871]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:31:10.642 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:31:10.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2024-09-21T02:31:10.681 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:10 smithi066 ceph-mon[33436]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T02:31:10.681 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:10 smithi066 ceph-mon[33436]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T02:31:10.681 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:10.682 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:10 smithi066 ceph-mon[33436]: from='osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:31:10.682 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:10 smithi066 ceph-mon[33436]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:31:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:10 smithi029 ceph-mon[31006]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T02:31:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:10 smithi029 ceph-mon[31006]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T02:31:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:10 smithi029 ceph-mon[31006]: from='osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:31:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:10 smithi029 ceph-mon[31006]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:31:10.930 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:31:11.259 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:10 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1[37383]: 2024-09-21T02:31:10.993+0000 7fc4fb86d640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T02:31:11.259 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:10 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1[37383]: 2024-09-21T02:31:10.997+0000 7fc4f3663640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:31:11.371 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 -- 172.21.15.190:0/4145059035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 msgr2=0x7f8384103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:11.371 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 --2- 172.21.15.190:0/4145059035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 0x7f8384103b00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8378009900 tx=0x7f837802a480 comp rx=0 tx=0).stop 2024-09-21T02:31:11.371 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 -- 172.21.15.190:0/4145059035 shutdown_connections 2024-09-21T02:31:11.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 --2- 172.21.15.190:0/4145059035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8384104090 0x7f838410aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:11.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 --2- 172.21.15.190:0/4145059035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 0x7f8384103b00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:11.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 --2- 172.21.15.190:0/4145059035 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8384102660 0x7f8384102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:11.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 -- 172.21.15.190:0/4145059035 >> 172.21.15.190:0/4145059035 conn(0x7f83840fdfd0 msgr2=0x7f8384100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:31:11.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 -- 172.21.15.190:0/4145059035 shutdown_connections 2024-09-21T02:31:11.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.370+0000 7f838abcd640 1 -- 172.21.15.190:0/4145059035 wait complete. 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838abcd640 1 Processor -- start 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838abcd640 1 -- start start 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838abcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8384102660 0x7f8384199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838abcd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 0x7f838419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838abcd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8384104090 0x7f838419b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838abcd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f838410f280 con 0x7f8384102660 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838abcd640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f838410f100 con 0x7f8384104090 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838abcd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f838410f400 con 0x7f83841036f0 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f8383fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 0x7f838419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f8383fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 0x7f838419a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:45302/0 (socket says 172.21.15.190:45302) 2024-09-21T02:31:11.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f8383fff640 1 -- 172.21.15.190:0/2844403682 learned_addr learned my addr 172.21.15.190:0/2844403682 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f838916a640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8384104090 0x7f838419b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.371+0000 7f8388969640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8384102660 0x7f8384199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8383fff640 1 -- 172.21.15.190:0/2844403682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8384104090 msgr2=0x7f838419b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8383fff640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8384104090 0x7f838419b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8383fff640 1 -- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8384102660 msgr2=0x7f8384199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8383fff640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8384102660 0x7f8384199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8383fff640 1 -- 172.21.15.190:0/2844403682 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838419b9c0 con 0x7f83841036f0 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8383fff640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 0x7f838419a600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8378009230 tx=0x7f83780042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8381ffb640 1 -- 172.21.15.190:0/2844403682 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837803f030 con 0x7f83841036f0 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8381ffb640 1 -- 172.21.15.190:0/2844403682 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8378003c90 con 0x7f83841036f0 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f838419bc50 con 0x7f83841036f0 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f8381ffb640 1 -- 172.21.15.190:0/2844403682 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378035070 con 0x7f83841036f0 2024-09-21T02:31:11.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.372+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83841a23b0 con 0x7f83841036f0 2024-09-21T02:31:11.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.373+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8384102ad0 con 0x7f83841036f0 2024-09-21T02:31:11.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.375+0000 7f8381ffb640 1 -- 172.21.15.190:0/2844403682 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f8378042030 con 0x7f83841036f0 2024-09-21T02:31:11.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.375+0000 7f8381ffb640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f836c0b6500 0x7f836c0b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:31:11.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.376+0000 7f8381ffb640 1 -- 172.21.15.190:0/2844403682 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2029+0+0 (secure 0 0 0) 0x7f83781063d0 con 0x7f83841036f0 2024-09-21T02:31:11.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.376+0000 7f8388969640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f836c0b6500 0x7f836c0b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:31:11.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.376+0000 7f8388969640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f836c0b6500 0x7f836c0b8a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8374005fd0 tx=0x7f8374005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:31:11.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.379+0000 7f8381ffb640 1 -- 172.21.15.190:0/2844403682 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f83780ce5d0 con 0x7f83841036f0 2024-09-21T02:31:11.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:11.563+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8384063800 con 0x7f836c0b6500 2024-09-21T02:31:11.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:11 smithi190 ceph-mon[33871]: from='osd.1 ' entity='osd.1' 2024-09-21T02:31:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:31:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T02:31:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:31:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:11.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:11 smithi029 ceph-mon[31006]: from='osd.1 ' entity='osd.1' 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:12.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:11 smithi066 ceph-mon[33436]: from='osd.1 ' entity='osd.1' 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi066 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi066 to 17088M 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] boot 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T02:31:12.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:12 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi066 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi066 to 17088M 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] boot 2024-09-21T02:31:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T02:31:12.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:12 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:13.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi066 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi066 to 17088M 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: osd.1 [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] boot 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T02:31:13.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:12 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T02:31:13.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:13 smithi190 ceph-mon[33871]: pgmap v79: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:13.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:13 smithi190 ceph-mon[33871]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T02:31:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:13 smithi029 ceph-mon[31006]: pgmap v79: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:13 smithi029 ceph-mon[31006]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T02:31:14.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:13 smithi066 ceph-mon[33436]: pgmap v79: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-09-21T02:31:14.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:13 smithi066 ceph-mon[33436]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T02:31:14.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:14 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/536241774' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"} : dispatch 2024-09-21T02:31:14.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:14 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"} : dispatch 2024-09-21T02:31:14.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:14 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"}]': finished 2024-09-21T02:31:14.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:14 smithi190 ceph-mon[33871]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T02:31:14.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:14.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:14 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/291235870' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:31:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:14 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/536241774' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"} : dispatch 2024-09-21T02:31:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:14 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"} : dispatch 2024-09-21T02:31:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:14 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"}]': finished 2024-09-21T02:31:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:14 smithi029 ceph-mon[31006]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T02:31:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:14 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/291235870' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:31:15.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:14 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/536241774' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"} : dispatch 2024-09-21T02:31:15.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:14 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"} : dispatch 2024-09-21T02:31:15.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:14 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492"}]': finished 2024-09-21T02:31:15.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:14 smithi066 ceph-mon[33436]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T02:31:15.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:15.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:14 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/291235870' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:31:15.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:15 smithi190 ceph-mon[33871]: pgmap v83: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:15 smithi029 ceph-mon[31006]: pgmap v83: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:16.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:15 smithi066 ceph-mon[33436]: pgmap v83: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:17.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:17 smithi190 ceph-mon[33871]: pgmap v84: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:17 smithi029 ceph-mon[31006]: pgmap v84: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:18.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:17 smithi066 ceph-mon[33436]: pgmap v84: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:19.825 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:19 smithi190 ceph-mon[33871]: pgmap v85: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:19 smithi029 ceph-mon[31006]: pgmap v85: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:19 smithi066 ceph-mon[33436]: pgmap v85: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:21.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:21 smithi190 ceph-mon[33871]: pgmap v86: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:21 smithi029 ceph-mon[31006]: pgmap v86: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:21 smithi066 ceph-mon[33436]: pgmap v86: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:22.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T02:31:22.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:22.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T02:31:22.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:23.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T02:31:23.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:23.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:23 smithi190 ceph-mon[33871]: Deploying daemon osd.2 on smithi190 2024-09-21T02:31:23.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:23 smithi190 ceph-mon[33871]: pgmap v87: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:23 smithi029 ceph-mon[31006]: Deploying daemon osd.2 on smithi190 2024-09-21T02:31:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:23 smithi029 ceph-mon[31006]: pgmap v87: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:23.910 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:23 smithi066 ceph-mon[33436]: Deploying daemon osd.2 on smithi190 2024-09-21T02:31:23.911 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:23 smithi066 ceph-mon[33436]: pgmap v87: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:25.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:25 smithi190 ceph-mon[33871]: pgmap v88: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:25 smithi029 ceph-mon[31006]: pgmap v88: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:25 smithi066 ceph-mon[33436]: pgmap v88: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:27.745 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:27 smithi190 ceph-mon[33871]: pgmap v89: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:27 smithi029 ceph-mon[31006]: pgmap v89: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:27 smithi066 ceph-mon[33436]: pgmap v89: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:28.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:28 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:28.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:28 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:28.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:28 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:28.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:28 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:28.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:28 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:28.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:28 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:29.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:28 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:29.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:28 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:29.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:28 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:29.727 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:29 smithi190 ceph-mon[33871]: pgmap v90: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:29 smithi029 ceph-mon[31006]: pgmap v90: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:29 smithi066 ceph-mon[33436]: pgmap v90: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:31.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:30 smithi190 ceph-mon[33871]: pgmap v91: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:30 smithi029 ceph-mon[31006]: pgmap v91: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:31.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:31.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:31.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:31.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:30 smithi066 ceph-mon[33436]: pgmap v91: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:31.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.950+0000 7f8381ffb640 1 -- 172.21.15.190:0/2844403682 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8384063800 con 0x7f836c0b6500 2024-09-21T02:31:31.952 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 2 on host 'smithi190' 2024-09-21T02:31:31.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f836c0b6500 msgr2=0x7f836c0b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:31.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f836c0b6500 0x7f836c0b8a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8374005fd0 tx=0x7f8374005850 comp rx=0 tx=0).stop 2024-09-21T02:31:31.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 msgr2=0x7f838419a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:31.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 0x7f838419a600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8378009230 tx=0x7f83780042f0 comp rx=0 tx=0).stop 2024-09-21T02:31:31.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 shutdown_connections 2024-09-21T02:31:31.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f836c0b6500 0x7f836c0b8a10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:31.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8384104090 0x7f838419b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:31.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83841036f0 0x7f838419a600 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:31.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 --2- 172.21.15.190:0/2844403682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8384102660 0x7f8384199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:31.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 >> 172.21.15.190:0/2844403682 conn(0x7f83840fdfd0 msgr2=0x7f83841069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:31:31.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 shutdown_connections 2024-09-21T02:31:31.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:31:31.953+0000 7f838abcd640 1 -- 172.21.15.190:0/2844403682 wait complete. 2024-09-21T02:31:32.561 DEBUG:teuthology.orchestra.run.smithi190:osd.2> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.2.service 2024-09-21T02:31:32.563 INFO:tasks.cephadm:Deploying osd.3 on smithi029 with /dev/vg_nvme/lv_3... 2024-09-21T02:31:32.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T02:31:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:31:33.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:32 smithi190 ceph-mon[33871]: pgmap v92: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:33.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:33.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:32 smithi029 ceph-mon[31006]: pgmap v92: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:33.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:33.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:33.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:33.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:32 smithi066 ceph-mon[33436]: pgmap v92: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:33.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:33.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:34.374 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:34 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2[38327]: 2024-09-21T02:31:34.033+0000 7f3f059e8640 -1 osd.2 0 log_to_monitors true 2024-09-21T02:31:35.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:34 smithi066 ceph-mon[33436]: pgmap v93: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:35.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:34 smithi066 ceph-mon[33436]: from='osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T02:31:35.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:34 smithi066 ceph-mon[33436]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T02:31:35.291 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:34 smithi190 ceph-mon[33871]: pgmap v93: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:35.291 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:34 smithi190 ceph-mon[33871]: from='osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T02:31:35.291 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:34 smithi190 ceph-mon[33871]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T02:31:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:34 smithi029 ceph-mon[31006]: pgmap v93: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:34 smithi029 ceph-mon[31006]: from='osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T02:31:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:34 smithi029 ceph-mon[31006]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T02:31:35.845 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:31:35.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-09-21T02:31:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T02:31:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T02:31:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:31:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:31:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi190 2024-09-21T02:31:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:31:36.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi190 to 17088M 2024-09-21T02:31:36.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:36.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:36.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.139 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:31:36.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi190 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi190 to 17088M 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:36.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi190 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi190 to 17088M 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:36.330 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:36.330 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:36.623 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2[38327]: 2024-09-21T02:31:36.327+0000 7f3f0196a640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T02:31:36.624 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2[38327]: 2024-09-21T02:31:36.333+0000 7f3ef9760640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:31:36.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 -- 172.21.15.29:0/891557257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 msgr2=0x7f0c94106440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 --2- 172.21.15.29:0/891557257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 0x7f0c94106440 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009900 tx=0x7f0c8402a410 comp rx=0 tx=0).stop 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 -- 172.21.15.29:0/891557257 shutdown_connections 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 --2- 172.21.15.29:0/891557257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0c94109490 0x7f0c9410b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 --2- 172.21.15.29:0/891557257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c94106980 0x7f0c94108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 --2- 172.21.15.29:0/891557257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 0x7f0c94106440 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 -- 172.21.15.29:0/891557257 >> 172.21.15.29:0/891557257 conn(0x7f0c940fdfe0 msgr2=0x7f0c94100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 -- 172.21.15.29:0/891557257 shutdown_connections 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 -- 172.21.15.29:0/891557257 wait complete. 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.644+0000 7f0c99588640 1 Processor -- start 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c99588640 1 -- start start 2024-09-21T02:31:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c99588640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 0x7f0c94199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c99588640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c94106980 0x7f0c9419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c99588640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0c94109490 0x7f0c9419b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c99588640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9410f240 con 0x7f0c9406c5c0 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c99588640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9410f0c0 con 0x7f0c94109490 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c99588640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9410f3c0 con 0x7f0c94106980 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 0x7f0c94199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 0x7f0c94199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60738/0 (socket says 172.21.15.29:60738) 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 -- 172.21.15.29:0/1926664588 learned_addr learned my addr 172.21.15.29:0/1926664588 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c937fe640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0c94109490 0x7f0c9419b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c927fc640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c94106980 0x7f0c9419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 -- 172.21.15.29:0/1926664588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c94106980 msgr2=0x7f0c9419a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c94106980 0x7f0c9419a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 -- 172.21.15.29:0/1926664588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0c94109490 msgr2=0x7f0c9419b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0c94109490 0x7f0c9419b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 -- 172.21.15.29:0/1926664588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9419b920 con 0x7f0c9406c5c0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c927fc640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c94106980 0x7f0c9419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.645+0000 7f0c92ffd640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 0x7f0c94199ef0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009230 tx=0x7f0c840040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.646+0000 7f0c7bfff640 1 -- 172.21.15.29:0/1926664588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8403f030 con 0x7f0c9406c5c0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.646+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c9419bbb0 con 0x7f0c9406c5c0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.646+0000 7f0c7bfff640 1 -- 172.21.15.29:0/1926664588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c840044c0 con 0x7f0c9406c5c0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.646+0000 7f0c7bfff640 1 -- 172.21.15.29:0/1926664588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84035070 con 0x7f0c9406c5c0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.646+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c941a2310 con 0x7f0c9406c5c0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.647+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c940ffb00 con 0x7f0c9406c5c0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.648+0000 7f0c7bfff640 1 -- 172.21.15.29:0/1926664588 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 154456+0+0 (secure 0 0 0) 0x7f0c84042030 con 0x7f0c9406c5c0 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.649+0000 7f0c7bfff640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0c7c0b6480 0x7f0c7c0b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:31:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.649+0000 7f0c927fc640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0c7c0b6480 0x7f0c7c0b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:31:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.649+0000 7f0c7bfff640 1 -- 172.21.15.29:0/1926664588 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2571+0+0 (secure 0 0 0) 0x7f0c84106670 con 0x7f0c9406c5c0 2024-09-21T02:31:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.650+0000 7f0c927fc640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0c7c0b6480 0x7f0c7c0b8990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009b50 tx=0x7f0c88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:31:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.652+0000 7f0c7bfff640 1 -- 172.21.15.29:0/1926664588 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0c840ce5b0 con 0x7f0c9406c5c0 2024-09-21T02:31:36.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:36.851+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0c94063800 con 0x7f0c7c0b6480 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: pgmap v95: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: from='osd.2 ' entity='osd.2' 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:31:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:37.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: pgmap v95: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:31:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T02:31:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: from='osd.2 ' entity='osd.2' 2024-09-21T02:31:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:31:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:31:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: pgmap v95: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:31:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T02:31:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: from='osd.2 ' entity='osd.2' 2024-09-21T02:31:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:31:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:31:37.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:38.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:37 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:31:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:37 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:31:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:37 smithi066 ceph-mon[33436]: from='client.14463 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:37 smithi066 ceph-mon[33436]: osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] boot 2024-09-21T02:31:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:37 smithi066 ceph-mon[33436]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T02:31:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:38.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:37 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:31:38.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:37 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:31:38.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:37 smithi029 ceph-mon[31006]: from='client.14463 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:38.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:38.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:37 smithi029 ceph-mon[31006]: osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] boot 2024-09-21T02:31:38.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:37 smithi029 ceph-mon[31006]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T02:31:38.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:38.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:37 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:31:38.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:37 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:31:38.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:37 smithi190 ceph-mon[33871]: from='client.14463 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:38.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:38.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:37 smithi190 ceph-mon[33871]: osd.2 [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] boot 2024-09-21T02:31:38.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:37 smithi190 ceph-mon[33871]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T02:31:38.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T02:31:39.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:38 smithi066 ceph-mon[33436]: pgmap v98: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:39.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:31:39.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:38 smithi190 ceph-mon[33871]: pgmap v98: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:39.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:31:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:38 smithi029 ceph-mon[31006]: pgmap v98: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-09-21T02:31:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:31:39.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50828]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T02:31:39.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50828]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50828]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50828]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50836]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T02:31:39.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50836]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50836]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.784 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50836]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50821]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T02:31:39.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50821]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50821]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50821]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50824]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T02:31:39.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50824]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50824]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 02:31:39 smithi029 sudo[50824]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.941 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42222]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T02:31:39.941 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42222]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.941 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42222]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.941 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42222]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.942 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42225]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T02:31:39.942 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42225]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.942 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42225]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.942 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42225]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.942 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42216]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T02:31:39.942 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42216]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.942 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42216]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.942 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42216]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.942 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42219]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T02:31:39.942 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.942 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42219]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.943 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 02:31:39 smithi190 sudo[42219]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.943 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41309]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T02:31:39.943 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41309]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.943 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41309]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.943 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41309]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.943 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41312]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T02:31:39.943 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41312]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.943 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41312]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.944 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41312]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41315]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41315]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41315]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41315]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41318]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41318]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41318]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 sudo[41318]: pam_unix(sudo:session): session closed for user root 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:31:39.944 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3013438422' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c4edcdb-857c-4f2a-a290-ba400aea7930"} : dispatch 2024-09-21T02:31:39.945 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T02:31:40.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T02:31:40.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T02:31:40.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:31:40.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3013438422' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c4edcdb-857c-4f2a-a290-ba400aea7930"} : dispatch 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3013438422' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c4edcdb-857c-4f2a-a290-ba400aea7930"}]': finished 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: osdmap e20: 4 total, 3 up, 4 in 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:39 smithi029 ceph-mon[31006]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3013438422' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c4edcdb-857c-4f2a-a290-ba400aea7930"}]': finished 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: osdmap e20: 4 total, 3 up, 4 in 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:39 smithi066 ceph-mon[33436]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3013438422' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c4edcdb-857c-4f2a-a290-ba400aea7930"} : dispatch 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3013438422' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c4edcdb-857c-4f2a-a290-ba400aea7930"}]': finished 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: osdmap e20: 4 total, 3 up, 4 in 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T02:31:40.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:39 smithi190 ceph-mon[33871]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T02:31:40.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:40.691+0000 7f0c7bfff640 1 -- 172.21.15.29:0/1926664588 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f0c840d5d10 con 0x7f0c9406c5c0 2024-09-21T02:31:40.943 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:40 smithi029 ceph-mon[31006]: pgmap v101: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:40.943 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:40 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4088338090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:31:40.943 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:40 smithi029 ceph-mon[31006]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T02:31:40.943 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:31:40.943 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:40 smithi029 ceph-mon[31006]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-09-21T02:31:41.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:40 smithi066 ceph-mon[33436]: pgmap v101: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:41.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:40 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4088338090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:31:41.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:40 smithi066 ceph-mon[33436]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T02:31:41.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:31:41.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:40 smithi066 ceph-mon[33436]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-09-21T02:31:41.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:40 smithi190 ceph-mon[33871]: pgmap v101: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:41.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:40 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4088338090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:31:41.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:40 smithi190 ceph-mon[33871]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T02:31:41.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:31:41.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:40 smithi190 ceph-mon[33871]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-09-21T02:31:43.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:42 smithi066 ceph-mon[33436]: pgmap v103: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:43.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:42 smithi190 ceph-mon[33871]: pgmap v103: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:42 smithi029 ceph-mon[31006]: pgmap v103: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:45.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:44 smithi066 ceph-mon[33436]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:45.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:44 smithi190 ceph-mon[33871]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:45.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:44 smithi029 ceph-mon[31006]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:47.222 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:46 smithi029 ceph-mon[31006]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:47.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:46 smithi066 ceph-mon[33436]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:47.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:46 smithi190 ceph-mon[33871]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:49.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:48 smithi029 ceph-mon[31006]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:49.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T02:31:49.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:49.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:48 smithi029 ceph-mon[31006]: Deploying daemon osd.3 on smithi029 2024-09-21T02:31:49.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:48 smithi066 ceph-mon[33436]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:49.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T02:31:49.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:49.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:48 smithi066 ceph-mon[33436]: Deploying daemon osd.3 on smithi029 2024-09-21T02:31:49.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:48 smithi190 ceph-mon[33871]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:49.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T02:31:49.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:49.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:48 smithi190 ceph-mon[33871]: Deploying daemon osd.3 on smithi029 2024-09-21T02:31:51.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:50 smithi066 ceph-mon[33436]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:51.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:50 smithi029 ceph-mon[31006]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:51.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:50 smithi190 ceph-mon[33871]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:53.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:52 smithi066 ceph-mon[33436]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:53.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:52 smithi190 ceph-mon[33871]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:53.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:52 smithi029 ceph-mon[31006]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:55.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:54 smithi066 ceph-mon[33436]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:55.271 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:54 smithi029 ceph-mon[31006]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:55.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:54 smithi190 ceph-mon[33871]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:56.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:55 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:56.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:55 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:56.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:55 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:56.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:55 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:56.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:55 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:56.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:55 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:56.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:55 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:31:56.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:55 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:56.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:55 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:57 smithi029 ceph-mon[31006]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:57.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:57 smithi190 ceph-mon[33871]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:57.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:57 smithi066 ceph-mon[33436]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:59 smithi066 ceph-mon[33436]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:31:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:59 smithi029 ceph-mon[31006]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:59.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:59.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:59.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:59.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:31:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:59.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.564+0000 7f0c7bfff640 1 -- 172.21.15.29:0/1926664588 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0c94063800 con 0x7f0c7c0b6480 2024-09-21T02:31:59.565 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 3 on host 'smithi029' 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0c7c0b6480 msgr2=0x7f0c7c0b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0c7c0b6480 0x7f0c7c0b8990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009b50 tx=0x7f0c88005f90 comp rx=0 tx=0).stop 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 msgr2=0x7f0c94199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 0x7f0c94199ef0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009230 tx=0x7f0c840040c0 comp rx=0 tx=0).stop 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 shutdown_connections 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0c7c0b6480 0x7f0c7c0b8990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0c94109490 0x7f0c9419b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c94106980 0x7f0c9419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 --2- 172.21.15.29:0/1926664588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c9406c5c0 0x7f0c94199ef0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:31:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.566+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 >> 172.21.15.29:0/1926664588 conn(0x7f0c940fdfe0 msgr2=0x7f0c94104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:31:59.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.567+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 shutdown_connections 2024-09-21T02:31:59.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:31:59.567+0000 7f0c99588640 1 -- 172.21.15.29:0/1926664588 wait complete. 2024-09-21T02:31:59.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:59 smithi190 ceph-mon[33871]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:31:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:31:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:31:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:31:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:31:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:00.269 DEBUG:teuthology.orchestra.run.smithi029:osd.3> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.3.service 2024-09-21T02:32:00.272 INFO:tasks.cephadm:Deploying osd.4 on smithi066 with /dev/vg_nvme/lv_3... 2024-09-21T02:32:00.272 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T02:32:00.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:00 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:00.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:00 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:00.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:00 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:00.536 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:32:00.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:00.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:00.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:00.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:00 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:00.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:00 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:00.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:00 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:01.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:01 smithi029 ceph-mon[31006]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:32:01.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:01 smithi029 ceph-mon[31006]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T02:32:01.632 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 02:32:01 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T02:32:01.327+0000 7f4ceb9c5640 -1 osd.3 0 log_to_monitors true 2024-09-21T02:32:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:01 smithi066 ceph-mon[33436]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:32:01.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:01 smithi066 ceph-mon[33436]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T02:32:01.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:01 smithi190 ceph-mon[33871]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:32:01.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:01 smithi190 ceph-mon[33871]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T02:32:02.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:02 smithi066 ceph-mon[33436]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T02:32:02.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:02 smithi066 ceph-mon[33436]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T02:32:02.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:02 smithi066 ceph-mon[33436]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:32:02.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:02 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:02.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:02 smithi066 ceph-mon[33436]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:32:02.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:02 smithi066 ceph-mon[33436]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T02:32:02.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:02 smithi190 ceph-mon[33871]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T02:32:02.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:02 smithi190 ceph-mon[33871]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T02:32:02.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:02 smithi190 ceph-mon[33871]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:32:02.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:02 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:02.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:02 smithi190 ceph-mon[33871]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:32:02.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:02 smithi190 ceph-mon[33871]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T02:32:02.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:02 smithi029 ceph-mon[31006]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T02:32:02.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:02 smithi029 ceph-mon[31006]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T02:32:02.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:02 smithi029 ceph-mon[31006]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:32:02.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:02 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:02.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:02 smithi029 ceph-mon[31006]: from='osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:32:02.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:02 smithi029 ceph-mon[31006]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T02:32:02.883 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 02:32:02 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T02:32:02.572+0000 7f4ce7947640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T02:32:02.883 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 02:32:02 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T02:32:02.579+0000 7f4ce3745640 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:32:03.238 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:32:03.268 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi066:vg_nvme/lv_3 2024-09-21T02:32:03.536 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] boot 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: osdmap e24: 4 total, 4 up, 4 in 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:03.819 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.820 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:03.820 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:03.820 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] boot 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: osdmap e24: 4 total, 4 up, 4 in 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:03.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:03.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:32:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: osd.3 [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] boot 2024-09-21T02:32:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: osdmap e24: 4 total, 4 up, 4 in 2024-09-21T02:32:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:32:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:03.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:03.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:03.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:03.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:03.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:04.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 -- 172.21.15.66:0/167714318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c105750 msgr2=0x7f287c107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 --2- 172.21.15.66:0/167714318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c105750 0x7f287c107c30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f286c009900 tx=0x7f286c02a410 comp rx=0 tx=0).stop 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 -- 172.21.15.66:0/167714318 shutdown_connections 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 --2- 172.21.15.66:0/167714318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287c108370 0x7f287c10a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 --2- 172.21.15.66:0/167714318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c105750 0x7f287c107c30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 --2- 172.21.15.66:0/167714318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f287c102c90 0x7f287c105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 -- 172.21.15.66:0/167714318 >> 172.21.15.66:0/167714318 conn(0x7f287c0fd770 msgr2=0x7f287c0ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 -- 172.21.15.66:0/167714318 shutdown_connections 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.020+0000 7f2881458640 1 -- 172.21.15.66:0/167714318 wait complete. 2024-09-21T02:32:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.021+0000 7f2881458640 1 Processor -- start 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.021+0000 7f2881458640 1 -- start start 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f2881458640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287c102c90 0x7f287c199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f2881458640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f287c105750 0x7f287c19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f2881458640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c108370 0x7f287c19b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f2881458640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f287c10f250 con 0x7f287c102c90 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f2881458640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f287c10f0d0 con 0x7f287c108370 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f2881458640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f287c10f3d0 con 0x7f287c105750 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c108370 0x7f287c19b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c108370 0x7f287c19b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.66:50448/0 (socket says 172.21.15.66:50448) 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 -- 172.21.15.66:0/1001245088 learned_addr learned my addr 172.21.15.66:0/1001245088 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-09-21T02:32:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287a7fc640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f287c105750 0x7f287c19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287affd640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287c102c90 0x7f287c199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 -- 172.21.15.66:0/1001245088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f287c105750 msgr2=0x7f287c19a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f287c105750 0x7f287c19a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 -- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287c102c90 msgr2=0x7f287c199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287c102c90 0x7f287c199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 -- 172.21.15.66:0/1001245088 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287c19b810 con 0x7f287c108370 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287affd640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287c102c90 0x7f287c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.022+0000 7f287b7fe640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c108370 0x7f287c19b080 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f287400cce0 tx=0x7f287400a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.023+0000 7f2873fff640 1 -- 172.21.15.66:0/1001245088 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287400f450 con 0x7f287c108370 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.023+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f287c19bb00 con 0x7f287c108370 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.023+0000 7f2873fff640 1 -- 172.21.15.66:0/1001245088 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f287400fa80 con 0x7f287c108370 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.023+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f287c1a2290 con 0x7f287c108370 2024-09-21T02:32:04.025 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.023+0000 7f2873fff640 1 -- 172.21.15.66:0/1001245088 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874018ad0 con 0x7f287c108370 2024-09-21T02:32:04.026 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.024+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f287c105c20 con 0x7f287c108370 2024-09-21T02:32:04.027 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.027+0000 7f2873fff640 1 -- 172.21.15.66:0/1001245088 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f287400f5f0 con 0x7f287c108370 2024-09-21T02:32:04.028 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.028+0000 7f2873fff640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f28680b6510 0x7f28680b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:04.028 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.028+0000 7f2873fff640 1 -- 172.21.15.66:0/1001245088 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 3669+0+0 (secure 0 0 0) 0x7f28740e6980 con 0x7f287c108370 2024-09-21T02:32:04.029 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.028+0000 7f287affd640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f28680b6510 0x7f28680b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:04.029 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.029+0000 7f287affd640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f28680b6510 0x7f28680b8a20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f286400abe0 tx=0x7f286400a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:32:04.031 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.031+0000 7f2873fff640 1 -- 172.21.15.66:0/1001245088 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2874014030 con 0x7f287c108370 2024-09-21T02:32:04.232 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:04.231+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f287c063800 con 0x7f28680b6510 2024-09-21T02:32:04.867 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:04 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:32:04.867 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:04 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:32:04.867 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:04 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi029 2024-09-21T02:32:04.867 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:04 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi029 to 8546M 2024-09-21T02:32:04.867 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:04 smithi066 ceph-mon[33436]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T02:32:04.867 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:04.867 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:04.868 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:04.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:04 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:04 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:04 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi029 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:04 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi029 to 8546M 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:04 smithi190 ceph-mon[33871]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:04 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:32:04.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:04 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:32:04.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:04 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi029 2024-09-21T02:32:04.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:04 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi029 to 8546M 2024-09-21T02:32:04.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:04 smithi029 ceph-mon[31006]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T02:32:04.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:04.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:04.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:05 smithi190 ceph-mon[33871]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:05 smithi190 ceph-mon[33871]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:05 smithi190 ceph-mon[33871]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T02:32:05.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:05 smithi029 ceph-mon[31006]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:05 smithi029 ceph-mon[31006]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:05 smithi029 ceph-mon[31006]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T02:32:05.931 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:05 smithi066 ceph-mon[33436]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:05.932 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:05 smithi066 ceph-mon[33436]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:05.932 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:05 smithi066 ceph-mon[33436]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T02:32:07.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:07 smithi190 ceph-mon[33871]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:07 smithi190 ceph-mon[33871]: from='client.? 172.21.15.66:0/3493378305' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"} : dispatch 2024-09-21T02:32:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:07 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"} : dispatch 2024-09-21T02:32:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:07 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"}]': finished 2024-09-21T02:32:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:07 smithi190 ceph-mon[33871]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T02:32:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:07 smithi190 ceph-mon[33871]: from='client.? 172.21.15.66:0/322367130' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:32:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:07 smithi029 ceph-mon[31006]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:07 smithi029 ceph-mon[31006]: from='client.? 172.21.15.66:0/3493378305' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"} : dispatch 2024-09-21T02:32:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:07 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"} : dispatch 2024-09-21T02:32:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:07 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"}]': finished 2024-09-21T02:32:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:07 smithi029 ceph-mon[31006]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T02:32:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:07 smithi029 ceph-mon[31006]: from='client.? 172.21.15.66:0/322367130' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:32:08.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:07 smithi066 ceph-mon[33436]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:07 smithi066 ceph-mon[33436]: from='client.? 172.21.15.66:0/3493378305' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"} : dispatch 2024-09-21T02:32:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:07 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"} : dispatch 2024-09-21T02:32:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:07 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3c143be-ba4f-4bd0-b14d-5c09661d58a4"}]': finished 2024-09-21T02:32:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:07 smithi066 ceph-mon[33436]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T02:32:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:07 smithi066 ceph-mon[33436]: from='client.? 172.21.15.66:0/322367130' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:32:09.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:09 smithi190 ceph-mon[33871]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-21T02:32:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:09 smithi029 ceph-mon[31006]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-21T02:32:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:09 smithi066 ceph-mon[33436]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-21T02:32:11.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:11 smithi190 ceph-mon[33871]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:32:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:11 smithi029 ceph-mon[31006]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:32:11.908 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:11 smithi066 ceph-mon[33436]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:32:13.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:13 smithi190 ceph-mon[33871]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:32:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:13 smithi029 ceph-mon[31006]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:32:13.968 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:13 smithi066 ceph-mon[33436]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:32:15.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:15 smithi029 ceph-mon[31006]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:32:15.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:15 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T02:32:15.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:15 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:15.838 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:15 smithi066 ceph-mon[33436]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:32:15.838 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:15 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T02:32:15.838 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:15 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:15.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:15 smithi190 ceph-mon[33871]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:32:15.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:15 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T02:32:15.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:15 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:16.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:16 smithi066 ceph-mon[33436]: Deploying daemon osd.4 on smithi066 2024-09-21T02:32:16.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:16 smithi190 ceph-mon[33871]: Deploying daemon osd.4 on smithi066 2024-09-21T02:32:16.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:16 smithi029 ceph-mon[31006]: Deploying daemon osd.4 on smithi066 2024-09-21T02:32:17.814 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:17 smithi066 ceph-mon[33436]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:32:17.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:17 smithi190 ceph-mon[33871]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:32:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:17 smithi029 ceph-mon[31006]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:32:19.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:19 smithi190 ceph-mon[33871]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-21T02:32:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:19 smithi029 ceph-mon[31006]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-21T02:32:19.916 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:19 smithi066 ceph-mon[33436]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-21T02:32:21.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:21 smithi190 ceph-mon[33871]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:21 smithi029 ceph-mon[31006]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:21 smithi066 ceph-mon[33436]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:22.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:22.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:22.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:22.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:22.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:22.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:22.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:22.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:22.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:23.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:23 smithi066 ceph-mon[33436]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:23.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:23 smithi190 ceph-mon[33871]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:23.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:23 smithi029 ceph-mon[31006]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:25 smithi066 ceph-mon[33436]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:25 smithi190 ceph-mon[33871]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:25.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:25.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:25.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:25 smithi029 ceph-mon[31006]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:25.903 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.901+0000 7f2873fff640 1 -- 172.21.15.66:0/1001245088 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f287c063800 con 0x7f28680b6510 2024-09-21T02:32:25.903 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 4 on host 'smithi066' 2024-09-21T02:32:25.905 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f28680b6510 msgr2=0x7f28680b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:25.905 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f28680b6510 0x7f28680b8a20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f286400abe0 tx=0x7f286400a5c0 comp rx=0 tx=0).stop 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c108370 msgr2=0x7f287c19b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c108370 0x7f287c19b080 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f287400cce0 tx=0x7f287400a620 comp rx=0 tx=0).stop 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 shutdown_connections 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f28680b6510 0x7f28680b8a20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f287c108370 0x7f287c19b080 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f287c105750 0x7f287c19a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 --2- 172.21.15.66:0/1001245088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287c102c90 0x7f287c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 >> 172.21.15.66:0/1001245088 conn(0x7f287c0fd770 msgr2=0x7f287c103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 shutdown_connections 2024-09-21T02:32:25.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:32:25.904+0000 7f2881458640 1 -- 172.21.15.66:0/1001245088 wait complete. 2024-09-21T02:32:26.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:26 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:26.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:26 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:26.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:26 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:26.542 DEBUG:teuthology.orchestra.run.smithi066:osd.4> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.4.service 2024-09-21T02:32:26.544 INFO:tasks.cephadm:Deploying osd.5 on smithi190 with /dev/vg_nvme/lv_3... 2024-09-21T02:32:26.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T02:32:26.572 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:26 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:26.573 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:26 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:26.573 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:26 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:26.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:26 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:26.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:26 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:26.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:26 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:26.796 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:32:27.580 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:27 smithi066 ceph-mon[33436]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:27.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:27 smithi190 ceph-mon[33871]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:27.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:27 smithi029 ceph-mon[31006]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:28.260 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 02:32:27 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4[43893]: 2024-09-21T02:32:27.828+0000 7ff88e2e2640 -1 osd.4 0 log_to_monitors true 2024-09-21T02:32:28.593 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:28 smithi190 ceph-mon[33871]: from='osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:32:28.593 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:28 smithi190 ceph-mon[33871]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:32:28.627 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:28 smithi066 ceph-mon[33436]: from='osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:32:28.627 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:28 smithi066 ceph-mon[33436]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:32:28.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:28 smithi029 ceph-mon[31006]: from='osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:32:28.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:28 smithi029 ceph-mon[31006]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:32:29.321 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:32:29.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2024-09-21T02:32:29.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:29 smithi190 ceph-mon[33871]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:29.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:29 smithi190 ceph-mon[33871]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T02:32:29.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:29 smithi190 ceph-mon[33871]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T02:32:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:29 smithi190 ceph-mon[33871]: from='osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:32:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:29 smithi190 ceph-mon[33871]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:32:29.627 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:32:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:29 smithi029 ceph-mon[31006]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:29 smithi029 ceph-mon[31006]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T02:32:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:29 smithi029 ceph-mon[31006]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T02:32:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:29 smithi029 ceph-mon[31006]: from='osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:32:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:29 smithi029 ceph-mon[31006]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:32:29.759 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 02:32:29 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4[43893]: 2024-09-21T02:32:29.278+0000 7ff88a264640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T02:32:29.759 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 02:32:29 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4[43893]: 2024-09-21T02:32:29.284+0000 7ff886062640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:32:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:29 smithi066 ceph-mon[33436]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:29 smithi066 ceph-mon[33436]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T02:32:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:29 smithi066 ceph-mon[33436]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T02:32:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:29 smithi066 ceph-mon[33436]: from='osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:32:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:29 smithi066 ceph-mon[33436]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.064+0000 7f0ddf87a640 1 -- 172.21.15.190:0/585914491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd806c660 msgr2=0x7f0dd81063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.064+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/585914491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd806c660 0x7f0dd81063c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009860 tx=0x7f0dcc02a340 comp rx=0 tx=0).stop 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 -- 172.21.15.190:0/585914491 shutdown_connections 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/585914491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0dd81094a0 0x7f0dd810b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/585914491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd8106900 0x7f0dd8108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/585914491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd806c660 0x7f0dd81063c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 -- 172.21.15.190:0/585914491 >> 172.21.15.190:0/585914491 conn(0x7f0dd80fdfe0 msgr2=0x7f0dd8100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 -- 172.21.15.190:0/585914491 shutdown_connections 2024-09-21T02:32:30.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 -- 172.21.15.190:0/585914491 wait complete. 2024-09-21T02:32:30.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 Processor -- start 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.065+0000 7f0ddf87a640 1 -- start start 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddf87a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0dd806c660 0x7f0dd8199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddf87a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd8106900 0x7f0dd819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddf87a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd81094a0 0x7f0dd819b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddf87a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810f260 con 0x7f0dd81094a0 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddf87a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810f0e0 con 0x7f0dd806c660 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddf87a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810f3e0 con 0x7f0dd8106900 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd8106900 0x7f0dd819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd8106900 0x7f0dd819a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:57256/0 (socket says 172.21.15.190:57256) 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 -- 172.21.15.190:0/4236485490 learned_addr learned my addr 172.21.15.190:0/4236485490 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddde17640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd81094a0 0x7f0dd819b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddd616640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0dd806c660 0x7f0dd8199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 -- 172.21.15.190:0/4236485490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0dd806c660 msgr2=0x7f0dd8199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0dd806c660 0x7f0dd8199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 -- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd81094a0 msgr2=0x7f0dd819b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd81094a0 0x7f0dd819b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 -- 172.21.15.190:0/4236485490 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd819b980 con 0x7f0dd8106900 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddde17640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd81094a0 0x7f0dd819b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0ddce15640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd8106900 0x7f0dd819a5c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400e9f0 tx=0x7f0dd400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.066+0000 7f0dca7fc640 1 -- 172.21.15.190:0/4236485490 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd400cd80 con 0x7f0dd8106900 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.067+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd819bc70 con 0x7f0dd8106900 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.067+0000 7f0dca7fc640 1 -- 172.21.15.190:0/4236485490 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dd40108c0 con 0x7f0dd8106900 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.067+0000 7f0dca7fc640 1 -- 172.21.15.190:0/4236485490 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd40187c0 con 0x7f0dd8106900 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.067+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd81a2370 con 0x7f0dd8106900 2024-09-21T02:32:30.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.068+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd80ffb00 con 0x7f0dd8106900 2024-09-21T02:32:30.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.069+0000 7f0dca7fc640 1 -- 172.21.15.190:0/4236485490 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f0dd4010430 con 0x7f0dd8106900 2024-09-21T02:32:30.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.070+0000 7f0dca7fc640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0dc40b6510 0x7f0dc40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:30.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.070+0000 7f0dca7fc640 1 -- 172.21.15.190:0/4236485490 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3821+0+0 (secure 0 0 0) 0x7f0dd40e6b00 con 0x7f0dd8106900 2024-09-21T02:32:30.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.070+0000 7f0ddd616640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0dc40b6510 0x7f0dc40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:30.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.071+0000 7f0ddd616640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0dc40b6510 0x7f0dc40b8a20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009830 tx=0x7f0dcc005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:32:30.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.074+0000 7f0dca7fc640 1 -- 172.21.15.190:0/4236485490 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0dd40ae500 con 0x7f0dd8106900 2024-09-21T02:32:30.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:30.260+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0dd8063800 con 0x7f0dc40b6510 2024-09-21T02:32:30.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi066 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi066 to 8544M 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:30.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:30.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:30.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:32:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T02:32:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi066 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi066 to 8544M 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:30.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi066 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi066 to 8544M 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:30.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:30.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:30.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:30.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:30.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:31.611 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:31 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:32:31.611 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:31 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:32:31.611 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:31 smithi190 ceph-mon[33871]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:31.611 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:31 smithi190 ceph-mon[33871]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:31.611 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:31 smithi190 ceph-mon[33871]: osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] boot 2024-09-21T02:32:31.611 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:31 smithi190 ceph-mon[33871]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T02:32:31.611 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:31 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:31 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:32:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:31 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:32:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:31 smithi029 ceph-mon[31006]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:31 smithi029 ceph-mon[31006]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:31 smithi029 ceph-mon[31006]: osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] boot 2024-09-21T02:32:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:31 smithi029 ceph-mon[31006]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T02:32:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:31 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:31 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:32:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:31 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:32:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:31 smithi066 ceph-mon[33436]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:32:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:31 smithi066 ceph-mon[33436]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:31 smithi066 ceph-mon[33436]: osd.4 [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] boot 2024-09-21T02:32:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:31 smithi066 ceph-mon[33436]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T02:32:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:31 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:32:32.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:32 smithi190 ceph-mon[33871]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T02:32:32.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:32 smithi029 ceph-mon[31006]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T02:32:32.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:32 smithi066 ceph-mon[33436]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T02:32:33.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:33 smithi190 ceph-mon[33871]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:33.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:33 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/2072658220' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"} : dispatch 2024-09-21T02:32:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:33 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"} : dispatch 2024-09-21T02:32:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:33 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"}]': finished 2024-09-21T02:32:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:33 smithi190 ceph-mon[33871]: osdmap e32: 6 total, 5 up, 6 in 2024-09-21T02:32:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:33 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/727324568' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:32:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:33 smithi029 ceph-mon[31006]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/2072658220' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"} : dispatch 2024-09-21T02:32:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:33 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"} : dispatch 2024-09-21T02:32:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:33 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"}]': finished 2024-09-21T02:32:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:33 smithi029 ceph-mon[31006]: osdmap e32: 6 total, 5 up, 6 in 2024-09-21T02:32:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/727324568' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:32:33.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:33 smithi066 ceph-mon[33436]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:33 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/2072658220' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"} : dispatch 2024-09-21T02:32:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:33 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"} : dispatch 2024-09-21T02:32:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:33 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d4db37-3682-40a4-ba57-c957525f8a09"}]': finished 2024-09-21T02:32:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:33 smithi066 ceph-mon[33436]: osdmap e32: 6 total, 5 up, 6 in 2024-09-21T02:32:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:33 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/727324568' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:32:35.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:35 smithi190 ceph-mon[33871]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:35 smithi029 ceph-mon[31006]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:35 smithi066 ceph-mon[33436]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:37.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:37 smithi190 ceph-mon[33871]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:37.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:37 smithi029 ceph-mon[31006]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:37.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:37 smithi066 ceph-mon[33436]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:39.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:39 smithi190 ceph-mon[33871]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:39 smithi029 ceph-mon[31006]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:39 smithi066 ceph-mon[33436]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:41 smithi190 ceph-mon[33871]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:41.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T02:32:41.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:41 smithi029 ceph-mon[31006]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T02:32:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:41.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:41 smithi066 ceph-mon[33436]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:41.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T02:32:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:42.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:42 smithi190 ceph-mon[33871]: Deploying daemon osd.5 on smithi190 2024-09-21T02:32:42.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:42 smithi029 ceph-mon[31006]: Deploying daemon osd.5 on smithi190 2024-09-21T02:32:42.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:42 smithi066 ceph-mon[33436]: Deploying daemon osd.5 on smithi190 2024-09-21T02:32:43.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:43 smithi190 ceph-mon[33871]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:43.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:43 smithi029 ceph-mon[31006]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:43.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:43 smithi066 ceph-mon[33436]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:45.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:45 smithi190 ceph-mon[33871]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:45 smithi029 ceph-mon[31006]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:45.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:45 smithi066 ceph-mon[33436]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:47.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:47 smithi190 ceph-mon[33871]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:47.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:47.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:47.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:47.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:47 smithi029 ceph-mon[31006]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:47.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:47.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:47.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:47 smithi066 ceph-mon[33436]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:49.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:49 smithi190 ceph-mon[33871]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:49.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:49 smithi029 ceph-mon[31006]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:49 smithi066 ceph-mon[33436]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:50.808 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.807+0000 7f0dca7fc640 1 -- 172.21.15.190:0/4236485490 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0dd8063800 con 0x7f0dc40b6510 2024-09-21T02:32:50.808 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 5 on host 'smithi190' 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.809+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0dc40b6510 msgr2=0x7f0dc40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.809+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0dc40b6510 0x7f0dc40b8a20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009830 tx=0x7f0dcc005c90 comp rx=0 tx=0).stop 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.809+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd8106900 msgr2=0x7f0dd819a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.809+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd8106900 0x7f0dd819a5c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400e9f0 tx=0x7f0dd400eeb0 comp rx=0 tx=0).stop 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.810+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 shutdown_connections 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.810+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0dc40b6510 0x7f0dc40b8a20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.810+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd81094a0 0x7f0dd819b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.810+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dd8106900 0x7f0dd819a5c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.810+0000 7f0ddf87a640 1 --2- 172.21.15.190:0/4236485490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0dd806c660 0x7f0dd8199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.810+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 >> 172.21.15.190:0/4236485490 conn(0x7f0dd80fdfe0 msgr2=0x7f0dd8104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.810+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 shutdown_connections 2024-09-21T02:32:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:32:50.810+0000 7f0ddf87a640 1 -- 172.21.15.190:0/4236485490 wait complete. 2024-09-21T02:32:51.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:50 smithi190 ceph-mon[33871]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:50 smithi029 ceph-mon[31006]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:51.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:51.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:51.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:51.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:50 smithi066 ceph-mon[33436]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:51.453 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.5.service 2024-09-21T02:32:51.456 INFO:tasks.cephadm:Deploying osd.6 on smithi029 with /dev/vg_nvme/lv_2... 2024-09-21T02:32:51.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T02:32:51.723 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:32:52.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:51 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:52.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:51 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:52.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:51 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:52.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:51 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:52.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:51 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:52.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:51 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:52.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:51 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:32:52.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:51 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:51 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:52.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:52 smithi190 ceph-mon[33871]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:52.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 02:32:52 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T02:32:52.827+0000 7f1a387ba640 -1 osd.5 0 log_to_monitors true 2024-09-21T02:32:53.081 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:52 smithi029 ceph-mon[31006]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:53.161 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:52 smithi066 ceph-mon[33436]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:53.926 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:53 smithi029 ceph-mon[31006]: from='osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:32:53.926 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:53 smithi029 ceph-mon[31006]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:32:54.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:53 smithi190 ceph-mon[33871]: from='osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:32:54.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:53 smithi190 ceph-mon[33871]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:32:54.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:53 smithi066 ceph-mon[33436]: from='osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:32:54.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:53 smithi066 ceph-mon[33436]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:32:54.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:32:54.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-09-21T02:32:55.057 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:32:55.123 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T02:32:54.819+0000 7f1a3473c640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T02:32:55.123 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T02:32:54.828+0000 7f1a3053a640 -1 osd.5 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi190 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi190 to 8544M 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:55.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:55.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:54 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi190 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi190 to 8544M 2024-09-21T02:32:55.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:55.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:55.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:54 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi190 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:55.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:32:55.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi190 to 8544M 2024-09-21T02:32:55.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:55.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:32:55.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:54 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:32:55.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 -- 172.21.15.29:0/1128214019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480104070 msgr2=0x7f448010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 --2- 172.21.15.29:0/1128214019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480104070 0x7f448010aa10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4478009ca0 tx=0x7f447802a950 comp rx=0 tx=0).stop 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 -- 172.21.15.29:0/1128214019 shutdown_connections 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 --2- 172.21.15.29:0/1128214019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480104070 0x7f448010aa10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 --2- 172.21.15.29:0/1128214019 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44801036d0 0x7f4480103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 --2- 172.21.15.29:0/1128214019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4480102640 0x7f4480102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 -- 172.21.15.29:0/1128214019 >> 172.21.15.29:0/1128214019 conn(0x7f44800fdfb0 msgr2=0x7f4480100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 -- 172.21.15.29:0/1128214019 shutdown_connections 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.555+0000 7f4485c79640 1 -- 172.21.15.29:0/1128214019 wait complete. 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.556+0000 7f4485c79640 1 Processor -- start 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.556+0000 7f4485c79640 1 -- start start 2024-09-21T02:32:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.556+0000 7f4485c79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480102640 0x7f4480106230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.556+0000 7f4485c79640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44801036d0 0x7f4480104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.556+0000 7f4485c79640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4480104070 0x7f4480105040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.556+0000 7f4485c79640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f448010e140 con 0x7f4480102640 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.556+0000 7f4485c79640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f448010dfc0 con 0x7f44801036d0 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.556+0000 7f4485c79640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f448010e2c0 con 0x7f4480104070 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480102640 0x7f4480106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480102640 0x7f4480106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43674/0 (socket says 172.21.15.29:43674) 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 -- 172.21.15.29:0/3785125619 learned_addr learned my addr 172.21.15.29:0/3785125619 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447effd640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44801036d0 0x7f4480104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447ffff640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4480104070 0x7f4480105040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 -- 172.21.15.29:0/3785125619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4480104070 msgr2=0x7f4480105040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4480104070 0x7f4480105040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 -- 172.21.15.29:0/3785125619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44801036d0 msgr2=0x7f4480104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44801036d0 0x7f4480104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 -- 172.21.15.29:0/3785125619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4480105740 con 0x7f4480102640 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447ffff640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4480104070 0x7f4480105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447effd640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44801036d0 0x7f4480104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447f7fe640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480102640 0x7f4480106230 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f447000e9f0 tx=0x7f447000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447cff9640 1 -- 172.21.15.29:0/3785125619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447000cd80 con 0x7f4480102640 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44801a3f50 con 0x7f4480102640 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447cff9640 1 -- 172.21.15.29:0/3785125619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44700054b0 con 0x7f4480102640 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f447cff9640 1 -- 172.21.15.29:0/3785125619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470010430 con 0x7f4480102640 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.557+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44801a42f0 con 0x7f4480102640 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.558+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4480102ab0 con 0x7f4480102640 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.561+0000 7f447cff9640 1 -- 172.21.15.29:0/3785125619 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f44700046c0 con 0x7f4480102640 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.562+0000 7f447cff9640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44680b6510 0x7f44680b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:32:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.562+0000 7f447effd640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44680b6510 0x7f44680b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:32:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.562+0000 7f447cff9640 1 -- 172.21.15.29:0/3785125619 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4253+0+0 (secure 0 0 0) 0x7f44700e69c0 con 0x7f4480102640 2024-09-21T02:32:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.562+0000 7f447effd640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44680b6510 0x7f44680b8a20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4474009c20 tx=0x7f4474005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:32:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.566+0000 7f447cff9640 1 -- 172.21.15.29:0/3785125619 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4470056710 con 0x7f4480102640 2024-09-21T02:32:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:32:55.786+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f44801a45b0 con 0x7f44680b6510 2024-09-21T02:32:56.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:55 smithi190 ceph-mon[33871]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:32:56.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:55 smithi190 ceph-mon[33871]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T02:32:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:55 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:55 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:55 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:55 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:55 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:55 smithi029 ceph-mon[31006]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:32:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:55 smithi029 ceph-mon[31006]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T02:32:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:55 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:55 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:55 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:55 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:55 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:56.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:55 smithi066 ceph-mon[33436]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:32:56.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:55 smithi066 ceph-mon[33436]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T02:32:56.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:55 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:56.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:55 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:56.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:55 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:32:56.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:55 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:32:56.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:55 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:32:57.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:56 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:32:57.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:56 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:32:57.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:56 smithi190 ceph-mon[33871]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:57.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:56 smithi190 ceph-mon[33871]: osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] boot 2024-09-21T02:32:57.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:56 smithi190 ceph-mon[33871]: osdmap e35: 6 total, 6 up, 6 in 2024-09-21T02:32:57.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:56 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:57.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:56 smithi190 ceph-mon[33871]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:57.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:56 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:32:57.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:56 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:32:57.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:56 smithi029 ceph-mon[31006]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:57.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:56 smithi029 ceph-mon[31006]: osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] boot 2024-09-21T02:32:57.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:56 smithi029 ceph-mon[31006]: osdmap e35: 6 total, 6 up, 6 in 2024-09-21T02:32:57.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:56 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:57.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:56 smithi029 ceph-mon[31006]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:57.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:56 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:32:57.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:56 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:32:57.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:56 smithi066 ceph-mon[33436]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:57.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:56 smithi066 ceph-mon[33436]: osd.5 [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] boot 2024-09-21T02:32:57.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:56 smithi066 ceph-mon[33436]: osdmap e35: 6 total, 6 up, 6 in 2024-09-21T02:32:57.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:56 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:32:57.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:56 smithi066 ceph-mon[33436]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:32:58.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:57 smithi029 ceph-mon[31006]: osdmap e36: 6 total, 6 up, 6 in 2024-09-21T02:32:58.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:57 smithi190 ceph-mon[33871]: osdmap e36: 6 total, 6 up, 6 in 2024-09-21T02:32:58.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:57 smithi066 ceph-mon[33436]: osdmap e36: 6 total, 6 up, 6 in 2024-09-21T02:32:59.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:58 smithi190 ceph-mon[33871]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T02:32:59.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:58 smithi190 ceph-mon[33871]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:32:59.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:58 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4119537347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "22b00286-ed8f-4f2d-954b-c055126544f4"} : dispatch 2024-09-21T02:32:59.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:58 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4119537347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22b00286-ed8f-4f2d-954b-c055126544f4"}]': finished 2024-09-21T02:32:59.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:58 smithi190 ceph-mon[33871]: osdmap e38: 7 total, 6 up, 7 in 2024-09-21T02:32:59.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:58 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:32:59.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:58 smithi029 ceph-mon[31006]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T02:32:59.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:58 smithi029 ceph-mon[31006]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:32:59.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:58 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4119537347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "22b00286-ed8f-4f2d-954b-c055126544f4"} : dispatch 2024-09-21T02:32:59.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:58 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4119537347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22b00286-ed8f-4f2d-954b-c055126544f4"}]': finished 2024-09-21T02:32:59.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:58 smithi029 ceph-mon[31006]: osdmap e38: 7 total, 6 up, 7 in 2024-09-21T02:32:59.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:58 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:32:59.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:58 smithi066 ceph-mon[33436]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T02:32:59.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:58 smithi066 ceph-mon[33436]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:32:59.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:58 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4119537347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "22b00286-ed8f-4f2d-954b-c055126544f4"} : dispatch 2024-09-21T02:32:59.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:58 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4119537347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22b00286-ed8f-4f2d-954b-c055126544f4"}]': finished 2024-09-21T02:32:59.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:58 smithi066 ceph-mon[33436]: osdmap e38: 7 total, 6 up, 7 in 2024-09-21T02:32:59.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:58 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:00.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:32:59 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2685461068' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:00.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:32:59 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2685461068' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:00.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:32:59 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2685461068' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:01.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:00 smithi190 ceph-mon[33871]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:01.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:00 smithi029 ceph-mon[31006]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:01.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:00 smithi066 ceph-mon[33436]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:03.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:02 smithi029 ceph-mon[31006]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:03.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:02 smithi066 ceph-mon[33436]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:03.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:02 smithi190 ceph-mon[33871]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:05.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:04 smithi029 ceph-mon[31006]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T02:33:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:04 smithi066 ceph-mon[33436]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T02:33:05.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:04 smithi190 ceph-mon[33871]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T02:33:07.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:06 smithi066 ceph-mon[33436]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:33:07.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:06 smithi029 ceph-mon[31006]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:33:07.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:06 smithi190 ceph-mon[33871]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:33:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T02:33:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:08.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T02:33:08.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:08.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T02:33:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:09.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:08 smithi066 ceph-mon[33436]: Deploying daemon osd.6 on smithi029 2024-09-21T02:33:09.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:08 smithi066 ceph-mon[33436]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:33:09.282 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:08 smithi029 ceph-mon[31006]: Deploying daemon osd.6 on smithi029 2024-09-21T02:33:09.283 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:08 smithi029 ceph-mon[31006]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:33:09.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:08 smithi190 ceph-mon[33871]: Deploying daemon osd.6 on smithi029 2024-09-21T02:33:09.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:08 smithi190 ceph-mon[33871]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:33:11.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:10 smithi066 ceph-mon[33436]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T02:33:11.318 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:10 smithi029 ceph-mon[31006]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T02:33:11.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:10 smithi190 ceph-mon[33871]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T02:33:13.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:12 smithi066 ceph-mon[33436]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:33:13.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:12 smithi029 ceph-mon[31006]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:33:13.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:12 smithi190 ceph-mon[33871]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:33:15.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:14 smithi066 ceph-mon[33436]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:33:15.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:15.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:15.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:15.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:14 smithi029 ceph-mon[31006]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:33:15.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:15.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:15.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:15.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:14 smithi190 ceph-mon[33871]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:33:15.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:15.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:15.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:17.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:16 smithi066 ceph-mon[33436]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:17.327 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:16 smithi029 ceph-mon[31006]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:17.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:16 smithi190 ceph-mon[33871]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:18.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.608+0000 7f447cff9640 1 -- 172.21.15.29:0/3785125619 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f44801a45b0 con 0x7f44680b6510 2024-09-21T02:33:18.610 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 6 on host 'smithi029' 2024-09-21T02:33:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44680b6510 msgr2=0x7f44680b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44680b6510 0x7f44680b8a20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4474009c20 tx=0x7f4474005f90 comp rx=0 tx=0).stop 2024-09-21T02:33:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480102640 msgr2=0x7f4480106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480102640 0x7f4480106230 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f447000e9f0 tx=0x7f447000edf0 comp rx=0 tx=0).stop 2024-09-21T02:33:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 shutdown_connections 2024-09-21T02:33:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44680b6510 0x7f44680b8a20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4480104070 0x7f4480105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44801036d0 0x7f4480104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 --2- 172.21.15.29:0/3785125619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4480102640 0x7f4480106230 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 >> 172.21.15.29:0/3785125619 conn(0x7f44800fdfb0 msgr2=0x7f44800ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:33:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 shutdown_connections 2024-09-21T02:33:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:33:18.611+0000 7f4485c79640 1 -- 172.21.15.29:0/3785125619 wait complete. 2024-09-21T02:33:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:18.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:18.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:19.359 DEBUG:teuthology.orchestra.run.smithi029:osd.6> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.6.service 2024-09-21T02:33:19.361 INFO:tasks.cephadm:Deploying osd.7 on smithi066 with /dev/vg_nvme/lv_2... 2024-09-21T02:33:19.361 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T02:33:19.628 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:33:20.632 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 02:33:20 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T02:33:20.356+0000 7ff4ad82b640 -1 osd.6 0 log_to_monitors true 2024-09-21T02:33:21.041 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:20 smithi066 ceph-mon[33436]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:21.041 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:20 smithi066 ceph-mon[33436]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T02:33:21.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:20 smithi029 ceph-mon[31006]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:21.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:20 smithi029 ceph-mon[31006]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T02:33:21.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:20 smithi190 ceph-mon[33871]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:21.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:20 smithi190 ceph-mon[33871]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T02:33:22.178 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:33:22.207 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:21 smithi066 ceph-mon[33436]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T02:33:22.207 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:21 smithi066 ceph-mon[33436]: osdmap e39: 7 total, 6 up, 7 in 2024-09-21T02:33:22.207 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:21 smithi066 ceph-mon[33436]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:33:22.207 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:21 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:22.209 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi066:vg_nvme/lv_2 2024-09-21T02:33:22.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:21 smithi190 ceph-mon[33871]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T02:33:22.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:21 smithi190 ceph-mon[33871]: osdmap e39: 7 total, 6 up, 7 in 2024-09-21T02:33:22.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:21 smithi190 ceph-mon[33871]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:33:22.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:22.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:21 smithi029 ceph-mon[31006]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T02:33:22.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:21 smithi029 ceph-mon[31006]: osdmap e39: 7 total, 6 up, 7 in 2024-09-21T02:33:22.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:21 smithi029 ceph-mon[31006]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:33:22.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:22.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 02:33:21 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T02:33:21.950+0000 7ff4a97ad640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T02:33:22.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 02:33:21 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T02:33:21.960+0000 7ff4a55ab640 -1 osd.6 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:33:22.474 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:33:22.946 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 -- 172.21.15.66:0/2361862962 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c06b7b0 msgr2=0x7ff81c108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:22.946 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 --2- 172.21.15.66:0/2361862962 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c06b7b0 0x7ff81c108790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff818009900 tx=0x7ff81802a410 comp rx=0 tx=0).stop 2024-09-21T02:33:22.946 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 -- 172.21.15.66:0/2361862962 shutdown_connections 2024-09-21T02:33:22.946 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 --2- 172.21.15.66:0/2361862962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff81c10b680 0x7ff81c10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:22.946 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 --2- 172.21.15.66:0/2361862962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81c108cd0 0x7ff81c10b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:22.946 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 --2- 172.21.15.66:0/2361862962 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c06b7b0 0x7ff81c108790 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:22.946 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 -- 172.21.15.66:0/2361862962 >> 172.21.15.66:0/2361862962 conn(0x7ff81c0fdfb0 msgr2=0x7ff81c100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 -- 172.21.15.66:0/2361862962 shutdown_connections 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.943+0000 7ff8236ad640 1 -- 172.21.15.66:0/2361862962 wait complete. 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff8236ad640 1 Processor -- start 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff8236ad640 1 -- start start 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff8236ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81c06b7b0 0x7ff81c19c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff8236ad640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c108cd0 0x7ff81c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff8236ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff81c10b680 0x7ff81c19d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff8236ad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c111430 con 0x7ff81c06b7b0 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff8236ad640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1112b0 con 0x7ff81c108cd0 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff8236ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1115b0 con 0x7ff81c10b680 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff820c48640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c108cd0 0x7ff81c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff820c48640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c108cd0 0x7ff81c19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.66:59620/0 (socket says 172.21.15.66:59620) 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff820c48640 1 -- 172.21.15.66:0/152185681 learned_addr learned my addr 172.21.15.66:0/152185681 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff821449640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81c06b7b0 0x7ff81c19c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:33:22.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.944+0000 7ff821c4a640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff81c10b680 0x7ff81c19d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff820c48640 1 -- 172.21.15.66:0/152185681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff81c10b680 msgr2=0x7ff81c19d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff820c48640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff81c10b680 0x7ff81c19d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff820c48640 1 -- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81c06b7b0 msgr2=0x7ff81c19c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff820c48640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81c06b7b0 0x7ff81c19c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff820c48640 1 -- 172.21.15.66:0/152185681 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c19dae0 con 0x7ff81c108cd0 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff820c48640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c108cd0 0x7ff81c19c720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00d950 tx=0x7ff80c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff8127fc640 1 -- 172.21.15.66:0/152185681 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c017070 con 0x7ff81c108cd0 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81c19ddd0 con 0x7ff81c108cd0 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff8127fc640 1 -- 172.21.15.66:0/152185681 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff80c009c30 con 0x7ff81c108cd0 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff8127fc640 1 -- 172.21.15.66:0/152185681 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c00f620 con 0x7ff81c108cd0 2024-09-21T02:33:22.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.945+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81c1a44d0 con 0x7ff81c108cd0 2024-09-21T02:33:22.949 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.947+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff81c0ffad0 con 0x7ff81c108cd0 2024-09-21T02:33:22.949 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.948+0000 7ff8127fc640 1 -- 172.21.15.66:0/152185681 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7ff80c005030 con 0x7ff81c108cd0 2024-09-21T02:33:22.949 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.949+0000 7ff8127fc640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8080b6510 0x7ff8080b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:33:22.949 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.949+0000 7ff821449640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8080b6510 0x7ff8080b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:33:22.950 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.949+0000 7ff8127fc640 1 -- 172.21.15.66:0/152185681 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4685+0+0 (secure 0 0 0) 0x7ff80c0e5fd0 con 0x7ff81c108cd0 2024-09-21T02:33:22.950 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.950+0000 7ff821449640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8080b6510 0x7ff8080b8a20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff8180057c0 tx=0x7ff818000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:33:22.952 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:22.952+0000 7ff8127fc640 1 -- 172.21.15.66:0/152185681 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff80c009de0 con 0x7ff81c108cd0 2024-09-21T02:33:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:33:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T02:33:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:23.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:23.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.151 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:23.150+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff81c063800 con 0x7ff8080b6510 2024-09-21T02:33:23.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:23.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:33:23.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T02:33:23.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:23.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:23.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:23.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:23.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:23.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:23.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:24.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi029 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi029 to 5697M 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] boot 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: osdmap e41: 7 total, 7 up, 7 in 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:33:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:23 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:24.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi029 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi029 to 5697M 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] boot 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: osdmap e41: 7 total, 7 up, 7 in 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:33:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:23 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:33:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:33:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi029 2024-09-21T02:33:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi029 to 5697M 2024-09-21T02:33:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: osd.6 [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] boot 2024-09-21T02:33:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: osdmap e41: 7 total, 7 up, 7 in 2024-09-21T02:33:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:33:24.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:24.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:33:24.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:33:24.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:23 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:25.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:24 smithi066 ceph-mon[33436]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:25.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:24 smithi066 ceph-mon[33436]: osdmap e42: 7 total, 7 up, 7 in 2024-09-21T02:33:25.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:24 smithi190 ceph-mon[33871]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:25.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:24 smithi190 ceph-mon[33871]: osdmap e42: 7 total, 7 up, 7 in 2024-09-21T02:33:25.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:24 smithi029 ceph-mon[31006]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:33:25.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:24 smithi029 ceph-mon[31006]: osdmap e42: 7 total, 7 up, 7 in 2024-09-21T02:33:26.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:25 smithi066 ceph-mon[33436]: from='client.? 172.21.15.66:0/3903872073' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"} : dispatch 2024-09-21T02:33:26.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:25 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"} : dispatch 2024-09-21T02:33:26.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:25 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"}]': finished 2024-09-21T02:33:26.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:25 smithi066 ceph-mon[33436]: osdmap e43: 8 total, 7 up, 8 in 2024-09-21T02:33:26.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:26.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:25 smithi190 ceph-mon[33871]: from='client.? 172.21.15.66:0/3903872073' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"} : dispatch 2024-09-21T02:33:26.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:25 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"} : dispatch 2024-09-21T02:33:26.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:25 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"}]': finished 2024-09-21T02:33:26.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:25 smithi190 ceph-mon[33871]: osdmap e43: 8 total, 7 up, 8 in 2024-09-21T02:33:26.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:26.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:25 smithi029 ceph-mon[31006]: from='client.? 172.21.15.66:0/3903872073' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"} : dispatch 2024-09-21T02:33:26.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:25 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"} : dispatch 2024-09-21T02:33:26.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:25 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b2ea8fa-8bdb-4065-9c6b-be4157c46f31"}]': finished 2024-09-21T02:33:26.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:25 smithi029 ceph-mon[31006]: osdmap e43: 8 total, 7 up, 8 in 2024-09-21T02:33:26.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:27.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:26 smithi066 ceph-mon[33436]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:27.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:26 smithi066 ceph-mon[33436]: from='client.? 172.21.15.66:0/3578899541' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:27.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:26 smithi190 ceph-mon[33871]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:27.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:26 smithi190 ceph-mon[33871]: from='client.? 172.21.15.66:0/3578899541' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:27.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:26 smithi029 ceph-mon[31006]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:27.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:26 smithi029 ceph-mon[31006]: from='client.? 172.21.15.66:0/3578899541' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:29.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:28 smithi066 ceph-mon[33436]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:29.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:28 smithi190 ceph-mon[33871]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:29.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:28 smithi029 ceph-mon[31006]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:31.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:30 smithi066 ceph-mon[33436]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:31.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:30 smithi190 ceph-mon[33871]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:31.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:30 smithi029 ceph-mon[31006]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:33.135 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:32 smithi066 ceph-mon[33436]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:33.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:32 smithi190 ceph-mon[33871]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:32 smithi029 ceph-mon[31006]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:35.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:34 smithi066 ceph-mon[33436]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:35.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T02:33:35.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:35.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:34 smithi066 ceph-mon[33436]: Deploying daemon osd.7 on smithi066 2024-09-21T02:33:35.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:34 smithi190 ceph-mon[33871]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:35.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T02:33:35.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:35.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:34 smithi190 ceph-mon[33871]: Deploying daemon osd.7 on smithi066 2024-09-21T02:33:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:34 smithi029 ceph-mon[31006]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T02:33:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:34 smithi029 ceph-mon[31006]: Deploying daemon osd.7 on smithi066 2024-09-21T02:33:37.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:36 smithi066 ceph-mon[33436]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:37.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:36 smithi190 ceph-mon[33871]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:37.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:36 smithi029 ceph-mon[31006]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:39.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:38 smithi066 ceph-mon[33436]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:39.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:38 smithi190 ceph-mon[33871]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:39.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:38 smithi029 ceph-mon[31006]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:41.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:40 smithi066 ceph-mon[33436]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:41.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:41.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:41.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:41.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:40 smithi190 ceph-mon[33871]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:41.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:41.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:41.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:40 smithi029 ceph-mon[31006]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:43.237 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:42 smithi066 ceph-mon[33436]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:43.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:42 smithi190 ceph-mon[33871]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:43.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:42 smithi029 ceph-mon[31006]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:45.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:45.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:45.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:44 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:44 smithi066 ceph-mon[33436]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:45.089 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 7 on host 'smithi066' 2024-09-21T02:33:45.090 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.088+0000 7ff8127fc640 1 -- 172.21.15.66:0/152185681 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff81c063800 con 0x7ff8080b6510 2024-09-21T02:33:45.091 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8080b6510 msgr2=0x7ff8080b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:45.091 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8080b6510 0x7ff8080b8a20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff8180057c0 tx=0x7ff818000bc0 comp rx=0 tx=0).stop 2024-09-21T02:33:45.091 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c108cd0 msgr2=0x7ff81c19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:45.091 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c108cd0 0x7ff81c19c720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00d950 tx=0x7ff80c00de10 comp rx=0 tx=0).stop 2024-09-21T02:33:45.091 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 shutdown_connections 2024-09-21T02:33:45.091 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8080b6510 0x7ff8080b8a20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:45.091 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff81c10b680 0x7ff81c19d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:45.091 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c108cd0 0x7ff81c19c720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:45.092 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 --2- 172.21.15.66:0/152185681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff81c06b7b0 0x7ff81c19c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:45.092 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 >> 172.21.15.66:0/152185681 conn(0x7ff81c0fdfb0 msgr2=0x7ff81c109c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:33:45.092 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 shutdown_connections 2024-09-21T02:33:45.092 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:33:45.090+0000 7ff8236ad640 1 -- 172.21.15.66:0/152185681 wait complete. 2024-09-21T02:33:45.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:45.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:45.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:44 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:44 smithi190 ceph-mon[33871]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:45.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:45.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:45.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:44 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:44 smithi029 ceph-mon[31006]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:45.729 DEBUG:teuthology.orchestra.run.smithi066:osd.7> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.7.service 2024-09-21T02:33:45.731 INFO:tasks.cephadm:Deploying osd.8 on smithi190 with /dev/vg_nvme/lv_2... 2024-09-21T02:33:45.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T02:33:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:45 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:45 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:45 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.982 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:45 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:45.983 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:45 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.983 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:45 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:45.989 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:33:46.110 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:45 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:33:46.110 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:45 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:46.110 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:45 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:46.947 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:46 smithi066 ceph-mon[33436]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:46.947 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:46 smithi066 ceph-mon[33436]: from='osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:33:46.947 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:46 smithi066 ceph-mon[33436]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:33:46.947 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 02:33:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T02:33:46.665+0000 7f6a6e668640 -1 osd.7 0 log_to_monitors true 2024-09-21T02:33:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:46 smithi190 ceph-mon[33871]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:46 smithi190 ceph-mon[33871]: from='osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:33:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:46 smithi190 ceph-mon[33871]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:33:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:46 smithi029 ceph-mon[31006]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:46 smithi029 ceph-mon[31006]: from='osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:33:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:46 smithi029 ceph-mon[31006]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:33:48.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:47 smithi190 ceph-mon[33871]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T02:33:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:47 smithi190 ceph-mon[33871]: osdmap e44: 8 total, 7 up, 8 in 2024-09-21T02:33:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:47 smithi190 ceph-mon[33871]: from='osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:33:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:47 smithi190 ceph-mon[33871]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:33:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:47 smithi029 ceph-mon[31006]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T02:33:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:47 smithi029 ceph-mon[31006]: osdmap e44: 8 total, 7 up, 8 in 2024-09-21T02:33:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:47 smithi029 ceph-mon[31006]: from='osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:33:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:47 smithi029 ceph-mon[31006]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:33:48.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:47 smithi066 ceph-mon[33436]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T02:33:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:47 smithi066 ceph-mon[33436]: osdmap e44: 8 total, 7 up, 8 in 2024-09-21T02:33:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:47 smithi066 ceph-mon[33436]: from='osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:33:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:47 smithi066 ceph-mon[33436]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:33:48.539 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:33:48.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2024-09-21T02:33:48.664 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T02:33:48.316+0000 7f6a6adff640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T02:33:48.664 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T02:33:48.326+0000 7f6a623e0640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:33:48.826 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='osd.7 ' entity='osd.7' 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:49.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:49.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.103 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:33:49.103 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='osd.7 ' entity='osd.7' 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:49.104 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:49.105 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='osd.7 ' entity='osd.7' 2024-09-21T02:33:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:33:49.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:49.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:33:49.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:33:49.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 -- 172.21.15.190:0/450544757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4075c70 msgr2=0x7fd2a4076080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:49.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 --2- 172.21.15.190:0/450544757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4075c70 0x7fd2a4076080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd298009900 tx=0x7fd29802a410 comp rx=0 tx=0).stop 2024-09-21T02:33:49.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 -- 172.21.15.190:0/450544757 shutdown_connections 2024-09-21T02:33:49.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 --2- 172.21.15.190:0/450544757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a410b810 0x7fd2a410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:49.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 --2- 172.21.15.190:0/450544757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4075c70 0x7fd2a4076080 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 --2- 172.21.15.190:0/450544757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2a4077060 0x7fd2a4075730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 -- 172.21.15.190:0/450544757 >> 172.21.15.190:0/450544757 conn(0x7fd2a40ff180 msgr2=0x7fd2a4101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 -- 172.21.15.190:0/450544757 shutdown_connections 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.292+0000 7fd2a8929640 1 -- 172.21.15.190:0/450544757 wait complete. 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.293+0000 7fd2a8929640 1 Processor -- start 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.293+0000 7fd2a8929640 1 -- start start 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.293+0000 7fd2a8929640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2a4075c70 0x7fd2a419c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.293+0000 7fd2a8929640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4077060 0x7fd2a419c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.293+0000 7fd2a8929640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a410b810 0x7fd2a419d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.293+0000 7fd2a8929640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4111450 con 0x7fd2a410b810 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.293+0000 7fd2a8929640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41112d0 con 0x7fd2a4075c70 2024-09-21T02:33:49.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.293+0000 7fd2a8929640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41115d0 con 0x7fd2a4077060 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4077060 0x7fd2a419c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4077060 0x7fd2a419c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:38058/0 (socket says 172.21.15.190:38058) 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 -- 172.21.15.190:0/3121150087 learned_addr learned my addr 172.21.15.190:0/3121150087 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a259c640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2a4075c70 0x7fd2a419c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a2d9d640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a410b810 0x7fd2a419d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 -- 172.21.15.190:0/3121150087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2a4075c70 msgr2=0x7fd2a419c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2a4075c70 0x7fd2a419c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 -- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a410b810 msgr2=0x7fd2a419d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a410b810 0x7fd2a419d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 -- 172.21.15.190:0/3121150087 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a419db70 con 0x7fd2a4077060 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a259c640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2a4075c70 0x7fd2a419c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a1d9b640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4077060 0x7fd2a419c7b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd298009230 tx=0x7fd2980042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2937fe640 1 -- 172.21.15.190:0/3121150087 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29803f030 con 0x7fd2a4077060 2024-09-21T02:33:49.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a419de00 con 0x7fd2a4077060 2024-09-21T02:33:49.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2937fe640 1 -- 172.21.15.190:0/3121150087 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd298003c90 con 0x7fd2a4077060 2024-09-21T02:33:49.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2937fe640 1 -- 172.21.15.190:0/3121150087 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd298035070 con 0x7fd2a4077060 2024-09-21T02:33:49.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.294+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a41a4560 con 0x7fd2a4077060 2024-09-21T02:33:49.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.296+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a4076080 con 0x7fd2a4077060 2024-09-21T02:33:49.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.298+0000 7fd2937fe640 1 -- 172.21.15.190:0/3121150087 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fd298042030 con 0x7fd2a4077060 2024-09-21T02:33:49.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.299+0000 7fd2937fe640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd28c0b6510 0x7fd28c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:33:49.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.299+0000 7fd2937fe640 1 -- 172.21.15.190:0/3121150087 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5430+0+0 (secure 0 0 0) 0x7fd298107170 con 0x7fd2a4077060 2024-09-21T02:33:49.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.299+0000 7fd2a259c640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd28c0b6510 0x7fd28c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:33:49.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.300+0000 7fd2a259c640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd28c0b6510 0x7fd28c0b8a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd294005d00 tx=0x7fd29400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:33:49.303 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.302+0000 7fd2937fe640 1 -- 172.21.15.190:0/3121150087 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd2980d2920 con 0x7fd2a4077060 2024-09-21T02:33:49.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:33:49.493+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd2a40ffde0 con 0x7fd28c0b6510 2024-09-21T02:33:50.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:33:50.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:33:50.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi066 2024-09-21T02:33:50.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi066 to 5696M 2024-09-21T02:33:50.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:50.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] boot 2024-09-21T02:33:50.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: osdmap e47: 8 total, 8 up, 8 in 2024-09-21T02:33:50.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:50.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:50.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:33:50.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:33:50.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi066 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi066 to 5696M 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] boot 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: osdmap e47: 8 total, 8 up, 8 in 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:33:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:50.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:33:50.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi066 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi066 to 5696M 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: osd.7 [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] boot 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: osdmap e47: 8 total, 8 up, 8 in 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:33:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:33:51.067 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:50 smithi190 ceph-mon[33871]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:51.067 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:50 smithi190 ceph-mon[33871]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T02:33:51.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:50 smithi066 ceph-mon[33436]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:51.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:50 smithi066 ceph-mon[33436]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T02:33:51.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:50 smithi029 ceph-mon[31006]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:51.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:50 smithi029 ceph-mon[31006]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T02:33:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:51 smithi066 ceph-mon[33436]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T02:33:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:51 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/516269170' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"} : dispatch 2024-09-21T02:33:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:51 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"} : dispatch 2024-09-21T02:33:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:51 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"}]': finished 2024-09-21T02:33:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:51 smithi066 ceph-mon[33436]: osdmap e50: 9 total, 8 up, 9 in 2024-09-21T02:33:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:51 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:33:52.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:51 smithi190 ceph-mon[33871]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T02:33:52.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:51 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/516269170' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"} : dispatch 2024-09-21T02:33:52.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:51 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"} : dispatch 2024-09-21T02:33:52.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:51 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"}]': finished 2024-09-21T02:33:52.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:51 smithi190 ceph-mon[33871]: osdmap e50: 9 total, 8 up, 9 in 2024-09-21T02:33:52.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:51 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:33:52.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:51 smithi029 ceph-mon[31006]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T02:33:52.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:51 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/516269170' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"} : dispatch 2024-09-21T02:33:52.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:51 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"} : dispatch 2024-09-21T02:33:52.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:51 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0995b89-dfe7-4dc5-b927-6d57c8e71d27"}]': finished 2024-09-21T02:33:52.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:51 smithi029 ceph-mon[31006]: osdmap e50: 9 total, 8 up, 9 in 2024-09-21T02:33:52.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:51 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:33:53.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:52 smithi066 ceph-mon[33436]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:53.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:52 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/2269160643' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:53.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:52 smithi190 ceph-mon[33871]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:53.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:52 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/2269160643' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:53.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:52 smithi029 ceph-mon[31006]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:53.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:52 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/2269160643' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:33:55.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:54 smithi066 ceph-mon[33436]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:55.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:54 smithi190 ceph-mon[33871]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:55.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:54 smithi029 ceph-mon[31006]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:33:57.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:56 smithi066 ceph-mon[33436]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:33:57.363 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:56 smithi190 ceph-mon[33871]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:33:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:56 smithi029 ceph-mon[31006]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:33:59.079 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:33:58 smithi190 ceph-mon[33871]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:33:59.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:33:58 smithi066 ceph-mon[33436]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:33:59.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:33:58 smithi029 ceph-mon[31006]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:34:01.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:01 smithi066 ceph-mon[33436]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:34:01.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:01 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T02:34:01.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:01 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:01.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:01 smithi066 ceph-mon[33436]: Deploying daemon osd.8 on smithi190 2024-09-21T02:34:01.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:01 smithi190 ceph-mon[33871]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:34:01.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:01 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T02:34:01.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:01 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:01.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:01 smithi190 ceph-mon[33871]: Deploying daemon osd.8 on smithi190 2024-09-21T02:34:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:01 smithi029 ceph-mon[31006]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:34:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:01 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T02:34:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:01 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:01 smithi029 ceph-mon[31006]: Deploying daemon osd.8 on smithi190 2024-09-21T02:34:03.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:03 smithi190 ceph-mon[33871]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:34:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:03 smithi029 ceph-mon[31006]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:34:03.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:03 smithi066 ceph-mon[33436]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:34:05.080 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:05 smithi190 ceph-mon[33871]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:05 smithi029 ceph-mon[31006]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:05.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:05 smithi066 ceph-mon[33436]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:07.234 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:07 smithi190 ceph-mon[33871]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:07.234 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:07.235 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:07.235 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:07 smithi029 ceph-mon[31006]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:07.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:07 smithi066 ceph-mon[33436]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:07.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:07.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:07.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.286 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:09 smithi190 ceph-mon[33871]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:09.286 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:09 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.286 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:09 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.286 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:09 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:09.286 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:09 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:09.286 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:09 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:09 smithi029 ceph-mon[31006]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:09 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:09 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:09 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:09 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:09 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:09 smithi066 ceph-mon[33436]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:09.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:09 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:09 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:09 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:09 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:09 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:10.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.047+0000 7fd2937fe640 1 -- 172.21.15.190:0/3121150087 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd2a40ffde0 con 0x7fd28c0b6510 2024-09-21T02:34:10.050 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 8 on host 'smithi190' 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd28c0b6510 msgr2=0x7fd28c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd28c0b6510 0x7fd28c0b8a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd294005d00 tx=0x7fd29400b410 comp rx=0 tx=0).stop 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4077060 msgr2=0x7fd2a419c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4077060 0x7fd2a419c7b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd298009230 tx=0x7fd2980042f0 comp rx=0 tx=0).stop 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 shutdown_connections 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd28c0b6510 0x7fd28c0b8a20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a410b810 0x7fd2a419d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a4077060 0x7fd2a419c7b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:10.052 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 --2- 172.21.15.190:0/3121150087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2a4075c70 0x7fd2a419c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:10.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 >> 172.21.15.190:0/3121150087 conn(0x7fd2a40ff180 msgr2=0x7fd2a40ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:34:10.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 shutdown_connections 2024-09-21T02:34:10.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:34:10.050+0000 7fd2a8929640 1 -- 172.21.15.190:0/3121150087 wait complete. 2024-09-21T02:34:10.866 DEBUG:teuthology.orchestra.run.smithi190:osd.8> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.8.service 2024-09-21T02:34:10.868 INFO:tasks.cephadm:Deploying osd.9 on smithi029 with /dev/vg_nvme/lv_1... 2024-09-21T02:34:10.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T02:34:11.115 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:11 smithi190 ceph-mon[33871]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:11.115 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:11.115 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:11.115 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:11 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:11.141 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:34:11.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:11 smithi029 ceph-mon[31006]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:11.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:11.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:11.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:11 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:11.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:11 smithi066 ceph-mon[33436]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:11.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:11.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:11.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:11 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:12.874 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 02:34:12 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T02:34:12.552+0000 7fd863337640 -1 osd.8 0 log_to_monitors true 2024-09-21T02:34:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:13 smithi029 ceph-mon[31006]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:13 smithi029 ceph-mon[31006]: from='osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:34:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:13 smithi029 ceph-mon[31006]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:34:13.351 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:13 smithi190 ceph-mon[33871]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:13.352 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:13 smithi190 ceph-mon[33871]: from='osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:34:13.352 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:13 smithi190 ceph-mon[33871]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:34:13.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:13 smithi066 ceph-mon[33436]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:13.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:13 smithi066 ceph-mon[33436]: from='osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:34:13.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:13 smithi066 ceph-mon[33436]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:34:14.045 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:14.046 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.081 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:34:14.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-09-21T02:34:14.373 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T02:34:14.047+0000 7fd85f2b9640 -1 osd.8 0 waiting for initial osdmap 2024-09-21T02:34:14.373 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 02:34:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T02:34:14.057+0000 7fd85b0b7640 -1 osd.8 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:34:14.381 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:34:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T02:34:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T02:34:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:34:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:34:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:14.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:14.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:14.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:14.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 -- 172.21.15.29:0/2845614216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 msgr2=0x7fe57c10aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 --2- 172.21.15.29:0/2845614216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 0x7fe57c10aa10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe574009900 tx=0x7fe57402a480 comp rx=0 tx=0).stop 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 -- 172.21.15.29:0/2845614216 shutdown_connections 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 --2- 172.21.15.29:0/2845614216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 0x7fe57c10aa10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 --2- 172.21.15.29:0/2845614216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe57c1036d0 0x7fe57c103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 --2- 172.21.15.29:0/2845614216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe57c102640 0x7fe57c102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 -- 172.21.15.29:0/2845614216 >> 172.21.15.29:0/2845614216 conn(0x7fe57c0fdfb0 msgr2=0x7fe57c100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 -- 172.21.15.29:0/2845614216 shutdown_connections 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.906+0000 7fe584315640 1 -- 172.21.15.29:0/2845614216 wait complete. 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe584315640 1 Processor -- start 2024-09-21T02:34:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe584315640 1 -- start start 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe584315640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe57c102640 0x7fe57c106230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe584315640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe57c1036d0 0x7fe57c104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe584315640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 0x7fe57c105040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe584315640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c10e140 con 0x7fe57c104070 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe584315640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c10dfc0 con 0x7fe57c102640 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe584315640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c10e2c0 con 0x7fe57c1036d0 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe5828b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 0x7fe57c105040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe5828b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 0x7fe57c105040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39606/0 (socket says 172.21.15.29:39606) 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe5828b2640 1 -- 172.21.15.29:0/2650444989 learned_addr learned my addr 172.21.15.29:0/2650444989 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe5820b1640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe57c102640 0x7fe57c106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.907+0000 7fe5818b0640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe57c1036d0 0x7fe57c104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5828b2640 1 -- 172.21.15.29:0/2650444989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe57c1036d0 msgr2=0x7fe57c104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5828b2640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe57c1036d0 0x7fe57c104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5828b2640 1 -- 172.21.15.29:0/2650444989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe57c102640 msgr2=0x7fe57c106230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5828b2640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe57c102640 0x7fe57c106230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5828b2640 1 -- 172.21.15.29:0/2650444989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c105740 con 0x7fe57c104070 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5818b0640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe57c1036d0 0x7fe57c104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5828b2640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 0x7fe57c105040 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe574005ab0 tx=0x7fe5740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5737fe640 1 -- 172.21.15.29:0/2650444989 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57403f030 con 0x7fe57c104070 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5737fe640 1 -- 172.21.15.29:0/2650444989 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe574003c90 con 0x7fe57c104070 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57c1a3f50 con 0x7fe57c104070 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe5737fe640 1 -- 172.21.15.29:0/2650444989 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574035070 con 0x7fe57c104070 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.908+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57c1a42f0 con 0x7fe57c104070 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.909+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe57c102ab0 con 0x7fe57c104070 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.911+0000 7fe5737fe640 1 -- 172.21.15.29:0/2650444989 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fe574042030 con 0x7fe57c104070 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.911+0000 7fe5737fe640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5680b6510 0x7fe5680b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.912+0000 7fe5820b1640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5680b6510 0x7fe5680b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:34:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.912+0000 7fe5737fe640 1 -- 172.21.15.29:0/2650444989 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5549+0+0 (secure 0 0 0) 0x7fe574107300 con 0x7fe57c104070 2024-09-21T02:34:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.912+0000 7fe5820b1640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5680b6510 0x7fe5680b8a20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe578005fd0 tx=0x7fe578009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:34:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:14.915+0000 7fe5737fe640 1 -- 172.21.15.29:0/2650444989 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe5740ce660 con 0x7fe57c104070 2024-09-21T02:34:15.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:15.117+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe57c0ffff0 con 0x7fe5680b6510 2024-09-21T02:34:15.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:15 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi190 2024-09-21T02:34:15.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:15 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi190 to 5696M 2024-09-21T02:34:15.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:15 smithi029 ceph-mon[31006]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:15.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:15 smithi029 ceph-mon[31006]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:34:15.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:15 smithi029 ceph-mon[31006]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T02:34:15.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:15 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:15.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:15 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:15.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:15 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi190 2024-09-21T02:34:15.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:15 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi190 to 5696M 2024-09-21T02:34:15.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:15 smithi190 ceph-mon[33871]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:15.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:15 smithi190 ceph-mon[33871]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:34:15.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:15 smithi190 ceph-mon[33871]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T02:34:15.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:15 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:15.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:15 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:15.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:15 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi190 2024-09-21T02:34:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:15 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi190 to 5696M 2024-09-21T02:34:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:15 smithi066 ceph-mon[33436]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:15 smithi066 ceph-mon[33436]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:34:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:15 smithi066 ceph-mon[33436]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T02:34:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:15 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:15 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:16.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] boot 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: osdmap e53: 9 total, 9 up, 9 in 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:34:16.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:16.376 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:34:16.376 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:34:16.376 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:16.376 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] boot 2024-09-21T02:34:16.376 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: osdmap e53: 9 total, 9 up, 9 in 2024-09-21T02:34:16.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:16.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:16.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:34:16.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:34:16.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:16.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: osd.8 [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] boot 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: osdmap e53: 9 total, 9 up, 9 in 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:34:16.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:17.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:17 smithi029 ceph-mon[31006]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:17.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:17 smithi029 ceph-mon[31006]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T02:34:17.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:17 smithi190 ceph-mon[33871]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:17.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:17 smithi190 ceph-mon[33871]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T02:34:17.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:17 smithi066 ceph-mon[33436]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:34:17.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:17 smithi066 ceph-mon[33436]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T02:34:18.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:18 smithi190 ceph-mon[33871]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T02:34:18.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:18 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3536352353' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7"} : dispatch 2024-09-21T02:34:18.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:18 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3536352353' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7"}]': finished 2024-09-21T02:34:18.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:18 smithi190 ceph-mon[33871]: osdmap e56: 10 total, 9 up, 10 in 2024-09-21T02:34:18.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:18.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:18 smithi029 ceph-mon[31006]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T02:34:18.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:18 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3536352353' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7"} : dispatch 2024-09-21T02:34:18.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:18 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3536352353' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7"}]': finished 2024-09-21T02:34:18.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:18 smithi029 ceph-mon[31006]: osdmap e56: 10 total, 9 up, 10 in 2024-09-21T02:34:18.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:18.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:18 smithi066 ceph-mon[33436]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T02:34:18.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:18 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3536352353' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7"} : dispatch 2024-09-21T02:34:18.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:18 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3536352353' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7"}]': finished 2024-09-21T02:34:18.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:18 smithi066 ceph-mon[33436]: osdmap e56: 10 total, 9 up, 10 in 2024-09-21T02:34:18.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:19.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:19 smithi190 ceph-mon[33871]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:19.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:19 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1921364041' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:34:19.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:19 smithi029 ceph-mon[31006]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:19.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:19 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1921364041' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:34:19.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:19 smithi066 ceph-mon[33436]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:19.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:19 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1921364041' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:34:21.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:21 smithi190 ceph-mon[33871]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-21T02:34:21.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:21 smithi029 ceph-mon[31006]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-21T02:34:21.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:21 smithi066 ceph-mon[33436]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-21T02:34:23.326 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:23 smithi066 ceph-mon[33436]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:34:23.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:23 smithi190 ceph-mon[33871]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:34:23.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:23 smithi029 ceph-mon[31006]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:34:25.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:25 smithi029 ceph-mon[31006]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-21T02:34:25.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:25 smithi190 ceph-mon[33871]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-21T02:34:25.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:25 smithi066 ceph-mon[33436]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-21T02:34:27.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:27 smithi190 ceph-mon[33871]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:34:27.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T02:34:27.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:27.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:27 smithi029 ceph-mon[31006]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:34:27.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T02:34:27.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:27.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:27 smithi066 ceph-mon[33436]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:34:27.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T02:34:27.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:28.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:28 smithi190 ceph-mon[33871]: Deploying daemon osd.9 on smithi029 2024-09-21T02:34:28.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:28 smithi029 ceph-mon[31006]: Deploying daemon osd.9 on smithi029 2024-09-21T02:34:28.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:28 smithi066 ceph-mon[33436]: Deploying daemon osd.9 on smithi029 2024-09-21T02:34:29.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:29 smithi190 ceph-mon[33871]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:34:29.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:29 smithi029 ceph-mon[31006]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:34:29.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:29 smithi066 ceph-mon[33436]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:34:31.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:31 smithi190 ceph-mon[33871]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:31.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:31 smithi029 ceph-mon[31006]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:31.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:31 smithi066 ceph-mon[33436]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:34:33.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:33 smithi029 ceph-mon[31006]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:33.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:33 smithi190 ceph-mon[33871]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:33.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:33 smithi066 ceph-mon[33436]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:34.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:34.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:34.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:34.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:34.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:34.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:34.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:34.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:34.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:35 smithi066 ceph-mon[33436]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:35.770 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:35 smithi029 ceph-mon[31006]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:35 smithi190 ceph-mon[33871]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:37.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.270+0000 7fe5737fe640 1 -- 172.21.15.29:0/2650444989 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe57c0ffff0 con 0x7fe5680b6510 2024-09-21T02:34:37.272 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 9 on host 'smithi029' 2024-09-21T02:34:37.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5680b6510 msgr2=0x7fe5680b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5680b6510 0x7fe5680b8a20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe578005fd0 tx=0x7fe578009680 comp rx=0 tx=0).stop 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 msgr2=0x7fe57c105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 0x7fe57c105040 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe574005ab0 tx=0x7fe5740042f0 comp rx=0 tx=0).stop 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 shutdown_connections 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5680b6510 0x7fe5680b8a20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe57c104070 0x7fe57c105040 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe57c1036d0 0x7fe57c104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 --2- 172.21.15.29:0/2650444989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe57c102640 0x7fe57c106230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 >> 172.21.15.29:0/2650444989 conn(0x7fe57c0fdfb0 msgr2=0x7fe57c0ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.272+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 shutdown_connections 2024-09-21T02:34:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:34:37.273+0000 7fe584315640 1 -- 172.21.15.29:0/2650444989 wait complete. 2024-09-21T02:34:37.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:37 smithi066 ceph-mon[33436]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:37.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:37.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:37 smithi190 ceph-mon[33871]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:37.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:37 smithi029 ceph-mon[31006]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:37.987 DEBUG:teuthology.orchestra.run.smithi029:osd.9> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.9.service 2024-09-21T02:34:37.990 INFO:tasks.cephadm:Deploying osd.10 on smithi066 with /dev/vg_nvme/lv_1... 2024-09-21T02:34:37.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T02:34:38.259 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:34:39.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:39 smithi190 ceph-mon[33871]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:39.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:39.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:39 smithi029 ceph-mon[31006]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:39.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:39 smithi066 ceph-mon[33436]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:39.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:39.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:39.863 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 02:34:39 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T02:34:39.526+0000 7f7c145aa640 -1 osd.9 0 log_to_monitors true 2024-09-21T02:34:40.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:40 smithi066 ceph-mon[33436]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T02:34:40.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:40 smithi029 ceph-mon[31006]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T02:34:40.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:40 smithi190 ceph-mon[33871]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T02:34:41.066 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-21T02:34:41.092 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi066:vg_nvme/lv_1 2024-09-21T02:34:41.350 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:34:41.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:41 smithi190 ceph-mon[33871]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:41.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:41 smithi190 ceph-mon[33871]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T02:34:41.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:41 smithi190 ceph-mon[33871]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T02:34:41.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:41 smithi190 ceph-mon[33871]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:34:41.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:41 smithi029 ceph-mon[31006]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:41 smithi029 ceph-mon[31006]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T02:34:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:41 smithi029 ceph-mon[31006]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T02:34:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:41 smithi029 ceph-mon[31006]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:34:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:41.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 02:34:41 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T02:34:41.095+0000 7f7c1052c640 -1 osd.9 0 waiting for initial osdmap 2024-09-21T02:34:41.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 02:34:41 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T02:34:41.107+0000 7f7c0c32a640 -1 osd.9 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:34:41.400 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:41 smithi066 ceph-mon[33436]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:41.400 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:41 smithi066 ceph-mon[33436]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T02:34:41.400 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:41 smithi066 ceph-mon[33436]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T02:34:41.400 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:41 smithi066 ceph-mon[33436]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]} : dispatch 2024-09-21T02:34:41.400 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:41.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.876+0000 7ff3f3042640 1 -- 172.21.15.66:0/3415706026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec1040b0 msgr2=0x7ff3ec10aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:41.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.876+0000 7ff3f3042640 1 --2- 172.21.15.66:0/3415706026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec1040b0 0x7ff3ec10aa50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009ca0 tx=0x7ff3e402a950 comp rx=0 tx=0).stop 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 -- 172.21.15.66:0/3415706026 shutdown_connections 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 --2- 172.21.15.66:0/3415706026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec1040b0 0x7ff3ec10aa50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 --2- 172.21.15.66:0/3415706026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ec103710 0x7ff3ec103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 --2- 172.21.15.66:0/3415706026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3ec102680 0x7ff3ec102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 -- 172.21.15.66:0/3415706026 >> 172.21.15.66:0/3415706026 conn(0x7ff3ec0fdff0 msgr2=0x7ff3ec1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 -- 172.21.15.66:0/3415706026 shutdown_connections 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 -- 172.21.15.66:0/3415706026 wait complete. 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 Processor -- start 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 -- start start 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.877+0000 7ff3f3042640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ec102680 0x7ff3ec199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3f3042640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec103710 0x7ff3ec19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3f3042640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3ec1040b0 0x7ff3ec19b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:34:41.883 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3f3042640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec10e1b0 con 0x7ff3ec102680 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3f3042640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec10e030 con 0x7ff3ec103710 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3f3042640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec10e330 con 0x7ff3ec1040b0 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec103710 0x7ff3ec19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec103710 0x7ff3ec19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.66:59850/0 (socket says 172.21.15.66:59850) 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 -- 172.21.15.66:0/69644138 learned_addr learned my addr 172.21.15.66:0/69644138 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3f0dde640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ec102680 0x7ff3ec199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3f15df640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3ec1040b0 0x7ff3ec19b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 -- 172.21.15.66:0/69644138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3ec1040b0 msgr2=0x7ff3ec19b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3ec1040b0 0x7ff3ec19b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 -- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ec102680 msgr2=0x7ff3ec199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ec102680 0x7ff3ec199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 -- 172.21.15.66:0/69644138 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3ec19b820 con 0x7ff3ec103710 2024-09-21T02:34:41.884 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3ebfff640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec103710 0x7ff3ec19a4f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400e9f0 tx=0x7ff3d400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:34:41.885 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3e9ffb640 1 -- 172.21.15.66:0/69644138 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d400ccc0 con 0x7ff3ec103710 2024-09-21T02:34:41.885 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3e9ffb640 1 -- 172.21.15.66:0/69644138 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3d40108c0 con 0x7ff3ec103710 2024-09-21T02:34:41.885 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3ec19bb10 con 0x7ff3ec103710 2024-09-21T02:34:41.885 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.878+0000 7ff3e9ffb640 1 -- 172.21.15.66:0/69644138 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4018750 con 0x7ff3ec103710 2024-09-21T02:34:41.885 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.879+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3ec1a22a0 con 0x7ff3ec103710 2024-09-21T02:34:41.885 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.880+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3ec102af0 con 0x7ff3ec103710 2024-09-21T02:34:41.886 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.881+0000 7ff3e9ffb640 1 -- 172.21.15.66:0/69644138 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7ff3d4010430 con 0x7ff3ec103710 2024-09-21T02:34:41.886 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.882+0000 7ff3e9ffb640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3d80b6510 0x7ff3d80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:34:41.886 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.882+0000 7ff3e9ffb640 1 -- 172.21.15.66:0/69644138 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5981+0+0 (secure 0 0 0) 0x7ff3d40e72a0 con 0x7ff3ec103710 2024-09-21T02:34:41.886 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.882+0000 7ff3f0dde640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3d80b6510 0x7ff3d80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:34:41.886 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.883+0000 7ff3f0dde640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3d80b6510 0x7ff3d80b8a20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00abe0 tx=0x7ff3dc00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:34:41.887 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:41.885+0000 7ff3e9ffb640 1 -- 172.21.15.66:0/69644138 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff3d40ae450 con 0x7ff3ec103710 2024-09-21T02:34:42.084 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:34:42.082+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff3ec0fff60 con 0x7ff3d80b6510 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi029 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi029 to 4273M 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:34:42.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi029 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi029 to 4273M 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:34:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:42.472 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi029 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi029 to 4273M 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:34:42.474 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:43.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:43 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:34:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:43 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:34:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:43 smithi190 ceph-mon[33871]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:43 smithi190 ceph-mon[33871]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:43 smithi190 ceph-mon[33871]: osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] boot 2024-09-21T02:34:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:43 smithi190 ceph-mon[33871]: osdmap e59: 10 total, 10 up, 10 in 2024-09-21T02:34:43.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:43 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:34:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:43 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:34:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:43 smithi029 ceph-mon[31006]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:43 smithi029 ceph-mon[31006]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:43 smithi029 ceph-mon[31006]: osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] boot 2024-09-21T02:34:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:43 smithi029 ceph-mon[31006]: osdmap e59: 10 total, 10 up, 10 in 2024-09-21T02:34:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:43.448 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:43 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:34:43.448 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:43 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:34:43.448 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:43 smithi066 ceph-mon[33436]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:34:43.448 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:43 smithi066 ceph-mon[33436]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:43.448 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:43 smithi066 ceph-mon[33436]: osd.9 [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] boot 2024-09-21T02:34:43.448 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:43 smithi066 ceph-mon[33436]: osdmap e59: 10 total, 10 up, 10 in 2024-09-21T02:34:43.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:34:44.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:44 smithi190 ceph-mon[33871]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T02:34:44.877 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:44 smithi066 ceph-mon[33436]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T02:34:44.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:44 smithi029 ceph-mon[31006]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T02:34:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:45 smithi190 ceph-mon[33871]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:45 smithi190 ceph-mon[33871]: from='client.? 172.21.15.66:0/1218182967' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"} : dispatch 2024-09-21T02:34:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:45 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"} : dispatch 2024-09-21T02:34:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:45 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"}]': finished 2024-09-21T02:34:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:45 smithi190 ceph-mon[33871]: osdmap e61: 11 total, 10 up, 11 in 2024-09-21T02:34:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:45 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:34:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:45 smithi190 ceph-mon[33871]: from='client.? 172.21.15.66:0/359940838' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:34:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:45 smithi029 ceph-mon[31006]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:45 smithi029 ceph-mon[31006]: from='client.? 172.21.15.66:0/1218182967' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"} : dispatch 2024-09-21T02:34:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:45 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"} : dispatch 2024-09-21T02:34:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:45 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"}]': finished 2024-09-21T02:34:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:45 smithi029 ceph-mon[31006]: osdmap e61: 11 total, 10 up, 11 in 2024-09-21T02:34:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:45 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:34:45.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:45 smithi029 ceph-mon[31006]: from='client.? 172.21.15.66:0/359940838' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:34:46.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:45 smithi066 ceph-mon[33436]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:45 smithi066 ceph-mon[33436]: from='client.? 172.21.15.66:0/1218182967' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"} : dispatch 2024-09-21T02:34:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:45 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"} : dispatch 2024-09-21T02:34:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:45 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6234f49-76e4-49a0-9468-5dba3e2f378a"}]': finished 2024-09-21T02:34:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:45 smithi066 ceph-mon[33436]: osdmap e61: 11 total, 10 up, 11 in 2024-09-21T02:34:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:45 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:34:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:45 smithi066 ceph-mon[33436]: from='client.? 172.21.15.66:0/359940838' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:34:47.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:47 smithi190 ceph-mon[33871]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:47 smithi029 ceph-mon[31006]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:48.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:47 smithi066 ceph-mon[33436]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:49 smithi066 ceph-mon[33436]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:49.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:49 smithi190 ceph-mon[33871]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:49 smithi029 ceph-mon[31006]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:51.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:51 smithi066 ceph-mon[33436]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:51.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:51 smithi190 ceph-mon[33871]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:51.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:51 smithi029 ceph-mon[31006]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:53.851 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:53 smithi066 ceph-mon[33436]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:53.851 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T02:34:53.851 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:53.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:53 smithi190 ceph-mon[33871]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:53.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T02:34:53.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:53 smithi029 ceph-mon[31006]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T02:34:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:34:54.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:54 smithi190 ceph-mon[33871]: Deploying daemon osd.10 on smithi066 2024-09-21T02:34:54.880 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:54 smithi066 ceph-mon[33436]: Deploying daemon osd.10 on smithi066 2024-09-21T02:34:54.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:54 smithi029 ceph-mon[31006]: Deploying daemon osd.10 on smithi066 2024-09-21T02:34:55.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:55 smithi190 ceph-mon[33871]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:55.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:55 smithi029 ceph-mon[31006]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:55.963 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:55 smithi066 ceph-mon[33436]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:57 smithi029 ceph-mon[31006]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:57 smithi066 ceph-mon[33436]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:58.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:57 smithi190 ceph-mon[33871]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:59 smithi029 ceph-mon[31006]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:34:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:59.895 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:59 smithi066 ceph-mon[33436]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:34:59.895 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:34:59.895 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:34:59.895 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:34:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:00.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:59 smithi190 ceph-mon[33871]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:34:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:01.825 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:01 smithi066 ceph-mon[33436]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:01 smithi029 ceph-mon[31006]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:02.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:01 smithi190 ceph-mon[33871]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:02.878 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 10 on host 'smithi066' 2024-09-21T02:35:02.878 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.876+0000 7ff3e9ffb640 1 -- 172.21.15.66:0/69644138 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff3ec0fff60 con 0x7ff3d80b6510 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3d80b6510 msgr2=0x7ff3d80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3d80b6510 0x7ff3d80b8a20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00abe0 tx=0x7ff3dc00a2d0 comp rx=0 tx=0).stop 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec103710 msgr2=0x7ff3ec19a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec103710 0x7ff3ec19a4f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400e9f0 tx=0x7ff3d400edf0 comp rx=0 tx=0).stop 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 shutdown_connections 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3d80b6510 0x7ff3d80b8a20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3ec1040b0 0x7ff3ec19b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3ec103710 0x7ff3ec19a4f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 --2- 172.21.15.66:0/69644138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ec102680 0x7ff3ec199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:02.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 >> 172.21.15.66:0/69644138 conn(0x7ff3ec0fdff0 msgr2=0x7ff3ec0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:02.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 shutdown_connections 2024-09-21T02:35:02.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-21T02:35:02.879+0000 7ff3f3042640 1 -- 172.21.15.66:0/69644138 wait complete. 2024-09-21T02:35:03.183 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:03.183 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.183 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.183 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:03.183 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:03.183 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.184 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:03.184 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.184 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:03 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:03.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:03.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:03.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:03.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:03 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:03 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:03.511 DEBUG:teuthology.orchestra.run.smithi066:osd.10> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.10.service 2024-09-21T02:35:03.513 INFO:tasks.cephadm:Deploying osd.11 on smithi190 with /dev/vg_nvme/lv_1... 2024-09-21T02:35:03.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T02:35:03.773 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:35:05.184 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:05 smithi190 ceph-mon[33871]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:05.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:05 smithi066 ceph-mon[33436]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:05 smithi029 ceph-mon[31006]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:05.636 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 02:35:05 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T02:35:05.335+0000 7f923f33b640 -1 osd.10 0 log_to_monitors true 2024-09-21T02:35:06.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:06 smithi190 ceph-mon[33871]: from='osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:35:06.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:06 smithi190 ceph-mon[33871]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:35:06.342 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:06 smithi066 ceph-mon[33436]: from='osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:35:06.342 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:06 smithi066 ceph-mon[33436]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:35:06.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:06 smithi029 ceph-mon[31006]: from='osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:35:06.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:06 smithi029 ceph-mon[31006]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:35:06.444 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-21T02:35:06.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi190:vg_nvme/lv_1 2024-09-21T02:35:06.738 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:35:07.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 -- 172.21.15.190:0/2637696560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 msgr2=0x7fb02c102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 --2- 172.21.15.190:0/2637696560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 0x7fb02c102ab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb024009900 tx=0x7fb02402a480 comp rx=0 tx=0).stop 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 -- 172.21.15.190:0/2637696560 shutdown_connections 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 --2- 172.21.15.190:0/2637696560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb02c104070 0x7fb02c10aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 --2- 172.21.15.190:0/2637696560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb02c1036d0 0x7fb02c103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 --2- 172.21.15.190:0/2637696560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 0x7fb02c102ab0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 -- 172.21.15.190:0/2637696560 >> 172.21.15.190:0/2637696560 conn(0x7fb02c0fdfb0 msgr2=0x7fb02c100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 -- 172.21.15.190:0/2637696560 shutdown_connections 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.241+0000 7fb0312e5640 1 -- 172.21.15.190:0/2637696560 wait complete. 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb0312e5640 1 Processor -- start 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb0312e5640 1 -- start start 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb0312e5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 0x7fb02c199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb0312e5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb02c1036d0 0x7fb02c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb0312e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb02c104070 0x7fb02c19b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb0312e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c10e200 con 0x7fb02c104070 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb0312e5640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c10e080 con 0x7fb02c1036d0 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb0312e5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c10e380 con 0x7fb02c102640 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb02affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 0x7fb02c199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb02affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 0x7fb02c199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:43926/0 (socket says 172.21.15.190:43926) 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb02a7fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb02c1036d0 0x7fb02c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.242+0000 7fb02affd640 1 -- 172.21.15.190:0/3779296877 learned_addr learned my addr 172.21.15.190:0/3779296877 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02b7fe640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb02c104070 0x7fb02c19b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02a7fc640 1 -- 172.21.15.190:0/3779296877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 msgr2=0x7fb02c199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02a7fc640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 0x7fb02c199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:07.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02a7fc640 1 -- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb02c104070 msgr2=0x7fb02c19b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02a7fc640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb02c104070 0x7fb02c19b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02a7fc640 1 -- 172.21.15.190:0/3779296877 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02c19b9d0 con 0x7fb02c1036d0 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02b7fe640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb02c104070 0x7fb02c19b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02affd640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 0x7fb02c199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb02a7fc640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb02c1036d0 0x7fb02c19a610 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb01400d720 tx=0x7fb01400dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb023fff640 1 -- 172.21.15.190:0/3779296877 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014017070 con 0x7fb02c1036d0 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02c19bcc0 con 0x7fb02c1036d0 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb023fff640 1 -- 172.21.15.190:0/3779296877 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb01400de30 con 0x7fb02c1036d0 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb023fff640 1 -- 172.21.15.190:0/3779296877 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0140047a0 con 0x7fb02c1036d0 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.243+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb02c1a23c0 con 0x7fb02c1036d0 2024-09-21T02:35:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.244+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb02c102ab0 con 0x7fb02c1036d0 2024-09-21T02:35:07.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.247+0000 7fb023fff640 1 -- 172.21.15.190:0/3779296877 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fb0140099d0 con 0x7fb02c1036d0 2024-09-21T02:35:07.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.248+0000 7fb023fff640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0180b6510 0x7fb0180b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:07.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.248+0000 7fb023fff640 1 -- 172.21.15.190:0/3779296877 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6414+0+0 (secure 0 0 0) 0x7fb0140e6c20 con 0x7fb02c1036d0 2024-09-21T02:35:07.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.248+0000 7fb02affd640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0180b6510 0x7fb0180b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:07.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.248+0000 7fb02affd640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0180b6510 0x7fb0180b8a20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb0240096c0 tx=0x7fb024031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:07.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.250+0000 7fb023fff640 1 -- 172.21.15.190:0/3779296877 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb014020020 con 0x7fb02c1036d0 2024-09-21T02:35:07.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi066 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi066 to 4272M 2024-09-21T02:35:07.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:07.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:07.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi066 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi066 to 4272M 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:07.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:07.446+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb02c0ffff0 con 0x7fb0180b6510 2024-09-21T02:35:07.509 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T02:35:07.074+0000 7f923b2bd640 -1 osd.10 0 waiting for initial osdmap 2024-09-21T02:35:07.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T02:35:07.087+0000 7f92370bb640 -1 osd.10 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]} : dispatch 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi066 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:07.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi066 to 4272M 2024-09-21T02:35:07.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:07.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:07.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:07.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] boot 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: osdmap e64: 11 total, 11 up, 11 in 2024-09-21T02:35:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:08 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] boot 2024-09-21T02:35:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: osdmap e64: 11 total, 11 up, 11 in 2024-09-21T02:35:08.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:08 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:08.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: osd.10 [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] boot 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: osdmap e64: 11 total, 11 up, 11 in 2024-09-21T02:35:08.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:08 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:35:09.079 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:09 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:35:09.079 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:09 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:35:09.348 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:09 smithi190 ceph-mon[33871]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:09.348 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:09 smithi190 ceph-mon[33871]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T02:35:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:09 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:35:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:09 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:35:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:09 smithi029 ceph-mon[31006]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:09.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:09 smithi029 ceph-mon[31006]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T02:35:09.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:09 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:35:09.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:09 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:35:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:09 smithi066 ceph-mon[33436]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:35:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:09 smithi066 ceph-mon[33436]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T02:35:10.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:10 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/4084541493' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"} : dispatch 2024-09-21T02:35:10.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:10 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"} : dispatch 2024-09-21T02:35:10.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:10 smithi190 ceph-mon[33871]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"}]': finished 2024-09-21T02:35:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:10 smithi190 ceph-mon[33871]: osdmap e66: 12 total, 11 up, 12 in 2024-09-21T02:35:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:10.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:10 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/4084541493' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"} : dispatch 2024-09-21T02:35:10.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:10 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"} : dispatch 2024-09-21T02:35:10.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:10 smithi029 ceph-mon[31006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"}]': finished 2024-09-21T02:35:10.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:10 smithi029 ceph-mon[31006]: osdmap e66: 12 total, 11 up, 12 in 2024-09-21T02:35:10.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:10.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:10 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/4084541493' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"} : dispatch 2024-09-21T02:35:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:10 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"} : dispatch 2024-09-21T02:35:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:10 smithi066 ceph-mon[33436]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75c747c5-0647-4baa-b2fa-a9651588604f"}]': finished 2024-09-21T02:35:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:10 smithi066 ceph-mon[33436]: osdmap e66: 12 total, 11 up, 12 in 2024-09-21T02:35:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:11.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:11 smithi190 ceph-mon[33871]: pgmap v252: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:11.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:11 smithi190 ceph-mon[33871]: from='client.? 172.21.15.190:0/571954597' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:35:11.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:11 smithi029 ceph-mon[31006]: pgmap v252: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:11.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:11 smithi029 ceph-mon[31006]: from='client.? 172.21.15.190:0/571954597' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:35:11.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:11 smithi066 ceph-mon[33436]: pgmap v252: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:11.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:11 smithi066 ceph-mon[33436]: from='client.? 172.21.15.190:0/571954597' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:35:13.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:13 smithi190 ceph-mon[33871]: pgmap v253: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:13.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:13 smithi029 ceph-mon[31006]: pgmap v253: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:13.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:13 smithi066 ceph-mon[33436]: pgmap v253: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:15.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:15 smithi190 ceph-mon[33871]: pgmap v254: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:15.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:15 smithi029 ceph-mon[31006]: pgmap v254: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:15.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:15 smithi066 ceph-mon[33436]: pgmap v254: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:17.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:17 smithi190 ceph-mon[33871]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:35:17.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:17 smithi029 ceph-mon[31006]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:35:17.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:17 smithi066 ceph-mon[33436]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-21T02:35:19.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:19 smithi190 ceph-mon[33871]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:35:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T02:35:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:19.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:19 smithi190 ceph-mon[33871]: Deploying daemon osd.11 on smithi190 2024-09-21T02:35:19.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:19 smithi029 ceph-mon[31006]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:35:19.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T02:35:19.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:19.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:19 smithi029 ceph-mon[31006]: Deploying daemon osd.11 on smithi190 2024-09-21T02:35:19.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:19 smithi066 ceph-mon[33436]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-21T02:35:19.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T02:35:19.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:19.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:19 smithi066 ceph-mon[33436]: Deploying daemon osd.11 on smithi190 2024-09-21T02:35:21.305 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:21 smithi190 ceph-mon[33871]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:35:21.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:21 smithi029 ceph-mon[31006]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:35:21.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:21 smithi066 ceph-mon[33436]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T02:35:23.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:23 smithi190 ceph-mon[33871]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:23.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:23 smithi029 ceph-mon[31006]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:23.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:23 smithi066 ceph-mon[33436]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:24.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:24 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:24 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:24.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:24 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:24 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:24 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:24.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:24 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:24.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:24 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:24.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:24 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:24.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:24 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:25.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:25 smithi190 ceph-mon[33871]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:25.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:25 smithi029 ceph-mon[31006]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:25.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:25 smithi066 ceph-mon[33436]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:27.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:27 smithi190 ceph-mon[33871]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:27.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:27.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:27.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:27 smithi029 ceph-mon[31006]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:27.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:27.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:27.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:27 smithi066 ceph-mon[33436]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:35:27.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:27.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:27.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:27.702 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.699+0000 7fb023fff640 1 -- 172.21.15.190:0/3779296877 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb02c0ffff0 con 0x7fb0180b6510 2024-09-21T02:35:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 11 on host 'smithi190' 2024-09-21T02:35:27.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0180b6510 msgr2=0x7fb0180b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:27.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0180b6510 0x7fb0180b8a20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb0240096c0 tx=0x7fb024031400 comp rx=0 tx=0).stop 2024-09-21T02:35:27.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb02c1036d0 msgr2=0x7fb02c19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:27.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb02c1036d0 0x7fb02c19a610 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb01400d720 tx=0x7fb01400dbe0 comp rx=0 tx=0).stop 2024-09-21T02:35:27.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 shutdown_connections 2024-09-21T02:35:27.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0180b6510 0x7fb0180b8a20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:27.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb02c104070 0x7fb02c19b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:27.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb02c1036d0 0x7fb02c19a610 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:27.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 --2- 172.21.15.190:0/3779296877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb02c102640 0x7fb02c199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:27.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 >> 172.21.15.190:0/3779296877 conn(0x7fb02c0fdfb0 msgr2=0x7fb02c0ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:27.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 shutdown_connections 2024-09-21T02:35:27.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:27.702+0000 7fb0312e5640 1 -- 172.21.15.190:0/3779296877 wait complete. 2024-09-21T02:35:28.347 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:28 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:28.347 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:28 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:28.347 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:28 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:28.380 DEBUG:teuthology.orchestra.run.smithi190:osd.11> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.11.service 2024-09-21T02:35:28.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:28 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:28.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:28 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:28.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:28 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:28.383 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-21T02:35:28.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T02:35:28.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:28 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:28.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:28 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:28.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:28 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:28.653 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 -- 172.21.15.29:0/1239009654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 msgr2=0x7fccbc108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/1239009654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 0x7fccbc108d60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009900 tx=0x7fccb002a410 comp rx=0 tx=0).stop 2024-09-21T02:35:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 -- 172.21.15.29:0/1239009654 shutdown_connections 2024-09-21T02:35:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/1239009654 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccbc1094a0 0x7fccbc10b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/1239009654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 0x7fccbc108d60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/1239009654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccbc101b70 0x7fccbc101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 -- 172.21.15.29:0/1239009654 >> 172.21.15.29:0/1239009654 conn(0x7fccbc0fd760 msgr2=0x7fccbc0ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 -- 172.21.15.29:0/1239009654 shutdown_connections 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.174+0000 7fccc2a4f640 1 -- 172.21.15.29:0/1239009654 wait complete. 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.175+0000 7fccc2a4f640 1 Processor -- start 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.175+0000 7fccc2a4f640 1 -- start start 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.175+0000 7fccc2a4f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccbc101b70 0x7fccbc199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.175+0000 7fccc2a4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 0x7fccbc19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.175+0000 7fccc2a4f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccbc1094a0 0x7fccbc19b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.175+0000 7fccc2a4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc10f2e0 con 0x7fccbc1025a0 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.175+0000 7fccc2a4f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc10f160 con 0x7fccbc1094a0 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.175+0000 7fccc2a4f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc10f460 con 0x7fccbc101b70 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 0x7fccbc19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 0x7fccbc19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51004/0 (socket says 172.21.15.29:51004) 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 -- 172.21.15.29:0/2175355281 learned_addr learned my addr 172.21.15.29:0/2175355281 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbbfff640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccbc101b70 0x7fccbc199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccc0fec640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccbc1094a0 0x7fccbc19b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 -- 172.21.15.29:0/2175355281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccbc101b70 msgr2=0x7fccbc199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccbc101b70 0x7fccbc199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 -- 172.21.15.29:0/2175355281 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccbc1094a0 msgr2=0x7fccbc19b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccbc1094a0 0x7fccbc19b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 -- 172.21.15.29:0/2175355281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc19b9d0 con 0x7fccbc1025a0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbbfff640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccbc101b70 0x7fccbc199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccbb7fe640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 0x7fccbc19a610 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009230 tx=0x7fccb00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccb97fa640 1 -- 172.21.15.29:0/2175355281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb003f030 con 0x7fccbc1025a0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccbc19bc60 con 0x7fccbc1025a0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccb97fa640 1 -- 172.21.15.29:0/2175355281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fccb00044c0 con 0x7fccbc1025a0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccb97fa640 1 -- 172.21.15.29:0/2175355281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0035070 con 0x7fccbc1025a0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.176+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccbc1a23c0 con 0x7fccbc1025a0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.177+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccbc101f80 con 0x7fccbc1025a0 2024-09-21T02:35:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.179+0000 7fccb97fa640 1 -- 172.21.15.29:0/2175355281 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fccb0042030 con 0x7fccbc1025a0 2024-09-21T02:35:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.180+0000 7fccb97fa640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcca40b6510 0x7fcca40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.180+0000 7fccbbfff640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcca40b6510 0x7fcca40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.180+0000 7fccb97fa640 1 -- 172.21.15.29:0/2175355281 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6809+0+0 (secure 0 0 0) 0x7fccb0107670 con 0x7fccbc1025a0 2024-09-21T02:35:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.180+0000 7fccbbfff640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcca40b6510 0x7fcca40b8a20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fccac009c20 tx=0x7fccac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.183+0000 7fccb97fa640 1 -- 172.21.15.29:0/2175355281 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fccb00ce4e0 con 0x7fccbc1025a0 2024-09-21T02:35:29.376 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:29 smithi029 ceph-mon[31006]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:29.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.375+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fccbc063fc0 con 0x7fccbc1025a0 2024-09-21T02:35:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:35:29.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.375+0000 7fccb97fa640 1 -- 172.21.15.29:0/2175355281 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v66) ==== 74+0+133 (secure 0 0 0) 0x7fccb00d28c0 con 0x7fccbc1025a0 2024-09-21T02:35:29.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.377+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcca40b6510 msgr2=0x7fcca40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:29.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcca40b6510 0x7fcca40b8a20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fccac009c20 tx=0x7fccac005f90 comp rx=0 tx=0).stop 2024-09-21T02:35:29.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 msgr2=0x7fccbc19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:29.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 0x7fccbc19a610 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009230 tx=0x7fccb00040c0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 shutdown_connections 2024-09-21T02:35:29.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcca40b6510 0x7fcca40b8a20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccbc1094a0 0x7fccbc19b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccbc1025a0 0x7fccbc19a610 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 --2- 172.21.15.29:0/2175355281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccbc101b70 0x7fccbc199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:29.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 >> 172.21.15.29:0/2175355281 conn(0x7fccbc0fd760 msgr2=0x7fccbc10a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:29.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 shutdown_connections 2024-09-21T02:35:29.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:29.378+0000 7fccc2a4f640 1 -- 172.21.15.29:0/2175355281 wait complete. 2024-09-21T02:35:29.504 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:29 smithi190 ceph-mon[33871]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:29.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:29 smithi066 ceph-mon[33436]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:30.191 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"num_osds":12,"num_up_osds":11,"osd_up_since":1726886107,"num_in_osds":12,"osd_in_since":1726886109,"num_remapped_pgs":0} 2024-09-21T02:35:30.505 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:30 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2175355281' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:30.505 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 02:35:30 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T02:35:30.307+0000 7fb61f163640 -1 osd.11 0 log_to_monitors true 2024-09-21T02:35:30.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:30 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2175355281' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:30 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2175355281' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:31.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T02:35:31.452 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:31.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:31 smithi066 ceph-mon[33436]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:31.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:31 smithi066 ceph-mon[33436]: from='osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:35:31.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:31 smithi066 ceph-mon[33436]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:35:31.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:31 smithi029 ceph-mon[31006]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:31.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:31 smithi029 ceph-mon[31006]: from='osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:35:31.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:31 smithi029 ceph-mon[31006]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:35:31.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:31 smithi190 ceph-mon[33871]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:31.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:31 smithi190 ceph-mon[33871]: from='osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:35:31.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:31 smithi190 ceph-mon[33871]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:35:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 -- 172.21.15.29:0/2700637184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 msgr2=0x7f4b6c108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 --2- 172.21.15.29:0/2700637184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 0x7f4b6c108820 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009900 tx=0x7f4b6802a410 comp rx=0 tx=0).stop 2024-09-21T02:35:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 -- 172.21.15.29:0/2700637184 shutdown_connections 2024-09-21T02:35:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 --2- 172.21.15.29:0/2700637184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c10b6c0 0x7f4b6c10dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 --2- 172.21.15.29:0/2700637184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b6c108d60 0x7f4b6c10b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 --2- 172.21.15.29:0/2700637184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 0x7f4b6c108820 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 -- 172.21.15.29:0/2700637184 >> 172.21.15.29:0/2700637184 conn(0x7f4b6c0fdfa0 msgr2=0x7f4b6c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 -- 172.21.15.29:0/2700637184 shutdown_connections 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.047+0000 7f4b73982640 1 -- 172.21.15.29:0/2700637184 wait complete. 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b73982640 1 Processor -- start 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b73982640 1 -- start start 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b73982640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 0x7f4b6c19c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b73982640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c108d60 0x7f4b6c19c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b73982640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b6c10b6c0 0x7f4b6c19d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b73982640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c111440 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b73982640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1112c0 con 0x7f4b6c10b6c0 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b73982640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1115c0 con 0x7f4b6c108d60 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b7171e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 0x7f4b6c19c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b7171e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 0x7f4b6c19c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51024/0 (socket says 172.21.15.29:51024) 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b7171e640 1 -- 172.21.15.29:0/2607318745 learned_addr learned my addr 172.21.15.29:0/2607318745 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b71f1f640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b6c10b6c0 0x7f4b6c19d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.048+0000 7f4b70f1d640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c108d60 0x7f4b6c19c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b7171e640 1 -- 172.21.15.29:0/2607318745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c108d60 msgr2=0x7f4b6c19c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b7171e640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c108d60 0x7f4b6c19c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b7171e640 1 -- 172.21.15.29:0/2607318745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b6c10b6c0 msgr2=0x7f4b6c19d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b7171e640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b6c10b6c0 0x7f4b6c19d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b7171e640 1 -- 172.21.15.29:0/2607318745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c19db30 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b70f1d640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c108d60 0x7f4b6c19c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b7171e640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 0x7f4b6c19c100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009230 tx=0x7f4b680042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b627fc640 1 -- 172.21.15.29:0/2607318745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6803f030 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c19ddc0 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b627fc640 1 -- 172.21.15.29:0/2607318745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4b68003c90 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b627fc640 1 -- 172.21.15.29:0/2607318745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68035070 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.049+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c1a4350 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.050+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b6c0ffac0 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.052+0000 7f4b627fc640 1 -- 172.21.15.29:0/2607318745 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f4b68042030 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.053+0000 7f4b627fc640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b580b6510 0x7f4b580b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:32.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.053+0000 7f4b70f1d640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b580b6510 0x7f4b580b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:32.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.053+0000 7f4b627fc640 1 -- 172.21.15.29:0/2607318745 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6831+0+0 (secure 0 0 0) 0x7f4b68107780 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.054+0000 7f4b70f1d640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b580b6510 0x7f4b580b8a20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0058c0 tx=0x7f4b5c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:32.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.056+0000 7f4b627fc640 1 -- 172.21.15.29:0/2607318745 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4b680ce5c0 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.247+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f4b6c063fc0 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: osdmap e67: 12 total, 11 up, 12 in 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: Detected new or changed devices on smithi190 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi190 to 4272M 2024-09-21T02:35:32.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:32.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:32.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.248+0000 7f4b627fc640 1 -- 172.21.15.29:0/2607318745 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v68) ==== 74+0+133 (secure 0 0 0) 0x7f4b680d29a0 con 0x7f4b6c06c3b0 2024-09-21T02:35:32.251 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:35:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b580b6510 msgr2=0x7f4b580b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b580b6510 0x7f4b580b8a20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0058c0 tx=0x7f4b5c005850 comp rx=0 tx=0).stop 2024-09-21T02:35:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 msgr2=0x7f4b6c19c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 0x7f4b6c19c100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009230 tx=0x7f4b680042f0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 shutdown_connections 2024-09-21T02:35:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b580b6510 0x7f4b580b8a20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b6c10b6c0 0x7f4b6c19d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c108d60 0x7f4b6c19c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 --2- 172.21.15.29:0/2607318745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b6c06c3b0 0x7f4b6c19c100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 >> 172.21.15.29:0/2607318745 conn(0x7f4b6c0fdfa0 msgr2=0x7f4b6c109cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 shutdown_connections 2024-09-21T02:35:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:32.251+0000 7f4b73982640 1 -- 172.21.15.29:0/2607318745 wait complete. 2024-09-21T02:35:32.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T02:35:32.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: osdmap e67: 12 total, 11 up, 12 in 2024-09-21T02:35:32.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:32.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:35:32.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:35:32.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: Detected new or changed devices on smithi190 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi190 to 4272M 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:32.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.623 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T02:35:32.164+0000 7fb61b0e5640 -1 osd.11 0 waiting for initial osdmap 2024-09-21T02:35:32.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T02:35:32.182+0000 7fb616ee3640 -1 osd.11 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: osdmap e67: 12 total, 11 up, 12 in 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]} : dispatch 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: Detected new or changed devices on smithi190 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:35:32.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi190 to 4272M 2024-09-21T02:35:32.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:32.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:32.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:32.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:33.113 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"num_osds":12,"num_up_osds":11,"osd_up_since":1726886107,"num_in_osds":12,"osd_in_since":1726886109,"num_remapped_pgs":0} 2024-09-21T02:35:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:33 smithi029 ceph-mon[31006]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:33 smithi029 ceph-mon[31006]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:35:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:33 smithi029 ceph-mon[31006]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T02:35:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2607318745' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:33.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:33 smithi066 ceph-mon[33436]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:33.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:33 smithi066 ceph-mon[33436]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:35:33.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:33 smithi066 ceph-mon[33436]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T02:35:33.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:33.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:33.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:33 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2607318745' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:33.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:33 smithi190 ceph-mon[33871]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:33.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:33 smithi190 ceph-mon[33871]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-09-21T02:35:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:33 smithi190 ceph-mon[33871]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T02:35:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:33 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2607318745' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:34.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T02:35:34.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:34 smithi029 ceph-mon[31006]: purged_snaps scrub starts 2024-09-21T02:35:34.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:34 smithi029 ceph-mon[31006]: purged_snaps scrub ok 2024-09-21T02:35:34.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:34 smithi029 ceph-mon[31006]: osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] boot 2024-09-21T02:35:34.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:34 smithi029 ceph-mon[31006]: osdmap e69: 12 total, 12 up, 12 in 2024-09-21T02:35:34.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:34.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:34 smithi190 ceph-mon[33871]: purged_snaps scrub starts 2024-09-21T02:35:34.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:34 smithi190 ceph-mon[33871]: purged_snaps scrub ok 2024-09-21T02:35:34.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:34 smithi190 ceph-mon[33871]: osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] boot 2024-09-21T02:35:34.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:34 smithi190 ceph-mon[33871]: osdmap e69: 12 total, 12 up, 12 in 2024-09-21T02:35:34.502 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:34.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:34 smithi066 ceph-mon[33436]: purged_snaps scrub starts 2024-09-21T02:35:34.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:34 smithi066 ceph-mon[33436]: purged_snaps scrub ok 2024-09-21T02:35:34.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:34 smithi066 ceph-mon[33436]: osd.11 [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] boot 2024-09-21T02:35:34.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:34 smithi066 ceph-mon[33436]: osdmap e69: 12 total, 12 up, 12 in 2024-09-21T02:35:34.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:35:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 -- 172.21.15.29:0/683696000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64102640 msgr2=0x7fae64102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 --2- 172.21.15.29:0/683696000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64102640 0x7fae64102ab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fae54009900 tx=0x7fae5402a480 comp rx=0 tx=0).stop 2024-09-21T02:35:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 -- 172.21.15.29:0/683696000 shutdown_connections 2024-09-21T02:35:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 --2- 172.21.15.29:0/683696000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae64104070 0x7fae6410aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 --2- 172.21.15.29:0/683696000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae641036d0 0x7fae64103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 --2- 172.21.15.29:0/683696000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64102640 0x7fae64102ab0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 -- 172.21.15.29:0/683696000 >> 172.21.15.29:0/683696000 conn(0x7fae640fdfb0 msgr2=0x7fae64100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 -- 172.21.15.29:0/683696000 shutdown_connections 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 -- 172.21.15.29:0/683696000 wait complete. 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.889+0000 7fae699c6640 1 Processor -- start 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae699c6640 1 -- start start 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae699c6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae64102640 0x7fae64199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae699c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae641036d0 0x7fae6419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae699c6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae64104070 0x7fae6419b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae699c6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae6410e180 con 0x7fae641036d0 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae699c6640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fae6410e000 con 0x7fae64102640 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae699c6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fae6410e300 con 0x7fae64104070 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae641036d0 0x7fae6419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae641036d0 0x7fae6419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45650/0 (socket says 172.21.15.29:45650) 2024-09-21T02:35:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae64102640 0x7fae64199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 -- 172.21.15.29:0/3404682493 learned_addr learned my addr 172.21.15.29:0/3404682493 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae637fe640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae64104070 0x7fae6419b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 -- 172.21.15.29:0/3404682493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae64104070 msgr2=0x7fae6419b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae64104070 0x7fae6419b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 -- 172.21.15.29:0/3404682493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae64102640 msgr2=0x7fae64199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae64102640 0x7fae64199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 -- 172.21.15.29:0/3404682493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6419b9a0 con 0x7fae641036d0 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae62ffd640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae64102640 0x7fae64199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.890+0000 7fae627fc640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae641036d0 0x7fae6419a5e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fae5800e9f0 tx=0x7fae5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.891+0000 7fae689c4640 1 -- 172.21.15.29:0/3404682493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5800cd80 con 0x7fae641036d0 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.891+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6419bc90 con 0x7fae641036d0 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.891+0000 7fae689c4640 1 -- 172.21.15.29:0/3404682493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fae580054b0 con 0x7fae641036d0 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.891+0000 7fae689c4640 1 -- 172.21.15.29:0/3404682493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae58010430 con 0x7fae641036d0 2024-09-21T02:35:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.891+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae641a2390 con 0x7fae641036d0 2024-09-21T02:35:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.892+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae641a2560 con 0x7fae641036d0 2024-09-21T02:35:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.893+0000 7fae689c4640 1 -- 172.21.15.29:0/3404682493 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fae580046c0 con 0x7fae641036d0 2024-09-21T02:35:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.894+0000 7fae689c4640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fae4c0b6510 0x7fae4c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.894+0000 7fae62ffd640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fae4c0b6510 0x7fae4c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.894+0000 7fae689c4640 1 -- 172.21.15.29:0/3404682493 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7fae580e7480 con 0x7fae641036d0 2024-09-21T02:35:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.895+0000 7fae62ffd640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fae4c0b6510 0x7fae4c0b8a20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae54005760 tx=0x7fae54031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:34.897+0000 7fae689c4640 1 -- 172.21.15.29:0/3404682493 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fae58056710 con 0x7fae641036d0 2024-09-21T02:35:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.087+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fae641a2730 con 0x7fae641036d0 2024-09-21T02:35:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.087+0000 7fae689c4640 1 -- 172.21.15.29:0/3404682493 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7fae580a9030 con 0x7fae641036d0 2024-09-21T02:35:35.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:35:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fae4c0b6510 msgr2=0x7fae4c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fae4c0b6510 0x7fae4c0b8a20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae54005760 tx=0x7fae54031400 comp rx=0 tx=0).stop 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae641036d0 msgr2=0x7fae6419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae641036d0 0x7fae6419a5e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fae5800e9f0 tx=0x7fae5800edf0 comp rx=0 tx=0).stop 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 shutdown_connections 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fae4c0b6510 0x7fae4c0b8a20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae64104070 0x7fae6419b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae641036d0 0x7fae6419a5e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 --2- 172.21.15.29:0/3404682493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae64102640 0x7fae64199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 >> 172.21.15.29:0/3404682493 conn(0x7fae640fdfb0 msgr2=0x7fae640ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 shutdown_connections 2024-09-21T02:35:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:35.090+0000 7fae699c6640 1 -- 172.21.15.29:0/3404682493 wait complete. 2024-09-21T02:35:35.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:35 smithi190 ceph-mon[33871]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:35.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:35 smithi190 ceph-mon[33871]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T02:35:35.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:35 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3404682493' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:35 smithi029 ceph-mon[31006]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:35 smithi029 ceph-mon[31006]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T02:35:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:35 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3404682493' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:35.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:35 smithi066 ceph-mon[33436]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:35:35.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:35 smithi066 ceph-mon[33436]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T02:35:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:35 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3404682493' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:35:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":70,"num_osds":12,"num_up_osds":12,"osd_up_since":1726886133,"num_in_osds":12,"osd_in_since":1726886109,"num_remapped_pgs":0} 2024-09-21T02:35:35.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:35:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:36.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 -- 172.21.15.29:0/3213547827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae0109df0 msgr2=0x7fbae0110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 --2- 172.21.15.29:0/3213547827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae0109df0 0x7fbae0110620 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009ca0 tx=0x7fbad802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 -- 172.21.15.29:0/3213547827 shutdown_connections 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 --2- 172.21.15.29:0/3213547827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae0109df0 0x7fbae0110620 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 --2- 172.21.15.29:0/3213547827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbae01094a0 0x7fbae01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 --2- 172.21.15.29:0/3213547827 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae0103b60 0x7fbae0103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 -- 172.21.15.29:0/3213547827 >> 172.21.15.29:0/3213547827 conn(0x7fbae007b850 msgr2=0x7fbae007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 -- 172.21.15.29:0/3213547827 shutdown_connections 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.652+0000 7fbae57b9640 1 -- 172.21.15.29:0/3213547827 wait complete. 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.653+0000 7fbae57b9640 1 Processor -- start 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.653+0000 7fbae57b9640 1 -- start start 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbae57b9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbae0103b60 0x7fbae019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbae57b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae01094a0 0x7fbae019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbae57b9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae0109df0 0x7fbae019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbae57b9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01139b0 con 0x7fbae01094a0 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbae57b9640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0113830 con 0x7fbae0109df0 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbae57b9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0113b30 con 0x7fbae0103b60 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae01094a0 0x7fbae019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae01094a0 0x7fbae019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45676/0 (socket says 172.21.15.29:45676) 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 -- 172.21.15.29:0/353564543 learned_addr learned my addr 172.21.15.29:0/353564543 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbadeffd640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbae0103b60 0x7fbae019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbadf7fe640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae0109df0 0x7fbae019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 -- 172.21.15.29:0/353564543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbae0103b60 msgr2=0x7fbae019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbae0103b60 0x7fbae019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 -- 172.21.15.29:0/353564543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae0109df0 msgr2=0x7fbae019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae0109df0 0x7fbae019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 -- 172.21.15.29:0/353564543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae01a0370 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbadf7fe640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae0109df0 0x7fbae019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbad67fc640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae01094a0 0x7fbae019aa60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbac800e9c0 tx=0x7fbac800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.654+0000 7fbadcff9640 1 -- 172.21.15.29:0/353564543 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac800ccc0 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.655+0000 7fbadcff9640 1 -- 172.21.15.29:0/353564543 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbac80054b0 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.655+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae01a0660 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.655+0000 7fbadcff9640 1 -- 172.21.15.29:0/353564543 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8010430 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.655+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae01a0b50 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.656+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae0103f70 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.658+0000 7fbadcff9640 1 -- 172.21.15.29:0/353564543 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fbac8004ee0 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.659+0000 7fbadcff9640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbacc0b6540 0x7fbacc0b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.659+0000 7fbadeffd640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbacc0b6540 0x7fbacc0b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.659+0000 7fbadcff9640 1 -- 172.21.15.29:0/353564543 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7fbac80e73f0 con 0x7fbae01094a0 2024-09-21T02:35:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.659+0000 7fbadeffd640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbacc0b6540 0x7fbacc0b8a50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbad0005e10 tx=0x7fbad0005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:36.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.662+0000 7fbadcff9640 1 -- 172.21.15.29:0/353564543 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbac8056710 con 0x7fbae01094a0 2024-09-21T02:35:36.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.849+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbae019b5f0 con 0x7fbae01094a0 2024-09-21T02:35:36.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.850+0000 7fbadcff9640 1 -- 172.21.15.29:0/353564543 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+18455 (secure 0 0 0) 0x7fbae01a0b50 con 0x7fbae01094a0 2024-09-21T02:35:36.853 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:35:36.853 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":70,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:35:34.164955+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:35:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbacc0b6540 msgr2=0x7fbacc0b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbacc0b6540 0x7fbacc0b8a50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbad0005e10 tx=0x7fbad0005da0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae01094a0 msgr2=0x7fbae019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae01094a0 0x7fbae019aa60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbac800e9c0 tx=0x7fbac800ee80 comp rx=0 tx=0).stop 2024-09-21T02:35:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 shutdown_connections 2024-09-21T02:35:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbacc0b6540 0x7fbacc0b8a50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae0109df0 0x7fbae019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae01094a0 0x7fbae019aa60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 --2- 172.21.15.29:0/353564543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbae0103b60 0x7fbae019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 >> 172.21.15.29:0/353564543 conn(0x7fbae007b850 msgr2=0x7fbae0107690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 shutdown_connections 2024-09-21T02:35:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:36.853+0000 7fbae57b9640 1 -- 172.21.15.29:0/353564543 wait complete. 2024-09-21T02:35:37.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:37 smithi066 ceph-mon[33436]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:37.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:37 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/353564543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:35:37.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:37 smithi190 ceph-mon[33871]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:37.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:37 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/353564543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:35:37.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:37 smithi029 ceph-mon[31006]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:37.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:37 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/353564543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:35:37.805 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T02:31:38.302181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T02:35:37.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T02:35:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.586+0000 7f440c80c640 1 -- 172.21.15.29:0/1826835615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4408102ae0 msgr2=0x7f4408102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.586+0000 7f440c80c640 1 --2- 172.21.15.29:0/1826835615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4408102ae0 0x7f4408102ef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009900 tx=0x7f43f802a3f0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.586+0000 7f440c80c640 1 -- 172.21.15.29:0/1826835615 shutdown_connections 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.586+0000 7f440c80c640 1 --2- 172.21.15.29:0/1826835615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4408108d20 0x7f440810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.586+0000 7f440c80c640 1 --2- 172.21.15.29:0/1826835615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44081083d0 0x7f44081087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.586+0000 7f440c80c640 1 --2- 172.21.15.29:0/1826835615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4408102ae0 0x7f4408102ef0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.586+0000 7f440c80c640 1 -- 172.21.15.29:0/1826835615 >> 172.21.15.29:0/1826835615 conn(0x7f4408079790 msgr2=0x7f4408079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.587+0000 7f440c80c640 1 -- 172.21.15.29:0/1826835615 shutdown_connections 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.587+0000 7f440c80c640 1 -- 172.21.15.29:0/1826835615 wait complete. 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.587+0000 7f440c80c640 1 Processor -- start 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440c80c640 1 -- start start 2024-09-21T02:35:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440c80c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4408102ae0 0x7f440819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440c80c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44081083d0 0x7f440819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440c80c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4408108d20 0x7f440819fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440c80c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44081139a0 con 0x7f44081083d0 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440c80c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4408113820 con 0x7f4408102ae0 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440c80c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4408113b20 con 0x7f4408108d20 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44081083d0 0x7f440819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44081083d0 0x7f440819aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45684/0 (socket says 172.21.15.29:45684) 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 -- 172.21.15.29:0/4108959692 learned_addr learned my addr 172.21.15.29:0/4108959692 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440659c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4408102ae0 0x7f440819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f4406d9d640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4408108d20 0x7f440819fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 -- 172.21.15.29:0/4108959692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4408108d20 msgr2=0x7f440819fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4408108d20 0x7f440819fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 -- 172.21.15.29:0/4108959692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4408102ae0 msgr2=0x7f440819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4408102ae0 0x7f440819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 -- 172.21.15.29:0/4108959692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44081a0460 con 0x7f44081083d0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f440659c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4408102ae0 0x7f440819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.588+0000 7f43fdd9b640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44081083d0 0x7f440819aa90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f43f000d950 tx=0x7f43f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.589+0000 7f43fffff640 1 -- 172.21.15.29:0/4108959692 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0017070 con 0x7f44081083d0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.589+0000 7f43fffff640 1 -- 172.21.15.29:0/4108959692 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43f0009c30 con 0x7f44081083d0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.589+0000 7f43fffff640 1 -- 172.21.15.29:0/4108959692 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f00047a0 con 0x7f44081083d0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.589+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44081a0750 con 0x7f44081083d0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.589+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44081a6eb0 con 0x7f44081083d0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.590+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4408102ef0 con 0x7f44081083d0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.591+0000 7f43fffff640 1 -- 172.21.15.29:0/4108959692 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f43f0004020 con 0x7f44081083d0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.592+0000 7f43fffff640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43f40b6590 0x7f43f40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.592+0000 7f440659c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43f40b6590 0x7f43f40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.592+0000 7f43fffff640 1 -- 172.21.15.29:0/4108959692 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7f43f00e7030 con 0x7f44081083d0 2024-09-21T02:35:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.593+0000 7f440659c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43f40b6590 0x7f43f40b8aa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43f8000c00 tx=0x7f43f8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.595+0000 7f43fffff640 1 -- 172.21.15.29:0/4108959692 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f43f0020020 con 0x7f44081083d0 2024-09-21T02:35:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.783+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f440802e640 con 0x7f44081083d0 2024-09-21T02:35:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.784+0000 7f43fffff640 1 -- 172.21.15.29:0/4108959692 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7f43f00add40 con 0x7f44081083d0 2024-09-21T02:35:38.786 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43f40b6590 msgr2=0x7f43f40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43f40b6590 0x7f43f40b8aa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43f8000c00 tx=0x7f43f8005a20 comp rx=0 tx=0).stop 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44081083d0 msgr2=0x7f440819aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44081083d0 0x7f440819aa90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f43f000d950 tx=0x7f43f000de10 comp rx=0 tx=0).stop 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 shutdown_connections 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43f40b6590 0x7f43f40b8aa0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4408108d20 0x7f440819fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44081083d0 0x7f440819aa90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 --2- 172.21.15.29:0/4108959692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4408102ae0 0x7f440819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:38.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 >> 172.21.15.29:0/4108959692 conn(0x7f4408079790 msgr2=0x7f44081065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:38.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 shutdown_connections 2024-09-21T02:35:38.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:38.786+0000 7f440c80c640 1 -- 172.21.15.29:0/4108959692 wait complete. 2024-09-21T02:35:39.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:39 smithi066 ceph-mon[33436]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:39.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:39 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4108959692' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:35:39.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:39 smithi190 ceph-mon[33871]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:39.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:39 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4108959692' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:35:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:39 smithi029 ceph-mon[31006]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:39.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:39 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4108959692' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:35:39.693 INFO:tasks.cephadm:Adding mds.a on smithi029 2024-09-21T02:35:39.693 INFO:tasks.cephadm:Adding mds.d on smithi029 2024-09-21T02:35:39.693 INFO:tasks.cephadm:Adding mds.g on smithi029 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.j on smithi029 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.b on smithi066 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.e on smithi066 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.h on smithi066 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.k on smithi066 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.c on smithi190 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.f on smithi190 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.i on smithi190 2024-09-21T02:35:39.694 INFO:tasks.cephadm:Adding mds.l on smithi190 2024-09-21T02:35:39.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph orch apply mds all '12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l' 2024-09-21T02:35:39.951 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 -- 172.21.15.190:0/68998802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 msgr2=0x7fcbc810aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/68998802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 0x7fcbc810aac0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009ca0 tx=0x7fcbc002a950 comp rx=0 tx=0).stop 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 -- 172.21.15.190:0/68998802 shutdown_connections 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/68998802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 0x7fcbc810aac0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/68998802 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc8103780 0x7fcbc8103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/68998802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc81026f0 0x7fcbc8102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 -- 172.21.15.190:0/68998802 >> 172.21.15.190:0/68998802 conn(0x7fcbc80fe080 msgr2=0x7fcbc8100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 -- 172.21.15.190:0/68998802 shutdown_connections 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.420+0000 7fcbcf8be640 1 -- 172.21.15.190:0/68998802 wait complete. 2024-09-21T02:35:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.421+0000 7fcbcf8be640 1 Processor -- start 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.421+0000 7fcbcf8be640 1 -- start start 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.421+0000 7fcbcf8be640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc81026f0 0x7fcbc81062e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.421+0000 7fcbcf8be640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc8103780 0x7fcbc8104bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.421+0000 7fcbcf8be640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 0x7fcbc81050f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.421+0000 7fcbcf8be640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc810e1e0 con 0x7fcbc8103780 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.421+0000 7fcbcf8be640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc810e060 con 0x7fcbc81026f0 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.421+0000 7fcbcf8be640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc810e360 con 0x7fcbc8104120 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 0x7fcbc81050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 0x7fcbc81050f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:46906/0 (socket says 172.21.15.190:46906) 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 -- 172.21.15.190:0/3183965279 learned_addr learned my addr 172.21.15.190:0/3183965279 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcd65a640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc81026f0 0x7fcbc81062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcce59640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc8103780 0x7fcbc8104bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 -- 172.21.15.190:0/3183965279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc81026f0 msgr2=0x7fcbc81062e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:40.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc81026f0 0x7fcbc81062e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 -- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc8103780 msgr2=0x7fcbc8104bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc8103780 0x7fcbc8104bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 -- 172.21.15.190:0/3183965279 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc81057f0 con 0x7fcbc8104120 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcd65a640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc81026f0 0x7fcbc81062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbcde5b640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 0x7fcbc81050f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcbc002f800 tx=0x7fcbc0003f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.422+0000 7fcbba7fc640 1 -- 172.21.15.190:0/3183965279 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0034040 con 0x7fcbc8104120 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.423+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc81a3f90 con 0x7fcbc8104120 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.423+0000 7fcbba7fc640 1 -- 172.21.15.190:0/3183965279 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbc002fdf0 con 0x7fcbc8104120 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.423+0000 7fcbba7fc640 1 -- 172.21.15.190:0/3183965279 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0004850 con 0x7fcbc8104120 2024-09-21T02:35:40.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.423+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc81a4330 con 0x7fcbc8104120 2024-09-21T02:35:40.427 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.424+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc81a4590 con 0x7fcbc8104120 2024-09-21T02:35:40.427 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.425+0000 7fcbba7fc640 1 -- 172.21.15.190:0/3183965279 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fcbc0037020 con 0x7fcbc8104120 2024-09-21T02:35:40.427 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.426+0000 7fcbba7fc640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcbb40b66b0 0x7fcbb40b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:40.427 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.426+0000 7fcbba7fc640 1 -- 172.21.15.190:0/3183965279 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7fcbc0107f90 con 0x7fcbc8104120 2024-09-21T02:35:40.427 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.426+0000 7fcbcd65a640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcbb40b66b0 0x7fcbb40b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:40.428 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.427+0000 7fcbcd65a640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcbb40b66b0 0x7fcbb40b8bc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40058c0 tx=0x7fcbc400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:40.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.429+0000 7fcbba7fc640 1 -- 172.21.15.190:0/3183965279 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcbc00098b0 con 0x7fcbc8104120 2024-09-21T02:35:40.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.626+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}) -- 0x7fcbc8100050 con 0x7fcbb40b66b0 2024-09-21T02:35:40.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.635+0000 7fcbba7fc640 1 -- 172.21.15.190:0/3183965279 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fcbc8100050 con 0x7fcbb40b66b0 2024-09-21T02:35:40.636 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mds.all update... 2024-09-21T02:35:40.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcbb40b66b0 msgr2=0x7fcbb40b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:40.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcbb40b66b0 0x7fcbb40b8bc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40058c0 tx=0x7fcbc400a2d0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 msgr2=0x7fcbc81050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:40.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 0x7fcbc81050f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcbc002f800 tx=0x7fcbc0003f20 comp rx=0 tx=0).stop 2024-09-21T02:35:40.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 shutdown_connections 2024-09-21T02:35:40.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcbb40b66b0 0x7fcbb40b8bc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbc8104120 0x7fcbc81050f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc8103780 0x7fcbc8104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 --2- 172.21.15.190:0/3183965279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc81026f0 0x7fcbc81062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:40.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 >> 172.21.15.190:0/3183965279 conn(0x7fcbc80fe080 msgr2=0x7fcbc80ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:40.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 shutdown_connections 2024-09-21T02:35:40.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-21T02:35:40.637+0000 7fcbcf8be640 1 -- 172.21.15.190:0/3183965279 wait complete. 2024-09-21T02:35:41.311 DEBUG:teuthology.orchestra.run.smithi029:mds.a> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.a.service 2024-09-21T02:35:41.314 DEBUG:teuthology.orchestra.run.smithi029:mds.d> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.d.service 2024-09-21T02:35:41.357 DEBUG:teuthology.orchestra.run.smithi029:mds.g> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.g.service 2024-09-21T02:35:41.400 DEBUG:teuthology.orchestra.run.smithi029:mds.j> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.j.service 2024-09-21T02:35:41.443 DEBUG:teuthology.orchestra.run.smithi066:mds.b> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.b.service 2024-09-21T02:35:41.446 DEBUG:teuthology.orchestra.run.smithi066:mds.e> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.e.service 2024-09-21T02:35:41.488 DEBUG:teuthology.orchestra.run.smithi066:mds.h> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.h.service 2024-09-21T02:35:41.531 DEBUG:teuthology.orchestra.run.smithi066:mds.k> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.k.service 2024-09-21T02:35:41.574 DEBUG:teuthology.orchestra.run.smithi190:mds.c> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.c.service 2024-09-21T02:35:41.576 DEBUG:teuthology.orchestra.run.smithi190:mds.f> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.f.service 2024-09-21T02:35:41.620 DEBUG:teuthology.orchestra.run.smithi190:mds.i> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.i.service 2024-09-21T02:35:41.663 DEBUG:teuthology.orchestra.run.smithi190:mds.l> sudo journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.l.service 2024-09-21T02:35:41.706 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-21T02:35:41.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:35:41.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2024-09-21T02:35:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:41.809 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:41.809 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:41.809 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:41.809 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:41.809 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:41.809 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:42.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:42.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:42.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/772709426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4103b90 msgr2=0x7ff6a4103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:42.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/772709426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4103b90 0x7ff6a4103fa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff694009900 tx=0x7ff69402a410 comp rx=0 tx=0).stop 2024-09-21T02:35:42.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/772709426 shutdown_connections 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/772709426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6a4109e20 0x7ff6a4110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/772709426 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6a41094d0 0x7ff6a41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/772709426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4103b90 0x7ff6a4103fa0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/772709426 >> 172.21.15.29:0/772709426 conn(0x7ff6a40ff600 msgr2=0x7ff6a4101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/772709426 shutdown_connections 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.465+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/772709426 wait complete. 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.466+0000 7ff6a8b2f640 1 Processor -- start 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.466+0000 7ff6a8b2f640 1 -- start start 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a8b2f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6a4103b90 0x7ff6a419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a8b2f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6a41094d0 0x7ff6a419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a8b2f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4109e20 0x7ff6a419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a8b2f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4113990 con 0x7ff6a4109e20 2024-09-21T02:35:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a8b2f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4113810 con 0x7ff6a41094d0 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a8b2f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4113b10 con 0x7ff6a4103b90 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4109e20 0x7ff6a419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4109e20 0x7ff6a419fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45708/0 (socket says 172.21.15.29:45708) 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 -- 172.21.15.29:0/315311879 learned_addr learned my addr 172.21.15.29:0/315311879 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a259c640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6a4103b90 0x7ff6a419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a1d9b640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6a41094d0 0x7ff6a419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 -- 172.21.15.29:0/315311879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6a4103b90 msgr2=0x7ff6a419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6a4103b90 0x7ff6a419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 -- 172.21.15.29:0/315311879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6a41094d0 msgr2=0x7ff6a419a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6a41094d0 0x7ff6a419a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 -- 172.21.15.29:0/315311879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a41a02b0 con 0x7ff6a4109e20 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a2d9d640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4109e20 0x7ff6a419fc40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009fd0 tx=0x7ff69c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6937fe640 1 -- 172.21.15.29:0/315311879 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c017070 con 0x7ff6a4109e20 2024-09-21T02:35:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6937fe640 1 -- 172.21.15.29:0/315311879 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff69c003e00 con 0x7ff6a4109e20 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6937fe640 1 -- 172.21.15.29:0/315311879 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c010460 con 0x7ff6a4109e20 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.467+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a41a05a0 con 0x7ff6a4109e20 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.468+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a41a6dc0 con 0x7ff6a4109e20 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.469+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a4104e10 con 0x7ff6a4109e20 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.471+0000 7ff6937fe640 1 -- 172.21.15.29:0/315311879 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7ff69c003970 con 0x7ff6a4109e20 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.472+0000 7ff6937fe640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff68c0b6540 0x7ff68c0b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.472+0000 7ff6a259c640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff68c0b6540 0x7ff68c0b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.472+0000 7ff6937fe640 1 -- 172.21.15.29:0/315311879 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7ff69c0e6c90 con 0x7ff6a4109e20 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.472+0000 7ff6a259c640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff68c0b6540 0x7ff68c0b8a50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff6940057c0 tx=0x7ff6940058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.475+0000 7ff6937fe640 1 -- 172.21.15.29:0/315311879 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff69c0ad920 con 0x7ff6a4109e20 2024-09-21T02:35:42.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.705+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6a410bbf0 con 0x7ff6a4109e20 2024-09-21T02:35:42.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.706+0000 7ff6937fe640 1 -- 172.21.15.29:0/315311879 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7ff69c0b1d00 con 0x7ff6a4109e20 2024-09-21T02:35:42.708 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:35:42.708 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"btime":"2024-09-21T02:27:47:924894+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-21T02:35:42.708 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 1 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff68c0b6540 msgr2=0x7ff68c0b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff68c0b6540 0x7ff68c0b8a50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff6940057c0 tx=0x7ff6940058d0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4109e20 msgr2=0x7ff6a419fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4109e20 0x7ff6a419fc40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009fd0 tx=0x7ff69c00b260 comp rx=0 tx=0).stop 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 shutdown_connections 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff68c0b6540 0x7ff68c0b8a50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a4109e20 0x7ff6a419fc40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6a41094d0 0x7ff6a419a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 --2- 172.21.15.29:0/315311879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6a4103b90 0x7ff6a419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 >> 172.21.15.29:0/315311879 conn(0x7ff6a40ff600 msgr2=0x7ff6a4101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:42.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 shutdown_connections 2024-09-21T02:35:42.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:42.708+0000 7ff6a8b2f640 1 -- 172.21.15.29:0/315311879 wait complete. 2024-09-21T02:35:42.785 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:42 smithi190 ceph-mon[33871]: Deploying daemon mds.l on smithi190 2024-09-21T02:35:43.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:42 smithi066 ceph-mon[33436]: Deploying daemon mds.l on smithi190 2024-09-21T02:35:43.124 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:42 smithi190 systemd[1]: Starting Ceph mds.l for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:35:43.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:42 smithi029 ceph-mon[31006]: Deploying daemon mds.l on smithi190 2024-09-21T02:35:43.429 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:43 smithi190 podman[61861]: 2024-09-21 02:35:43.079249903 +0000 UTC m=+0.015610487 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:35:43.429 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:43 smithi190 podman[61861]: 2024-09-21 02:35:43.181167632 +0000 UTC m=+0.117528258 container create 7019d1c01fae166566e17ca56bbfc5ca988a59e3ec7d375e9664618cd9c18fa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T02:35:43.429 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:43 smithi190 podman[61861]: 2024-09-21 02:35:43.392178066 +0000 UTC m=+0.328538650 container init 7019d1c01fae166566e17ca56bbfc5ca988a59e3ec7d375e9664618cd9c18fa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T02:35:43.429 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:43 smithi190 podman[61861]: 2024-09-21 02:35:43.395151578 +0000 UTC m=+0.331512163 container start 7019d1c01fae166566e17ca56bbfc5ca988a59e3ec7d375e9664618cd9c18fa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T02:35:43.429 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mds[61881]: -- 172.21.15.190:0/3611596476 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b24e911860 con 0x55b252ab9180 2024-09-21T02:35:43.874 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l[61877]: starting mds.l at 2024-09-21T02:35:43.874 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:43 smithi190 bash[61861]: 7019d1c01fae166566e17ca56bbfc5ca988a59e3ec7d375e9664618cd9c18fa8 2024-09-21T02:35:43.874 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 02:35:43 smithi190 systemd[1]: Started Ceph mds.l for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:35:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/315311879' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6832/3478087331,v1:172.21.15.190:6833/3478087331] up:boot 2024-09-21T02:35:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: fsmap 1 up:standby 2024-09-21T02:35:43.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:35:43.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:43.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:43.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:43.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:43.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:43.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/315311879' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6832/3478087331,v1:172.21.15.190:6833/3478087331] up:boot 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: fsmap 1 up:standby 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:43.956 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/315311879' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6832/3478087331,v1:172.21.15.190:6833/3478087331] up:boot 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: fsmap 1 up:standby 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:44.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:45.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:44 smithi066 ceph-mon[33436]: Deploying daemon mds.k on smithi066 2024-09-21T02:35:45.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:44 smithi190 ceph-mon[33871]: Deploying daemon mds.k on smithi066 2024-09-21T02:35:45.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:44 smithi029 ceph-mon[31006]: Deploying daemon mds.k on smithi066 2024-09-21T02:35:45.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:35:45.945 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:45 smithi066 ceph-mon[33436]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:45.945 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:45 smithi066 systemd[1]: Starting Ceph mds.k for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:35:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:46.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:45 smithi029 ceph-mon[31006]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:46.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:45 smithi190 ceph-mon[33871]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:46.214 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:45 smithi066 podman[60683]: 2024-09-21 02:35:45.847665044 +0000 UTC m=+0.017194771 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:35:46.214 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:45 smithi066 podman[60683]: 2024-09-21 02:35:45.968633303 +0000 UTC m=+0.138163057 container create 328ec19fddf2132b2b468281bc5e45a2576cc433e5088acbe7e767a09fed6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:35:46.214 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:46 smithi066 podman[60683]: 2024-09-21 02:35:46.213095552 +0000 UTC m=+0.382625271 container init 328ec19fddf2132b2b468281bc5e45a2576cc433e5088acbe7e767a09fed6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T02:35:46.509 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:46 smithi066 podman[60683]: 2024-09-21 02:35:46.216481191 +0000 UTC m=+0.386010903 container start 328ec19fddf2132b2b468281bc5e45a2576cc433e5088acbe7e767a09fed6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T02:35:46.510 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:46 smithi066 ceph-mds[60702]: -- 172.21.15.66:0/96029789 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c688cd860 con 0x564c6ca75600 2024-09-21T02:35:46.510 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k[60698]: starting mds.k at 2024-09-21T02:35:46.510 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:46 smithi066 bash[60683]: 328ec19fddf2132b2b468281bc5e45a2576cc433e5088acbe7e767a09fed6948 2024-09-21T02:35:46.510 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 02:35:46 smithi066 systemd[1]: Started Ceph mds.k for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:35:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 -- 172.21.15.29:0/786716094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 msgr2=0x7fd0ac102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/786716094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 0x7fd0ac102460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009900 tx=0x7fd09c02a410 comp rx=0 tx=0).stop 2024-09-21T02:35:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 -- 172.21.15.29:0/786716094 shutdown_connections 2024-09-21T02:35:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/786716094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0ac106bf0 0x7fd0ac110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/786716094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0ac1062a0 0x7fd0ac1066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/786716094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 0x7fd0ac102460 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 -- 172.21.15.29:0/786716094 >> 172.21.15.29:0/786716094 conn(0x7fd0ac077c60 msgr2=0x7fd0ac0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 -- 172.21.15.29:0/786716094 shutdown_connections 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 -- 172.21.15.29:0/786716094 wait complete. 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.504+0000 7fd0b21ca640 1 Processor -- start 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0b21ca640 1 -- start start 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0b21ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 0x7fd0ac102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0b21ca640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0ac1062a0 0x7fd0ac103530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0b21ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0ac106bf0 0x7fd0ac103a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0b21ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac113950 con 0x7fd0ac102050 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0b21ca640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac1137d0 con 0x7fd0ac1062a0 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0b21ca640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac113ad0 con 0x7fd0ac106bf0 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 0x7fd0ac102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 0x7fd0ac102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47666/0 (socket says 172.21.15.29:47666) 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0ab7fe640 1 -- 172.21.15.29:0/2944692941 learned_addr learned my addr 172.21.15.29:0/2944692941 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0abfff640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0ac106bf0 0x7fd0ac103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0aaffd640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0ac1062a0 0x7fd0ac103530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0ab7fe640 1 -- 172.21.15.29:0/2944692941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0ac106bf0 msgr2=0x7fd0ac103a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0ab7fe640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0ac106bf0 0x7fd0ac103a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0ab7fe640 1 -- 172.21.15.29:0/2944692941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0ac1062a0 msgr2=0x7fd0ac103530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0ab7fe640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0ac1062a0 0x7fd0ac103530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0ab7fe640 1 -- 172.21.15.29:0/2944692941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac1a8700 con 0x7fd0ac102050 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0abfff640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0ac106bf0 0x7fd0ac103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.505+0000 7fd0aaffd640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0ac1062a0 0x7fd0ac103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.506+0000 7fd0ab7fe640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 0x7fd0ac102ff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd09c005940 tx=0x7fd09c032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.506+0000 7fd0a8ff9640 1 -- 172.21.15.29:0/2944692941 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c03f030 con 0x7fd0ac102050 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.506+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ac1a8990 con 0x7fd0ac102050 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.506+0000 7fd0a8ff9640 1 -- 172.21.15.29:0/2944692941 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd09c004450 con 0x7fd0ac102050 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.506+0000 7fd0a8ff9640 1 -- 172.21.15.29:0/2944692941 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c035070 con 0x7fd0ac102050 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.506+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0ac1a8ec0 con 0x7fd0ac102050 2024-09-21T02:35:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.507+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0ac102460 con 0x7fd0ac102050 2024-09-21T02:35:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.509+0000 7fd0a8ff9640 1 -- 172.21.15.29:0/2944692941 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fd09c042030 con 0x7fd0ac102050 2024-09-21T02:35:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.510+0000 7fd0a8ff9640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0940b6510 0x7fd0940b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.510+0000 7fd0aaffd640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0940b6510 0x7fd0940b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.510+0000 7fd0a8ff9640 1 -- 172.21.15.29:0/2944692941 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7fd09c107820 con 0x7fd0ac102050 2024-09-21T02:35:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.510+0000 7fd0aaffd640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0940b6510 0x7fd0940b8a20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac109220 tx=0x7fd0a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.517+0000 7fd0a8ff9640 1 -- 172.21.15.29:0/2944692941 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd09c0ce550 con 0x7fd0ac102050 2024-09-21T02:35:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.759+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0ac1042f0 con 0x7fd0ac102050 2024-09-21T02:35:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.760+0000 7fd0a8ff9640 1 -- 172.21.15.29:0/2944692941 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2263 (secure 0 0 0) 0x7fd0ac1042f0 con 0x7fd0ac102050 2024-09-21T02:35:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:35:46.762 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"btime":"2024-09-21T02:35:46:649127+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T02:35:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2024-09-21T02:35:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0940b6510 msgr2=0x7fd0940b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0940b6510 0x7fd0940b8a20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac109220 tx=0x7fd0a0005f90 comp rx=0 tx=0).stop 2024-09-21T02:35:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 msgr2=0x7fd0ac102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 0x7fd0ac102ff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd09c005940 tx=0x7fd09c032f90 comp rx=0 tx=0).stop 2024-09-21T02:35:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 shutdown_connections 2024-09-21T02:35:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0940b6510 0x7fd0940b8a20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0ac106bf0 0x7fd0ac103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0ac1062a0 0x7fd0ac103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 --2- 172.21.15.29:0/2944692941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0ac102050 0x7fd0ac102ff0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 >> 172.21.15.29:0/2944692941 conn(0x7fd0ac077c60 msgr2=0x7fd0ac1075b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 shutdown_connections 2024-09-21T02:35:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:46.762+0000 7fd0b21ca640 1 -- 172.21.15.29:0/2944692941 wait complete. 2024-09-21T02:35:47.672 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: Deploying daemon mds.j on smithi029 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6832/2291646688,v1:172.21.15.66:6833/2291646688] up:boot 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: fsmap 2 up:standby 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:35:47.673 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:47 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2944692941' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: Deploying daemon mds.j on smithi029 2024-09-21T02:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6832/2291646688,v1:172.21.15.66:6833/2291646688] up:boot 2024-09-21T02:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: fsmap 2 up:standby 2024-09-21T02:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:47 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2944692941' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: Deploying daemon mds.j on smithi029 2024-09-21T02:35:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6832/2291646688,v1:172.21.15.66:6833/2291646688] up:boot 2024-09-21T02:35:47.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: fsmap 2 up:standby 2024-09-21T02:35:47.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:35:47.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:47 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2944692941' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:48.882 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:48 smithi029 systemd[1]: Starting Ceph mds.j for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:35:49.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:49 smithi029 ceph-mon[31006]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:49.582 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:49 smithi029 podman[70702]: 2024-09-21 02:35:49.16238586 +0000 UTC m=+0.016465930 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:35:49.582 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:49 smithi029 podman[70702]: 2024-09-21 02:35:49.282931636 +0000 UTC m=+0.137011701 container create a110a80bdb156d1c8ba01ea86a242860a31a592abf9dab37b0d26c47e294df07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:35:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:49 smithi066 ceph-mon[33436]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:49.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:49 smithi190 ceph-mon[33871]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:49.882 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:49 smithi029 podman[70702]: 2024-09-21 02:35:49.581282562 +0000 UTC m=+0.435362634 container init a110a80bdb156d1c8ba01ea86a242860a31a592abf9dab37b0d26c47e294df07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS) 2024-09-21T02:35:49.882 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:49 smithi029 podman[70702]: 2024-09-21 02:35:49.584378506 +0000 UTC m=+0.438458566 container start a110a80bdb156d1c8ba01ea86a242860a31a592abf9dab37b0d26c47e294df07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T02:35:49.882 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:49 smithi029 ceph-mds[70721]: -- 172.21.15.29:0/2784818030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564059541860 con 0x56405d6e9a80 2024-09-21T02:35:49.882 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:49 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j[70717]: starting mds.j at 2024-09-21T02:35:49.882 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:49 smithi029 bash[70702]: a110a80bdb156d1c8ba01ea86a242860a31a592abf9dab37b0d26c47e294df07 2024-09-21T02:35:49.882 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 02:35:49 smithi029 systemd[1]: Started Ceph mds.j for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:35:51.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: Deploying daemon mds.d on smithi029 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.29:6834/502969019,v1:172.21.15.29:6835/502969019] up:boot 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: fsmap 3 up:standby 2024-09-21T02:35:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T02:35:51.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:35:51.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:51.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:51.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:51.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: Deploying daemon mds.d on smithi029 2024-09-21T02:35:51.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:51.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.29:6834/502969019,v1:172.21.15.29:6835/502969019] up:boot 2024-09-21T02:35:51.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: fsmap 3 up:standby 2024-09-21T02:35:51.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T02:35:51.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: Deploying daemon mds.d on smithi029 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.29:6834/502969019,v1:172.21.15.29:6835/502969019] up:boot 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: fsmap 3 up:standby 2024-09-21T02:35:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T02:35:51.311 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:52.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 -- 172.21.15.29:0/466421109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8284109df0 msgr2=0x7f8284110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:52.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 --2- 172.21.15.29:0/466421109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8284109df0 0x7f8284110620 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f827c009ca0 tx=0x7f827c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 -- 172.21.15.29:0/466421109 shutdown_connections 2024-09-21T02:35:52.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 --2- 172.21.15.29:0/466421109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8284109df0 0x7f8284110620 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 --2- 172.21.15.29:0/466421109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82841094a0 0x7f82841098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 --2- 172.21.15.29:0/466421109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284103b60 0x7f8284103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 -- 172.21.15.29:0/466421109 >> 172.21.15.29:0/466421109 conn(0x7f828407b850 msgr2=0x7f828407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 -- 172.21.15.29:0/466421109 shutdown_connections 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.974+0000 7f8289cb2640 1 -- 172.21.15.29:0/466421109 wait complete. 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8289cb2640 1 Processor -- start 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8289cb2640 1 -- start start 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8289cb2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284103b60 0x7f828419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8289cb2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82841094a0 0x7f828419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8289cb2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8284109df0 0x7f828419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8289cb2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82841139b0 con 0x7f82841094a0 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8289cb2640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8284113830 con 0x7f8284109df0 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8289cb2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8284113b30 con 0x7f8284103b60 2024-09-21T02:35:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8282ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82841094a0 0x7f828419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8282ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82841094a0 0x7f828419a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47712/0 (socket says 172.21.15.29:47712) 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f8282ffd640 1 -- 172.21.15.29:0/1168363851 learned_addr learned my addr 172.21.15.29:0/1168363851 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.975+0000 7f82837fe640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284103b60 0x7f828419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8283fff640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8284109df0 0x7f828419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8282ffd640 1 -- 172.21.15.29:0/1168363851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284103b60 msgr2=0x7f828419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8282ffd640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284103b60 0x7f828419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8282ffd640 1 -- 172.21.15.29:0/1168363851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8284109df0 msgr2=0x7f828419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8282ffd640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8284109df0 0x7f828419fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8282ffd640 1 -- 172.21.15.29:0/1168363851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82841a0370 con 0x7f82841094a0 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8282ffd640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82841094a0 0x7f828419a9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f827800e9f0 tx=0x7f827800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8280ff9640 1 -- 172.21.15.29:0/1168363851 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827800ccc0 con 0x7f82841094a0 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82841a0660 con 0x7f82841094a0 2024-09-21T02:35:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8280ff9640 1 -- 172.21.15.29:0/1168363851 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82780108c0 con 0x7f82841094a0 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8280ff9640 1 -- 172.21.15.29:0/1168363851 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8278018660 con 0x7f82841094a0 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.976+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82841a6dc0 con 0x7f82841094a0 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.977+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8284103f70 con 0x7f82841094a0 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.979+0000 7f8280ff9640 1 -- 172.21.15.29:0/1168363851 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f8278010430 con 0x7f82841094a0 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.980+0000 7f8280ff9640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f826c0b6510 0x7f826c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.980+0000 7f82837fe640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f826c0b6510 0x7f826c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.980+0000 7f8280ff9640 1 -- 172.21.15.29:0/1168363851 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7f82780e76f0 con 0x7f82841094a0 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.980+0000 7f82837fe640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f826c0b6510 0x7f826c0b8a20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f82740095f0 tx=0x7f8274005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:51.983+0000 7f8280ff9640 1 -- 172.21.15.29:0/1168363851 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f82780b2760 con 0x7f82841094a0 2024-09-21T02:35:52.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.216+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f828410bc10 con 0x7f82841094a0 2024-09-21T02:35:52.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.216+0000 7f8280ff9640 1 -- 172.21.15.29:0/1168363851 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3085 (secure 0 0 0) 0x7f82780b76a0 con 0x7f82841094a0 2024-09-21T02:35:52.218 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:35:52.218 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":4,"btime":"2024-09-21T02:35:50:393555+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T02:35:52.219 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 4 2024-09-21T02:35:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f826c0b6510 msgr2=0x7f826c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f826c0b6510 0x7f826c0b8a20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f82740095f0 tx=0x7f8274005f90 comp rx=0 tx=0).stop 2024-09-21T02:35:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82841094a0 msgr2=0x7f828419a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82841094a0 0x7f828419a9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f827800e9f0 tx=0x7f827800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 shutdown_connections 2024-09-21T02:35:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f826c0b6510 0x7f826c0b8a20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8284109df0 0x7f828419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82841094a0 0x7f828419a9a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 --2- 172.21.15.29:0/1168363851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284103b60 0x7f828419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 >> 172.21.15.29:0/1168363851 conn(0x7f828407b850 msgr2=0x7f8284107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 shutdown_connections 2024-09-21T02:35:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:52.219+0000 7f8289cb2640 1 -- 172.21.15.29:0/1168363851 wait complete. 2024-09-21T02:35:52.951 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 02:35:52 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-d[71096]: starting mds.d at 2024-09-21T02:35:53.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:52 smithi066 ceph-mon[33436]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:53.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:52 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1168363851' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:53.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:52 smithi029 ceph-mon[31006]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:53.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:52 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1168363851' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:53.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:52 smithi190 ceph-mon[33871]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:53.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:52 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1168363851' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:53.632 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 02:35:53 smithi029 bash[71065]: a77e9981ac91fb72f566b34f78f43d8c402adf674bc8ce058835a887f5d4387f 2024-09-21T02:35:53.632 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 02:35:53 smithi029 systemd[1]: Started Ceph mds.d for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:35:54.196 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.29:6836/2096055950,v1:172.21.15.29:6837/2096055950] up:boot 2024-09-21T02:35:54.196 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: fsmap 4 up:standby 2024-09-21T02:35:54.196 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T02:35:54.196 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.196 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.196 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.196 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:54.197 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:54.197 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:54.197 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:53 smithi066 ceph-mon[33436]: Deploying daemon mds.e on smithi066 2024-09-21T02:35:54.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.29:6836/2096055950,v1:172.21.15.29:6837/2096055950] up:boot 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: fsmap 4 up:standby 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:54.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:53 smithi190 ceph-mon[33871]: Deploying daemon mds.e on smithi066 2024-09-21T02:35:54.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.29:6836/2096055950,v1:172.21.15.29:6837/2096055950] up:boot 2024-09-21T02:35:54.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: fsmap 4 up:standby 2024-09-21T02:35:54.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T02:35:54.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:54.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:54.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:54.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:54.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:53 smithi029 ceph-mon[31006]: Deploying daemon mds.e on smithi066 2024-09-21T02:35:55.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:35:55.593 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:55.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:55 smithi029 ceph-mon[31006]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:55.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:55 smithi066 ceph-mon[33436]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:55.760 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:55 smithi066 systemd[1]: Starting Ceph mds.e for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:35:55.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:55 smithi190 ceph-mon[33871]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:56.069 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:55 smithi066 podman[60965]: 2024-09-21 02:35:55.67902203 +0000 UTC m=+0.016342837 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:35:56.069 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:55 smithi066 podman[60965]: 2024-09-21 02:35:55.7998864 +0000 UTC m=+0.137207233 container create 00e9dcb7646e829478b2fe8fe4ecbd822835e4a9e7e3042526060f1f908026b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:35:56.069 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:56 smithi066 podman[60965]: 2024-09-21 02:35:56.069074877 +0000 UTC m=+0.406395678 container init 00e9dcb7646e829478b2fe8fe4ecbd822835e4a9e7e3042526060f1f908026b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731) 2024-09-21T02:35:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 -- 172.21.15.29:0/913187020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c1094a0 msgr2=0x7efd5c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 --2- 172.21.15.29:0/913187020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c1094a0 0x7efd5c1098b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efd50009900 tx=0x7efd5002a410 comp rx=0 tx=0).stop 2024-09-21T02:35:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 -- 172.21.15.29:0/913187020 shutdown_connections 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 --2- 172.21.15.29:0/913187020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c109df0 0x7efd5c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 --2- 172.21.15.29:0/913187020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c1094a0 0x7efd5c1098b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 --2- 172.21.15.29:0/913187020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c103b60 0x7efd5c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 -- 172.21.15.29:0/913187020 >> 172.21.15.29:0/913187020 conn(0x7efd5c07b850 msgr2=0x7efd5c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 -- 172.21.15.29:0/913187020 shutdown_connections 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.101+0000 7efd61007640 1 -- 172.21.15.29:0/913187020 wait complete. 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd61007640 1 Processor -- start 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd61007640 1 -- start start 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd61007640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c103b60 0x7efd5c19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd61007640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c1094a0 0x7efd5c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd61007640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c109df0 0x7efd5c1a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd61007640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd5c113960 con 0x7efd5c103b60 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd61007640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7efd5c1137e0 con 0x7efd5c1094a0 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd61007640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efd5c113ae0 con 0x7efd5c109df0 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.102+0000 7efd5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c103b60 0x7efd5c19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c103b60 0x7efd5c19c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57350/0 (socket says 172.21.15.29:57350) 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5ad9d640 1 -- 172.21.15.29:0/294940611 learned_addr learned my addr 172.21.15.29:0/294940611 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5a59c640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c1094a0 0x7efd5c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5b59e640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c109df0 0x7efd5c1a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5ad9d640 1 -- 172.21.15.29:0/294940611 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c109df0 msgr2=0x7efd5c1a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5ad9d640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c109df0 0x7efd5c1a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5ad9d640 1 -- 172.21.15.29:0/294940611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c1094a0 msgr2=0x7efd5c19cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5ad9d640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c1094a0 0x7efd5c19cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5ad9d640 1 -- 172.21.15.29:0/294940611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5c1a2520 con 0x7efd5c103b60 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5a59c640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c1094a0 0x7efd5c19cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd5ad9d640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c103b60 0x7efd5c19c5b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efd4c00e9f0 tx=0x7efd4c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd43fff640 1 -- 172.21.15.29:0/294940611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c00cd80 con 0x7efd5c103b60 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd43fff640 1 -- 172.21.15.29:0/294940611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd4c0108c0 con 0x7efd5c103b60 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd43fff640 1 -- 172.21.15.29:0/294940611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c0187c0 con 0x7efd5c103b60 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd5c1a2810 con 0x7efd5c103b60 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.103+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd5c1a8f70 con 0x7efd5c103b60 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.104+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd5c103f70 con 0x7efd5c103b60 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.106+0000 7efd43fff640 1 -- 172.21.15.29:0/294940611 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7efd4c010430 con 0x7efd5c103b60 2024-09-21T02:35:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.107+0000 7efd43fff640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd440b6590 0x7efd440b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.107+0000 7efd5a59c640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd440b6590 0x7efd440b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.107+0000 7efd43fff640 1 -- 172.21.15.29:0/294940611 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7efd4c0e75f0 con 0x7efd5c103b60 2024-09-21T02:35:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.108+0000 7efd5a59c640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd440b6590 0x7efd440b8aa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd50005940 tx=0x7efd500058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.110+0000 7efd43fff640 1 -- 172.21.15.29:0/294940611 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7efd4c0b2700 con 0x7efd5c103b60 2024-09-21T02:35:56.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.339+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd5c1a92a0 con 0x7efd5c103b60 2024-09-21T02:35:56.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.339+0000 7efd43fff640 1 -- 172.21.15.29:0/294940611 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3910 (secure 0 0 0) 0x7efd4c0b7640 con 0x7efd5c103b60 2024-09-21T02:35:56.340 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:35:56.341 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"btime":"2024-09-21T02:35:52:969324+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T02:35:56.341 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 5 2024-09-21T02:35:56.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd440b6590 msgr2=0x7efd440b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:56.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd440b6590 0x7efd440b8aa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd50005940 tx=0x7efd500058d0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c103b60 msgr2=0x7efd5c19c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:56.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c103b60 0x7efd5c19c5b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efd4c00e9f0 tx=0x7efd4c00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 shutdown_connections 2024-09-21T02:35:56.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd440b6590 0x7efd440b8aa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c109df0 0x7efd5c1a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c1094a0 0x7efd5c19cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 --2- 172.21.15.29:0/294940611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c103b60 0x7efd5c19c5b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:56.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 >> 172.21.15.29:0/294940611 conn(0x7efd5c07b850 msgr2=0x7efd5c1052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:56.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 shutdown_connections 2024-09-21T02:35:56.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:56.343+0000 7efd61007640 1 -- 172.21.15.29:0/294940611 wait complete. 2024-09-21T02:35:56.509 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:56 smithi066 podman[60965]: 2024-09-21 02:35:56.072169718 +0000 UTC m=+0.409490506 container start 00e9dcb7646e829478b2fe8fe4ecbd822835e4a9e7e3042526060f1f908026b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T02:35:56.510 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:56 smithi066 ceph-mds[60984]: -- 172.21.15.66:0/3575093147 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f30d751860 con 0x55f3118f9600 2024-09-21T02:35:56.510 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:56 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: starting mds.e at 2024-09-21T02:35:56.510 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:56 smithi066 bash[60965]: 00e9dcb7646e829478b2fe8fe4ecbd822835e4a9e7e3042526060f1f908026b6 2024-09-21T02:35:56.510 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 02:35:56 smithi066 systemd[1]: Started Ceph mds.e for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:35:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:57.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: Deploying daemon mds.f on smithi190 2024-09-21T02:35:57.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/294940611' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:57.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6834/4067532455,v1:172.21.15.66:6835/4067532455] up:boot 2024-09-21T02:35:57.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: fsmap 5 up:standby 2024-09-21T02:35:57.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:35:57.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: Deploying daemon mds.f on smithi190 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/294940611' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6834/4067532455,v1:172.21.15.66:6835/4067532455] up:boot 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: fsmap 5 up:standby 2024-09-21T02:35:57.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:35:57.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: Deploying daemon mds.f on smithi190 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/294940611' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6834/4067532455,v1:172.21.15.66:6835/4067532455] up:boot 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: fsmap 5 up:standby 2024-09-21T02:35:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:35:58.344 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 systemd[1]: Starting Ceph mds.f for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:35:58.605 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 podman[62137]: 2024-09-21 02:35:58.245740092 +0000 UTC m=+0.015700369 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:35:58.605 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 podman[62137]: 2024-09-21 02:35:58.368879587 +0000 UTC m=+0.138839867 container create a231aab13ca903f0db603b1da8a98321d76231ec2218709ae2a31287da5b0e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T02:35:58.605 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 podman[62137]: 2024-09-21 02:35:58.604929797 +0000 UTC m=+0.374890075 container init a231aab13ca903f0db603b1da8a98321d76231ec2218709ae2a31287da5b0e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:35:58.873 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 podman[62137]: 2024-09-21 02:35:58.607729203 +0000 UTC m=+0.377689481 container start a231aab13ca903f0db603b1da8a98321d76231ec2218709ae2a31287da5b0e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T02:35:58.874 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 ceph-mds[62155]: -- 172.21.15.190:0/1147129789 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563ec47d1860 con 0x563ec8979180 2024-09-21T02:35:58.874 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f[62151]: starting mds.f at 2024-09-21T02:35:58.874 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 bash[62137]: a231aab13ca903f0db603b1da8a98321d76231ec2218709ae2a31287da5b0e06 2024-09-21T02:35:58.874 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 02:35:58 smithi190 systemd[1]: Started Ceph mds.f for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:35:59.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:35:59.400 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:35:59.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:59 smithi190 ceph-mon[33871]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:59.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:59.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:59.501 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:35:59 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:59.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:59 smithi066 ceph-mon[33436]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:59.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:59.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:35:59 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:59 smithi029 ceph-mon[31006]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:35:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:35:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:35:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:35:59.521 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:35:59 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:35:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 -- 172.21.15.29:0/2008618001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd164108cf0 msgr2=0x7fd16410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 --2- 172.21.15.29:0/2008618001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd164108cf0 0x7fd16410f520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009ca0 tx=0x7fd15c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 -- 172.21.15.29:0/2008618001 shutdown_connections 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 --2- 172.21.15.29:0/2008618001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd164108cf0 0x7fd16410f520 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 --2- 172.21.15.29:0/2008618001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1641083a0 0x7fd1641087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 --2- 172.21.15.29:0/2008618001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd164102a60 0x7fd164102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 -- 172.21.15.29:0/2008618001 >> 172.21.15.29:0/2008618001 conn(0x7fd164079730 msgr2=0x7fd164079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 -- 172.21.15.29:0/2008618001 shutdown_connections 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 -- 172.21.15.29:0/2008618001 wait complete. 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.978+0000 7fd16c264640 1 Processor -- start 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16c264640 1 -- start start 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16c264640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd164102a60 0x7fd1640ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16c264640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1641083a0 0x7fd1640ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16c264640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd164108cf0 0x7fd1641000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16c264640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd164112860 con 0x7fd1641083a0 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16c264640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd1641126e0 con 0x7fd164108cf0 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16c264640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd1641129e0 con 0x7fd164102a60 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd1697ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1641083a0 0x7fd1640ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd1697ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1641083a0 0x7fd1640ffbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57368/0 (socket says 172.21.15.29:57368) 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd1697ff640 1 -- 172.21.15.29:0/3270856649 learned_addr learned my addr 172.21.15.29:0/3270856649 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16a000640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd164102a60 0x7fd1640ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16a801640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd164108cf0 0x7fd1641000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd1697ff640 1 -- 172.21.15.29:0/3270856649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd164102a60 msgr2=0x7fd1640ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd1697ff640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd164102a60 0x7fd1640ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd1697ff640 1 -- 172.21.15.29:0/3270856649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd164108cf0 msgr2=0x7fd1641000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd1697ff640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd164108cf0 0x7fd1641000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd1697ff640 1 -- 172.21.15.29:0/3270856649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1641a8660 con 0x7fd1641083a0 2024-09-21T02:35:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.979+0000 7fd16a801640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd164108cf0 0x7fd1641000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.980+0000 7fd1697ff640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1641083a0 0x7fd1640ffbb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd15800e9c0 tx=0x7fd15800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.980+0000 7fd156ffd640 1 -- 172.21.15.29:0/3270856649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15800cd80 con 0x7fd1641083a0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.980+0000 7fd156ffd640 1 -- 172.21.15.29:0/3270856649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1580054b0 con 0x7fd1641083a0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.980+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1641a8950 con 0x7fd1641083a0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.980+0000 7fd156ffd640 1 -- 172.21.15.29:0/3270856649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd158010430 con 0x7fd1641083a0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.980+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1641a8e90 con 0x7fd1641083a0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.981+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd164102e70 con 0x7fd1641083a0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.983+0000 7fd156ffd640 1 -- 172.21.15.29:0/3270856649 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fd158004f20 con 0x7fd1641083a0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.983+0000 7fd156ffd640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1500b6510 0x7fd1500b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.984+0000 7fd16a000640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1500b6510 0x7fd1500b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.984+0000 7fd156ffd640 1 -- 172.21.15.29:0/3270856649 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7fd1580e7430 con 0x7fd1641083a0 2024-09-21T02:35:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.984+0000 7fd16a000640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1500b6510 0x7fd1500b8a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd1641010c0 tx=0x7fd160005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:35:59.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:35:59.988+0000 7fd156ffd640 1 -- 172.21.15.29:0/3270856649 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd1580ae160 con 0x7fd1641083a0 2024-09-21T02:36:00.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.217+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd16402e640 con 0x7fd1641083a0 2024-09-21T02:36:00.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.217+0000 7fd156ffd640 1 -- 172.21.15.29:0/3270856649 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5560 (secure 0 0 0) 0x7fd1580d10e0 con 0x7fd1641083a0 2024-09-21T02:36:00.219 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:00.219 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":7,"btime":"2024-09-21T02:35:59:229680+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24587,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24613,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-09-21T02:36:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 7 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.219+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1500b6510 msgr2=0x7fd1500b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.219+0000 7fd16c264640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1500b6510 0x7fd1500b8a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd1641010c0 tx=0x7fd160005f90 comp rx=0 tx=0).stop 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.219+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1641083a0 msgr2=0x7fd1640ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.219+0000 7fd16c264640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1641083a0 0x7fd1640ffbb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd15800e9c0 tx=0x7fd15800ee80 comp rx=0 tx=0).stop 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.220+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 shutdown_connections 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.220+0000 7fd16c264640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1500b6510 0x7fd1500b8a20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.220+0000 7fd16c264640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd164108cf0 0x7fd1641000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.220+0000 7fd16c264640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1641083a0 0x7fd1640ffbb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.220+0000 7fd16c264640 1 --2- 172.21.15.29:0/3270856649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd164102a60 0x7fd1640ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.220+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 >> 172.21.15.29:0/3270856649 conn(0x7fd164079730 msgr2=0x7fd16406b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.220+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 shutdown_connections 2024-09-21T02:36:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:00.220+0000 7fd16c264640 1 -- 172.21.15.29:0/3270856649 wait complete. 2024-09-21T02:36:00.457 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:00 smithi190 ceph-mon[33871]: Deploying daemon mds.c on smithi190 2024-09-21T02:36:00.457 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:00 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] up:boot 2024-09-21T02:36:00.457 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:00 smithi190 ceph-mon[33871]: fsmap 6 up:standby 2024-09-21T02:36:00.457 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:00 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:36:00.457 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:00 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3270856649' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:00.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:00 smithi066 ceph-mon[33436]: Deploying daemon mds.c on smithi190 2024-09-21T02:36:00.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:00 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] up:boot 2024-09-21T02:36:00.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:00 smithi066 ceph-mon[33436]: fsmap 6 up:standby 2024-09-21T02:36:00.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:00 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:36:00.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:00 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3270856649' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:00 smithi029 ceph-mon[31006]: Deploying daemon mds.c on smithi190 2024-09-21T02:36:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:00 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] up:boot 2024-09-21T02:36:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:00 smithi029 ceph-mon[31006]: fsmap 6 up:standby 2024-09-21T02:36:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:00 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:36:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:00 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3270856649' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:00.848 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:00 smithi190 systemd[1]: Starting Ceph mds.c for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:36:01.108 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:00 smithi190 podman[62408]: 2024-09-21 02:36:00.750910166 +0000 UTC m=+0.015885632 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:36:01.109 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:00 smithi190 podman[62408]: 2024-09-21 02:36:00.855646571 +0000 UTC m=+0.120622036 container create 783b37f837dc826a97638b02cba5ba98f22419457bb6c6a879d456c4fcab42eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:36:01.109 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:01 smithi190 podman[62408]: 2024-09-21 02:36:01.108436277 +0000 UTC m=+0.373411747 container init 783b37f837dc826a97638b02cba5ba98f22419457bb6c6a879d456c4fcab42eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T02:36:01.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:01 smithi190 ceph-mon[33871]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:01.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:01 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:01.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:01 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:01.374 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:01 smithi190 podman[62408]: 2024-09-21 02:36:01.111295943 +0000 UTC m=+0.376271408 container start 783b37f837dc826a97638b02cba5ba98f22419457bb6c6a879d456c4fcab42eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T02:36:01.374 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:01 smithi190 ceph-mds[62427]: -- 172.21.15.190:0/1429881460 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f07901860 con 0x561f0baa9a80 2024-09-21T02:36:01.374 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c[62423]: starting mds.c at 2024-09-21T02:36:01.374 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:01 smithi190 bash[62408]: 783b37f837dc826a97638b02cba5ba98f22419457bb6c6a879d456c4fcab42eb 2024-09-21T02:36:01.374 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 02:36:01 smithi190 systemd[1]: Started Ceph mds.c for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:36:01.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:01 smithi029 ceph-mon[31006]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:01 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:01 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:01.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:01 smithi066 ceph-mon[33436]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:01.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:01 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:01.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:01 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:02.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:02 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:02.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:02 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:02.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:02 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] up:boot 2024-09-21T02:36:02.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:02 smithi066 ceph-mon[33436]: fsmap 7 up:standby 2024-09-21T02:36:02.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:02 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:36:02.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:02 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:02.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:02 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:02.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:02 smithi066 ceph-mon[33436]: Deploying daemon mds.h on smithi066 2024-09-21T02:36:02.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:02 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:02.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:02 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:02.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:02 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] up:boot 2024-09-21T02:36:02.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:02 smithi190 ceph-mon[33871]: fsmap 7 up:standby 2024-09-21T02:36:02.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:02 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:36:02.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:02 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:02.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:02 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:02.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:02 smithi190 ceph-mon[33871]: Deploying daemon mds.h on smithi066 2024-09-21T02:36:02.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:02 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:02.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:02 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:02.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:02 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] up:boot 2024-09-21T02:36:02.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:02 smithi029 ceph-mon[31006]: fsmap 7 up:standby 2024-09-21T02:36:02.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:02 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:36:02.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:02 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:02.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:02 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:02.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:02 smithi029 ceph-mon[31006]: Deploying daemon mds.h on smithi066 2024-09-21T02:36:03.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:36:03.371 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:03.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:03 smithi029 ceph-mon[31006]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:03.461 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:03 smithi066 ceph-mon[33436]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:03.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:03 smithi190 ceph-mon[33871]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 -- 172.21.15.29:0/1371616523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870103b60 msgr2=0x7f6870103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 --2- 172.21.15.29:0/1371616523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870103b60 0x7f6870103f70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f686c009900 tx=0x7f686c02a410 comp rx=0 tx=0).stop 2024-09-21T02:36:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 -- 172.21.15.29:0/1371616523 shutdown_connections 2024-09-21T02:36:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 --2- 172.21.15.29:0/1371616523 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6870109df0 0x7f6870110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 --2- 172.21.15.29:0/1371616523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68701094a0 0x7f68701098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 --2- 172.21.15.29:0/1371616523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870103b60 0x7f6870103f70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 -- 172.21.15.29:0/1371616523 >> 172.21.15.29:0/1371616523 conn(0x7f687007b850 msgr2=0x7f687007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 -- 172.21.15.29:0/1371616523 shutdown_connections 2024-09-21T02:36:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.875+0000 7f68779ee640 1 -- 172.21.15.29:0/1371616523 wait complete. 2024-09-21T02:36:03.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f68779ee640 1 Processor -- start 2024-09-21T02:36:03.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f68779ee640 1 -- start start 2024-09-21T02:36:03.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f68779ee640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6870103b60 0x7f687019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f68779ee640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68701094a0 0x7f687019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f68779ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870109df0 0x7f687019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f68779ee640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6870113960 con 0x7f6870109df0 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f68779ee640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f68701137e0 con 0x7f6870103b60 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f68779ee640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6870113ae0 con 0x7f68701094a0 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f6875f8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870109df0 0x7f687019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f6875f8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870109df0 0x7f687019fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57384/0 (socket says 172.21.15.29:57384) 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f6875f8b640 1 -- 172.21.15.29:0/2220156388 learned_addr learned my addr 172.21.15.29:0/2220156388 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f687578a640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6870103b60 0x7f687019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.876+0000 7f6874f89640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68701094a0 0x7f687019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f6875f8b640 1 -- 172.21.15.29:0/2220156388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68701094a0 msgr2=0x7f687019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f6875f8b640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68701094a0 0x7f687019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f6875f8b640 1 -- 172.21.15.29:0/2220156388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6870103b60 msgr2=0x7f687019a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f6875f8b640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6870103b60 0x7f687019a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f6875f8b640 1 -- 172.21.15.29:0/2220156388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68701a0320 con 0x7f6870109df0 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f6875f8b640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870109df0 0x7f687019fbf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f686800eaa0 tx=0x7f686800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f68667fc640 1 -- 172.21.15.29:0/2220156388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686800cdb0 con 0x7f6870109df0 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f68667fc640 1 -- 172.21.15.29:0/2220156388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6868005020 con 0x7f6870109df0 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68701a0610 con 0x7f6870109df0 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f68667fc640 1 -- 172.21.15.29:0/2220156388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68680116d0 con 0x7f6870109df0 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.877+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68701a6d70 con 0x7f6870109df0 2024-09-21T02:36:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.878+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6870103f70 con 0x7f6870109df0 2024-09-21T02:36:03.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.880+0000 7f68667fc640 1 -- 172.21.15.29:0/2220156388 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f68680051d0 con 0x7f6870109df0 2024-09-21T02:36:03.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.881+0000 7f68667fc640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f685c0b6590 0x7f685c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:03.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.881+0000 7f687578a640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f685c0b6590 0x7f685c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:03.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.881+0000 7f68667fc640 1 -- 172.21.15.29:0/2220156388 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7f68680e7250 con 0x7f6870109df0 2024-09-21T02:36:03.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.881+0000 7f687578a640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f685c0b6590 0x7f685c0b8aa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f686c0057c0 tx=0x7f686c0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:03.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:03.884+0000 7f68667fc640 1 -- 172.21.15.29:0/2220156388 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6868016030 con 0x7f6870109df0 2024-09-21T02:36:04.009 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 02:36:03 smithi066 systemd[1]: Started Ceph mds.h for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:36:04.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.112+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f687019b4e0 con 0x7f6870109df0 2024-09-21T02:36:04.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.113+0000 7f68667fc640 1 -- 172.21.15.29:0/2220156388 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6388 (secure 0 0 0) 0x7f687019b4e0 con 0x7f6870109df0 2024-09-21T02:36:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:04.114 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"btime":"2024-09-21T02:36:01:243425+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24587,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24613,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-21T02:36:04.114 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-09-21T02:36:04.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f685c0b6590 msgr2=0x7f685c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:04.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f685c0b6590 0x7f685c0b8aa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f686c0057c0 tx=0x7f686c0058d0 comp rx=0 tx=0).stop 2024-09-21T02:36:04.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870109df0 msgr2=0x7f687019fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:04.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870109df0 0x7f687019fbf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f686800eaa0 tx=0x7f686800ef60 comp rx=0 tx=0).stop 2024-09-21T02:36:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 shutdown_connections 2024-09-21T02:36:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f685c0b6590 0x7f685c0b8aa0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6870109df0 0x7f687019fbf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68701094a0 0x7f687019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 --2- 172.21.15.29:0/2220156388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6870103b60 0x7f687019a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 >> 172.21.15.29:0/2220156388 conn(0x7f687007b850 msgr2=0x7f68701075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 shutdown_connections 2024-09-21T02:36:04.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:04.115+0000 7f68779ee640 1 -- 172.21.15.29:0/2220156388 wait complete. 2024-09-21T02:36:05.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:05.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:05.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:05.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: Deploying daemon mds.i on smithi190 2024-09-21T02:36:05.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:05.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2220156388' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:05.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] up:boot 2024-09-21T02:36:05.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: fsmap 8 up:standby 2024-09-21T02:36:05.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:04 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:36:05.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: Deploying daemon mds.i on smithi190 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2220156388' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] up:boot 2024-09-21T02:36:05.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: fsmap 8 up:standby 2024-09-21T02:36:05.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:04 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:36:05.313 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: Deploying daemon mds.i on smithi190 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2220156388' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] up:boot 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: fsmap 8 up:standby 2024-09-21T02:36:05.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:04 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:36:05.874 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:05 smithi190 systemd[1]: Starting Ceph mds.i for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:36:06.256 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:05 smithi190 podman[62679]: 2024-09-21 02:36:05.899355036 +0000 UTC m=+0.015779776 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:36:06.256 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:06 smithi190 podman[62679]: 2024-09-21 02:36:06.010924903 +0000 UTC m=+0.127349633 container create bc6ea4992bafb0117474ec924701e072345834330b93b20c6d28e27f492b943b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:36:06.256 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:06 smithi190 podman[62679]: 2024-09-21 02:36:06.255486186 +0000 UTC m=+0.371910919 container init bc6ea4992bafb0117474ec924701e072345834330b93b20c6d28e27f492b943b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:36:06.624 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:06 smithi190 podman[62679]: 2024-09-21 02:36:06.258564246 +0000 UTC m=+0.374988978 container start bc6ea4992bafb0117474ec924701e072345834330b93b20c6d28e27f492b943b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux ) 2024-09-21T02:36:06.624 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:06 smithi190 ceph-mds[62699]: -- 172.21.15.190:0/177860032 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557fd787b860 con 0x557fdba23180 2024-09-21T02:36:06.624 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:06 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: starting mds.i at 2024-09-21T02:36:06.624 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:06 smithi190 bash[62679]: bc6ea4992bafb0117474ec924701e072345834330b93b20c6d28e27f492b943b 2024-09-21T02:36:06.624 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:06 smithi190 systemd[1]: Started Ceph mds.i for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:36:06.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: Deploying daemon mds.g on smithi029 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] up:boot 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: fsmap 9 up:standby 2024-09-21T02:36:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:07 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:36:07.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:07.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:07.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:07.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:07.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: Deploying daemon mds.g on smithi029 2024-09-21T02:36:07.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] up:boot 2024-09-21T02:36:07.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: fsmap 9 up:standby 2024-09-21T02:36:07.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:07 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:36:07.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: Deploying daemon mds.g on smithi029 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] up:boot 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: fsmap 9 up:standby 2024-09-21T02:36:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:07 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:36:08.089 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 -- 172.21.15.29:0/2400401970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768104a30 msgr2=0x7fe768104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 --2- 172.21.15.29:0/2400401970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768104a30 0x7fe768104e40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe758009900 tx=0x7fe75802a410 comp rx=0 tx=0).stop 2024-09-21T02:36:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 -- 172.21.15.29:0/2400401970 shutdown_connections 2024-09-21T02:36:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 --2- 172.21.15.29:0/2400401970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe76810de60 0x7fe768110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 --2- 172.21.15.29:0/2400401970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe768105380 0x7fe76810d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 --2- 172.21.15.29:0/2400401970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768104a30 0x7fe768104e40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 -- 172.21.15.29:0/2400401970 >> 172.21.15.29:0/2400401970 conn(0x7fe768079760 msgr2=0x7fe768079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 -- 172.21.15.29:0/2400401970 shutdown_connections 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.835+0000 7fe76dff0640 1 -- 172.21.15.29:0/2400401970 wait complete. 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.836+0000 7fe76dff0640 1 Processor -- start 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.836+0000 7fe76dff0640 1 -- start start 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe76dff0640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe768104a30 0x7fe76819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe76dff0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768105380 0x7fe76819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe76dff0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76810de60 0x7fe76819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe76dff0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe768113970 con 0x7fe768105380 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe76dff0640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe7681137f0 con 0x7fe768104a30 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe76dff0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe768113af0 con 0x7fe76810de60 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768105380 0x7fe76819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768105380 0x7fe76819aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56564/0 (socket says 172.21.15.29:56564) 2024-09-21T02:36:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 -- 172.21.15.29:0/3495639395 learned_addr learned my addr 172.21.15.29:0/3495639395 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe767fff640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76810de60 0x7fe76819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe7677fe640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe768104a30 0x7fe76819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 -- 172.21.15.29:0/3495639395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76810de60 msgr2=0x7fe76819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76810de60 0x7fe76819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 -- 172.21.15.29:0/3495639395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe768104a30 msgr2=0x7fe76819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe768104a30 0x7fe76819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 -- 172.21.15.29:0/3495639395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7681a0310 con 0x7fe768105380 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe7677fe640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe768104a30 0x7fe76819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe75ffff640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768105380 0x7fe76819aa00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe75000d750 tx=0x7fe75000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.837+0000 7fe7657fa640 1 -- 172.21.15.29:0/3495639395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe750017070 con 0x7fe768105380 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.838+0000 7fe7657fa640 1 -- 172.21.15.29:0/3495639395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7500054b0 con 0x7fe768105380 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.838+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7681a0600 con 0x7fe768105380 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.838+0000 7fe7657fa640 1 -- 172.21.15.29:0/3495639395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe750004bc0 con 0x7fe768105380 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.838+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7681a6e20 con 0x7fe768105380 2024-09-21T02:36:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.839+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe76806c1f0 con 0x7fe768105380 2024-09-21T02:36:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.840+0000 7fe7657fa640 1 -- 172.21.15.29:0/3495639395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fe750009c30 con 0x7fe768105380 2024-09-21T02:36:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.841+0000 7fe7657fa640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe7540b6540 0x7fe7540b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.841+0000 7fe7677fe640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe7540b6540 0x7fe7540b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.841+0000 7fe7657fa640 1 -- 172.21.15.29:0/3495639395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7fe7500e68a0 con 0x7fe768105380 2024-09-21T02:36:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.841+0000 7fe7677fe640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe7540b6540 0x7fe7540b8a50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe758006010 tx=0x7fe758005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:08.845+0000 7fe7657fa640 1 -- 172.21.15.29:0/3495639395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe7500ad530 con 0x7fe768105380 2024-09-21T02:36:09.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.078+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe768104e40 con 0x7fe768105380 2024-09-21T02:36:09.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.078+0000 7fe7657fa640 1 -- 172.21.15.29:0/3495639395 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8043 (secure 0 0 0) 0x7fe7500b1910 con 0x7fe768105380 2024-09-21T02:36:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:09.080 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"btime":"2024-09-21T02:36:06:972250+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24587,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24599,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24613,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24637,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-21T02:36:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 10 2024-09-21T02:36:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe7540b6540 msgr2=0x7fe7540b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe7540b6540 0x7fe7540b8a50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe758006010 tx=0x7fe758005de0 comp rx=0 tx=0).stop 2024-09-21T02:36:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768105380 msgr2=0x7fe76819aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768105380 0x7fe76819aa00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe75000d750 tx=0x7fe75000dc10 comp rx=0 tx=0).stop 2024-09-21T02:36:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 shutdown_connections 2024-09-21T02:36:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe7540b6540 0x7fe7540b8a50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76810de60 0x7fe76819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:09.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe768105380 0x7fe76819aa00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:09.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 --2- 172.21.15.29:0/3495639395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe768104a30 0x7fe76819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:09.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 >> 172.21.15.29:0/3495639395 conn(0x7fe768079760 msgr2=0x7fe7680ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:09.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 shutdown_connections 2024-09-21T02:36:09.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:09.081+0000 7fe76dff0640 1 -- 172.21.15.29:0/3495639395 wait complete. 2024-09-21T02:36:09.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:09 smithi029 ceph-mon[31006]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:09.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:09 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3495639395' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:09.624 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 02:36:09 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g[71834]: starting mds.g at 2024-09-21T02:36:09.624 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 02:36:09 smithi029 bash[71805]: 23c9bd050707f68df772a36f0502c3f0f22afdb0df7f1a36ea038bad848adf14 2024-09-21T02:36:09.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:09 smithi066 ceph-mon[33436]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:09.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:09 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3495639395' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:09 smithi190 ceph-mon[33871]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:09 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3495639395' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:09.882 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 02:36:09 smithi029 systemd[1]: Started Ceph mds.g for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:36:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.29:6838/886485890,v1:172.21.15.29:6839/886485890] up:boot 2024-09-21T02:36:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: fsmap 10 up:standby 2024-09-21T02:36:10.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:10 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.29:6838/886485890,v1:172.21.15.29:6839/886485890] up:boot 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: fsmap 10 up:standby 2024-09-21T02:36:11.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:10 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:36:11.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.29:6838/886485890,v1:172.21.15.29:6839/886485890] up:boot 2024-09-21T02:36:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: fsmap 10 up:standby 2024-09-21T02:36:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:10 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:36:11.832 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:11 smithi029 ceph-mon[31006]: Deploying daemon mds.a on smithi029 2024-09-21T02:36:11.832 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:11 smithi029 ceph-mon[31006]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:12.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:11 smithi066 ceph-mon[33436]: Deploying daemon mds.a on smithi029 2024-09-21T02:36:12.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:11 smithi066 ceph-mon[33436]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:12.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:11 smithi190 ceph-mon[33871]: Deploying daemon mds.a on smithi029 2024-09-21T02:36:12.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:11 smithi190 ceph-mon[33871]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:12.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:36:12.475 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:12 smithi029 systemd[1]: Starting Ceph mds.a for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T02:36:12.749 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:12 smithi029 podman[72100]: 2024-09-21 02:36:12.482492999 +0000 UTC m=+0.017342626 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:36:12.750 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:12 smithi029 podman[72100]: 2024-09-21 02:36:12.613250425 +0000 UTC m=+0.148100039 container create 53953a68aa93f7be5d00b7c8f1b74097316ea7fd466ee48535d6ce3372f4b74a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:36:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:13.167 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:12 smithi029 podman[72100]: 2024-09-21 02:36:12.88560835 +0000 UTC m=+0.420457969 container init 53953a68aa93f7be5d00b7c8f1b74097316ea7fd466ee48535d6ce3372f4b74a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:36:13.167 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:12 smithi029 podman[72100]: 2024-09-21 02:36:12.888638084 +0000 UTC m=+0.423487700 container start 53953a68aa93f7be5d00b7c8f1b74097316ea7fd466ee48535d6ce3372f4b74a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T02:36:13.167 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:12 smithi029 ceph-mds[72154]: -- 172.21.15.29:0/1139040675 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b18a88bc20 con 0x55b18ea33180 2024-09-21T02:36:13.167 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:12 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a[72143]: starting mds.a at 2024-09-21T02:36:13.167 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:13 smithi029 bash[72100]: 53953a68aa93f7be5d00b7c8f1b74097316ea7fd466ee48535d6ce3372f4b74a 2024-09-21T02:36:13.167 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 02:36:13 smithi029 systemd[1]: Started Ceph mds.a for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:36:13.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 -- 172.21.15.29:0/3570033777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa8102ae0 msgr2=0x7f7fa8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 --2- 172.21.15.29:0/3570033777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa8102ae0 0x7f7fa8102ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7f98009900 tx=0x7f7f9802a410 comp rx=0 tx=0).stop 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 -- 172.21.15.29:0/3570033777 shutdown_connections 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 --2- 172.21.15.29:0/3570033777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa8108d20 0x7f7fa810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 --2- 172.21.15.29:0/3570033777 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fa81083d0 0x7f7fa81087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 --2- 172.21.15.29:0/3570033777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa8102ae0 0x7f7fa8102ef0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 -- 172.21.15.29:0/3570033777 >> 172.21.15.29:0/3570033777 conn(0x7f7fa8079760 msgr2=0x7f7fa8079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 -- 172.21.15.29:0/3570033777 shutdown_connections 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.547+0000 7f7faed0f640 1 -- 172.21.15.29:0/3570033777 wait complete. 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7faed0f640 1 Processor -- start 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7faed0f640 1 -- start start 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7faed0f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa8102ae0 0x7f7fa819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7faed0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa81083d0 0x7f7fa819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7faed0f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fa8108d20 0x7f7fa819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7faed0f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa8112840 con 0x7f7fa81083d0 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7faed0f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa81126c0 con 0x7f7fa8108d20 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7faed0f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa81129c0 con 0x7f7fa8102ae0 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.548+0000 7f7fa7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa81083d0 0x7f7fa819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa81083d0 0x7f7fa819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56614/0 (socket says 172.21.15.29:56614) 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa7fff640 1 -- 172.21.15.29:0/11978374 learned_addr learned my addr 172.21.15.29:0/11978374 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7facaab640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa8102ae0 0x7f7fa819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fad2ac640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fa8108d20 0x7f7fa819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa7fff640 1 -- 172.21.15.29:0/11978374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa8102ae0 msgr2=0x7f7fa819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa7fff640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa8102ae0 0x7f7fa819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa7fff640 1 -- 172.21.15.29:0/11978374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fa8108d20 msgr2=0x7f7fa819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa7fff640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fa8108d20 0x7f7fa819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa7fff640 1 -- 172.21.15.29:0/11978374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa81a02f0 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa7fff640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa81083d0 0x7f7fa819a920 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000e9c0 tx=0x7f7f9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa5ffb640 1 -- 172.21.15.29:0/11978374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9000cd80 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa5ffb640 1 -- 172.21.15.29:0/11978374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f900054b0 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa81a05e0 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7fa5ffb640 1 -- 172.21.15.29:0/11978374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90010430 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.549+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa81a6d40 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.551+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa8103d10 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.552+0000 7f7fa5ffb640 1 -- 172.21.15.29:0/11978374 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f7f90004f20 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.553+0000 7f7fa5ffb640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7f940b6590 0x7f7f940b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.553+0000 7f7facaab640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7f940b6590 0x7f7f940b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.553+0000 7f7fa5ffb640 1 -- 172.21.15.29:0/11978374 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7f7f900e7590 con 0x7f7fa81083d0 2024-09-21T02:36:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.553+0000 7f7facaab640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7f940b6590 0x7f7f940b8aa0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f98000c00 tx=0x7f7f98009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.557+0000 7f7fa5ffb640 1 -- 172.21.15.29:0/11978374 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7f900ae220 con 0x7f7fa81083d0 2024-09-21T02:36:13.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:13 smithi066 ceph-mon[33436]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:13 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:13 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:13 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:13 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:13 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:13 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:13.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.790+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fa81a7050 con 0x7f7fa81083d0 2024-09-21T02:36:13.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.790+0000 7f7fa5ffb640 1 -- 172.21.15.29:0/11978374 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9692 (secure 0 0 0) 0x7f7f900b2600 con 0x7f7fa81083d0 2024-09-21T02:36:13.792 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:13.792 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":12,"btime":"2024-09-21T02:36:13:677477+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14781,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14787,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24587,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24599,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24613,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24637,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-21T02:36:13.792 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 12 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7f940b6590 msgr2=0x7f7f940b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7f940b6590 0x7f7f940b8aa0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f98000c00 tx=0x7f7f98009680 comp rx=0 tx=0).stop 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa81083d0 msgr2=0x7f7fa819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa81083d0 0x7f7fa819a920 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000e9c0 tx=0x7f7f9000ee80 comp rx=0 tx=0).stop 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 shutdown_connections 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7f940b6590 0x7f7f940b8aa0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fa8108d20 0x7f7fa819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa81083d0 0x7f7fa819a920 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 --2- 172.21.15.29:0/11978374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa8102ae0 0x7f7fa819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:13.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 >> 172.21.15.29:0/11978374 conn(0x7f7fa8079760 msgr2=0x7f7fa8101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:13.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 shutdown_connections 2024-09-21T02:36:13.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:13.793+0000 7f7faed0f640 1 -- 172.21.15.29:0/11978374 wait complete. 2024-09-21T02:36:14.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:13 smithi190 ceph-mon[33871]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:13 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:13 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:13 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:13 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:13 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:13 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:13 smithi029 ceph-mon[31006]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:13 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:13 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:13 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:13 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:36:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:13 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:36:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:13 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:14.841 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:14 smithi066 ceph-mon[33436]: Deploying daemon mds.b on smithi066 2024-09-21T02:36:14.841 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:14 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.29:6840/1488187270,v1:172.21.15.29:6841/1488187270] up:boot 2024-09-21T02:36:14.841 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:14 smithi066 ceph-mon[33436]: fsmap 11 up:standby 2024-09-21T02:36:14.841 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:14 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:36:14.841 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:14 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/11978374' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:14 smithi029 ceph-mon[31006]: Deploying daemon mds.b on smithi066 2024-09-21T02:36:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:14 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.29:6840/1488187270,v1:172.21.15.29:6841/1488187270] up:boot 2024-09-21T02:36:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:14 smithi029 ceph-mon[31006]: fsmap 11 up:standby 2024-09-21T02:36:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:14 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:36:14.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:14 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/11978374' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:15.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:14 smithi190 ceph-mon[33871]: Deploying daemon mds.b on smithi066 2024-09-21T02:36:15.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:14 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.29:6840/1488187270,v1:172.21.15.29:6841/1488187270] up:boot 2024-09-21T02:36:15.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:14 smithi190 ceph-mon[33871]: fsmap 11 up:standby 2024-09-21T02:36:15.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:14 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:36:15.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:14 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/11978374' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:16.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:15 smithi066 ceph-mon[33436]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:16.009 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 02:36:15 smithi066 systemd[1]: Started Ceph mds.b for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T02:36:16.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:15 smithi190 ceph-mon[33871]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:16.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:15 smithi029 ceph-mon[31006]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:16.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:36:17.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] up:boot 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: fsmap 12 up:standby 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:36:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:16 smithi066 ceph-mon[33436]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:17.060 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:17.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] up:boot 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: fsmap 12 up:standby 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:36:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:16 smithi190 ceph-mon[33871]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:17.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] up:boot 2024-09-21T02:36:17.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: fsmap 12 up:standby 2024-09-21T02:36:17.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:36:17.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:17.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:36:17.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:16 smithi029 ceph-mon[31006]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 -- 172.21.15.29:0/946521676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028108cf0 msgr2=0x7fd02810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 --2- 172.21.15.29:0/946521676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028108cf0 0x7fd02810f520 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd020009ca0 tx=0x7fd02002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 -- 172.21.15.29:0/946521676 shutdown_connections 2024-09-21T02:36:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 --2- 172.21.15.29:0/946521676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028108cf0 0x7fd02810f520 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 --2- 172.21.15.29:0/946521676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0281083a0 0x7fd0281087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 --2- 172.21.15.29:0/946521676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd028102a60 0x7fd028102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 -- 172.21.15.29:0/946521676 >> 172.21.15.29:0/946521676 conn(0x7fd028079730 msgr2=0x7fd028079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 -- 172.21.15.29:0/946521676 shutdown_connections 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.600+0000 7fd02ffea640 1 -- 172.21.15.29:0/946521676 wait complete. 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02ffea640 1 Processor -- start 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02ffea640 1 -- start start 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02ffea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028102a60 0x7fd02819b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02ffea640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0281083a0 0x7fd02819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02ffea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd028108cf0 0x7fd02819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02ffea640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd028112860 con 0x7fd028102a60 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02ffea640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd0281126e0 con 0x7fd0281083a0 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02ffea640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd0281129e0 con 0x7fd028108cf0 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02dd86640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028102a60 0x7fd02819b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02dd86640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028102a60 0x7fd02819b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36118/0 (socket says 172.21.15.29:36118) 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02dd86640 1 -- 172.21.15.29:0/816233504 learned_addr learned my addr 172.21.15.29:0/816233504 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02d585640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0281083a0 0x7fd02819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.601+0000 7fd02e587640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd028108cf0 0x7fd02819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02dd86640 1 -- 172.21.15.29:0/816233504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd028108cf0 msgr2=0x7fd02819bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02dd86640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd028108cf0 0x7fd02819bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02dd86640 1 -- 172.21.15.29:0/816233504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0281083a0 msgr2=0x7fd02819b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02dd86640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0281083a0 0x7fd02819b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02dd86640 1 -- 172.21.15.29:0/816233504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0281a6540 con 0x7fd028102a60 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02d585640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0281083a0 0x7fd02819b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02dd86640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028102a60 0x7fd02819b2b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd02400e9f0 tx=0x7fd02400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd01effd640 1 -- 172.21.15.29:0/816233504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02400cd80 con 0x7fd028102a60 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0281a6830 con 0x7fd028102a60 2024-09-21T02:36:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd01effd640 1 -- 172.21.15.29:0/816233504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0240054b0 con 0x7fd028102a60 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd01effd640 1 -- 172.21.15.29:0/816233504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024010430 con 0x7fd028102a60 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.602+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0281a6d70 con 0x7fd028102a60 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.603+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd028103ce0 con 0x7fd028102a60 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.605+0000 7fd01effd640 1 -- 172.21.15.29:0/816233504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fd0240046c0 con 0x7fd028102a60 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.605+0000 7fd01effd640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0140b6540 0x7fd0140b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.605+0000 7fd02d585640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0140b6540 0x7fd0140b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.606+0000 7fd01effd640 1 -- 172.21.15.29:0/816233504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7fd0240e7350 con 0x7fd028102a60 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.606+0000 7fd02d585640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0140b6540 0x7fd0140b8a50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd028105f60 tx=0x7fd018005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.609+0000 7fd01effd640 1 -- 172.21.15.29:0/816233504 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd0240ae080 con 0x7fd028102a60 2024-09-21T02:36:17.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.846+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd028102e70 con 0x7fd028102a60 2024-09-21T02:36:17.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.847+0000 7fd01effd640 1 -- 172.21.15.29:0/816233504 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10518 (secure 0 0 0) 0x7fd028102e70 con 0x7fd028102a60 2024-09-21T02:36:17.848 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:17.849 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":13,"btime":"2024-09-21T02:36:15:691029+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14781,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14787,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24587,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24599,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24613,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24632,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24637,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-21T02:36:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 13 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0140b6540 msgr2=0x7fd0140b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0140b6540 0x7fd0140b8a50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd028105f60 tx=0x7fd018005f90 comp rx=0 tx=0).stop 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028102a60 msgr2=0x7fd02819b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028102a60 0x7fd02819b2b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd02400e9f0 tx=0x7fd02400edf0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 shutdown_connections 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0140b6540 0x7fd0140b8a50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd028108cf0 0x7fd02819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0281083a0 0x7fd02819b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 --2- 172.21.15.29:0/816233504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd028102a60 0x7fd02819b2b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:17.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 >> 172.21.15.29:0/816233504 conn(0x7fd028079730 msgr2=0x7fd0281044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:17.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 shutdown_connections 2024-09-21T02:36:17.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:17.850+0000 7fd02ffea640 1 -- 172.21.15.29:0/816233504 wait complete. 2024-09-21T02:36:18.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:18 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:18 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/816233504' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:18.617 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.617 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.617 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.617 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:18 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:18 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/816233504' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:18.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:18 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:18.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:18 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/816233504' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:18.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph --format=json-pretty osd lspools 2024-09-21T02:36:18.918 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:19.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:19 smithi029 ceph-mon[31006]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:19.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:19.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:36:19.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:19 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:36:19.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 -- 172.21.15.29:0/3353815174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c81083a0 msgr2=0x7f68c81087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:19.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/3353815174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c81083a0 0x7f68c81087b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009900 tx=0x7f68bc02a410 comp rx=0 tx=0).stop 2024-09-21T02:36:19.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 -- 172.21.15.29:0/3353815174 shutdown_connections 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/3353815174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8108cf0 0x7f68c810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/3353815174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c81083a0 0x7f68c81087b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/3353815174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c8102a60 0x7f68c8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 -- 172.21.15.29:0/3353815174 >> 172.21.15.29:0/3353815174 conn(0x7f68c8079730 msgr2=0x7f68c8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 -- 172.21.15.29:0/3353815174 shutdown_connections 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.445+0000 7f68cd4bd640 1 -- 172.21.15.29:0/3353815174 wait complete. 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68cd4bd640 1 Processor -- start 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68cd4bd640 1 -- start start 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68cd4bd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8102a60 0x7f68c80ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68cd4bd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c81083a0 0x7f68c80ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68cd4bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c8108cf0 0x7f68c81000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68cd4bd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f68c8112810 con 0x7f68c8108cf0 2024-09-21T02:36:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68cd4bd640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f68c8112690 con 0x7f68c81083a0 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68cd4bd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f68c8112990 con 0x7f68c8102a60 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.446+0000 7f68c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c8108cf0 0x7f68c81000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8102a60 0x7f68c80ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c67fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c81083a0 0x7f68c80ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c8108cf0 0x7f68c81000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36128/0 (socket says 172.21.15.29:36128) 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c77fe640 1 -- 172.21.15.29:0/1859818474 learned_addr learned my addr 172.21.15.29:0/1859818474 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8102a60 0x7f68c80ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44506/0 (socket says 172.21.15.29:44506) 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c77fe640 1 -- 172.21.15.29:0/1859818474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8102a60 msgr2=0x7f68c80ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c77fe640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8102a60 0x7f68c80ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c77fe640 1 -- 172.21.15.29:0/1859818474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c81083a0 msgr2=0x7f68c80ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c77fe640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c81083a0 0x7f68c80ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c77fe640 1 -- 172.21.15.29:0/1859818474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c81a8690 con 0x7f68c8108cf0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c6ffd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8102a60 0x7f68c80ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c67fc640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c81083a0 0x7f68c80ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68c77fe640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c8108cf0 0x7f68c81000f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009fd0 tx=0x7f68c000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68affff640 1 -- 172.21.15.29:0/1859818474 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0017070 con 0x7f68c8108cf0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68affff640 1 -- 172.21.15.29:0/1859818474 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68c0003e00 con 0x7f68c8108cf0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c81a8980 con 0x7f68c8108cf0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68affff640 1 -- 172.21.15.29:0/1859818474 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0010460 con 0x7f68c8108cf0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.447+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c81a8e40 con 0x7f68c8108cf0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.448+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c8102e70 con 0x7f68c8108cf0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.450+0000 7f68affff640 1 -- 172.21.15.29:0/1859818474 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f68c0003970 con 0x7f68c8108cf0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.451+0000 7f68affff640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f68b00b6510 0x7f68b00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.451+0000 7f68c6ffd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f68b00b6510 0x7f68b00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.451+0000 7f68affff640 1 -- 172.21.15.29:0/1859818474 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7f68c00e6f70 con 0x7f68c8108cf0 2024-09-21T02:36:19.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.451+0000 7f68c6ffd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f68b00b6510 0x7f68b00b8a20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f68c81010c0 tx=0x7f68b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:19.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.454+0000 7f68affff640 1 -- 172.21.15.29:0/1859818474 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f68c00b1fe0 con 0x7f68c8108cf0 2024-09-21T02:36:19.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:19 smithi190 ceph-mon[33871]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:19.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:19.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:36:19.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:19 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:36:19.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.672+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f68c8100970 con 0x7f68c8108cf0 2024-09-21T02:36:19.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.673+0000 7f68affff640 1 -- 172.21.15.29:0/1859818474 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v70) ==== 84+0+65 (secure 0 0 0) 0x7f68c8100970 con 0x7f68c8108cf0 2024-09-21T02:36:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:19.674 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2024-09-21T02:36:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T02:36:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2024-09-21T02:36:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2024-09-21T02:36:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:36:19.675 INFO:teuthology.orchestra.run.smithi029.stdout:] 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f68b00b6510 msgr2=0x7f68b00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f68b00b6510 0x7f68b00b8a20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f68c81010c0 tx=0x7f68b8005f90 comp rx=0 tx=0).stop 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c8108cf0 msgr2=0x7f68c81000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c8108cf0 0x7f68c81000f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009fd0 tx=0x7f68c000b260 comp rx=0 tx=0).stop 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 shutdown_connections 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f68b00b6510 0x7f68b00b8a20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68c8108cf0 0x7f68c81000f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c81083a0 0x7f68c80ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 --2- 172.21.15.29:0/1859818474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8102a60 0x7f68c80ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.675+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 >> 172.21.15.29:0/1859818474 conn(0x7f68c8079730 msgr2=0x7f68c806b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.676+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 shutdown_connections 2024-09-21T02:36:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:19.676+0000 7f68cd4bd640 1 -- 172.21.15.29:0/1859818474 wait complete. 2024-09-21T02:36:19.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:19 smithi066 ceph-mon[33436]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:19.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:19.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:36:19.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:19.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:19 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:36:20.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:20 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1859818474' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T02:36:20.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:20 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1859818474' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T02:36:20.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:20 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1859818474' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T02:36:21.109 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-21T02:36:21.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-21T02:36:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:21.805 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:21 smithi029 ceph-mon[31006]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:21.805 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:21.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:21.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:21.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:21 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:21.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:21 smithi190 ceph-mon[33871]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:21.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:21 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:21.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 -- 172.21.15.29:0/180959634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 msgr2=0x7f0af8102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:21.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 --2- 172.21.15.29:0/180959634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 0x7f0af8102e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009900 tx=0x7f0af402a410 comp rx=0 tx=0).stop 2024-09-21T02:36:21.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 -- 172.21.15.29:0/180959634 shutdown_connections 2024-09-21T02:36:21.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 --2- 172.21.15.29:0/180959634 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0af8108cf0 0x7f0af810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:21.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 --2- 172.21.15.29:0/180959634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0af81083a0 0x7f0af81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 --2- 172.21.15.29:0/180959634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 0x7f0af8102e70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 -- 172.21.15.29:0/180959634 >> 172.21.15.29:0/180959634 conn(0x7f0af8079730 msgr2=0x7f0af8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 -- 172.21.15.29:0/180959634 shutdown_connections 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.900+0000 7f0afff87640 1 -- 172.21.15.29:0/180959634 wait complete. 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.901+0000 7f0afff87640 1 Processor -- start 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.901+0000 7f0afff87640 1 -- start start 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afff87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 0x7f0af819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afff87640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0af81083a0 0x7f0af819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afff87640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0af8108cf0 0x7f0af819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afff87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0af8112810 con 0x7f0af8102a60 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afff87640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0af8112690 con 0x7f0af8108cf0 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afff87640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0af8112990 con 0x7f0af81083a0 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 0x7f0af819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 0x7f0af819a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36156/0 (socket says 172.21.15.29:36156) 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 -- 172.21.15.29:0/1041098486 learned_addr learned my addr 172.21.15.29:0/1041098486 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afe524640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0af8108cf0 0x7f0af819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afd522640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0af81083a0 0x7f0af819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 -- 172.21.15.29:0/1041098486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0af81083a0 msgr2=0x7f0af819a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0af81083a0 0x7f0af819a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 -- 172.21.15.29:0/1041098486 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0af8108cf0 msgr2=0x7f0af819fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0af8108cf0 0x7f0af819fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 -- 172.21.15.29:0/1041098486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af81a0300 con 0x7f0af8102a60 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.902+0000 7f0afdd23640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 0x7f0af819a390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009230 tx=0x7f0af4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.903+0000 7f0aeeffd640 1 -- 172.21.15.29:0/1041098486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af403f030 con 0x7f0af8102a60 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.903+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af81a0590 con 0x7f0af8102a60 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.903+0000 7f0aeeffd640 1 -- 172.21.15.29:0/1041098486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0af4003a80 con 0x7f0af8102a60 2024-09-21T02:36:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.903+0000 7f0aeeffd640 1 -- 172.21.15.29:0/1041098486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af4035070 con 0x7f0af8102a60 2024-09-21T02:36:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.903+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af81a6d50 con 0x7f0af8102a60 2024-09-21T02:36:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.904+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0af8102e70 con 0x7f0af8102a60 2024-09-21T02:36:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.905+0000 7f0aeeffd640 1 -- 172.21.15.29:0/1041098486 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f0af4042030 con 0x7f0af8102a60 2024-09-21T02:36:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.906+0000 7f0aeeffd640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0ae40b6510 0x7f0ae40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.906+0000 7f0afd522640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0ae40b6510 0x7f0ae40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.906+0000 7f0aeeffd640 1 -- 172.21.15.29:0/1041098486 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7127+0+0 (secure 0 0 0) 0x7f0af4107a80 con 0x7f0af8102a60 2024-09-21T02:36:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.907+0000 7f0afd522640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0ae40b6510 0x7f0ae40b8a20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8005fd0 tx=0x7f0ae8005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:21.909+0000 7f0aeeffd640 1 -- 172.21.15.29:0/1041098486 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0af40ce710 con 0x7f0af8102a60 2024-09-21T02:36:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:21 smithi066 ceph-mon[33436]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:21 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:21 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:21 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:21 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:22.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.098+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0af819b4c0 con 0x7f0af8102a60 2024-09-21T02:36:22.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.712+0000 7f0aeeffd640 1 -- 172.21.15.29:0/1041098486 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v71) ==== 138+0+0 (secure 0 0 0) 0x7f0af40d2af0 con 0x7f0af8102a60 2024-09-21T02:36:22.714 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2024-09-21T02:36:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0ae40b6510 msgr2=0x7f0ae40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0ae40b6510 0x7f0ae40b8a20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8005fd0 tx=0x7f0ae8005e20 comp rx=0 tx=0).stop 2024-09-21T02:36:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 msgr2=0x7f0af819a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 0x7f0af819a390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009230 tx=0x7f0af4004270 comp rx=0 tx=0).stop 2024-09-21T02:36:22.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 shutdown_connections 2024-09-21T02:36:22.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0ae40b6510 0x7f0ae40b8a20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:22.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0af8108cf0 0x7f0af819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:22.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0af81083a0 0x7f0af819a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:22.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 --2- 172.21.15.29:0/1041098486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0af8102a60 0x7f0af819a390 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:22.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.716+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 >> 172.21.15.29:0/1041098486 conn(0x7f0af8079730 msgr2=0x7f0af8101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:22.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.717+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 shutdown_connections 2024-09-21T02:36:22.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:22.717+0000 7f0afff87640 1 -- 172.21.15.29:0/1041098486 wait complete. 2024-09-21T02:36:23.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:23.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:36:23.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:22 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1041098486' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T02:36:23.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:23.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:36:23.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:22 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1041098486' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T02:36:23.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:36:23.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:36:23.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:36:23.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:22 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1041098486' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T02:36:23.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-21T02:36:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:23 smithi029 ceph-mon[31006]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:23 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1041098486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T02:36:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:23 smithi029 ceph-mon[31006]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T02:36:24.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:23 smithi066 ceph-mon[33436]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:24.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:23 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1041098486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T02:36:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:23 smithi066 ceph-mon[33436]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T02:36:24.114 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:23 smithi190 ceph-mon[33871]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:24.114 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:23 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1041098486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T02:36:24.114 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:23 smithi190 ceph-mon[33871]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T02:36:24.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2453374910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a810de10 msgr2=0x7f56a8110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2453374910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a810de10 0x7f56a8110300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5698009ca0 tx=0x7f569802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2453374910 shutdown_connections 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2453374910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a810de10 0x7f56a8110300 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2453374910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56a8105300 0x7f56a810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2453374910 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56a81049b0 0x7f56a8104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2453374910 >> 172.21.15.29:0/2453374910 conn(0x7f56a8079730 msgr2=0x7f56a8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2453374910 shutdown_connections 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.343+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2453374910 wait complete. 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56ad6b4640 1 Processor -- start 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56ad6b4640 1 -- start start 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56ad6b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a81049b0 0x7f56a819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56ad6b4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56a8105300 0x7f56a819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56ad6b4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56a810de10 0x7f56a819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56ad6b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f56a8113990 con 0x7f56a81049b0 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56ad6b4640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f56a8113810 con 0x7f56a810de10 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56ad6b4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f56a8113b10 con 0x7f56a8105300 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56a6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a81049b0 0x7f56a819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56a6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a81049b0 0x7f56a819a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54794/0 (socket says 172.21.15.29:54794) 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.344+0000 7f56a6ffd640 1 -- 172.21.15.29:0/2151093168 learned_addr learned my addr 172.21.15.29:0/2151093168 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56a77fe640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56a810de10 0x7f56a819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56a67fc640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56a8105300 0x7f56a819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56a6ffd640 1 -- 172.21.15.29:0/2151093168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56a8105300 msgr2=0x7f56a819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56a6ffd640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56a8105300 0x7f56a819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56a6ffd640 1 -- 172.21.15.29:0/2151093168 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56a810de10 msgr2=0x7f56a819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56a6ffd640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56a810de10 0x7f56a819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56a6ffd640 1 -- 172.21.15.29:0/2151093168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a81a02f0 con 0x7f56a81049b0 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56a6ffd640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a81049b0 0x7f56a819a380 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f56a000d770 tx=0x7f56a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f569ffff640 1 -- 172.21.15.29:0/2151093168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0017070 con 0x7f56a81049b0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f569ffff640 1 -- 172.21.15.29:0/2151093168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56a0009c30 con 0x7f56a81049b0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a81a05e0 con 0x7f56a81049b0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f569ffff640 1 -- 172.21.15.29:0/2151093168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0003c50 con 0x7f56a81049b0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.345+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a81a6d40 con 0x7f56a81049b0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.346+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56a8104dc0 con 0x7f56a81049b0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.348+0000 7f569ffff640 1 -- 172.21.15.29:0/2151093168 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f56a0003df0 con 0x7f56a81049b0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.349+0000 7f569ffff640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56940b6590 0x7f56940b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.349+0000 7f56a67fc640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56940b6590 0x7f56940b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.349+0000 7f569ffff640 1 -- 172.21.15.29:0/2151093168 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7497+0+0 (secure 0 0 0) 0x7f56a00e6d20 con 0x7f56a81049b0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.349+0000 7f56a67fc640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56940b6590 0x7f56940b8aa0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f56a807a030 tx=0x7f5690005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.352+0000 7f569ffff640 1 -- 172.21.15.29:0/2151093168 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f56a0009de0 con 0x7f56a81049b0 2024-09-21T02:36:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.542+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f56a802e640 con 0x7f56a81049b0 2024-09-21T02:36:24.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.720+0000 7f569ffff640 1 -- 172.21.15.29:0/2151093168 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v73) ==== 171+0+0 (secure 0 0 0) 0x7f56a00b1ca0 con 0x7f56a81049b0 2024-09-21T02:36:24.721 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2024-09-21T02:36:24.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56940b6590 msgr2=0x7f56940b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:24.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56940b6590 0x7f56940b8aa0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f56a807a030 tx=0x7f5690005f90 comp rx=0 tx=0).stop 2024-09-21T02:36:24.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a81049b0 msgr2=0x7f56a819a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:24.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a81049b0 0x7f56a819a380 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f56a000d770 tx=0x7f56a000dc30 comp rx=0 tx=0).stop 2024-09-21T02:36:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 shutdown_connections 2024-09-21T02:36:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56940b6590 0x7f56940b8aa0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56a810de10 0x7f56a819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56a8105300 0x7f56a819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 --2- 172.21.15.29:0/2151093168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a81049b0 0x7f56a819a380 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 >> 172.21.15.29:0/2151093168 conn(0x7f56a8079730 msgr2=0x7f56a8100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 shutdown_connections 2024-09-21T02:36:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:24.723+0000 7f56ad6b4640 1 -- 172.21.15.29:0/2151093168 wait complete. 2024-09-21T02:36:24.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:24 smithi029 ceph-mon[31006]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T02:36:24.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:24 smithi029 ceph-mon[31006]: pgmap v295: 33 pgs: 11 creating+peering, 21 unknown, 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:24.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:24 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2151093168' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T02:36:25.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:24 smithi066 ceph-mon[33436]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T02:36:25.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:24 smithi066 ceph-mon[33436]: pgmap v295: 33 pgs: 11 creating+peering, 21 unknown, 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:25.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:24 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2151093168' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T02:36:25.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:24 smithi190 ceph-mon[33871]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T02:36:25.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:24 smithi190 ceph-mon[33871]: pgmap v295: 33 pgs: 11 creating+peering, 21 unknown, 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:25.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:24 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2151093168' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T02:36:25.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-21T02:36:25.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:25 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2151093168' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T02:36:25.797 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:25 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2151093168' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T02:36:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:25 smithi066 ceph-mon[33436]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T02:36:26.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:25 smithi029 ceph-mon[31006]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T02:36:26.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:25 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2151093168' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T02:36:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:25 smithi190 ceph-mon[33871]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T02:36:26.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 -- 172.21.15.29:0/4285678350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c103b60 msgr2=0x7f136c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 --2- 172.21.15.29:0/4285678350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c103b60 0x7f136c103f70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f135c009900 tx=0x7f135c02a410 comp rx=0 tx=0).stop 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 -- 172.21.15.29:0/4285678350 shutdown_connections 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 --2- 172.21.15.29:0/4285678350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f136c109df0 0x7f136c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 --2- 172.21.15.29:0/4285678350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f136c1094a0 0x7f136c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 --2- 172.21.15.29:0/4285678350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c103b60 0x7f136c103f70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 -- 172.21.15.29:0/4285678350 >> 172.21.15.29:0/4285678350 conn(0x7f136c07b850 msgr2=0x7f136c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 -- 172.21.15.29:0/4285678350 shutdown_connections 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 -- 172.21.15.29:0/4285678350 wait complete. 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.325+0000 7f1371d2c640 1 Processor -- start 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f1371d2c640 1 -- start start 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f1371d2c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f136c103b60 0x7f136c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f1371d2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c1094a0 0x7f136c19a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f1371d2c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f136c109df0 0x7f136c19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f1371d2c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1139c0 con 0x7f136c1094a0 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f1371d2c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f136c113840 con 0x7f136c103b60 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f1371d2c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f136c113b40 con 0x7f136c109df0 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c1094a0 0x7f136c19a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c1094a0 0x7f136c19a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54820/0 (socket says 172.21.15.29:54820) 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136affd640 1 -- 172.21.15.29:0/791008313 learned_addr learned my addr 172.21.15.29:0/791008313 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136bfff640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f136c109df0 0x7f136c19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136b7fe640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f136c103b60 0x7f136c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136affd640 1 -- 172.21.15.29:0/791008313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f136c109df0 msgr2=0x7f136c19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136affd640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f136c109df0 0x7f136c19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136affd640 1 -- 172.21.15.29:0/791008313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f136c103b60 msgr2=0x7f136c19a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136affd640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f136c103b60 0x7f136c19a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.326+0000 7f136affd640 1 -- 172.21.15.29:0/791008313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c1a03a0 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.327+0000 7f136affd640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c1094a0 0x7f136c19a9d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f136000d740 tx=0x7f136000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.327+0000 7f1368ff9640 1 -- 172.21.15.29:0/791008313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360017070 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.327+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c1a0690 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.327+0000 7f1368ff9640 1 -- 172.21.15.29:0/791008313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f136000de50 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.327+0000 7f1368ff9640 1 -- 172.21.15.29:0/791008313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136000f620 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.327+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c1a6df0 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.328+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f136c103f70 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.330+0000 7f1368ff9640 1 -- 172.21.15.29:0/791008313 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f136001a070 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.331+0000 7f1368ff9640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f13540b6510 0x7f13540b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.331+0000 7f136b7fe640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f13540b6510 0x7f13540b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.331+0000 7f1368ff9640 1 -- 172.21.15.29:0/791008313 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7879+0+0 (secure 0 0 0) 0x7f13600e6b60 con 0x7f136c1094a0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.332+0000 7f136b7fe640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f13540b6510 0x7f13540b8a20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f135c000c00 tx=0x7f135c009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.335+0000 7f1368ff9640 1 -- 172.21.15.29:0/791008313 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f13600ad580 con 0x7f136c1094a0 2024-09-21T02:36:26.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.561+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f136c19b560 con 0x7f136c1094a0 2024-09-21T02:36:26.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.732+0000 7f1368ff9640 1 -- 172.21.15.29:0/791008313 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f136c19b560 con 0x7f136c1094a0 2024-09-21T02:36:26.734 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2024-09-21T02:36:26.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f13540b6510 msgr2=0x7f13540b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:26.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f13540b6510 0x7f13540b8a20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f135c000c00 tx=0x7f135c009680 comp rx=0 tx=0).stop 2024-09-21T02:36:26.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c1094a0 msgr2=0x7f136c19a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c1094a0 0x7f136c19a9d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f136000d740 tx=0x7f136000dc00 comp rx=0 tx=0).stop 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 shutdown_connections 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f13540b6510 0x7f13540b8a20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f136c109df0 0x7f136c19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c1094a0 0x7f136c19a9d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 --2- 172.21.15.29:0/791008313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f136c103b60 0x7f136c19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.735+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 >> 172.21.15.29:0/791008313 conn(0x7f136c07b850 msgr2=0x7f136c107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.736+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 shutdown_connections 2024-09-21T02:36:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:26.736+0000 7f1371d2c640 1 -- 172.21.15.29:0/791008313 wait complete. 2024-09-21T02:36:26.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:26 smithi029 ceph-mon[31006]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T02:36:26.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:26 smithi029 ceph-mon[31006]: pgmap v298: 97 pgs: 21 creating+peering, 66 unknown, 10 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:26.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:26 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/791008313' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T02:36:26.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:26 smithi029 ceph-mon[31006]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-21T02:36:27.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:26 smithi066 ceph-mon[33436]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T02:36:27.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:26 smithi066 ceph-mon[33436]: pgmap v298: 97 pgs: 21 creating+peering, 66 unknown, 10 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:27.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:26 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/791008313' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T02:36:27.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:26 smithi066 ceph-mon[33436]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-21T02:36:27.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:26 smithi190 ceph-mon[33871]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T02:36:27.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:26 smithi190 ceph-mon[33871]: pgmap v298: 97 pgs: 21 creating+peering, 66 unknown, 10 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:27.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:26 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/791008313' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T02:36:27.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:26 smithi190 ceph-mon[33871]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-21T02:36:27.580 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-21T02:36:27.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-21T02:36:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:27 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/791008313' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T02:36:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:27 smithi029 ceph-mon[31006]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T02:36:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:27 smithi029 ceph-mon[31006]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-21T02:36:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:27 smithi029 ceph-mon[31006]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-21T02:36:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:27 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/791008313' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T02:36:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:27 smithi066 ceph-mon[33436]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T02:36:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:27 smithi066 ceph-mon[33436]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-21T02:36:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:27 smithi066 ceph-mon[33436]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-21T02:36:28.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:27 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/791008313' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T02:36:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:27 smithi190 ceph-mon[33871]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T02:36:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:27 smithi190 ceph-mon[33871]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-21T02:36:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:27 smithi190 ceph-mon[33871]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-21T02:36:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/602652057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58108cf0 msgr2=0x7f5e5810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/602652057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58108cf0 0x7f5e5810f520 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009ca0 tx=0x7f5e5002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/602652057 shutdown_connections 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/602652057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58108cf0 0x7f5e5810f520 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/602652057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e581083a0 0x7f5e581087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/602652057 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e58102a60 0x7f5e58102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/602652057 >> 172.21.15.29:0/602652057 conn(0x7f5e58079730 msgr2=0x7f5e58079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/602652057 shutdown_connections 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.431+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/602652057 wait complete. 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5ec5a640 1 Processor -- start 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5ec5a640 1 -- start start 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5ec5a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58102a60 0x7f5e58199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5ec5a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e581083a0 0x7f5e581997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5ec5a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e58108cf0 0x7f5e5819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5ec5a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5e58112860 con 0x7f5e58102a60 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5ec5a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581126e0 con 0x7f5e58108cf0 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5ec5a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581129e0 con 0x7f5e581083a0 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5c9f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58102a60 0x7f5e58199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5c9f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58102a60 0x7f5e58199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54844/0 (socket says 172.21.15.29:54844) 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e5c9f6640 1 -- 172.21.15.29:0/713067036 learned_addr learned my addr 172.21.15.29:0/713067036 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.432+0000 7f5e57fff640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e581083a0 0x7f5e581997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5d1f7640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e58108cf0 0x7f5e5819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5c9f6640 1 -- 172.21.15.29:0/713067036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e581083a0 msgr2=0x7f5e581997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5c9f6640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e581083a0 0x7f5e581997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5c9f6640 1 -- 172.21.15.29:0/713067036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e58108cf0 msgr2=0x7f5e5819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5c9f6640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e58108cf0 0x7f5e5819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5c9f6640 1 -- 172.21.15.29:0/713067036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5819f0c0 con 0x7f5e58102a60 2024-09-21T02:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5c9f6640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58102a60 0x7f5e58199210 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800d950 tx=0x7f5e4800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e55ffb640 1 -- 172.21.15.29:0/713067036 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48017070 con 0x7f5e58102a60 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e55ffb640 1 -- 172.21.15.29:0/713067036 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e4800de80 con 0x7f5e58102a60 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e5819f3b0 con 0x7f5e58102a60 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e55ffb640 1 -- 172.21.15.29:0/713067036 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4800f620 con 0x7f5e58102a60 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.433+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e581a5bd0 con 0x7f5e58102a60 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.434+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e58102e70 con 0x7f5e58102a60 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.436+0000 7f5e55ffb640 1 -- 172.21.15.29:0/713067036 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f5e48009c30 con 0x7f5e58102a60 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.437+0000 7f5e55ffb640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e400b6540 0x7f5e400b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.437+0000 7f5e57fff640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e400b6540 0x7f5e400b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.437+0000 7f5e55ffb640 1 -- 172.21.15.29:0/713067036 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f5e480e6dd0 con 0x7f5e58102a60 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.437+0000 7f5e57fff640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e400b6540 0x7f5e400b8a50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e58105eb0 tx=0x7f5e4c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.440+0000 7f5e55ffb640 1 -- 172.21.15.29:0/713067036 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5e480ad710 con 0x7f5e58102a60 2024-09-21T02:36:28.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.641+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f5e5819a340 con 0x7f5e58102a60 2024-09-21T02:36:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.761+0000 7f5e55ffb640 1 -- 172.21.15.29:0/713067036 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v76) ==== 157+0+0 (secure 0 0 0) 0x7f5e480b1af0 con 0x7f5e58102a60 2024-09-21T02:36:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e400b6540 msgr2=0x7f5e400b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e400b6540 0x7f5e400b8a50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e58105eb0 tx=0x7f5e4c00a2d0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58102a60 msgr2=0x7f5e58199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58102a60 0x7f5e58199210 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800d950 tx=0x7f5e4800dd50 comp rx=0 tx=0).stop 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 shutdown_connections 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e400b6540 0x7f5e400b8a50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e58108cf0 0x7f5e5819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e581083a0 0x7f5e581997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 --2- 172.21.15.29:0/713067036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e58102a60 0x7f5e58199210 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.764+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 >> 172.21.15.29:0/713067036 conn(0x7f5e58079730 msgr2=0x7f5e581043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.765+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 shutdown_connections 2024-09-21T02:36:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:28.765+0000 7f5e5ec5a640 1 -- 172.21.15.29:0/713067036 wait complete. 2024-09-21T02:36:29.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:28 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] up:active 2024-09-21T02:36:29.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:28 smithi066 ceph-mon[33436]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:29.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:28 smithi066 ceph-mon[33436]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:29.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:28 smithi066 ceph-mon[33436]: pgmap v300: 97 pgs: 21 creating+peering, 31 unknown, 45 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 486 B/s wr, 1 op/s 2024-09-21T02:36:29.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:28 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/713067036' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T02:36:29.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:28 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] up:active 2024-09-21T02:36:29.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:28 smithi190 ceph-mon[33871]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:29.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:28 smithi190 ceph-mon[33871]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:29.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:28 smithi190 ceph-mon[33871]: pgmap v300: 97 pgs: 21 creating+peering, 31 unknown, 45 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 486 B/s wr, 1 op/s 2024-09-21T02:36:29.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:28 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/713067036' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T02:36:29.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:28 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] up:active 2024-09-21T02:36:29.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:28 smithi029 ceph-mon[31006]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:29.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:28 smithi029 ceph-mon[31006]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:29.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:28 smithi029 ceph-mon[31006]: pgmap v300: 97 pgs: 21 creating+peering, 31 unknown, 45 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 486 B/s wr, 1 op/s 2024-09-21T02:36:29.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:28 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/713067036' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T02:36:29.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-21T02:36:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:29 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/713067036' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T02:36:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:29 smithi029 ceph-mon[31006]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T02:36:29.914 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:30.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:29 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/713067036' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T02:36:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:29 smithi190 ceph-mon[33871]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T02:36:30.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:29 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/713067036' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T02:36:30.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:29 smithi066 ceph-mon[33436]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.453+0000 7fee2b6b6640 1 -- 172.21.15.29:0/1333646968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 msgr2=0x7fee24107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.453+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/1333646968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 0x7fee24107f50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fee14009900 tx=0x7fee1402a410 comp rx=0 tx=0).stop 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.453+0000 7fee2b6b6640 1 -- 172.21.15.29:0/1333646968 shutdown_connections 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.453+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/1333646968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee24108490 0x7fee2410ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.453+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/1333646968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 0x7fee24107f50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.453+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/1333646968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee24102200 0x7fee24102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.453+0000 7fee2b6b6640 1 -- 172.21.15.29:0/1333646968 >> 172.21.15.29:0/1333646968 conn(0x7fee24079f50 msgr2=0x7fee2407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.454+0000 7fee2b6b6640 1 -- 172.21.15.29:0/1333646968 shutdown_connections 2024-09-21T02:36:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.454+0000 7fee2b6b6640 1 -- 172.21.15.29:0/1333646968 wait complete. 2024-09-21T02:36:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.454+0000 7fee2b6b6640 1 Processor -- start 2024-09-21T02:36:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.454+0000 7fee2b6b6640 1 -- start start 2024-09-21T02:36:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee2b6b6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee24102200 0x7fee2419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee2b6b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 0x7fee2419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee2b6b6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee24108490 0x7fee2419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee2b6b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fee24111780 con 0x7fee24107b40 2024-09-21T02:36:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee2b6b6640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fee24111600 con 0x7fee24102200 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee2b6b6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee24111900 con 0x7fee24108490 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 0x7fee2419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 0x7fee2419a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54866/0 (socket says 172.21.15.29:54866) 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 -- 172.21.15.29:0/797993452 learned_addr learned my addr 172.21.15.29:0/797993452 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee29c53640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee24108490 0x7fee2419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee29452640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee24102200 0x7fee2419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 -- 172.21.15.29:0/797993452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee24108490 msgr2=0x7fee2419fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee24108490 0x7fee2419fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 -- 172.21.15.29:0/797993452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee24102200 msgr2=0x7fee2419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee24102200 0x7fee2419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 -- 172.21.15.29:0/797993452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee241a0270 con 0x7fee24107b40 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee29452640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee24102200 0x7fee2419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:36:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee29c53640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee24108490 0x7fee2419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:36:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee28c51640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 0x7fee2419a960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fee14005940 tx=0x7fee140040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.455+0000 7fee1a7fc640 1 -- 172.21.15.29:0/797993452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1403f030 con 0x7fee24107b40 2024-09-21T02:36:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.456+0000 7fee1a7fc640 1 -- 172.21.15.29:0/797993452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee14003a80 con 0x7fee24107b40 2024-09-21T02:36:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.456+0000 7fee1a7fc640 1 -- 172.21.15.29:0/797993452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14035070 con 0x7fee24107b40 2024-09-21T02:36:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.456+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee241a0500 con 0x7fee24107b40 2024-09-21T02:36:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.456+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee241a0990 con 0x7fee24107b40 2024-09-21T02:36:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.457+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee24102610 con 0x7fee24107b40 2024-09-21T02:36:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.458+0000 7fee1a7fc640 1 -- 172.21.15.29:0/797993452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fee14042030 con 0x7fee24107b40 2024-09-21T02:36:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.459+0000 7fee1a7fc640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee100b65c0 0x7fee100b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.459+0000 7fee29452640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee100b65c0 0x7fee100b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.459+0000 7fee1a7fc640 1 -- 172.21.15.29:0/797993452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 8277+0+0 (secure 0 0 0) 0x7fee14107cb0 con 0x7fee24107b40 2024-09-21T02:36:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.460+0000 7fee29452640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee100b65c0 0x7fee100b8ad0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fee2000baa0 tx=0x7fee20005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.462+0000 7fee1a7fc640 1 -- 172.21.15.29:0/797993452 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fee140ce4b0 con 0x7fee24107b40 2024-09-21T02:36:30.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:30.654+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fee2419b4f0 con 0x7fee24107b40 2024-09-21T02:36:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:30 smithi190 ceph-mon[33871]: pgmap v302: 97 pgs: 2 creating+peering, 95 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-09-21T02:36:31.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:30 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/797993452' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T02:36:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:30 smithi029 ceph-mon[31006]: pgmap v302: 97 pgs: 2 creating+peering, 95 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-09-21T02:36:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:30 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/797993452' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T02:36:31.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:30 smithi066 ceph-mon[33436]: pgmap v302: 97 pgs: 2 creating+peering, 95 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-09-21T02:36:31.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:30 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/797993452' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T02:36:31.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.775+0000 7fee1a7fc640 1 -- 172.21.15.29:0/797993452 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v78) ==== 228+0+0 (secure 0 0 0) 0x7fee2419b4f0 con 0x7fee24107b40 2024-09-21T02:36:31.777 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data_ec' created 2024-09-21T02:36:31.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.778+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee100b65c0 msgr2=0x7fee100b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:31.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.778+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee100b65c0 0x7fee100b8ad0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fee2000baa0 tx=0x7fee20005d90 comp rx=0 tx=0).stop 2024-09-21T02:36:31.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.778+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 msgr2=0x7fee2419a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:31.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.778+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 0x7fee2419a960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fee14005940 tx=0x7fee140040e0 comp rx=0 tx=0).stop 2024-09-21T02:36:31.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.779+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 shutdown_connections 2024-09-21T02:36:31.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.779+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee100b65c0 0x7fee100b8ad0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.779+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee24108490 0x7fee2419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.779+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee24107b40 0x7fee2419a960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.779+0000 7fee2b6b6640 1 --2- 172.21.15.29:0/797993452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee24102200 0x7fee2419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.779+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 >> 172.21.15.29:0/797993452 conn(0x7fee24079f50 msgr2=0x7fee24105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.779+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 shutdown_connections 2024-09-21T02:36:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:31.779+0000 7fee2b6b6640 1 -- 172.21.15.29:0/797993452 wait complete. 2024-09-21T02:36:32.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:31 smithi190 ceph-mon[33871]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T02:36:32.124 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:36:31 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: 2024-09-21T02:36:31.981+0000 7f34d21df640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-21T02:36:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:31 smithi029 ceph-mon[31006]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T02:36:32.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:31 smithi066 ceph-mon[33436]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T02:36:32.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-21T02:36:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:33.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:32 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/797993452' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T02:36:33.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:32 smithi029 ceph-mon[31006]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T02:36:33.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:32 smithi029 ceph-mon[31006]: pgmap v305: 129 pgs: 4 creating+peering, 28 unknown, 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-09-21T02:36:33.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:32 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/797993452' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T02:36:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:32 smithi190 ceph-mon[33871]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T02:36:33.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:32 smithi190 ceph-mon[33871]: pgmap v305: 129 pgs: 4 creating+peering, 28 unknown, 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-09-21T02:36:33.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:32 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/797993452' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T02:36:33.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:32 smithi066 ceph-mon[33436]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T02:36:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:32 smithi066 ceph-mon[33436]: pgmap v305: 129 pgs: 4 creating+peering, 28 unknown, 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 -- 172.21.15.29:0/2944836313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4758103b60 msgr2=0x7f4758103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 --2- 172.21.15.29:0/2944836313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4758103b60 0x7f4758103f70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4754009900 tx=0x7f475402a410 comp rx=0 tx=0).stop 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 -- 172.21.15.29:0/2944836313 shutdown_connections 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 --2- 172.21.15.29:0/2944836313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4758109df0 0x7f4758110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 --2- 172.21.15.29:0/2944836313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47581094a0 0x7f47581098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 --2- 172.21.15.29:0/2944836313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4758103b60 0x7f4758103f70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 -- 172.21.15.29:0/2944836313 >> 172.21.15.29:0/2944836313 conn(0x7f475807b850 msgr2=0x7f475807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 -- 172.21.15.29:0/2944836313 shutdown_connections 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.310+0000 7f476033d640 1 -- 172.21.15.29:0/2944836313 wait complete. 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f476033d640 1 Processor -- start 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f476033d640 1 -- start start 2024-09-21T02:36:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f476033d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4758103b60 0x7f475819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f476033d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47581094a0 0x7f475819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f476033d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4758109df0 0x7f475819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f476033d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4758113960 con 0x7f47581094a0 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f476033d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f47581137e0 con 0x7f4758103b60 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f476033d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4758113ae0 con 0x7f4758109df0 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f475d8d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47581094a0 0x7f475819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f475d8d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47581094a0 0x7f475819a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54898/0 (socket says 172.21.15.29:54898) 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f475e0d9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4758103b60 0x7f475819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.311+0000 7f475d8d8640 1 -- 172.21.15.29:0/1684357698 learned_addr learned my addr 172.21.15.29:0/1684357698 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f475e8da640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4758109df0 0x7f475819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f475d8d8640 1 -- 172.21.15.29:0/1684357698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4758109df0 msgr2=0x7f475819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f475d8d8640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4758109df0 0x7f475819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f475d8d8640 1 -- 172.21.15.29:0/1684357698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4758103b60 msgr2=0x7f475819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f475d8d8640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4758103b60 0x7f475819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f475d8d8640 1 -- 172.21.15.29:0/1684357698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47581a0320 con 0x7f47581094a0 2024-09-21T02:36:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f475d8d8640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47581094a0 0x7f475819a950 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f474800d740 tx=0x7f474800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f474f7fe640 1 -- 172.21.15.29:0/1684357698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748017070 con 0x7f47581094a0 2024-09-21T02:36:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47581a0610 con 0x7f47581094a0 2024-09-21T02:36:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f474f7fe640 1 -- 172.21.15.29:0/1684357698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f474800de50 con 0x7f47581094a0 2024-09-21T02:36:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f474f7fe640 1 -- 172.21.15.29:0/1684357698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474800f620 con 0x7f47581094a0 2024-09-21T02:36:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.312+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47581a6d70 con 0x7f47581094a0 2024-09-21T02:36:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.313+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4758103f70 con 0x7f47581094a0 2024-09-21T02:36:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.316+0000 7f474f7fe640 1 -- 172.21.15.29:0/1684357698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f474801a070 con 0x7f47581094a0 2024-09-21T02:36:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.317+0000 7f474f7fe640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47440b6510 0x7f47440b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.317+0000 7f475e0d9640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47440b6510 0x7f47440b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.317+0000 7f474f7fe640 1 -- 172.21.15.29:0/1684357698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8770+0+0 (secure 0 0 0) 0x7f47480e6fb0 con 0x7f47581094a0 2024-09-21T02:36:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.318+0000 7f475e0d9640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47440b6510 0x7f47440b8a20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f47540057c0 tx=0x7f47540058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.320+0000 7f474f7fe640 1 -- 172.21.15.29:0/1684357698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f47480ad5c0 con 0x7f47581094a0 2024-09-21T02:36:33.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.509+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f475810bc20 con 0x7f47581094a0 2024-09-21T02:36:33.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.787+0000 7f474f7fe640 1 -- 172.21.15.29:0/1684357698 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v80) ==== 169+0+0 (secure 0 0 0) 0x7f47480b19a0 con 0x7f47581094a0 2024-09-21T02:36:33.788 INFO:teuthology.orchestra.run.smithi029.stderr:set pool 4 allow_ec_overwrites to true 2024-09-21T02:36:33.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47440b6510 msgr2=0x7f47440b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47440b6510 0x7f47440b8a20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f47540057c0 tx=0x7f47540058d0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47581094a0 msgr2=0x7f475819a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47581094a0 0x7f475819a950 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f474800d740 tx=0x7f474800dc00 comp rx=0 tx=0).stop 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 shutdown_connections 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47440b6510 0x7f47440b8a20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4758109df0 0x7f475819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47581094a0 0x7f475819a950 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 --2- 172.21.15.29:0/1684357698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4758103b60 0x7f475819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.790+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 >> 172.21.15.29:0/1684357698 conn(0x7f475807b850 msgr2=0x7f47581075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.791+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 shutdown_connections 2024-09-21T02:36:33.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:33.791+0000 7f476033d640 1 -- 172.21.15.29:0/1684357698 wait complete. 2024-09-21T02:36:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:33 smithi190 ceph-mon[33871]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T02:36:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:33 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1684357698' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T02:36:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:33 smithi029 ceph-mon[31006]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T02:36:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1684357698' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T02:36:34.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:33 smithi066 ceph-mon[33436]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T02:36:34.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:33 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1684357698' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T02:36:34.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-21T02:36:34.953 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:35.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:34 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1684357698' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T02:36:35.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:34 smithi029 ceph-mon[31006]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T02:36:35.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:34 smithi029 ceph-mon[31006]: pgmap v308: 129 pgs: 5 creating+peering, 17 unknown, 107 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-09-21T02:36:35.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:34 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1684357698' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T02:36:35.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:34 smithi190 ceph-mon[33871]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T02:36:35.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:34 smithi190 ceph-mon[33871]: pgmap v308: 129 pgs: 5 creating+peering, 17 unknown, 107 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-09-21T02:36:35.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:34 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1684357698' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T02:36:35.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:34 smithi066 ceph-mon[33436]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T02:36:35.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:34 smithi066 ceph-mon[33436]: pgmap v308: 129 pgs: 5 creating+peering, 17 unknown, 107 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-09-21T02:36:35.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 -- 172.21.15.29:0/955201587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334102a80 msgr2=0x7f8334102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:35.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 --2- 172.21.15.29:0/955201587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334102a80 0x7f8334102e90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8324009900 tx=0x7f832402a410 comp rx=0 tx=0).stop 2024-09-21T02:36:35.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 -- 172.21.15.29:0/955201587 shutdown_connections 2024-09-21T02:36:35.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 --2- 172.21.15.29:0/955201587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8334108d10 0x7f833410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 --2- 172.21.15.29:0/955201587 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83341083c0 0x7f83341087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 --2- 172.21.15.29:0/955201587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334102a80 0x7f8334102e90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 -- 172.21.15.29:0/955201587 >> 172.21.15.29:0/955201587 conn(0x7f8334079740 msgr2=0x7f8334079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 -- 172.21.15.29:0/955201587 shutdown_connections 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.606+0000 7f8338baa640 1 -- 172.21.15.29:0/955201587 wait complete. 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.607+0000 7f8338baa640 1 Processor -- start 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.607+0000 7f8338baa640 1 -- start start 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8338baa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8334102a80 0x7f833419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8338baa640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83341083c0 0x7f833419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8338baa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334108d10 0x7f833419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8338baa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8334113950 con 0x7f8334108d10 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8338baa640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f83341137d0 con 0x7f83341083c0 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8338baa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8334113ad0 con 0x7f8334102a80 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334108d10 0x7f833419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334108d10 0x7f833419fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41332/0 (socket says 172.21.15.29:41332) 2024-09-21T02:36:35.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 -- 172.21.15.29:0/2899183325 learned_addr learned my addr 172.21.15.29:0/2899183325 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f833259c640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8334102a80 0x7f833419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8331d9b640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83341083c0 0x7f833419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 -- 172.21.15.29:0/2899183325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8334102a80 msgr2=0x7f833419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8334102a80 0x7f833419a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 -- 172.21.15.29:0/2899183325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83341083c0 msgr2=0x7f833419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83341083c0 0x7f833419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 -- 172.21.15.29:0/2899183325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83341a0210 con 0x7f8334108d10 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.608+0000 7f8332d9d640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334108d10 0x7f833419fba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f832c00eaa0 tx=0x7f832c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.609+0000 7f83237fe640 1 -- 172.21.15.29:0/2899183325 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832c00cdb0 con 0x7f8334108d10 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.609+0000 7f83237fe640 1 -- 172.21.15.29:0/2899183325 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f832c005020 con 0x7f8334108d10 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.609+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83341a0500 con 0x7f8334108d10 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.609+0000 7f83237fe640 1 -- 172.21.15.29:0/2899183325 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832c0116d0 con 0x7f8334108d10 2024-09-21T02:36:35.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.609+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83341a6d20 con 0x7f8334108d10 2024-09-21T02:36:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.610+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8334103d00 con 0x7f8334108d10 2024-09-21T02:36:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.612+0000 7f83237fe640 1 -- 172.21.15.29:0/2899183325 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f832c0051d0 con 0x7f8334108d10 2024-09-21T02:36:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.613+0000 7f83237fe640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f831c0b6540 0x7f831c0b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.613+0000 7f833259c640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f831c0b6540 0x7f831c0b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.613+0000 7f83237fe640 1 -- 172.21.15.29:0/2899183325 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8770+0+0 (secure 0 0 0) 0x7f832c0e7b90 con 0x7f8334108d10 2024-09-21T02:36:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.614+0000 7f833259c640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f831c0b6540 0x7f831c0b8a50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8324000c00 tx=0x7f8324009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.616+0000 7f83237fe640 1 -- 172.21.15.29:0/2899183325 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f832c016030 con 0x7f8334108d10 2024-09-21T02:36:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.843+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f8334102e90 con 0x7f8334108d10 2024-09-21T02:36:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.895+0000 7f83237fe640 1 -- 172.21.15.29:0/2899183325 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f8334102e90 con 0x7f8334108d10 2024-09-21T02:36:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:added data pool 4 to fsmap 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.898+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f831c0b6540 msgr2=0x7f831c0b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.898+0000 7f8338baa640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f831c0b6540 0x7f831c0b8a50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8324000c00 tx=0x7f8324009680 comp rx=0 tx=0).stop 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.898+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334108d10 msgr2=0x7f833419fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.898+0000 7f8338baa640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334108d10 0x7f833419fba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f832c00eaa0 tx=0x7f832c00ef60 comp rx=0 tx=0).stop 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.899+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 shutdown_connections 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.899+0000 7f8338baa640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f831c0b6540 0x7f831c0b8a50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.899+0000 7f8338baa640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8334108d10 0x7f833419fba0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.899+0000 7f8338baa640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83341083c0 0x7f833419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.899+0000 7f8338baa640 1 --2- 172.21.15.29:0/2899183325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8334102a80 0x7f833419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.899+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 >> 172.21.15.29:0/2899183325 conn(0x7f8334079740 msgr2=0x7f8334101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.899+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 shutdown_connections 2024-09-21T02:36:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:35.899+0000 7f8338baa640 1 -- 172.21.15.29:0/2899183325 wait complete. 2024-09-21T02:36:36.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:35 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2899183325' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T02:36:36.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:35 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2899183325' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T02:36:36.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:35 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2899183325' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T02:36:36.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:36:37.113 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:37.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:36 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2899183325' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T02:36:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:36 smithi029 ceph-mon[31006]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T02:36:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:36 smithi029 ceph-mon[31006]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:36 smithi029 ceph-mon[31006]: pgmap v310: 129 pgs: 5 creating+peering, 3 unknown, 121 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:37.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:36 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2899183325' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T02:36:37.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:36 smithi066 ceph-mon[33436]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T02:36:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:36 smithi066 ceph-mon[33436]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:36 smithi066 ceph-mon[33436]: pgmap v310: 129 pgs: 5 creating+peering, 3 unknown, 121 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:37.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:36 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2899183325' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T02:36:37.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:36 smithi190 ceph-mon[33871]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T02:36:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:36 smithi190 ceph-mon[33871]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:36:37.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:36 smithi190 ceph-mon[33871]: pgmap v310: 129 pgs: 5 creating+peering, 3 unknown, 121 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 -- 172.21.15.29:0/4200939947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c108d10 msgr2=0x7f618c10f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 --2- 172.21.15.29:0/4200939947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c108d10 0x7f618c10f540 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6184009ca0 tx=0x7f618402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 -- 172.21.15.29:0/4200939947 shutdown_connections 2024-09-21T02:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 --2- 172.21.15.29:0/4200939947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c108d10 0x7f618c10f540 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 --2- 172.21.15.29:0/4200939947 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f618c1083c0 0x7f618c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 --2- 172.21.15.29:0/4200939947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c102a80 0x7f618c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 -- 172.21.15.29:0/4200939947 >> 172.21.15.29:0/4200939947 conn(0x7f618c079740 msgr2=0x7f618c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 -- 172.21.15.29:0/4200939947 shutdown_connections 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.619+0000 7f61935a9640 1 -- 172.21.15.29:0/4200939947 wait complete. 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.620+0000 7f61935a9640 1 Processor -- start 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.620+0000 7f61935a9640 1 -- start start 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f61935a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c102a80 0x7f618c199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f61935a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c1083c0 0x7f618c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f61935a9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f618c108d10 0x7f618c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f61935a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f618c112880 con 0x7f618c1083c0 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f61935a9640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f618c112700 con 0x7f618c108d10 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f61935a9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f618c112a00 con 0x7f618c102a80 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c1083c0 0x7f618c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c1083c0 0x7f618c1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41350/0 (socket says 172.21.15.29:41350) 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 -- 172.21.15.29:0/3660187752 learned_addr learned my addr 172.21.15.29:0/3660187752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6191b46640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f618c108d10 0x7f618c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6191345640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c102a80 0x7f618c199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 -- 172.21.15.29:0/3660187752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c102a80 msgr2=0x7f618c199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c102a80 0x7f618c199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 -- 172.21.15.29:0/3660187752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f618c108d10 msgr2=0x7f618c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f618c108d10 0x7f618c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 -- 172.21.15.29:0/3660187752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c19f180 con 0x7f618c1083c0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.621+0000 7f6190b44640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c1083c0 0x7f618c1997b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6180005fd0 tx=0x7f618000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.622+0000 7f617e7fc640 1 -- 172.21.15.29:0/3660187752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180017070 con 0x7f618c1083c0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.622+0000 7f617e7fc640 1 -- 172.21.15.29:0/3660187752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6180009c30 con 0x7f618c1083c0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.622+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c19f470 con 0x7f618c1083c0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.622+0000 7f617e7fc640 1 -- 172.21.15.29:0/3660187752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180003c70 con 0x7f618c1083c0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.622+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c1a5bd0 con 0x7f618c1083c0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.623+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f618c103d00 con 0x7f618c1083c0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.624+0000 7f617e7fc640 1 -- 172.21.15.29:0/3660187752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f6180003e10 con 0x7f618c1083c0 2024-09-21T02:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.625+0000 7f617e7fc640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61780b6590 0x7f61780b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.625+0000 7f6191345640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61780b6590 0x7f61780b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.625+0000 7f617e7fc640 1 -- 172.21.15.29:0/3660187752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f61800e7950 con 0x7f618c1083c0 2024-09-21T02:36:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.625+0000 7f6191345640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61780b6590 0x7f61780b8aa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6188005fd0 tx=0x7f6188005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.628+0000 7f617e7fc640 1 -- 172.21.15.29:0/3660187752 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6180020030 con 0x7f618c1083c0 2024-09-21T02:36:37.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.856+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f618c102e90 con 0x7f618c1083c0 2024-09-21T02:36:37.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.857+0000 7f617e7fc640 1 -- 172.21.15.29:0/3660187752 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11836 (secure 0 0 0) 0x7f618c102e90 con 0x7f618c1083c0 2024-09-21T02:36:37.858 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:37.859 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"btime":"2024-09-21T02:36:35:894793+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14781,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14787,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24587,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24599,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24613,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24632,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:36:35.844548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24637,"qdb_cluster":[24637]},"id":1}]} 2024-09-21T02:36:37.859 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2024-09-21T02:36:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.859+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61780b6590 msgr2=0x7f61780b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.859+0000 7f61935a9640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61780b6590 0x7f61780b8aa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6188005fd0 tx=0x7f6188005e20 comp rx=0 tx=0).stop 2024-09-21T02:36:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.859+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c1083c0 msgr2=0x7f618c1997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.859+0000 7f61935a9640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c1083c0 0x7f618c1997b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6180005fd0 tx=0x7f618000dde0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.860+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 shutdown_connections 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.860+0000 7f61935a9640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61780b6590 0x7f61780b8aa0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.860+0000 7f61935a9640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f618c108d10 0x7f618c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.860+0000 7f61935a9640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f618c1083c0 0x7f618c1997b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.860+0000 7f61935a9640 1 --2- 172.21.15.29:0/3660187752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c102a80 0x7f618c199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.860+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 >> 172.21.15.29:0/3660187752 conn(0x7f618c079740 msgr2=0x7f618c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.860+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 shutdown_connections 2024-09-21T02:36:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:37.860+0000 7f61935a9640 1 -- 172.21.15.29:0/3660187752 wait complete. 2024-09-21T02:36:38.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:37 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3660187752' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:38.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:37 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3660187752' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:38.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:37 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3660187752' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:38.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:36:38.987 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:39.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:38 smithi029 ceph-mon[31006]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:39.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:38 smithi066 ceph-mon[33436]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:39.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:38 smithi190 ceph-mon[33871]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:39.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 -- 172.21.15.29:0/2695041348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81020a0 msgr2=0x7f4dc81024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/2695041348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81020a0 0x7f4dc81024b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009900 tx=0x7f4db802a410 comp rx=0 tx=0).stop 2024-09-21T02:36:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 -- 172.21.15.29:0/2695041348 shutdown_connections 2024-09-21T02:36:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/2695041348 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc8106c40 0x7f4dc81106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/2695041348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dc81062f0 0x7f4dc8106700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/2695041348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81020a0 0x7f4dc81024b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 -- 172.21.15.29:0/2695041348 >> 172.21.15.29:0/2695041348 conn(0x7f4dc8079f90 msgr2=0x7f4dc807a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 -- 172.21.15.29:0/2695041348 shutdown_connections 2024-09-21T02:36:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.574+0000 7f4dce9ea640 1 -- 172.21.15.29:0/2695041348 wait complete. 2024-09-21T02:36:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dce9ea640 1 Processor -- start 2024-09-21T02:36:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dce9ea640 1 -- start start 2024-09-21T02:36:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dce9ea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dc81020a0 0x7f4dc8198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dce9ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81062f0 0x7f4dc81993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dce9ea640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc8106c40 0x7f4dc819fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dce9ea640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81139a0 con 0x7f4dc81062f0 2024-09-21T02:36:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dce9ea640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc8113820 con 0x7f4dc8106c40 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dce9ea640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc8113b20 con 0x7f4dc81020a0 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.575+0000 7f4dc77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81062f0 0x7f4dc81993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81062f0 0x7f4dc81993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41360/0 (socket says 172.21.15.29:41360) 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc77fe640 1 -- 172.21.15.29:0/3330413716 learned_addr learned my addr 172.21.15.29:0/3330413716 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dccf87640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc8106c40 0x7f4dc819fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc7fff640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dc81020a0 0x7f4dc8198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc77fe640 1 -- 172.21.15.29:0/3330413716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dc81020a0 msgr2=0x7f4dc8198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc77fe640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dc81020a0 0x7f4dc8198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc77fe640 1 -- 172.21.15.29:0/3330413716 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc8106c40 msgr2=0x7f4dc819fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc77fe640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc8106c40 0x7f4dc819fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc77fe640 1 -- 172.21.15.29:0/3330413716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc81a0410 con 0x7f4dc81062f0 2024-09-21T02:36:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc77fe640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81062f0 0x7f4dc81993b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00c8c0 tx=0x7f4dbc00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc57fa640 1 -- 172.21.15.29:0/3330413716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc007bb0 con 0x7f4dc81062f0 2024-09-21T02:36:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc81a0700 con 0x7f4dc81062f0 2024-09-21T02:36:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dc57fa640 1 -- 172.21.15.29:0/3330413716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4dbc007d50 con 0x7f4dc81062f0 2024-09-21T02:36:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.576+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc81a0c60 con 0x7f4dc81062f0 2024-09-21T02:36:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.577+0000 7f4dc57fa640 1 -- 172.21.15.29:0/3330413716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc018660 con 0x7f4dc81062f0 2024-09-21T02:36:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.578+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db4005290 con 0x7f4dc81062f0 2024-09-21T02:36:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.579+0000 7f4dc57fa640 1 -- 172.21.15.29:0/3330413716 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f4dbc0188e0 con 0x7f4dc81062f0 2024-09-21T02:36:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.580+0000 7f4dc57fa640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4db00b6510 0x7f4db00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.581+0000 7f4dc7fff640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4db00b6510 0x7f4db00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.581+0000 7f4dc57fa640 1 -- 172.21.15.29:0/3330413716 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f4dbc0e7520 con 0x7f4dc81062f0 2024-09-21T02:36:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.581+0000 7f4dc7fff640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4db00b6510 0x7f4db00b8a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4db8005ec0 tx=0x7f4db80058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.584+0000 7f4dc57fa640 1 -- 172.21.15.29:0/3330413716 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4dbc014030 con 0x7f4dc81062f0 2024-09-21T02:36:39.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.773+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4db4005850 con 0x7f4dc81062f0 2024-09-21T02:36:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.774+0000 7f4dc57fa640 1 -- 172.21.15.29:0/3330413716 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24199 (secure 0 0 0) 0x7f4dbc0adc00 con 0x7f4dc81062f0 2024-09-21T02:36:39.776 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:39.776 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:36:35.844398+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:36:39.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4db00b6510 msgr2=0x7f4db00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:39.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4db00b6510 0x7f4db00b8a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4db8005ec0 tx=0x7f4db80058d0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81062f0 msgr2=0x7f4dc81993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81062f0 0x7f4dc81993b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00c8c0 tx=0x7f4dbc00cd80 comp rx=0 tx=0).stop 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 shutdown_connections 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4db00b6510 0x7f4db00b8a20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc8106c40 0x7f4dc819fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc81062f0 0x7f4dc81993b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 --2- 172.21.15.29:0/3330413716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dc81020a0 0x7f4dc8198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 >> 172.21.15.29:0/3330413716 conn(0x7f4dc8079f90 msgr2=0x7f4dc81073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 shutdown_connections 2024-09-21T02:36:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:39.777+0000 7f4dce9ea640 1 -- 172.21.15.29:0/3330413716 wait complete. 2024-09-21T02:36:40.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:39 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3330413716' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:40.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:39 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3330413716' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:40.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:39 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3330413716' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:40.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:36:40.828 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:41.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:40 smithi029 ceph-mon[31006]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:41.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:40 smithi066 ceph-mon[33436]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 -- 172.21.15.29:0/1032518259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda401094a0 msgr2=0x7fda401098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 --2- 172.21.15.29:0/1032518259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda401094a0 0x7fda401098b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fda34009900 tx=0x7fda3402a410 comp rx=0 tx=0).stop 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 -- 172.21.15.29:0/1032518259 shutdown_connections 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 --2- 172.21.15.29:0/1032518259 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda40109df0 0x7fda40110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 --2- 172.21.15.29:0/1032518259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda401094a0 0x7fda401098b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 --2- 172.21.15.29:0/1032518259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda40103b60 0x7fda40103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 -- 172.21.15.29:0/1032518259 >> 172.21.15.29:0/1032518259 conn(0x7fda4007b850 msgr2=0x7fda4007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 -- 172.21.15.29:0/1032518259 shutdown_connections 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.318+0000 7fda4559b640 1 -- 172.21.15.29:0/1032518259 wait complete. 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.319+0000 7fda4559b640 1 Processor -- start 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.319+0000 7fda4559b640 1 -- start start 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda4559b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda40103b60 0x7fda4019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda4559b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda401094a0 0x7fda4019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda4559b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda40109df0 0x7fda4019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:41.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda4559b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda40113960 con 0x7fda40109df0 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda4559b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fda401137e0 con 0x7fda40103b60 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda4559b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fda40113ae0 con 0x7fda401094a0 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda40109df0 0x7fda4019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda40109df0 0x7fda4019fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41370/0 (socket says 172.21.15.29:41370) 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 -- 172.21.15.29:0/2919543018 learned_addr learned my addr 172.21.15.29:0/2919543018 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3effd640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda40103b60 0x7fda4019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3e7fc640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda401094a0 0x7fda4019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 -- 172.21.15.29:0/2919543018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda401094a0 msgr2=0x7fda4019a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda401094a0 0x7fda4019a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 -- 172.21.15.29:0/2919543018 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda40103b60 msgr2=0x7fda4019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda40103b60 0x7fda4019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 -- 172.21.15.29:0/2919543018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda401a03a0 con 0x7fda40109df0 2024-09-21T02:36:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.320+0000 7fda3f7fe640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda40109df0 0x7fda4019fc70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fda38009fd0 tx=0x7fda3800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.321+0000 7fda27fff640 1 -- 172.21.15.29:0/2919543018 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38017070 con 0x7fda40109df0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.321+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda401a0690 con 0x7fda40109df0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.321+0000 7fda27fff640 1 -- 172.21.15.29:0/2919543018 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fda38003e00 con 0x7fda40109df0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.321+0000 7fda27fff640 1 -- 172.21.15.29:0/2919543018 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38010460 con 0x7fda40109df0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.321+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda401a6df0 con 0x7fda40109df0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.322+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda40103f70 con 0x7fda40109df0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.323+0000 7fda27fff640 1 -- 172.21.15.29:0/2919543018 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fda38003970 con 0x7fda40109df0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.324+0000 7fda27fff640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fda280b6510 0x7fda280b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.324+0000 7fda3effd640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fda280b6510 0x7fda280b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.324+0000 7fda27fff640 1 -- 172.21.15.29:0/2919543018 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fda380e75a0 con 0x7fda40109df0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.325+0000 7fda3effd640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fda280b6510 0x7fda280b8a20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda30005d00 tx=0x7fda3000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:41.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.327+0000 7fda27fff640 1 -- 172.21.15.29:0/2919543018 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fda380b1f70 con 0x7fda40109df0 2024-09-21T02:36:41.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:40 smithi190 ceph-mon[33871]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-21T02:36:41.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.515+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda4002e640 con 0x7fda40109df0 2024-09-21T02:36:41.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.516+0000 7fda27fff640 1 -- 172.21.15.29:0/2919543018 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24199 (secure 0 0 0) 0x7fda380b6eb0 con 0x7fda40109df0 2024-09-21T02:36:41.518 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:41.519 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:36:35.844398+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fda280b6510 msgr2=0x7fda280b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fda280b6510 0x7fda280b8a20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda30005d00 tx=0x7fda3000b410 comp rx=0 tx=0).stop 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda40109df0 msgr2=0x7fda4019fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda40109df0 0x7fda4019fc70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fda38009fd0 tx=0x7fda3800b260 comp rx=0 tx=0).stop 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 shutdown_connections 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fda280b6510 0x7fda280b8a20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda40109df0 0x7fda4019fc70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda401094a0 0x7fda4019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 --2- 172.21.15.29:0/2919543018 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda40103b60 0x7fda4019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.518+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 >> 172.21.15.29:0/2919543018 conn(0x7fda4007b850 msgr2=0x7fda40105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.519+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 shutdown_connections 2024-09-21T02:36:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:41.519+0000 7fda4559b640 1 -- 172.21.15.29:0/2919543018 wait complete. 2024-09-21T02:36:42.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:41 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2919543018' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:42.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:41 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2919543018' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:42.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:36:42.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:41 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2919543018' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:42.581 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:43.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:42 smithi029 ceph-mon[31006]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.109+0000 7f2daa232640 1 -- 172.21.15.29:0/641868136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 msgr2=0x7f2da4110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.109+0000 7f2daa232640 1 --2- 172.21.15.29:0/641868136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 0x7f2da4110330 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009ca0 tx=0x7f2d9c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.110+0000 7f2daa232640 1 -- 172.21.15.29:0/641868136 shutdown_connections 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.110+0000 7f2daa232640 1 --2- 172.21.15.29:0/641868136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 0x7f2da4110330 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.110+0000 7f2daa232640 1 --2- 172.21.15.29:0/641868136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da4105380 0x7f2da410d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.110+0000 7f2daa232640 1 --2- 172.21.15.29:0/641868136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2da4104a30 0x7f2da4104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.110+0000 7f2daa232640 1 -- 172.21.15.29:0/641868136 >> 172.21.15.29:0/641868136 conn(0x7f2da40797b0 msgr2=0x7f2da4079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.110+0000 7f2daa232640 1 -- 172.21.15.29:0/641868136 shutdown_connections 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.110+0000 7f2daa232640 1 -- 172.21.15.29:0/641868136 wait complete. 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.110+0000 7f2daa232640 1 Processor -- start 2024-09-21T02:36:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2daa232640 1 -- start start 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2daa232640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da4104a30 0x7f2da419a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2daa232640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2da4105380 0x7f2da419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2daa232640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 0x7f2da419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2daa232640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2da41139c0 con 0x7f2da410de60 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2daa232640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2da4113840 con 0x7f2da4105380 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2daa232640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2da4113b40 con 0x7f2da4104a30 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2da3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 0x7f2da419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2da3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 0x7f2da419fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41394/0 (socket says 172.21.15.29:41394) 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2da3fff640 1 -- 172.21.15.29:0/280312977 learned_addr learned my addr 172.21.15.29:0/280312977 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2da2ffd640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2da4105380 0x7f2da419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.111+0000 7f2da37fe640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da4104a30 0x7f2da419a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da3fff640 1 -- 172.21.15.29:0/280312977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da4104a30 msgr2=0x7f2da419a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:43.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da3fff640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da4104a30 0x7f2da419a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da3fff640 1 -- 172.21.15.29:0/280312977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2da4105380 msgr2=0x7f2da419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da3fff640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2da4105380 0x7f2da419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da3fff640 1 -- 172.21.15.29:0/280312977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da41a0360 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da3fff640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 0x7f2da419fc30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c031040 tx=0x7f2d9c003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da0ff9640 1 -- 172.21.15.29:0/280312977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c032c90 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da41a05f0 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da0ff9640 1 -- 172.21.15.29:0/280312977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2d9c032e30 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2da0ff9640 1 -- 172.21.15.29:0/280312977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c030b30 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.112+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da41a6db0 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.114+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da4104e40 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.115+0000 7f2da0ff9640 1 -- 172.21.15.29:0/280312977 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f2d9c030cd0 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.116+0000 7f2da0ff9640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2d8c0b6510 0x7f2d8c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.117+0000 7f2da0ff9640 1 -- 172.21.15.29:0/280312977 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f2d9c108620 con 0x7f2da410de60 2024-09-21T02:36:43.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.117+0000 7f2da37fe640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2d8c0b6510 0x7f2d8c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:43.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.117+0000 7f2da37fe640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2d8c0b6510 0x7f2d8c0b8a20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2d94005d00 tx=0x7f2d9400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:43.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.121+0000 7f2da0ff9640 1 -- 172.21.15.29:0/280312977 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2d9c0cec70 con 0x7f2da410de60 2024-09-21T02:36:43.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:42 smithi066 ceph-mon[33436]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:43.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.350+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2da4108f40 con 0x7f2da410de60 2024-09-21T02:36:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.351+0000 7f2da0ff9640 1 -- 172.21.15.29:0/280312977 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11836 (secure 0 0 0) 0x7f2d9c0d3050 con 0x7f2da410de60 2024-09-21T02:36:43.353 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:43.353 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"btime":"2024-09-21T02:36:35:894793+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14781,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14787,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24560,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24574,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24587,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24599,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24613,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24632,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:36:35.844548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24637,"qdb_cluster":[24637]},"id":1}]} 2024-09-21T02:36:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2024-09-21T02:36:43.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2d8c0b6510 msgr2=0x7f2d8c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2d8c0b6510 0x7f2d8c0b8a20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2d94005d00 tx=0x7f2d9400b410 comp rx=0 tx=0).stop 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 msgr2=0x7f2da419fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 0x7f2da419fc30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c031040 tx=0x7f2d9c003f80 comp rx=0 tx=0).stop 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 shutdown_connections 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2d8c0b6510 0x7f2d8c0b8a20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2da410de60 0x7f2da419fc30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2da4105380 0x7f2da419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 --2- 172.21.15.29:0/280312977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2da4104a30 0x7f2da419a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 >> 172.21.15.29:0/280312977 conn(0x7f2da40797b0 msgr2=0x7f2da4100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.353+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 shutdown_connections 2024-09-21T02:36:43.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:43.354+0000 7f2daa232640 1 -- 172.21.15.29:0/280312977 wait complete. 2024-09-21T02:36:43.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:42 smithi190 ceph-mon[33871]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:44.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:43 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/280312977' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:44.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:36:44.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:43 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/280312977' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:44.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:43 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/280312977' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:36:44.388 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.897+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/4162888793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98103b60 msgr2=0x7f6a98103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.897+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/4162888793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98103b60 0x7f6a98103f70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009900 tx=0x7f6a9402a410 comp rx=0 tx=0).stop 2024-09-21T02:36:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.897+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/4162888793 shutdown_connections 2024-09-21T02:36:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.897+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/4162888793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a98109df0 0x7f6a98110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.897+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/4162888793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a981094a0 0x7f6a981098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.897+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/4162888793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98103b60 0x7f6a98103f70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.897+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/4162888793 >> 172.21.15.29:0/4162888793 conn(0x7f6a9807b850 msgr2=0x7f6a9807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.898+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/4162888793 shutdown_connections 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.898+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/4162888793 wait complete. 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.898+0000 7f6a9f7b3640 1 Processor -- start 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.898+0000 7f6a9f7b3640 1 -- start start 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9f7b3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a98103b60 0x7f6a9819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9f7b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a981094a0 0x7f6a9819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9f7b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98109df0 0x7f6a9819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9f7b3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6a98113960 con 0x7f6a98109df0 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9f7b3640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6a981137e0 con 0x7f6a98103b60 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9f7b3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6a98113ae0 con 0x7f6a981094a0 2024-09-21T02:36:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98109df0 0x7f6a9819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98109df0 0x7f6a9819fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45830/0 (socket says 172.21.15.29:45830) 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 -- 172.21.15.29:0/2955826077 learned_addr learned my addr 172.21.15.29:0/2955826077 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9cd4e640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a981094a0 0x7f6a9819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9d54f640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a98103b60 0x7f6a9819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 -- 172.21.15.29:0/2955826077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a981094a0 msgr2=0x7f6a9819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a981094a0 0x7f6a9819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 -- 172.21.15.29:0/2955826077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a98103b60 msgr2=0x7f6a9819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a98103b60 0x7f6a9819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 -- 172.21.15.29:0/2955826077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a981a0300 con 0x7f6a98109df0 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9cd4e640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a981094a0 0x7f6a9819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:36:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9d54f640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a98103b60 0x7f6a9819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9dd50640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98109df0 0x7f6a9819fc90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000cce0 tx=0x7f6a9000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a8e7fc640 1 -- 172.21.15.29:0/2955826077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9000f450 con 0x7f6a98109df0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a8e7fc640 1 -- 172.21.15.29:0/2955826077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a9000fa80 con 0x7f6a98109df0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a981a05f0 con 0x7f6a98109df0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a8e7fc640 1 -- 172.21.15.29:0/2955826077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90018ad0 con 0x7f6a98109df0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.899+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a981a6e10 con 0x7f6a98109df0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.900+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a98103f70 con 0x7f6a98109df0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.902+0000 7f6a8e7fc640 1 -- 172.21.15.29:0/2955826077 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f6a9000f5f0 con 0x7f6a98109df0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.903+0000 7f6a8e7fc640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a840b65c0 0x7f6a840b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.903+0000 7f6a9d54f640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a840b65c0 0x7f6a840b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.903+0000 7f6a8e7fc640 1 -- 172.21.15.29:0/2955826077 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f6a900e7db0 con 0x7f6a98109df0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.904+0000 7f6a9d54f640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a840b65c0 0x7f6a840b8ad0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a94006010 tx=0x7f6a94000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:44.906+0000 7f6a8e7fc640 1 -- 172.21.15.29:0/2955826077 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6a90014030 con 0x7f6a98109df0 2024-09-21T02:36:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.100+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a9819b580 con 0x7f6a98109df0 2024-09-21T02:36:45.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:44 smithi029 ceph-mon[31006]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:45.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.101+0000 7f6a8e7fc640 1 -- 172.21.15.29:0/2955826077 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24199 (secure 0 0 0) 0x7f6a9819b580 con 0x7f6a98109df0 2024-09-21T02:36:45.104 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:45.104 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:36:35.844398+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:36:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a840b65c0 msgr2=0x7f6a840b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a840b65c0 0x7f6a840b8ad0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a94006010 tx=0x7f6a94000bc0 comp rx=0 tx=0).stop 2024-09-21T02:36:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98109df0 msgr2=0x7f6a9819fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98109df0 0x7f6a9819fc90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000cce0 tx=0x7f6a9000a620 comp rx=0 tx=0).stop 2024-09-21T02:36:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 shutdown_connections 2024-09-21T02:36:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a840b65c0 0x7f6a840b8ad0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a98109df0 0x7f6a9819fc90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a981094a0 0x7f6a9819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 --2- 172.21.15.29:0/2955826077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a98103b60 0x7f6a9819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 >> 172.21.15.29:0/2955826077 conn(0x7f6a9807b850 msgr2=0x7f6a98107650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 shutdown_connections 2024-09-21T02:36:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:45.104+0000 7f6a9f7b3640 1 -- 172.21.15.29:0/2955826077 wait complete. 2024-09-21T02:36:45.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:44 smithi066 ceph-mon[33436]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:45.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:44 smithi190 ceph-mon[33871]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:46.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-21T02:36:26.562695+0000', 'modified': '2024-09-21T02:36:35.844548+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24637': {'gid': 24637, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.190:6839/3083654868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 3083654868}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 3083654868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24637, 'qdb_cluster': [24637]} 2024-09-21T02:36:46.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-21T02:36:46.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T02:36:46.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:45 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2955826077' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:46.299 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:46.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:45 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2955826077' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:46.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:45 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2955826077' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 -- 172.21.15.29:0/2047345442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db0108d20 msgr2=0x7f3db010f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2047345442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db0108d20 0x7f3db010f550 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009ca0 tx=0x7f3da802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 -- 172.21.15.29:0/2047345442 shutdown_connections 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2047345442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db0108d20 0x7f3db010f550 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2047345442 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3db01083d0 0x7f3db01087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2047345442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db0102ae0 0x7f3db0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 -- 172.21.15.29:0/2047345442 >> 172.21.15.29:0/2047345442 conn(0x7f3db0079760 msgr2=0x7f3db0079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 -- 172.21.15.29:0/2047345442 shutdown_connections 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.805+0000 7f3db5f96640 1 -- 172.21.15.29:0/2047345442 wait complete. 2024-09-21T02:36:46.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3db5f96640 1 Processor -- start 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3db5f96640 1 -- start start 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3db5f96640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3db0102ae0 0x7f3db019b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3db5f96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01083d0 0x7f3db019b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3db5f96640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db0108d20 0x7f3db019bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3db5f96640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3db0112890 con 0x7f3db01083d0 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3db5f96640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3db0112710 con 0x7f3db0102ae0 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3db5f96640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3db0112a10 con 0x7f3db0108d20 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.806+0000 7f3daeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01083d0 0x7f3db019b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01083d0 0x7f3db019b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45846/0 (socket says 172.21.15.29:45846) 2024-09-21T02:36:46.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daeffd640 1 -- 172.21.15.29:0/2515192923 learned_addr learned my addr 172.21.15.29:0/2515192923 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daffff640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db0108d20 0x7f3db019bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3db0102ae0 0x7f3db019b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daeffd640 1 -- 172.21.15.29:0/2515192923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db0108d20 msgr2=0x7f3db019bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daeffd640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db0108d20 0x7f3db019bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daeffd640 1 -- 172.21.15.29:0/2515192923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3db0102ae0 msgr2=0x7f3db019b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daeffd640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3db0102ae0 0x7f3db019b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daeffd640 1 -- 172.21.15.29:0/2515192923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db01a6370 con 0x7f3db01083d0 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daffff640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db0108d20 0x7f3db019bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3db0102ae0 0x7f3db019b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3daeffd640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01083d0 0x7f3db019b680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3da400c970 tx=0x7f3da400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:46.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3dacff9640 1 -- 172.21.15.29:0/2515192923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4014070 con 0x7f3db01083d0 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3db5f96640 1 -- 172.21.15.29:0/2515192923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db01a6660 con 0x7f3db01083d0 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3dacff9640 1 -- 172.21.15.29:0/2515192923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3da4004440 con 0x7f3db01083d0 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3dacff9640 1 -- 172.21.15.29:0/2515192923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4003d40 con 0x7f3db01083d0 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.807+0000 7f3db5f96640 1 -- 172.21.15.29:0/2515192923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db01a6bf0 con 0x7f3db01083d0 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dbc50a740 1 Processor -- start 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dbc50a740 1 -- start start 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dacff9640 1 -- 172.21.15.29:0/2515192923 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f3da4017070 con 0x7f3db01083d0 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 0x55e8e61b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5de50a0 0x55e8e61b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e8e612bc10 con 0x55e8e5ded280 2024-09-21T02:36:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e8e611b040 con 0x55e8e5de50a0 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.810+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e8e611ee50 con 0x55e8e5e0df60 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45862/0 (socket says 172.21.15.29:45862) 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 -- 172.21.15.29:0/4160656684 learned_addr learned my addr 172.21.15.29:0/4160656684 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daffff640 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 0x55e8e61b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5de50a0 0x55e8e61b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 -- 172.21.15.29:0/4160656684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 msgr2=0x55e8e61b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 0x55e8e61b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 -- 172.21.15.29:0/4160656684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5de50a0 msgr2=0x55e8e61b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5de50a0 0x55e8e61b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 -- 172.21.15.29:0/4160656684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8e5e79540 con 0x55e8e5ded280 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daeffd640 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61b4980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3da40ed690 tx=0x7f3da40edb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c804aa42b47c9c61 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3dacff9640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3d980b6510 0x7f3d980b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3d95ffb640 1 -- 172.21.15.29:0/4160656684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da40fa070 con 0x55e8e5ded280 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3daffff640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3d980b6510 0x7f3d980b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3d95ffb640 1 -- 172.21.15.29:0/4160656684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3da40acea0 con 0x55e8e5ded280 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.811+0000 7f3d95ffb640 1 -- 172.21.15.29:0/4160656684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da40acce0 con 0x55e8e5ded280 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dacff9640 1 -- 172.21.15.29:0/2515192923 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f3da40e7440 con 0x7f3db01083d0 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4160656684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 msgr2=0x55e8e61b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3daffff640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3d980b6510 0x7f3d980b8a20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3da8004af0 tx=0x7f3da8005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61b4980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3da40ed690 tx=0x7f3da40edb50 comp rx=0 tx=0).stop 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4160656684 shutdown_connections 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61b4980 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5de50a0 0x55e8e61b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4160656684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 0x55e8e61b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4160656684 >> 172.21.15.29:0/4160656684 conn(0x55e8e5e02350 msgr2=0x55e8e61b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4160656684 shutdown_connections 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.812+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4160656684 wait complete. 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.813+0000 7f3dbc50a740 1 -- start start 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.813+0000 7f3dbc50a740 1 Processor -- start 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.813+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61bf770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.813+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 0x55e8e61bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.813+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 0x55e8e61cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.813+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e8e5f7c5f0 con 0x55e8e5de50a0 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.813+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e8e60b6010 con 0x55e8e5ded280 2024-09-21T02:36:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.813+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e8e61c26e0 con 0x55e8e5e0df60 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3dbc50a740 10 client.0 _ll_register_callbacks cb 0x55e8e6115340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61bf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45872/0 (socket says 172.21.15.29:45872) 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 -- 172.21.15.29:0/2602526787 learned_addr learned my addr 172.21.15.29:0/2602526787 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daeffd640 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 0x55e8e61bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daffff640 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 0x55e8e61cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 -- 172.21.15.29:0/2602526787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 msgr2=0x55e8e61cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 0x55e8e61cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 -- 172.21.15.29:0/2602526787 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 msgr2=0x55e8e61bfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 0x55e8e61bfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 -- 172.21.15.29:0/2602526787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8e60e3a00 con 0x55e8e5de50a0 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61bf770 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009ed0 tx=0x7f3da00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca038867b3d5c41f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3d8effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3d8effd640 1 -- 172.21.15.29:0/2602526787 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0004220 con 0x55e8e5de50a0 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3d8effd640 1 -- 172.21.15.29:0/2602526787 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f3da00043c0 con 0x55e8e5de50a0 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2602526787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da40acea0 con 0x55e8e5de50a0 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.814+0000 7f3d8effd640 1 -- 172.21.15.29:0/2602526787 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3da0012720 con 0x55e8e5de50a0 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3dbc50a740 10 client.14892 fetch_fsmap learned FSMap version 17 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2602526787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55e8e61ce630 con 0x55e8e5de50a0 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3d8effd640 1 -- 172.21.15.29:0/2602526787 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3da40acea0 con 0x55e8e5de50a0 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3d8effd640 1 -- 172.21.15.29:0/2602526787 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f3da000f040 con 0x55e8e5de50a0 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3dbc50a740 10 client.14892 fetch_fsmap finished waiting for FSMap version 17 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3dbc50a740 10 client.14892 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3dbc50a740 20 client.14892 populate_metadata read hostname 'smithi029' 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e8e5dcc8e0 0x55e8e61d2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3dbc50a740 4 client.14892 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2602526787 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8e612db60 con 0x55e8e5dcc8e0 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.815+0000 7f3daeffd640 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e8e5dcc8e0 0x55e8e61d2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.816+0000 7f3daeffd640 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e8e5dcc8e0 0x55e8e61d2160 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f81ea62ce7e8e0a7 server_cookie=ab5d3cfb3f2ace85 in_seq=0 out_seq=0 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.816+0000 7f3d8effd640 10 client.14892 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3d8effd640 1 -- 172.21.15.29:0/2602526787 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e8e612db60 con 0x55e8e5dcc8e0 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3d8effd640 10 client.14892 handle_command_reply: tid=0 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 client.14892 shutdown 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2602526787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 msgr2=0x55e8e61bf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61bf770 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009ed0 tx=0x7f3da00117b0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2602526787 shutdown_connections 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5e0df60 0x55e8e61cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 0x55e8e61bfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e8e5dcc8e0 0x55e8e61d2160 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2602526787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61bf770 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3d8effd640 0 client.14892 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:36:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2602526787 >> 172.21.15.29:0/2602526787 conn(0x55e8e5e02350 msgr2=0x55e8e61b7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2602526787 shutdown_connections 2024-09-21T02:36:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2602526787 wait complete. 2024-09-21T02:36:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.817+0000 7f3dbc50a740 20 client.14892 trim_cache size 0 max 16384 2024-09-21T02:36:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3dbc50a740 1 Processor -- start 2024-09-21T02:36:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3dbc50a740 1 -- start start 2024-09-21T02:36:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5dcc8e0 0x55e8e61bf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5de50a0 0x55e8e61cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e8e61d04a0 con 0x55e8e5ded280 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e8e61d0b70 con 0x55e8e5dcc8e0 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e8e61d1240 con 0x55e8e5de50a0 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45876/0 (socket says 172.21.15.29:45876) 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daffff640 1 -- 172.21.15.29:0/2936677529 learned_addr learned my addr 172.21.15.29:0/2936677529 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daeffd640 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5de50a0 0x55e8e61cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5dcc8e0 0x55e8e61bf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daffff640 1 -- 172.21.15.29:0/2936677529 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5de50a0 msgr2=0x55e8e61cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daffff640 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5de50a0 0x55e8e61cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daffff640 1 -- 172.21.15.29:0/2936677529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5dcc8e0 msgr2=0x55e8e61bf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:36:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daffff640 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5dcc8e0 0x55e8e61bf520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.836+0000 7f3daffff640 1 -- 172.21.15.29:0/2936677529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8e61c0290 con 0x55e8e5ded280 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.837+0000 7f3daffff640 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61cbeb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009ca0 tx=0x7f3da8043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e55c93feadcb06 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.837+0000 7f3d957fa640 1 -- 172.21.15.29:0/2936677529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8003c90 con 0x55e8e5ded280 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.837+0000 7f3d957fa640 1 -- 172.21.15.29:0/2936677529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3da8036050 con 0x55e8e5ded280 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.837+0000 7f3d957fa640 1 -- 172.21.15.29:0/2936677529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8030dd0 con 0x55e8e5ded280 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2936677529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 msgr2=0x55e8e61cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61cbeb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009ca0 tx=0x7f3da8043000 comp rx=0 tx=0).stop 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2936677529 shutdown_connections 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5ded280 0x55e8e61cbeb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5de50a0 0x55e8e61cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/2936677529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5dcc8e0 0x55e8e61bf520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2936677529 >> 172.21.15.29:0/2936677529 conn(0x55e8e5e02350 msgr2=0x55e8e61ba920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2936677529 shutdown_connections 2024-09-21T02:36:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 -- 172.21.15.29:0/2936677529 wait complete. 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 -- start start 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.838+0000 7f3dbc50a740 1 Processor -- start 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5dcc8e0 0x55e8e61dadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61d42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3dbc50a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 0x55e8e61d49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e8e61d1fe0 con 0x55e8e5de50a0 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e8e61d1910 con 0x55e8e5ded280 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3dbc50a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e8e61d26b0 con 0x55e8e5dcc8e0 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3dbc50a740 10 client.0 _ll_register_callbacks cb 0x7f3da00097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61d42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61d42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45892/0 (socket says 172.21.15.29:45892) 2024-09-21T02:36:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 -- 172.21.15.29:0/4137998727 learned_addr learned my addr 172.21.15.29:0/4137998727 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5dcc8e0 0x55e8e61dadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daffff640 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 0x55e8e61d49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 -- 172.21.15.29:0/4137998727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5dcc8e0 msgr2=0x55e8e61dadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5dcc8e0 0x55e8e61dadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 -- 172.21.15.29:0/4137998727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 msgr2=0x55e8e61d49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 0x55e8e61d49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 -- 172.21.15.29:0/4137998727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8e61d9a10 con 0x55e8e5de50a0 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3daeffd640 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61d42a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3da40ad090 tx=0x7f3da4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff8bc76a658d8da server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3d8effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.839+0000 7f3d8effd640 1 -- 172.21.15.29:0/4137998727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4094270 con 0x55e8e5de50a0 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3d8effd640 1 -- 172.21.15.29:0/4137998727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f3da4096490 con 0x55e8e5de50a0 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4137998727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da8036050 con 0x55e8e5de50a0 2024-09-21T02:36:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3d8effd640 1 -- 172.21.15.29:0/4137998727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3da4094440 con 0x55e8e5de50a0 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 10 client.14898 fetch_fsmap learned FSMap version 17 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4137998727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55e8e623d7b0 con 0x55e8e5de50a0 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3d8effd640 1 -- 172.21.15.29:0/4137998727 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3da8036050 con 0x55e8e5de50a0 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3d8effd640 1 -- 172.21.15.29:0/4137998727 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f3da40a9600 con 0x55e8e5de50a0 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 10 client.14898 fetch_fsmap finished waiting for FSMap version 17 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 10 client.14898 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 20 client.14898 populate_metadata read hostname 'smithi029' 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e8e5e0df60 0x55e8e6240b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 4 client.14898 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.840+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4137998727 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "status"}) -- 0x55e8e6129610 con 0x55e8e5e0df60 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e8e5e0df60 0x55e8e6240b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3daf7fe640 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e8e5e0df60 0x55e8e6240b00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d47f85e12e628225 server_cookie=f00e5a3dbc5756a6 in_seq=0 out_seq=0 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3d8effd640 10 client.14898 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:36:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3d8effd640 1 -- 172.21.15.29:0/4137998727 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e8e6129610 con 0x55e8e5e0df60 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3d8effd640 10 client.14898 handle_command_reply: tid=0 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3dbc50a740 1 client.14898 shutdown 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4137998727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 msgr2=0x55e8e61d42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61d42a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3da40ad090 tx=0x7f3da4000bc0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4137998727 shutdown_connections 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e8e5ded280 0x55e8e61d49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.841+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e8e5e0df60 0x55e8e6240b00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e8e5de50a0 0x55e8e61d42a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3dbc50a740 1 --2- 172.21.15.29:0/4137998727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e8e5dcc8e0 0x55e8e61dadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4137998727 >> 172.21.15.29:0/4137998727 conn(0x55e8e5e02350 msgr2=0x55e8e61b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3d8effd640 0 client.14898 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4137998727 shutdown_connections 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3dbc50a740 1 -- 172.21.15.29:0/4137998727 wait complete. 2024-09-21T02:36:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3dbc50a740 20 client.14898 trim_cache size 0 max 16384 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24637, 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 20.108050941999998, 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 17, 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 81, 2024-09-21T02:36:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 80, 2024-09-21T02:36:46.855 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 40.548320144999998 2024-09-21T02:36:46.855 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:36:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 -- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3d980b6510 msgr2=0x7f3d980b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3d980b6510 0x7f3d980b8a20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3da8004af0 tx=0x7f3da8005d80 comp rx=0 tx=0).stop 2024-09-21T02:36:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 -- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01083d0 msgr2=0x7f3db019b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01083d0 0x7f3db019b680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3da400c970 tx=0x7f3da400ce30 comp rx=0 tx=0).stop 2024-09-21T02:36:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 -- 172.21.15.29:0/2515192923 shutdown_connections 2024-09-21T02:36:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3d980b6510 0x7f3d980b8a20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db0108d20 0x7f3db019bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01083d0 0x7f3db019b680 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 --2- 172.21.15.29:0/2515192923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3db0102ae0 0x7f3db019b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.842+0000 7f3db5f96640 1 -- 172.21.15.29:0/2515192923 >> 172.21.15.29:0/2515192923 conn(0x7f3db0079760 msgr2=0x7f3db0104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.843+0000 7f3db5f96640 1 -- 172.21.15.29:0/2515192923 shutdown_connections 2024-09-21T02:36:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:46.843+0000 7f3db5f96640 1 -- 172.21.15.29:0/2515192923 wait complete. 2024-09-21T02:36:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:46 smithi029 ceph-mon[31006]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:47.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:46 smithi066 ceph-mon[33436]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:47.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:46 smithi190 ceph-mon[33871]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:47.815 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T02:36:47.815 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:47.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T02:36:48.081 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 -- 172.21.15.29:0/191293737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 msgr2=0x7fed84110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 --2- 172.21.15.29:0/191293737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 0x7fed84110330 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009ca0 tx=0x7fed7c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 -- 172.21.15.29:0/191293737 shutdown_connections 2024-09-21T02:36:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 --2- 172.21.15.29:0/191293737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 0x7fed84110330 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 --2- 172.21.15.29:0/191293737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed84105380 0x7fed8410d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 --2- 172.21.15.29:0/191293737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed84104a30 0x7fed84104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 -- 172.21.15.29:0/191293737 >> 172.21.15.29:0/191293737 conn(0x7fed84079760 msgr2=0x7fed84079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 -- 172.21.15.29:0/191293737 shutdown_connections 2024-09-21T02:36:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 -- 172.21.15.29:0/191293737 wait complete. 2024-09-21T02:36:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.620+0000 7fed8a231640 1 Processor -- start 2024-09-21T02:36:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed8a231640 1 -- start start 2024-09-21T02:36:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed8a231640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed84104a30 0x7fed8419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed8a231640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed84105380 0x7fed8419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed8a231640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 0x7fed8419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed8a231640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fed841139c0 con 0x7fed8410de60 2024-09-21T02:36:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed8a231640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fed84113840 con 0x7fed84105380 2024-09-21T02:36:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed8a231640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fed84113b40 con 0x7fed84104a30 2024-09-21T02:36:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 0x7fed8419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 0x7fed8419fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45914/0 (socket says 172.21.15.29:45914) 2024-09-21T02:36:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed83fff640 1 -- 172.21.15.29:0/2199391834 learned_addr learned my addr 172.21.15.29:0/2199391834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed837fe640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed84104a30 0x7fed8419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed82ffd640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed84105380 0x7fed8419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed83fff640 1 -- 172.21.15.29:0/2199391834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed84104a30 msgr2=0x7fed8419a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed83fff640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed84104a30 0x7fed8419a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed83fff640 1 -- 172.21.15.29:0/2199391834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed84105380 msgr2=0x7fed8419a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed83fff640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed84105380 0x7fed8419a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed83fff640 1 -- 172.21.15.29:0/2199391834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed841a0300 con 0x7fed8410de60 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.621+0000 7fed82ffd640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed84105380 0x7fed8419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.622+0000 7fed83fff640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 0x7fed8419fbd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fed7c005e30 tx=0x7fed7c0093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.622+0000 7fed80ff9640 1 -- 172.21.15.29:0/2199391834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c032d10 con 0x7fed8410de60 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.622+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed841a0590 con 0x7fed8410de60 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.622+0000 7fed80ff9640 1 -- 172.21.15.29:0/2199391834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed7c032eb0 con 0x7fed8410de60 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.622+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed841a6d50 con 0x7fed8410de60 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.622+0000 7fed80ff9640 1 -- 172.21.15.29:0/2199391834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c030b20 con 0x7fed8410de60 2024-09-21T02:36:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.623+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed8406c1f0 con 0x7fed8410de60 2024-09-21T02:36:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.624+0000 7fed80ff9640 1 -- 172.21.15.29:0/2199391834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fed7c030cc0 con 0x7fed8410de60 2024-09-21T02:36:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.625+0000 7fed80ff9640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed6c0b6510 0x7fed6c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.625+0000 7fed837fe640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed6c0b6510 0x7fed6c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.625+0000 7fed80ff9640 1 -- 172.21.15.29:0/2199391834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fed7c1087e0 con 0x7fed8410de60 2024-09-21T02:36:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.626+0000 7fed837fe640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed6c0b6510 0x7fed6c0b8a20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fed8406c8f0 tx=0x7fed74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.630+0000 7fed80ff9640 1 -- 172.21.15.29:0/2199391834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fed7c0cedd0 con 0x7fed8410de60 2024-09-21T02:36:48.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.860+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fed84108f40 con 0x7fed8410de60 2024-09-21T02:36:48.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.860+0000 7fed80ff9640 1 -- 172.21.15.29:0/2199391834 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fed7c0d31b0 con 0x7fed8410de60 2024-09-21T02:36:48.863 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-21T02:36:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed6c0b6510 msgr2=0x7fed6c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed6c0b6510 0x7fed6c0b8a20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fed8406c8f0 tx=0x7fed74005f90 comp rx=0 tx=0).stop 2024-09-21T02:36:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 msgr2=0x7fed8419fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 0x7fed8419fbd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fed7c005e30 tx=0x7fed7c0093d0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 shutdown_connections 2024-09-21T02:36:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed6c0b6510 0x7fed6c0b8a20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed8410de60 0x7fed8419fbd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed84105380 0x7fed8419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 --2- 172.21.15.29:0/2199391834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed84104a30 0x7fed8419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 >> 172.21.15.29:0/2199391834 conn(0x7fed84079760 msgr2=0x7fed84100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 shutdown_connections 2024-09-21T02:36:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:48.863+0000 7fed8a231640 1 -- 172.21.15.29:0/2199391834 wait complete. 2024-09-21T02:36:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:48 smithi029 ceph-mon[31006]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:48 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2199391834' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:36:49.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:48 smithi066 ceph-mon[33436]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:49.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:48 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2199391834' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:36:49.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:48 smithi190 ceph-mon[33871]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:49.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:48 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2199391834' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:36:49.726 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-21T02:36:49.726 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T02:36:49.727 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-09-21T02:36:49.727 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-21T02:36:49.727 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:49.727 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-21T02:36:49.727 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T02:36:49.727 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T02:36:49.727 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-21T02:36:49.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:49.727 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-09-21T02:36:49.747 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T02:36:49.747 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T02:36:49.747 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T02:36:49.747 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:36:49.747 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-09-21T02:36:49.747 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:36:49.747 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:36:49.747 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42158sec preferred_lft 42158sec 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-09-21T02:36:49.748 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-09-21T02:36:49.749 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T02:36:49.749 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:36:49.749 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:36:49.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-09-21T02:36:49.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-09-21T02:36:49.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-09-21T02:36:49.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T02:36:49.750 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:36:49.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:36:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:36:49.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:49.962 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T02:36:49.988 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-09-21T02:36:49.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:49.990 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-09-21T02:36:50.063 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-09-21T02:36:50.063 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T02:36:50.063 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-21T02:36:50.063 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-21T02:36:50.063 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T02:36:50.064 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:36:50.064 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:36:50.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-21T02:36:50.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-21T02:36:50.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-21T02:36:50.064 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:36:50.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:36:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:36:50.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:50.372 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-09-21T02:36:50.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:50.389 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-09-21T02:36:50.445 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:36:50.445 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:36:50.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:36:50.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-21T02:36:50.445 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:36:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:36:50.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:36:50.565 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-21T02:36:50.565 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:36:50.565 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:36:50.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-21T02:36:50.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T02:36:50.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-21T02:36:50.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-21T02:36:50.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-21T02:36:50.566 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:36:50.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:36:50.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:36:50.697 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:36:50.697 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:36:50.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-09-21T02:36:50.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T02:36:50.697 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:36:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:36:50.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:36:50.776 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-21T02:36:50.776 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T02:36:50.776 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:50.792 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-21T02:36:50.793 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T02:36:50.793 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:50.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-09-21T02:36:50.916 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:36:50.974 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-09-21T02:36:50.974 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-09-21T02:36:50.974 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-09-21T02:36:50.975 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-09-21T02:36:50.975 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-09-21T02:36:50.975 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-09-21T02:36:50.975 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-09-21T02:36:50.975 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-09-21T02:36:50.975 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-09-21T02:36:50.975 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-09-21T02:36:50.976 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/65e94114351b2767a54725ea472225cf3493330a42fa5f3c5371e4195d48759f/merged 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9e18fad01859c31c997a0b7a85bd005aecfa2b6c148bf35f11fb68fe505072b3/merged 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:36:50.977 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae089362fa72be0b3ed38359504dfbbac148a50d357690d53ba08ac2d4afa09f/merged 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae9c5dd64f8d38c950d8e898c4f62b3c1f190b52b226a3e4ecd6dfbadfbe5a71/merged 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/8b2d054545a75eb0f5b709ecf46b8aa54abe476f2b6c79737325518cb5a40565/merged 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/380069ce9b157bf5c5e4b90386b11fcf43e7f52a57f8e087c88af902e8a81235/merged 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b966f50adb464157d55eca43102cf7570966d672cdf1c05427ac3e97c5702866/merged 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/5fde1bef43735c21d5b405c73400f7aeeadbb1648b9d09d11052e12b8473fe7b/merged 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b2ce62941dd7f4417fd6703a397ae2a8497e5ebbac5686951e2e8c8bed23ed87/merged 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ec963248cbfe4528a5340d7a3068cb9447df56f02903ca4cf26504d73424e9c7/merged 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:36:50.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:50.979 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-09-21T02:36:51.031 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-21T02:36:51.031 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-21T02:36:51.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-09-21T02:36:51.101 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:36:51.157 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-09-21T02:36:51.157 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-09-21T02:36:51.157 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-09-21T02:36:51.158 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/65e94114351b2767a54725ea472225cf3493330a42fa5f3c5371e4195d48759f/merged 2024-09-21T02:36:51.159 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9e18fad01859c31c997a0b7a85bd005aecfa2b6c148bf35f11fb68fe505072b3/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae089362fa72be0b3ed38359504dfbbac148a50d357690d53ba08ac2d4afa09f/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae9c5dd64f8d38c950d8e898c4f62b3c1f190b52b226a3e4ecd6dfbadfbe5a71/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/8b2d054545a75eb0f5b709ecf46b8aa54abe476f2b6c79737325518cb5a40565/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/380069ce9b157bf5c5e4b90386b11fcf43e7f52a57f8e087c88af902e8a81235/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b966f50adb464157d55eca43102cf7570966d672cdf1c05427ac3e97c5702866/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/5fde1bef43735c21d5b405c73400f7aeeadbb1648b9d09d11052e12b8473fe7b/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b2ce62941dd7f4417fd6703a397ae2a8497e5ebbac5686951e2e8c8bed23ed87/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ec963248cbfe4528a5340d7a3068cb9447df56f02903ca4cf26504d73424e9c7/merged 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:36:51.160 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:36:51.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:51.161 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-09-21T02:36:51.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:2024-09-21T02:36:51.205+0000 7fa8cb417480 -1 init, newargv = 0x7fa8b80046a0 newargc=15 2024-09-21T02:36:51.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[75238]: starting ceph client 2024-09-21T02:36:51.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[75238]: starting fuse 2024-09-21T02:36:51.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:50 smithi066 ceph-mon[33436]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:51.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:50 smithi190 ceph-mon[33871]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:51.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:50 smithi029 ceph-mon[31006]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:52.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-09-21T02:36:52.349 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-09-21T02:36:52.408 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:36:52.409 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/65e94114351b2767a54725ea472225cf3493330a42fa5f3c5371e4195d48759f/merged 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9e18fad01859c31c997a0b7a85bd005aecfa2b6c148bf35f11fb68fe505072b3/merged 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae089362fa72be0b3ed38359504dfbbac148a50d357690d53ba08ac2d4afa09f/merged 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae9c5dd64f8d38c950d8e898c4f62b3c1f190b52b226a3e4ecd6dfbadfbe5a71/merged 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/8b2d054545a75eb0f5b709ecf46b8aa54abe476f2b6c79737325518cb5a40565/merged 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/380069ce9b157bf5c5e4b90386b11fcf43e7f52a57f8e087c88af902e8a81235/merged 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b966f50adb464157d55eca43102cf7570966d672cdf1c05427ac3e97c5702866/merged 2024-09-21T02:36:52.410 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/5fde1bef43735c21d5b405c73400f7aeeadbb1648b9d09d11052e12b8473fe7b/merged 2024-09-21T02:36:52.411 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b2ce62941dd7f4417fd6703a397ae2a8497e5ebbac5686951e2e8c8bed23ed87/merged 2024-09-21T02:36:52.411 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ec963248cbfe4528a5340d7a3068cb9447df56f02903ca4cf26504d73424e9c7/merged 2024-09-21T02:36:52.411 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-09-21T02:36:52.411 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:36:52.411 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:36:52.411 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:52.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:52.412 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-09-21T02:36:52.470 INFO:teuthology.orchestra.run.smithi029.stdout:120 2024-09-21T02:36:52.470 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo stdin-killer -- python3 -c ' 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> import glob 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> import re 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> import os 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> import subprocess 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> def _find_admin_socket(client_name): 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> files = glob.glob(asok_path) 2024-09-21T02:36:52.471 DEBUG:teuthology.orchestra.run.smithi029:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> # Given a non-glob path, it better be there 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> if "*" not in asok_path: 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> assert(len(files) == 1) 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> return files[0] 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> for f in files: 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> contents = proc_f.read() 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> if mountpoint in contents: 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> return f 2024-09-21T02:36:52.472 DEBUG:teuthology.orchestra.run.smithi029:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-21T02:36:52.473 DEBUG:teuthology.orchestra.run.smithi029:> 2024-09-21T02:36:52.473 DEBUG:teuthology.orchestra.run.smithi029:> print(_find_admin_socket("client.admin")) 2024-09-21T02:36:52.473 DEBUG:teuthology.orchestra.run.smithi029:> ' 2024-09-21T02:36:52.599 INFO:teuthology.orchestra.run.smithi029.stdout:/var/run/ceph/ceph-client.admin.75238.asok 2024-09-21T02:36:52.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:36:52.611 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75238.asok 2024-09-21T02:36:52.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:52.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75238.asok status 2024-09-21T02:36:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:36:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "metadata": { 2024-09-21T02:36:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_sha1": "1843221895adccfb3b692060251dd6a0183bc410", 2024-09-21T02:36:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T02:36:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "entity_id": "admin", 2024-09-21T02:36:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": "smithi029", 2024-09-21T02:36:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-21T02:36:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "pid": "75238", 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "root": "/" 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_count": 0, 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_pinned_count": 0, 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 14910, 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "inst": { 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "name": { 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "client", 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "num": 14910 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:36:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "192.168.0.1:0", 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 4130789189 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "192.168.0.1:0", 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 4130789189 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "inst_str": "client.14910 192.168.0.1:0/4130789189", 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "addr_str": "192.168.0.1:0/4130789189", 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "inode_count": 1, 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_epoch": 17, 2024-09-21T02:36:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch": 81, 2024-09-21T02:36:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch_barrier": 0, 2024-09-21T02:36:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklisted": false, 2024-09-21T02:36:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs" 2024-09-21T02:36:52.761 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:36:52.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:52.767 DEBUG:teuthology.orchestra.run.smithi029:> stat --file-system '--printf=%T 2024-09-21T02:36:52.767 DEBUG:teuthology.orchestra.run.smithi029:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:52.783 INFO:teuthology.orchestra.run.smithi029.stdout:fuseblk 2024-09-21T02:36:52.783 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:52.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:52.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:52.851 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-21T02:36:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T02:36:52.968 INFO:teuthology.orchestra.run.smithi029.stdout:# file: . 2024-09-21T02:36:52.968 INFO:teuthology.orchestra.run.smithi029.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-21T02:36:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:52.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:36:52.974 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:36:52.975 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T02:36:52.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T02:36:53.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:52 smithi066 ceph-mon[33436]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:53.290 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:53.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:52 smithi190 ceph-mon[33871]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:53.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:52 smithi029 ceph-mon[31006]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:36:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 -- 172.21.15.29:0/2576180652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c1083a0 msgr2=0x7f5e9c1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/2576180652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c1083a0 0x7f5e9c1087b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009900 tx=0x7f5e9002a410 comp rx=0 tx=0).stop 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 -- 172.21.15.29:0/2576180652 shutdown_connections 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/2576180652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e9c108cf0 0x7f5e9c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/2576180652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c1083a0 0x7f5e9c1087b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/2576180652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9c102a60 0x7f5e9c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 -- 172.21.15.29:0/2576180652 >> 172.21.15.29:0/2576180652 conn(0x7f5e9c079730 msgr2=0x7f5e9c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 -- 172.21.15.29:0/2576180652 shutdown_connections 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.995+0000 7f5ea11ef640 1 -- 172.21.15.29:0/2576180652 wait complete. 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.996+0000 7f5ea11ef640 1 Processor -- start 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.996+0000 7f5ea11ef640 1 -- start start 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5ea11ef640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e9c102a60 0x7f5e9c19b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5ea11ef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9c1083a0 0x7f5e9c19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5ea11ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c108cf0 0x7f5e9c19bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5ea11ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c112850 con 0x7f5e9c108cf0 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5ea11ef640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c1126d0 con 0x7f5e9c102a60 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5ea11ef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c1129d0 con 0x7f5e9c1083a0 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c108cf0 0x7f5e9c19bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c108cf0 0x7f5e9c19bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45938/0 (socket says 172.21.15.29:45938) 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 -- 172.21.15.29:0/3397275276 learned_addr learned my addr 172.21.15.29:0/3397275276 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9ad9d640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e9c102a60 0x7f5e9c19b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9a59c640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9c1083a0 0x7f5e9c19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 -- 172.21.15.29:0/3397275276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9c1083a0 msgr2=0x7f5e9c19b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9c1083a0 0x7f5e9c19b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 -- 172.21.15.29:0/3397275276 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e9c102a60 msgr2=0x7f5e9c19b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e9c102a60 0x7f5e9c19b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 -- 172.21.15.29:0/3397275276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9c1a64d0 con 0x7f5e9c108cf0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.997+0000 7f5e9b59e640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c108cf0 0x7f5e9c19bcc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400cad0 tx=0x7f5e9400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.998+0000 7f5e83fff640 1 -- 172.21.15.29:0/3397275276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9400ad60 con 0x7f5e9c108cf0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.998+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9c1a67c0 con 0x7f5e9c108cf0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.998+0000 7f5e83fff640 1 -- 172.21.15.29:0/3397275276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e94005020 con 0x7f5e9c108cf0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.998+0000 7f5e83fff640 1 -- 172.21.15.29:0/3397275276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9400f6b0 con 0x7f5e9c108cf0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.998+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9c1a6d50 con 0x7f5e9c108cf0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:53.999+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e9c102e70 con 0x7f5e9c108cf0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.000+0000 7f5e83fff640 1 -- 172.21.15.29:0/3397275276 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f5e940051d0 con 0x7f5e9c108cf0 2024-09-21T02:36:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.001+0000 7f5e83fff640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e840b6540 0x7f5e840b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.001+0000 7f5e83fff640 1 -- 172.21.15.29:0/3397275276 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f5e940e7810 con 0x7f5e9c108cf0 2024-09-21T02:36:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.001+0000 7f5e9ad9d640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e840b6540 0x7f5e840b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.002+0000 7f5e9ad9d640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e840b6540 0x7f5e840b8a50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c06b9c0 tx=0x7f5e8c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.005+0000 7f5e83fff640 1 -- 172.21.15.29:0/3397275276 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5e94014030 con 0x7f5e9c108cf0 2024-09-21T02:36:54.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.208+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5e9c06b270 con 0x7f5e9c108cf0 2024-09-21T02:36:54.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.208+0000 7f5e83fff640 1 -- 172.21.15.29:0/3397275276 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) ==== 81+0+589 (secure 0 0 0) 0x7f5e9c06b270 con 0x7f5e9c108cf0 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3488265372 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3273694575 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1830887273 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/313500774 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/2882183575 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/2882183575 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3241279793 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/595746591 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1927307007 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:54.210 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/595746591 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:54.211 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2024-09-21T02:36:54.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e840b6540 msgr2=0x7f5e840b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:54.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e840b6540 0x7f5e840b8a50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c06b9c0 tx=0x7f5e8c009110 comp rx=0 tx=0).stop 2024-09-21T02:36:54.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c108cf0 msgr2=0x7f5e9c19bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:54.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c108cf0 0x7f5e9c19bcc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400cad0 tx=0x7f5e9400ced0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 shutdown_connections 2024-09-21T02:36:54.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5e840b6540 0x7f5e840b8a50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e9c108cf0 0x7f5e9c19bcc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9c1083a0 0x7f5e9c19b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 --2- 172.21.15.29:0/3397275276 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5e9c102a60 0x7f5e9c19b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:54.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 >> 172.21.15.29:0/3397275276 conn(0x7f5e9c079730 msgr2=0x7f5e9c1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:54.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 shutdown_connections 2024-09-21T02:36:54.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:54.211+0000 7f5ea11ef640 1 -- 172.21.15.29:0/3397275276 wait complete. 2024-09-21T02:36:55.099 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:36:55.099 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T02:36:55.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T02:36:55.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:54 smithi066 ceph-mon[33436]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:55.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:54 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3397275276' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:36:55.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:54 smithi190 ceph-mon[33871]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:55.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:54 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3397275276' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:36:55.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:54 smithi029 ceph-mon[31006]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:55.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:54 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3397275276' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:36:55.424 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 -- 172.21.15.29:0/1216404420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 msgr2=0x7f19c01087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 --2- 172.21.15.29:0/1216404420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 0x7f19c01087c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009900 tx=0x7f19b402a410 comp rx=0 tx=0).stop 2024-09-21T02:36:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 -- 172.21.15.29:0/1216404420 shutdown_connections 2024-09-21T02:36:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 --2- 172.21.15.29:0/1216404420 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c0108d00 0x7f19c010f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 --2- 172.21.15.29:0/1216404420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 0x7f19c01087c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 --2- 172.21.15.29:0/1216404420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c0102a70 0x7f19c0102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 -- 172.21.15.29:0/1216404420 >> 172.21.15.29:0/1216404420 conn(0x7f19c0079730 msgr2=0x7f19c0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 -- 172.21.15.29:0/1216404420 shutdown_connections 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.958+0000 7f19c7474640 1 -- 172.21.15.29:0/1216404420 wait complete. 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.959+0000 7f19c7474640 1 Processor -- start 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.959+0000 7f19c7474640 1 -- start start 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c7474640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c0102a70 0x7f19c019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c7474640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 0x7f19c019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c7474640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c0108d00 0x7f19c019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c7474640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f19c0113940 con 0x7f19c01083b0 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c7474640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f19c01137c0 con 0x7f19c0108d00 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c7474640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f19c0113ac0 con 0x7f19c0102a70 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c4a0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 0x7f19c019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c4a0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 0x7f19c019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37042/0 (socket says 172.21.15.29:37042) 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c4a0f640 1 -- 172.21.15.29:0/2490931968 learned_addr learned my addr 172.21.15.29:0/2490931968 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c5210640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c0102a70 0x7f19c019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c5a11640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c0108d00 0x7f19c019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c4a0f640 1 -- 172.21.15.29:0/2490931968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c0102a70 msgr2=0x7f19c019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c4a0f640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c0102a70 0x7f19c019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c4a0f640 1 -- 172.21.15.29:0/2490931968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c0108d00 msgr2=0x7f19c019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c4a0f640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c0108d00 0x7f19c019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c4a0f640 1 -- 172.21.15.29:0/2490931968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c01a0340 con 0x7f19c01083b0 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c5210640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c0102a70 0x7f19c019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.960+0000 7f19c5a11640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c0108d00 0x7f19c019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.961+0000 7f19c4a0f640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 0x7f19c019a970 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f19b4005940 tx=0x7f19b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.961+0000 7f19b27fc640 1 -- 172.21.15.29:0/2490931968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b403f030 con 0x7f19c01083b0 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.961+0000 7f19b27fc640 1 -- 172.21.15.29:0/2490931968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19b4003c90 con 0x7f19c01083b0 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.961+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19c01a05d0 con 0x7f19c01083b0 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.961+0000 7f19b27fc640 1 -- 172.21.15.29:0/2490931968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4035070 con 0x7f19c01083b0 2024-09-21T02:36:55.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.961+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19c01a6d90 con 0x7f19c01083b0 2024-09-21T02:36:55.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.962+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19c0103cf0 con 0x7f19c01083b0 2024-09-21T02:36:55.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.963+0000 7f19b27fc640 1 -- 172.21.15.29:0/2490931968 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f19b4042030 con 0x7f19c01083b0 2024-09-21T02:36:55.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.964+0000 7f19b27fc640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f19ac0b6510 0x7f19ac0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:55.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.964+0000 7f19c5210640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f19ac0b6510 0x7f19ac0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:55.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.964+0000 7f19b27fc640 1 -- 172.21.15.29:0/2490931968 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f19b4107f20 con 0x7f19c01083b0 2024-09-21T02:36:55.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.965+0000 7f19c5210640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f19ac0b6510 0x7f19ac0b8a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0058c0 tx=0x7f19bc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:55.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:55.967+0000 7f19b27fc640 1 -- 172.21.15.29:0/2490931968 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f19b40ce570 con 0x7f19c01083b0 2024-09-21T02:36:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.170+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f19c0102e80 con 0x7f19c01083b0 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.170+0000 7f19b27fc640 1 -- 172.21.15.29:0/2490931968 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) ==== 81+0+589 (secure 0 0 0) 0x7f19b40d2950 con 0x7f19c01083b0 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3488265372 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3273694575 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1830887273 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/313500774 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/2882183575 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/2882183575 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3241279793 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/595746591 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1927307007 2024-09-22T02:28:37.826591+0000 2024-09-21T02:36:56.172 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/595746591 2024-09-22T02:28:10.590358+0000 2024-09-21T02:36:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f19ac0b6510 msgr2=0x7f19ac0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f19ac0b6510 0x7f19ac0b8a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0058c0 tx=0x7f19bc005850 comp rx=0 tx=0).stop 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 msgr2=0x7f19c019a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 0x7f19c019a970 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f19b4005940 tx=0x7f19b40042f0 comp rx=0 tx=0).stop 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 shutdown_connections 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f19ac0b6510 0x7f19ac0b8a20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c0108d00 0x7f19c019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19c01083b0 0x7f19c019a970 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 --2- 172.21.15.29:0/2490931968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c0102a70 0x7f19c019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:56.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 >> 172.21.15.29:0/2490931968 conn(0x7f19c0079730 msgr2=0x7f19c01044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:56.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 shutdown_connections 2024-09-21T02:36:56.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:56.173+0000 7f19c7474640 1 -- 172.21.15.29:0/2490931968 wait complete. 2024-09-21T02:36:57.090 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi029.front.sepia.ceph.com... 2024-09-21T02:36:57.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:57.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:36:57.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[75238]: fuse finished with error 0 and tester_r 0 2024-09-21T02:36:57.122 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T02:36:57.176 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T02:36:57.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:36:57.176 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-21T02:36:57.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:36:57.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:56 smithi066 ceph-mon[33436]: pgmap v320: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:57.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:56 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2490931968' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:36:57.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:56 smithi029 ceph-mon[31006]: pgmap v320: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:57.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:56 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2490931968' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:36:57.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:56 smithi190 ceph-mon[33871]: pgmap v320: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:57.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:56 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2490931968' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:36:57.502 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:36:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.040+0000 7fa69e105640 1 -- 172.21.15.29:0/547464551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698105b90 msgr2=0x7fa69810f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.040+0000 7fa69e105640 1 --2- 172.21.15.29:0/547464551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698105b90 0x7fa69810f5e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa690009ca0 tx=0x7fa69002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.040+0000 7fa69e105640 1 -- 172.21.15.29:0/547464551 shutdown_connections 2024-09-21T02:36:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.040+0000 7fa69e105640 1 --2- 172.21.15.29:0/547464551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698105b90 0x7fa69810f5e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.040+0000 7fa69e105640 1 --2- 172.21.15.29:0/547464551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa698105240 0x7fa698105650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.040+0000 7fa69e105640 1 --2- 172.21.15.29:0/547464551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa698103270 0x7fa698103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.040+0000 7fa69e105640 1 -- 172.21.15.29:0/547464551 >> 172.21.15.29:0/547464551 conn(0x7fa698079ec0 msgr2=0x7fa69807a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.041+0000 7fa69e105640 1 -- 172.21.15.29:0/547464551 shutdown_connections 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.041+0000 7fa69e105640 1 -- 172.21.15.29:0/547464551 wait complete. 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.041+0000 7fa69e105640 1 Processor -- start 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.041+0000 7fa69e105640 1 -- start start 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa69e105640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698103270 0x7fa69819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa69e105640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa698105240 0x7fa69819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa69e105640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa698105b90 0x7fa69819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa69e105640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa698113a40 con 0x7fa698103270 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa69e105640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa6981138c0 con 0x7fa698105240 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa69e105640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa698113bc0 con 0x7fa698105b90 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698103270 0x7fa69819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698103270 0x7fa69819a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37060/0 (socket says 172.21.15.29:37060) 2024-09-21T02:36:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 -- 172.21.15.29:0/2013273569 learned_addr learned my addr 172.21.15.29:0/2013273569 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa696ffd640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa698105240 0x7fa69819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa697fff640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa698105b90 0x7fa69819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 -- 172.21.15.29:0/2013273569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa698105b90 msgr2=0x7fa69819fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa698105b90 0x7fa69819fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 -- 172.21.15.29:0/2013273569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa698105240 msgr2=0x7fa69819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa698105240 0x7fa69819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 -- 172.21.15.29:0/2013273569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6981a02e0 con 0x7fa698103270 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.042+0000 7fa6977fe640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698103270 0x7fa69819a370 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa68800d770 tx=0x7fa68800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.043+0000 7fa694ff9640 1 -- 172.21.15.29:0/2013273569 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688017070 con 0x7fa698103270 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.043+0000 7fa694ff9640 1 -- 172.21.15.29:0/2013273569 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6880054b0 con 0x7fa698103270 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.043+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6981a05d0 con 0x7fa698103270 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.043+0000 7fa694ff9640 1 -- 172.21.15.29:0/2013273569 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688004c60 con 0x7fa698103270 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.043+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6981a6d30 con 0x7fa698103270 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.044+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa698103680 con 0x7fa698103270 2024-09-21T02:36:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.046+0000 7fa694ff9640 1 -- 172.21.15.29:0/2013273569 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fa688004e00 con 0x7fa698103270 2024-09-21T02:36:58.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.047+0000 7fa694ff9640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa6800b6590 0x7fa6800b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:36:58.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.047+0000 7fa696ffd640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa6800b6590 0x7fa6800b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:36:58.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.047+0000 7fa694ff9640 1 -- 172.21.15.29:0/2013273569 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fa6880e72f0 con 0x7fa698103270 2024-09-21T02:36:58.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.048+0000 7fa696ffd640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa6800b6590 0x7fa6800b8aa0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa698109030 tx=0x7fa68c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:36:58.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.050+0000 7fa694ff9640 1 -- 172.21.15.29:0/2013273569 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa6880b1d40 con 0x7fa698103270 2024-09-21T02:36:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.239+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa69819b4a0 con 0x7fa698103270 2024-09-21T02:36:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.241+0000 7fa694ff9640 1 -- 172.21.15.29:0/2013273569 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24199 (secure 0 0 0) 0x7fa6880b6c80 con 0x7fa698103270 2024-09-21T02:36:58.242 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:36:58.242 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:36:35.844398+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:36:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa6800b6590 msgr2=0x7fa6800b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa6800b6590 0x7fa6800b8aa0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa698109030 tx=0x7fa68c00a2d0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698103270 msgr2=0x7fa69819a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:36:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698103270 0x7fa69819a370 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa68800d770 tx=0x7fa68800dc30 comp rx=0 tx=0).stop 2024-09-21T02:36:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 shutdown_connections 2024-09-21T02:36:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa6800b6590 0x7fa6800b8aa0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa698105b90 0x7fa69819fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa698105240 0x7fa69819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 --2- 172.21.15.29:0/2013273569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698103270 0x7fa69819a370 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:36:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 >> 172.21.15.29:0/2013273569 conn(0x7fa698079ec0 msgr2=0x7fa698106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:36:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 shutdown_connections 2024-09-21T02:36:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:36:58.243+0000 7fa69e105640 1 -- 172.21.15.29:0/2013273569 wait complete. 2024-09-21T02:36:59.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:36:59.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:58 smithi029 ceph-mon[31006]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:59.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:36:58 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2013273569' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:59.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:58 smithi066 ceph-mon[33436]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:59.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:36:58 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2013273569' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:59.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:58 smithi190 ceph-mon[33871]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:36:59.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:36:58 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2013273569' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:36:59.492 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:00.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 -- 172.21.15.29:0/2411187222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d7410de10 msgr2=0x7f5d74110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:00.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2411187222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d7410de10 0x7f5d74110300 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009ca0 tx=0x7f5d6c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 -- 172.21.15.29:0/2411187222 shutdown_connections 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2411187222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d7410de10 0x7f5d74110300 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2411187222 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d74105300 0x7f5d7410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2411187222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d741049b0 0x7f5d74104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 -- 172.21.15.29:0/2411187222 >> 172.21.15.29:0/2411187222 conn(0x7f5d74079730 msgr2=0x7f5d74079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 -- 172.21.15.29:0/2411187222 shutdown_connections 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 -- 172.21.15.29:0/2411187222 wait complete. 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.027+0000 7f5d7b489640 1 Processor -- start 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d7b489640 1 -- start start 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d7b489640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d741049b0 0x7f5d7419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d7b489640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d74105300 0x7f5d7419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d7b489640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d7410de10 0x7f5d7419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d7b489640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74113990 con 0x7f5d741049b0 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d7b489640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74113810 con 0x7f5d7410de10 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d7b489640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74113b10 con 0x7f5d74105300 2024-09-21T02:37:00.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d741049b0 0x7f5d7419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d741049b0 0x7f5d7419a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37080/0 (socket says 172.21.15.29:37080) 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 -- 172.21.15.29:0/2519819954 learned_addr learned my addr 172.21.15.29:0/2519819954 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d78a24640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d74105300 0x7f5d7419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 -- 172.21.15.29:0/2519819954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d74105300 msgr2=0x7f5d7419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d74105300 0x7f5d7419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79a26640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d7410de10 0x7f5d7419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 -- 172.21.15.29:0/2519819954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d7410de10 msgr2=0x7f5d7419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d7410de10 0x7f5d7419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 -- 172.21.15.29:0/2519819954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d741a02d0 con 0x7f5d741049b0 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d78a24640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d74105300 0x7f5d7419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79a26640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d7410de10 0x7f5d7419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.028+0000 7f5d79225640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d741049b0 0x7f5d7419a360 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000d950 tx=0x7f5d7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:00.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.029+0000 7f5d6a7fc640 1 -- 172.21.15.29:0/2519819954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70017070 con 0x7f5d741049b0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.029+0000 7f5d6a7fc640 1 -- 172.21.15.29:0/2519819954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d70009c30 con 0x7f5d741049b0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.029+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d741a05c0 con 0x7f5d741049b0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.029+0000 7f5d6a7fc640 1 -- 172.21.15.29:0/2519819954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7000f620 con 0x7f5d741049b0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.029+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d741a6d20 con 0x7f5d741049b0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.030+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d74104dc0 con 0x7f5d741049b0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.032+0000 7f5d6a7fc640 1 -- 172.21.15.29:0/2519819954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f5d7001a070 con 0x7f5d741049b0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.032+0000 7f5d6a7fc640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d600b6590 0x7f5d600b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.032+0000 7f5d78a24640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d600b6590 0x7f5d600b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.033+0000 7f5d6a7fc640 1 -- 172.21.15.29:0/2519819954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f5d700e6f80 con 0x7f5d741049b0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.033+0000 7f5d78a24640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d600b6590 0x7f5d600b8aa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5d640058c0 tx=0x7f5d64005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:00.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.036+0000 7f5d6a7fc640 1 -- 172.21.15.29:0/2519819954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5d70009de0 con 0x7f5d741049b0 2024-09-21T02:37:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.225+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d741a7040 con 0x7f5d741049b0 2024-09-21T02:37:00.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.227+0000 7f5d6a7fc640 1 -- 172.21.15.29:0/2519819954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24199 (secure 0 0 0) 0x7f5d700ad650 con 0x7f5d741049b0 2024-09-21T02:37:00.228 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:00.228 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:36:35.844398+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:37:00.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d600b6590 msgr2=0x7f5d600b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:00.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d600b6590 0x7f5d600b8aa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5d640058c0 tx=0x7f5d64005850 comp rx=0 tx=0).stop 2024-09-21T02:37:00.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d741049b0 msgr2=0x7f5d7419a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d741049b0 0x7f5d7419a360 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000d950 tx=0x7f5d7000de10 comp rx=0 tx=0).stop 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 shutdown_connections 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d600b6590 0x7f5d600b8aa0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d7410de10 0x7f5d7419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d74105300 0x7f5d7419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 --2- 172.21.15.29:0/2519819954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d741049b0 0x7f5d7419a360 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.230+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 >> 172.21.15.29:0/2519819954 conn(0x7f5d74079730 msgr2=0x7f5d740ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.231+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 shutdown_connections 2024-09-21T02:37:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:00.231+0000 7f5d7b489640 1 -- 172.21.15.29:0/2519819954 wait complete. 2024-09-21T02:37:01.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs set cephfs standby_count_wanted 0 2024-09-21T02:37:01.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:00 smithi066 ceph-mon[33436]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:01.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:00 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2519819954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:01.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:00 smithi190 ceph-mon[33871]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:01.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:00 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2519819954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:00 smithi029 ceph-mon[31006]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:00 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2519819954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:01.453 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:02.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 -- 172.21.15.29:0/1122095888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 msgr2=0x7fe44010d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:02.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 --2- 172.21.15.29:0/1122095888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 0x7fe44010d850 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe428009900 tx=0x7fe42802a410 comp rx=0 tx=0).stop 2024-09-21T02:37:02.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 -- 172.21.15.29:0/1122095888 shutdown_connections 2024-09-21T02:37:02.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 --2- 172.21.15.29:0/1122095888 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe44010de10 0x7fe440110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 --2- 172.21.15.29:0/1122095888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 0x7fe44010d850 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 --2- 172.21.15.29:0/1122095888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4401049b0 0x7fe440104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 -- 172.21.15.29:0/1122095888 >> 172.21.15.29:0/1122095888 conn(0x7fe440079730 msgr2=0x7fe440079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 -- 172.21.15.29:0/1122095888 shutdown_connections 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.031+0000 7fe445eef640 1 -- 172.21.15.29:0/1122095888 wait complete. 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.032+0000 7fe445eef640 1 Processor -- start 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe445eef640 1 -- start start 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe445eef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4401049b0 0x7fe44019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe445eef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 0x7fe44019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe445eef640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe44010de10 0x7fe44019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe445eef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe440113920 con 0x7fe440105300 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe445eef640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe4401137a0 con 0x7fe44010de10 2024-09-21T02:37:02.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe445eef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe440113aa0 con 0x7fe4401049b0 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 0x7fe44019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 0x7fe44019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37100/0 (socket says 172.21.15.29:37100) 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43effd640 1 -- 172.21.15.29:0/3022556194 learned_addr learned my addr 172.21.15.29:0/3022556194 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43f7fe640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4401049b0 0x7fe44019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43ffff640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe44010de10 0x7fe44019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43effd640 1 -- 172.21.15.29:0/3022556194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4401049b0 msgr2=0x7fe44019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43effd640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4401049b0 0x7fe44019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43effd640 1 -- 172.21.15.29:0/3022556194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe44010de10 msgr2=0x7fe44019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43effd640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe44010de10 0x7fe44019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.033+0000 7fe43effd640 1 -- 172.21.15.29:0/3022556194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4401a0340 con 0x7fe440105300 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.034+0000 7fe43effd640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 0x7fe44019a970 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe428005940 tx=0x7fe428005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.034+0000 7fe43cff9640 1 -- 172.21.15.29:0/3022556194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42803f030 con 0x7fe440105300 2024-09-21T02:37:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.034+0000 7fe43cff9640 1 -- 172.21.15.29:0/3022556194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4280043d0 con 0x7fe440105300 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.034+0000 7fe43cff9640 1 -- 172.21.15.29:0/3022556194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428035070 con 0x7fe440105300 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.034+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4401a05d0 con 0x7fe440105300 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.034+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4401a6d90 con 0x7fe440105300 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.035+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe44006b560 con 0x7fe440105300 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.036+0000 7fe43cff9640 1 -- 172.21.15.29:0/3022556194 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fe428042030 con 0x7fe440105300 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.038+0000 7fe43cff9640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe42c0b6510 0x7fe42c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.038+0000 7fe43f7fe640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe42c0b6510 0x7fe42c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.038+0000 7fe43cff9640 1 -- 172.21.15.29:0/3022556194 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fe428107cb0 con 0x7fe440105300 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.038+0000 7fe43f7fe640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe42c0b6510 0x7fe42c0b8a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe44006bc60 tx=0x7fe430009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.041+0000 7fe43cff9640 1 -- 172.21.15.29:0/3022556194 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe4280ce300 con 0x7fe440105300 2024-09-21T02:37:02.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.268+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe440104dc0 con 0x7fe440105300 2024-09-21T02:37:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.983+0000 7fe43cff9640 1 -- 172.21.15.29:0/3022556194 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fe440104dc0 con 0x7fe440105300 2024-09-21T02:37:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe42c0b6510 msgr2=0x7fe42c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe42c0b6510 0x7fe42c0b8a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe44006bc60 tx=0x7fe430009500 comp rx=0 tx=0).stop 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 msgr2=0x7fe44019a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 0x7fe44019a970 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe428005940 tx=0x7fe428005e10 comp rx=0 tx=0).stop 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 shutdown_connections 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe42c0b6510 0x7fe42c0b8a20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe44010de10 0x7fe44019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe440105300 0x7fe44019a970 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 --2- 172.21.15.29:0/3022556194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4401049b0 0x7fe44019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.985+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 >> 172.21.15.29:0/3022556194 conn(0x7fe440079730 msgr2=0x7fe440100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.986+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 shutdown_connections 2024-09-21T02:37:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:02.986+0000 7fe445eef640 1 -- 172.21.15.29:0/3022556194 wait complete. 2024-09-21T02:37:03.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:02 smithi029 ceph-mon[31006]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:03.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:02 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3022556194' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T02:37:03.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:02 smithi066 ceph-mon[33436]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:03.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:02 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3022556194' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T02:37:03.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:02 smithi190 ceph-mon[33871]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:03.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:02 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3022556194' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T02:37:03.925 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-09-21T02:37:03.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs set cephfs max_mds 5 2024-09-21T02:37:04.196 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:03 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3022556194' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T02:37:04.196 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:03 smithi029 ceph-mon[31006]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:37:04.197 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:04.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:03 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3022556194' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T02:37:04.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:03 smithi066 ceph-mon[33436]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:37:04.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:03 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3022556194' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T02:37:04.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:03 smithi190 ceph-mon[33871]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:37:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 -- 172.21.15.29:0/3711352336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 msgr2=0x7fce2c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 --2- 172.21.15.29:0/3711352336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 0x7fce2c103f70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fce28009900 tx=0x7fce2802a410 comp rx=0 tx=0).stop 2024-09-21T02:37:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 -- 172.21.15.29:0/3711352336 shutdown_connections 2024-09-21T02:37:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 --2- 172.21.15.29:0/3711352336 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2c109df0 0x7fce2c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 --2- 172.21.15.29:0/3711352336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce2c1094a0 0x7fce2c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 --2- 172.21.15.29:0/3711352336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 0x7fce2c103f70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 -- 172.21.15.29:0/3711352336 >> 172.21.15.29:0/3711352336 conn(0x7fce2c07b850 msgr2=0x7fce2c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 -- 172.21.15.29:0/3711352336 shutdown_connections 2024-09-21T02:37:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.759+0000 7fce3329f640 1 -- 172.21.15.29:0/3711352336 wait complete. 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3329f640 1 Processor -- start 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3329f640 1 -- start start 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3329f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 0x7fce2c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3329f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2c1094a0 0x7fce2c19a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3329f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce2c109df0 0x7fce2c19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3329f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce2c113960 con 0x7fce2c103b60 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3329f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fce2c1137e0 con 0x7fce2c1094a0 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3329f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fce2c113ae0 con 0x7fce2c109df0 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3103b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 0x7fce2c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3103b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 0x7fce2c19a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45408/0 (socket says 172.21.15.29:45408) 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3103b640 1 -- 172.21.15.29:0/3388142165 learned_addr learned my addr 172.21.15.29:0/3388142165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3083a640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2c1094a0 0x7fce2c19a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3183c640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce2c109df0 0x7fce2c19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3103b640 1 -- 172.21.15.29:0/3388142165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce2c109df0 msgr2=0x7fce2c19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3103b640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce2c109df0 0x7fce2c19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3103b640 1 -- 172.21.15.29:0/3388142165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2c1094a0 msgr2=0x7fce2c19a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3103b640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2c1094a0 0x7fce2c19a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3103b640 1 -- 172.21.15.29:0/3388142165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2c1a03a0 con 0x7fce2c103b60 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.760+0000 7fce3083a640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2c1094a0 0x7fce2c19a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.761+0000 7fce3103b640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 0x7fce2c19a430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fce28009230 tx=0x7fce28004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.761+0000 7fce227fc640 1 -- 172.21.15.29:0/3388142165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2803f030 con 0x7fce2c103b60 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.761+0000 7fce227fc640 1 -- 172.21.15.29:0/3388142165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce28003a80 con 0x7fce2c103b60 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.761+0000 7fce227fc640 1 -- 172.21.15.29:0/3388142165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce28035070 con 0x7fce2c103b60 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.761+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce2c1a0630 con 0x7fce2c103b60 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.761+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce2c1a6df0 con 0x7fce2c103b60 2024-09-21T02:37:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.762+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce2c103f70 con 0x7fce2c103b60 2024-09-21T02:37:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.764+0000 7fce227fc640 1 -- 172.21.15.29:0/3388142165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fce28042030 con 0x7fce2c103b60 2024-09-21T02:37:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.764+0000 7fce227fc640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce180b6590 0x7fce180b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.765+0000 7fce227fc640 1 -- 172.21.15.29:0/3388142165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fce28108140 con 0x7fce2c103b60 2024-09-21T02:37:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.765+0000 7fce3083a640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce180b6590 0x7fce180b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.765+0000 7fce3083a640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce180b6590 0x7fce180b8aa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fce1c0058c0 tx=0x7fce1c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:04.768+0000 7fce227fc640 1 -- 172.21.15.29:0/3388142165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fce280ce790 con 0x7fce2c103b60 2024-09-21T02:37:05.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:05.002+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fce2c10bc20 con 0x7fce2c103b60 2024-09-21T02:37:05.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:04 smithi029 ceph-mon[31006]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:05.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:04 smithi066 ceph-mon[33436]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:05.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:04 smithi190 ceph-mon[33871]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:37:05.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:05.997+0000 7fce227fc640 1 -- 172.21.15.29:0/3388142165 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fce280d2b70 con 0x7fce2c103b60 2024-09-21T02:37:06.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce180b6590 msgr2=0x7fce180b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce180b6590 0x7fce180b8aa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fce1c0058c0 tx=0x7fce1c005850 comp rx=0 tx=0).stop 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 msgr2=0x7fce2c19a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 0x7fce2c19a430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fce28009230 tx=0x7fce28004270 comp rx=0 tx=0).stop 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 shutdown_connections 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce180b6590 0x7fce180b8aa0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce2c109df0 0x7fce2c19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2c1094a0 0x7fce2c19a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 --2- 172.21.15.29:0/3388142165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2c103b60 0x7fce2c19a430 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 >> 172.21.15.29:0/3388142165 conn(0x7fce2c07b850 msgr2=0x7fce2c107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 shutdown_connections 2024-09-21T02:37:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:06.001+0000 7fce3329f640 1 -- 172.21.15.29:0/3388142165 wait complete. 2024-09-21T02:37:06.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:05 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3388142165' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T02:37:06.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:05 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3388142165' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T02:37:06.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:05 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3388142165' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T02:37:06.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs set cephfs allow_standby_replay true 2024-09-21T02:37:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:07.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:07 smithi066 ceph-mon[33436]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T02:37:07.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:07 smithi066 ceph-mon[33436]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T02:37:07.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:07 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3388142165' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T02:37:07.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:07 smithi066 ceph-mon[33436]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T02:37:07.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:07 smithi066 ceph-mon[33436]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:37:07.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:07 smithi066 ceph-mon[33436]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-21T02:37:07.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:07 smithi066 ceph-mon[33436]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-21T02:37:07.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:06 smithi029 ceph-mon[31006]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T02:37:07.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:06 smithi029 ceph-mon[31006]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T02:37:07.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:06 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3388142165' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T02:37:07.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:06 smithi029 ceph-mon[31006]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T02:37:07.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:06 smithi029 ceph-mon[31006]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:37:07.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:07 smithi029 ceph-mon[31006]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-21T02:37:07.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:07 smithi029 ceph-mon[31006]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-21T02:37:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:07 smithi190 ceph-mon[33871]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T02:37:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:07 smithi190 ceph-mon[33871]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T02:37:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:07 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3388142165' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T02:37:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:07 smithi190 ceph-mon[33871]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T02:37:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:07 smithi190 ceph-mon[33871]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T02:37:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:07 smithi190 ceph-mon[33871]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-21T02:37:07.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:07 smithi190 ceph-mon[33871]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-21T02:37:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 -- 172.21.15.29:0/2972305034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61281094c0 msgr2=0x7f61281098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 --2- 172.21.15.29:0/2972305034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61281094c0 0x7f61281098d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f611c009900 tx=0x7f611c02a410 comp rx=0 tx=0).stop 2024-09-21T02:37:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 -- 172.21.15.29:0/2972305034 shutdown_connections 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 --2- 172.21.15.29:0/2972305034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6128109e10 0x7f6128110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 --2- 172.21.15.29:0/2972305034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61281094c0 0x7f61281098d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 --2- 172.21.15.29:0/2972305034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6128103b80 0x7f6128103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 -- 172.21.15.29:0/2972305034 >> 172.21.15.29:0/2972305034 conn(0x7f61280ff610 msgr2=0x7f6128101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 -- 172.21.15.29:0/2972305034 shutdown_connections 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.594+0000 7f612ef41640 1 -- 172.21.15.29:0/2972305034 wait complete. 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.595+0000 7f612ef41640 1 Processor -- start 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.595+0000 7f612ef41640 1 -- start start 2024-09-21T02:37:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ef41640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6128103b80 0x7f612819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ef41640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61281094c0 0x7f612819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ef41640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6128109e10 0x7f612819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ef41640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6128113980 con 0x7f6128103b80 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ef41640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6128113800 con 0x7f6128109e10 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ef41640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6128113b00 con 0x7f61281094c0 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6128103b80 0x7f612819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6128103b80 0x7f612819a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45430/0 (socket says 172.21.15.29:45430) 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 -- 172.21.15.29:0/3410745074 learned_addr learned my addr 172.21.15.29:0/3410745074 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f6127fff640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61281094c0 0x7f612819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612d4de640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6128109e10 0x7f612819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 -- 172.21.15.29:0/3410745074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61281094c0 msgr2=0x7f612819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61281094c0 0x7f612819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 -- 172.21.15.29:0/3410745074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6128109e10 msgr2=0x7f612819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6128109e10 0x7f612819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 -- 172.21.15.29:0/3410745074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61281a03c0 con 0x7f6128103b80 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612d4de640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6128109e10 0x7f612819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.596+0000 7f612ccdd640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6128103b80 0x7f612819a450 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f611800d950 tx=0x7f611800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.597+0000 7f6125ffb640 1 -- 172.21.15.29:0/3410745074 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6118017030 con 0x7f6128103b80 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.597+0000 7f6125ffb640 1 -- 172.21.15.29:0/3410745074 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6118004020 con 0x7f6128103b80 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.597+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61281a06b0 con 0x7f6128103b80 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.597+0000 7f6125ffb640 1 -- 172.21.15.29:0/3410745074 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61180055d0 con 0x7f6128103b80 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.597+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61281a6e10 con 0x7f6128103b80 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.598+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6128103f90 con 0x7f6128103b80 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.600+0000 7f6125ffb640 1 -- 172.21.15.29:0/3410745074 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7f6118009a30 con 0x7f6128103b80 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.600+0000 7f6125ffb640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61100b6590 0x7f61100b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.600+0000 7f6127fff640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61100b6590 0x7f61100b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:07.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.601+0000 7f6125ffb640 1 -- 172.21.15.29:0/3410745074 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f61180e7520 con 0x7f6128103b80 2024-09-21T02:37:07.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.601+0000 7f6127fff640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61100b6590 0x7f61100b8aa0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f611c005940 tx=0x7f611c0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:07.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.604+0000 7f6125ffb640 1 -- 172.21.15.29:0/3410745074 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f61180adbf0 con 0x7f6128103b80 2024-09-21T02:37:07.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:07.835+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f612802e640 con 0x7f6128103b80 2024-09-21T02:37:08.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.017+0000 7f6125ffb640 1 -- 172.21.15.29:0/3410745074 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f61180b1fd0 con 0x7f6128103b80 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61100b6590 msgr2=0x7f61100b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61100b6590 0x7f61100b8aa0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f611c005940 tx=0x7f611c0058d0 comp rx=0 tx=0).stop 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6128103b80 msgr2=0x7f612819a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6128103b80 0x7f612819a450 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f611800d950 tx=0x7f611800de10 comp rx=0 tx=0).stop 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 shutdown_connections 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61100b6590 0x7f61100b8aa0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6128109e10 0x7f612819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61281094c0 0x7f612819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 --2- 172.21.15.29:0/3410745074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6128103b80 0x7f612819a450 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 >> 172.21.15.29:0/3410745074 conn(0x7f61280ff610 msgr2=0x7f6128101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 shutdown_connections 2024-09-21T02:37:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:08.019+0000 7f612ef41640 1 -- 172.21.15.29:0/3410745074 wait complete. 2024-09-21T02:37:08.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:08 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] up:active 2024-09-21T02:37:08.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:08 smithi066 ceph-mon[33436]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T02:37:08.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:08 smithi066 ceph-mon[33436]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-21T02:37:08.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:08 smithi066 ceph-mon[33436]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-21T02:37:08.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:08 smithi066 ceph-mon[33436]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-21T02:37:08.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:08 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3410745074' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T02:37:08.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:08 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] up:active 2024-09-21T02:37:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:08 smithi190 ceph-mon[33871]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T02:37:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:08 smithi190 ceph-mon[33871]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-21T02:37:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:08 smithi190 ceph-mon[33871]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-21T02:37:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:08 smithi190 ceph-mon[33871]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-21T02:37:08.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:08 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3410745074' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T02:37:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:08 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] up:active 2024-09-21T02:37:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:08 smithi029 ceph-mon[31006]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T02:37:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:08 smithi029 ceph-mon[31006]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-21T02:37:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:08 smithi029 ceph-mon[31006]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-21T02:37:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:08 smithi029 ceph-mon[31006]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-21T02:37:08.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:08 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3410745074' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T02:37:08.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs set cephfs session_timeout 300 2024-09-21T02:37:09.088 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:09.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:09 smithi029 ceph-mon[31006]: pgmap v326: 129 pgs: 129 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-09-21T02:37:09.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:09 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3410745074' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T02:37:09.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:09 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] up:active 2024-09-21T02:37:09.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:09 smithi029 ceph-mon[31006]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T02:37:09.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:09 smithi029 ceph-mon[31006]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-21T02:37:09.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:09 smithi029 ceph-mon[31006]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-21T02:37:09.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:09 smithi029 ceph-mon[31006]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-21T02:37:09.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:09 smithi190 ceph-mon[33871]: pgmap v326: 129 pgs: 129 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-09-21T02:37:09.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:09 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3410745074' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T02:37:09.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:09 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] up:active 2024-09-21T02:37:09.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:09 smithi190 ceph-mon[33871]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T02:37:09.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:09 smithi190 ceph-mon[33871]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-21T02:37:09.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:09 smithi190 ceph-mon[33871]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-21T02:37:09.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:09 smithi190 ceph-mon[33871]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-21T02:37:09.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:09 smithi066 ceph-mon[33436]: pgmap v326: 129 pgs: 129 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-09-21T02:37:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:09 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3410745074' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T02:37:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:09 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] up:active 2024-09-21T02:37:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:09 smithi066 ceph-mon[33436]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T02:37:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:09 smithi066 ceph-mon[33436]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-21T02:37:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:09 smithi066 ceph-mon[33436]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-21T02:37:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:09 smithi066 ceph-mon[33436]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-21T02:37:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.620+0000 7fa306139640 1 -- 172.21.15.29:0/3964955852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300102ae0 msgr2=0x7fa300102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 --2- 172.21.15.29:0/3964955852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300102ae0 0x7fa300102ef0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009900 tx=0x7fa2f002a410 comp rx=0 tx=0).stop 2024-09-21T02:37:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 -- 172.21.15.29:0/3964955852 shutdown_connections 2024-09-21T02:37:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 --2- 172.21.15.29:0/3964955852 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa300108d20 0x7fa30010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 --2- 172.21.15.29:0/3964955852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3001083d0 0x7fa3001087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 --2- 172.21.15.29:0/3964955852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300102ae0 0x7fa300102ef0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 -- 172.21.15.29:0/3964955852 >> 172.21.15.29:0/3964955852 conn(0x7fa300079760 msgr2=0x7fa300079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 -- 172.21.15.29:0/3964955852 shutdown_connections 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 -- 172.21.15.29:0/3964955852 wait complete. 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.621+0000 7fa306139640 1 Processor -- start 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa306139640 1 -- start start 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa306139640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa300102ae0 0x7fa30019b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa306139640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3001083d0 0x7fa30019b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa306139640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300108d20 0x7fa30019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa306139640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa3001128a0 con 0x7fa300108d20 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa306139640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa300112720 con 0x7fa300102ae0 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa306139640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa300112a20 con 0x7fa3001083d0 2024-09-21T02:37:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2fffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300108d20 0x7fa30019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2fffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300108d20 0x7fa30019bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45442/0 (socket says 172.21.15.29:45442) 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2fffff640 1 -- 172.21.15.29:0/1514969510 learned_addr learned my addr 172.21.15.29:0/1514969510 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2ff7fe640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa300102ae0 0x7fa30019b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2feffd640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3001083d0 0x7fa30019b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2fffff640 1 -- 172.21.15.29:0/1514969510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3001083d0 msgr2=0x7fa30019b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2fffff640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3001083d0 0x7fa30019b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2fffff640 1 -- 172.21.15.29:0/1514969510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa300102ae0 msgr2=0x7fa30019b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2fffff640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa300102ae0 0x7fa30019b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.622+0000 7fa2fffff640 1 -- 172.21.15.29:0/1514969510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3001a64a0 con 0x7fa300108d20 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.623+0000 7fa2fffff640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300108d20 0x7fa30019bc90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800cad0 tx=0x7fa2f800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.623+0000 7fa2fcff9640 1 -- 172.21.15.29:0/1514969510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f800ad60 con 0x7fa300108d20 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.623+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3001a6790 con 0x7fa300108d20 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.623+0000 7fa2fcff9640 1 -- 172.21.15.29:0/1514969510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2f8005020 con 0x7fa300108d20 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.623+0000 7fa2fcff9640 1 -- 172.21.15.29:0/1514969510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f800f6b0 con 0x7fa300108d20 2024-09-21T02:37:09.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.623+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3001a6d00 con 0x7fa300108d20 2024-09-21T02:37:09.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.624+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa300103d10 con 0x7fa300108d20 2024-09-21T02:37:09.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.625+0000 7fa2fcff9640 1 -- 172.21.15.29:0/1514969510 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7fa2f80051d0 con 0x7fa300108d20 2024-09-21T02:37:09.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.626+0000 7fa2fcff9640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2e80b6540 0x7fa2e80b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:09.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.626+0000 7fa2ff7fe640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2e80b6540 0x7fa2e80b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:09.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.626+0000 7fa2fcff9640 1 -- 172.21.15.29:0/1514969510 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fa2f80e7810 con 0x7fa300108d20 2024-09-21T02:37:09.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.627+0000 7fa2ff7fe640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2e80b6540 0x7fa2e80b8a50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa30006c6a0 tx=0x7fa2f00058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:09.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.630+0000 7fa2fcff9640 1 -- 172.21.15.29:0/1514969510 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa2f8014030 con 0x7fa300108d20 2024-09-21T02:37:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:09.856+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fa300102ef0 con 0x7fa300108d20 2024-09-21T02:37:10.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.035+0000 7fa2fcff9640 1 -- 172.21.15.29:0/1514969510 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7fa2f80ade60 con 0x7fa300108d20 2024-09-21T02:37:10.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2e80b6540 msgr2=0x7fa2e80b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2e80b6540 0x7fa2e80b8a50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa30006c6a0 tx=0x7fa2f00058d0 comp rx=0 tx=0).stop 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300108d20 msgr2=0x7fa30019bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300108d20 0x7fa30019bc90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800cad0 tx=0x7fa2f800ced0 comp rx=0 tx=0).stop 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 shutdown_connections 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2e80b6540 0x7fa2e80b8a50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa300108d20 0x7fa30019bc90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3001083d0 0x7fa30019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 --2- 172.21.15.29:0/1514969510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa300102ae0 0x7fa30019b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.037+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 >> 172.21.15.29:0/1514969510 conn(0x7fa300079760 msgr2=0x7fa300101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:10.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.038+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 shutdown_connections 2024-09-21T02:37:10.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:10.038+0000 7fa306139640 1 -- 172.21.15.29:0/1514969510 wait complete. 2024-09-21T02:37:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:10 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] up:active 2024-09-21T02:37:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:10 smithi029 ceph-mon[31006]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T02:37:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:10 smithi029 ceph-mon[31006]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T02:37:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:10 smithi029 ceph-mon[31006]: Cluster is now healthy 2024-09-21T02:37:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:10 smithi029 ceph-mon[31006]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-21T02:37:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:10 smithi029 ceph-mon[31006]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T02:37:10.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:10 smithi029 ceph-mon[31006]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-21T02:37:10.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:10 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1514969510' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T02:37:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:10 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] up:active 2024-09-21T02:37:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:10 smithi190 ceph-mon[33871]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T02:37:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:10 smithi190 ceph-mon[33871]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T02:37:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:10 smithi190 ceph-mon[33871]: Cluster is now healthy 2024-09-21T02:37:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:10 smithi190 ceph-mon[33871]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-21T02:37:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:10 smithi190 ceph-mon[33871]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T02:37:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:10 smithi190 ceph-mon[33871]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-21T02:37:10.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:10 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1514969510' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T02:37:10.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:10 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] up:active 2024-09-21T02:37:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:10 smithi066 ceph-mon[33436]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T02:37:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:10 smithi066 ceph-mon[33436]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T02:37:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:10 smithi066 ceph-mon[33436]: Cluster is now healthy 2024-09-21T02:37:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:10 smithi066 ceph-mon[33436]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-21T02:37:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:10 smithi066 ceph-mon[33436]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T02:37:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:10 smithi066 ceph-mon[33436]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-21T02:37:10.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:10 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1514969510' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T02:37:10.842 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-09-21T02:37:10.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs subvolumegroup create cephfs qa 2024-09-21T02:37:11.109 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-mon[31006]: pgmap v327: 129 pgs: 129 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 7 op/s 2024-09-21T02:37:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1514969510' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T02:37:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] up:active 2024-09-21T02:37:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-mon[31006]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-21T02:37:11.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:37:11.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:11 smithi190 ceph-mon[33871]: pgmap v327: 129 pgs: 129 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 7 op/s 2024-09-21T02:37:11.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:11 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1514969510' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T02:37:11.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:11 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] up:active 2024-09-21T02:37:11.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:11 smithi190 ceph-mon[33871]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-21T02:37:11.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:11 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:37:11.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:11 smithi066 ceph-mon[33436]: pgmap v327: 129 pgs: 129 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 7 op/s 2024-09-21T02:37:11.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:11 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1514969510' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T02:37:11.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:11 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] up:active 2024-09-21T02:37:11.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:11 smithi066 ceph-mon[33436]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-21T02:37:11.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:11 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:37:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 -- 172.21.15.29:0/947151209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc981031b0 msgr2=0x7efc981035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 --2- 172.21.15.29:0/947151209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc981031b0 0x7efc981035c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efc88009900 tx=0x7efc8802a410 comp rx=0 tx=0).stop 2024-09-21T02:37:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 -- 172.21.15.29:0/947151209 shutdown_connections 2024-09-21T02:37:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 --2- 172.21.15.29:0/947151209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc98105b10 0x7efc9810f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 --2- 172.21.15.29:0/947151209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc981051c0 0x7efc981055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 --2- 172.21.15.29:0/947151209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc981031b0 0x7efc981035c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 -- 172.21.15.29:0/947151209 >> 172.21.15.29:0/947151209 conn(0x7efc98079e40 msgr2=0x7efc9807a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 -- 172.21.15.29:0/947151209 shutdown_connections 2024-09-21T02:37:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.633+0000 7efc9d071640 1 -- 172.21.15.29:0/947151209 wait complete. 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9d071640 1 Processor -- start 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9d071640 1 -- start start 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9d071640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc981031b0 0x7efc9819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9d071640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc981051c0 0x7efc9819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9d071640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc98105b10 0x7efc9819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9d071640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efc98112910 con 0x7efc98105b10 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9d071640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7efc98112790 con 0x7efc981031b0 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9d071640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efc98112a90 con 0x7efc981051c0 2024-09-21T02:37:11.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc98105b10 0x7efc9819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc98105b10 0x7efc9819fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45460/0 (socket says 172.21.15.29:45460) 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.634+0000 7efc9759e640 1 -- 172.21.15.29:0/2212772103 learned_addr learned my addr 172.21.15.29:0/2212772103 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9659c640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc981051c0 0x7efc9819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc96d9d640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc981031b0 0x7efc9819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9759e640 1 -- 172.21.15.29:0/2212772103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc981051c0 msgr2=0x7efc9819aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9759e640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc981051c0 0x7efc9819aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9759e640 1 -- 172.21.15.29:0/2212772103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc981031b0 msgr2=0x7efc9819a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9759e640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc981031b0 0x7efc9819a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9759e640 1 -- 172.21.15.29:0/2212772103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc981a0400 con 0x7efc98105b10 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9659c640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc981051c0 0x7efc9819aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9759e640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc98105b10 0x7efc9819fcd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efc9000eaa0 tx=0x7efc9000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc7ffff640 1 -- 172.21.15.29:0/2212772103 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9000cdb0 con 0x7efc98105b10 2024-09-21T02:37:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc981a06f0 con 0x7efc98105b10 2024-09-21T02:37:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc7ffff640 1 -- 172.21.15.29:0/2212772103 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc90005020 con 0x7efc98105b10 2024-09-21T02:37:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc7ffff640 1 -- 172.21.15.29:0/2212772103 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc900116d0 con 0x7efc98105b10 2024-09-21T02:37:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.635+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc981a6ed0 con 0x7efc98105b10 2024-09-21T02:37:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.636+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc981035c0 con 0x7efc98105b10 2024-09-21T02:37:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.638+0000 7efc7ffff640 1 -- 172.21.15.29:0/2212772103 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 154512+0+0 (secure 0 0 0) 0x7efc90004a40 con 0x7efc98105b10 2024-09-21T02:37:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.639+0000 7efc7ffff640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efc800b6510 0x7efc800b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.639+0000 7efc96d9d640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efc800b6510 0x7efc800b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.639+0000 7efc7ffff640 1 -- 172.21.15.29:0/2212772103 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7efc900e7ad0 con 0x7efc98105b10 2024-09-21T02:37:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.639+0000 7efc96d9d640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efc800b6510 0x7efc800b8a20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efc88004770 tx=0x7efc880058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.642+0000 7efc7ffff640 1 -- 172.21.15.29:0/2212772103 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7efc90016030 con 0x7efc98105b10 2024-09-21T02:37:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.857+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7efc98063800 con 0x7efc800b6510 2024-09-21T02:37:11.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.892+0000 7efc7ffff640 1 -- 172.21.15.29:0/2212772103 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efc98063800 con 0x7efc800b6510 2024-09-21T02:37:11.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efc800b6510 msgr2=0x7efc800b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:11.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efc800b6510 0x7efc800b8a20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efc88004770 tx=0x7efc880058d0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc98105b10 msgr2=0x7efc9819fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc98105b10 0x7efc9819fcd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efc9000eaa0 tx=0x7efc9000ef60 comp rx=0 tx=0).stop 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 shutdown_connections 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efc800b6510 0x7efc800b8a20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc98105b10 0x7efc9819fcd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc981051c0 0x7efc9819aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 --2- 172.21.15.29:0/2212772103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc981031b0 0x7efc9819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 >> 172.21.15.29:0/2212772103 conn(0x7efc98079e40 msgr2=0x7efc98101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 shutdown_connections 2024-09-21T02:37:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:11.894+0000 7efc9d071640 1 -- 172.21.15.29:0/2212772103 wait complete. 2024-09-21T02:37:12.132 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:37:11.863+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:37:12.132 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:37:11.863+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:37:12.132 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:37:11.863+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:37:12.132 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:37:11.863+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:37:12.132 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:37:11 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:37:11.863+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:37:12.374 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:37:11 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: 2024-09-21T02:37:11.985+0000 7f34d21df640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-21T02:37:12.374 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:37:11 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: 2024-09-21T02:37:11.985+0000 7f34d21df640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-09-21T02:37:12.374 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 02:37:11 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: 2024-09-21T02:37:11.985+0000 7f34d21df640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-09-21T02:37:12.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-21T02:37:13.008 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:13.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:13 smithi029 ceph-mon[31006]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:13 smithi029 ceph-mon[31006]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.2 KiB/s wr, 13 op/s 2024-09-21T02:37:13.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:13 smithi029 ceph-mon[31006]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-09-21T02:37:13.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:13 smithi190 ceph-mon[33871]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:13 smithi190 ceph-mon[33871]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.2 KiB/s wr, 13 op/s 2024-09-21T02:37:13.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:13 smithi190 ceph-mon[33871]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-09-21T02:37:13.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:13 smithi066 ceph-mon[33436]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:13 smithi066 ceph-mon[33436]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.2 KiB/s wr, 13 op/s 2024-09-21T02:37:13.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:13 smithi066 ceph-mon[33436]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-09-21T02:37:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.509+0000 7fe4e946f640 1 -- 172.21.15.29:0/2841372012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e41094b0 msgr2=0x7fe4e41098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.509+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2841372012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e41094b0 0x7fe4e41098c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009900 tx=0x7fe4cc02a410 comp rx=0 tx=0).stop 2024-09-21T02:37:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 -- 172.21.15.29:0/2841372012 shutdown_connections 2024-09-21T02:37:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2841372012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4e4109e00 0x7fe4e4110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2841372012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e41094b0 0x7fe4e41098c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2841372012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4e4103bc0 0x7fe4e4103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 -- 172.21.15.29:0/2841372012 >> 172.21.15.29:0/2841372012 conn(0x7fe4e407b910 msgr2=0x7fe4e407bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 -- 172.21.15.29:0/2841372012 shutdown_connections 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 -- 172.21.15.29:0/2841372012 wait complete. 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 Processor -- start 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.510+0000 7fe4e946f640 1 -- start start 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e946f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4e4103bc0 0x7fe4e419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e946f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4e41094b0 0x7fe4e419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e946f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e4109e00 0x7fe4e419fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e946f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4113990 con 0x7fe4e4109e00 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e946f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4113810 con 0x7fe4e41094b0 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e946f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4113b10 con 0x7fe4e4103bc0 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e4109e00 0x7fe4e419fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e4109e00 0x7fe4e419fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45494/0 (socket says 172.21.15.29:45494) 2024-09-21T02:37:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e37fe640 1 -- 172.21.15.29:0/2773585248 learned_addr learned my addr 172.21.15.29:0/2773585248 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e2ffd640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4e4103bc0 0x7fe4e419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e27fc640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4e41094b0 0x7fe4e419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e37fe640 1 -- 172.21.15.29:0/2773585248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4e4103bc0 msgr2=0x7fe4e419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e37fe640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4e4103bc0 0x7fe4e419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e37fe640 1 -- 172.21.15.29:0/2773585248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4e41094b0 msgr2=0x7fe4e419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e37fe640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4e41094b0 0x7fe4e419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.511+0000 7fe4e37fe640 1 -- 172.21.15.29:0/2773585248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e41a02f0 con 0x7fe4e4109e00 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.512+0000 7fe4e37fe640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e4109e00 0x7fe4e419fc80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400cce0 tx=0x7fe4d400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.512+0000 7fe4dbfff640 1 -- 172.21.15.29:0/2773585248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d400f450 con 0x7fe4e4109e00 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.512+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e41a05e0 con 0x7fe4e4109e00 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.512+0000 7fe4dbfff640 1 -- 172.21.15.29:0/2773585248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4d400fa80 con 0x7fe4e4109e00 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.512+0000 7fe4dbfff640 1 -- 172.21.15.29:0/2773585248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d40189c0 con 0x7fe4e4109e00 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.512+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e41a6e00 con 0x7fe4e4109e00 2024-09-21T02:37:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.513+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e4104df0 con 0x7fe4e4109e00 2024-09-21T02:37:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.515+0000 7fe4dbfff640 1 -- 172.21.15.29:0/2773585248 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fe4d400fc30 con 0x7fe4e4109e00 2024-09-21T02:37:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.516+0000 7fe4dbfff640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4d00b65d0 0x7fe4d00b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.516+0000 7fe4e2ffd640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4d00b65d0 0x7fe4d00b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.516+0000 7fe4dbfff640 1 -- 172.21.15.29:0/2773585248 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fe4d40e7d80 con 0x7fe4e4109e00 2024-09-21T02:37:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.517+0000 7fe4e2ffd640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4d00b65d0 0x7fe4d00b8ae0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009990 tx=0x7fe4dc009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.519+0000 7fe4dbfff640 1 -- 172.21.15.29:0/2773585248 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe4d4056710 con 0x7fe4e4109e00 2024-09-21T02:37:13.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.737+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe4e410bc30 con 0x7fe4d00b65d0 2024-09-21T02:37:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.768+0000 7fe4dbfff640 1 -- 172.21.15.29:0/2773585248 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe4e410bc30 con 0x7fe4d00b65d0 2024-09-21T02:37:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4d00b65d0 msgr2=0x7fe4d00b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4d00b65d0 0x7fe4d00b8ae0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009990 tx=0x7fe4dc009110 comp rx=0 tx=0).stop 2024-09-21T02:37:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e4109e00 msgr2=0x7fe4e419fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e4109e00 0x7fe4e419fc80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400cce0 tx=0x7fe4d400a620 comp rx=0 tx=0).stop 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 shutdown_connections 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4d00b65d0 0x7fe4d00b8ae0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e4109e00 0x7fe4e419fc80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4e41094b0 0x7fe4e419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 --2- 172.21.15.29:0/2773585248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4e4103bc0 0x7fe4e419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 >> 172.21.15.29:0/2773585248 conn(0x7fe4e407b910 msgr2=0x7fe4e41075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 shutdown_connections 2024-09-21T02:37:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:13.770+0000 7fe4e946f640 1 -- 172.21.15.29:0/2773585248 wait complete. 2024-09-21T02:37:14.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-21T02:37:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:15.287 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:15 smithi029 ceph-mon[31006]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:15.287 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:15 smithi029 ceph-mon[31006]: pgmap v329: 129 pgs: 129 active+clean; 593 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.3 KiB/s wr, 18 op/s 2024-09-21T02:37:15.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:15 smithi190 ceph-mon[33871]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:15.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:15 smithi190 ceph-mon[33871]: pgmap v329: 129 pgs: 129 active+clean; 593 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.3 KiB/s wr, 18 op/s 2024-09-21T02:37:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 -- 172.21.15.29:0/942597209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45041094a0 msgr2=0x7f45041098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 --2- 172.21.15.29:0/942597209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45041094a0 0x7f45041098b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009900 tx=0x7f44f402a410 comp rx=0 tx=0).stop 2024-09-21T02:37:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 -- 172.21.15.29:0/942597209 shutdown_connections 2024-09-21T02:37:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 --2- 172.21.15.29:0/942597209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4504109df0 0x7f4504110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 --2- 172.21.15.29:0/942597209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45041094a0 0x7f45041098b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 --2- 172.21.15.29:0/942597209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4504103b60 0x7f4504103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 -- 172.21.15.29:0/942597209 >> 172.21.15.29:0/942597209 conn(0x7f450407b850 msgr2=0x7f450407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 -- 172.21.15.29:0/942597209 shutdown_connections 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.383+0000 7f450c3cb640 1 -- 172.21.15.29:0/942597209 wait complete. 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450c3cb640 1 Processor -- start 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450c3cb640 1 -- start start 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450c3cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4504103b60 0x7f450419a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450c3cb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45041094a0 0x7f450419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450c3cb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4504109df0 0x7f450419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450c3cb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4504113960 con 0x7f4504103b60 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450c3cb640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f45041137e0 con 0x7f45041094a0 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450c3cb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4504113ae0 con 0x7f4504109df0 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450a167640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4504103b60 0x7f450419a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450a167640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4504103b60 0x7f450419a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42086/0 (socket says 172.21.15.29:42086) 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.384+0000 7f450a167640 1 -- 172.21.15.29:0/2256959688 learned_addr learned my addr 172.21.15.29:0/2256959688 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f4509966640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45041094a0 0x7f450419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450a968640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4504109df0 0x7f450419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450a167640 1 -- 172.21.15.29:0/2256959688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4504109df0 msgr2=0x7f450419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450a167640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4504109df0 0x7f450419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450a167640 1 -- 172.21.15.29:0/2256959688 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45041094a0 msgr2=0x7f450419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450a167640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45041094a0 0x7f450419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450a167640 1 -- 172.21.15.29:0/2256959688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45041a0360 con 0x7f4504103b60 2024-09-21T02:37:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f4509966640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45041094a0 0x7f450419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450a167640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4504103b60 0x7f450419a3f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f450000d770 tx=0x7f450000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f44fb7fe640 1 -- 172.21.15.29:0/2256959688 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500017070 con 0x7f4504103b60 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45041a0650 con 0x7f4504103b60 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f44fb7fe640 1 -- 172.21.15.29:0/2256959688 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45000054b0 con 0x7f4504103b60 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f44fb7fe640 1 -- 172.21.15.29:0/2256959688 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500004c60 con 0x7f4504103b60 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.385+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45041a6db0 con 0x7f4504103b60 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.386+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4504103f70 con 0x7f4504103b60 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.388+0000 7f44fb7fe640 1 -- 172.21.15.29:0/2256959688 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f4500004e00 con 0x7f4504103b60 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.389+0000 7f44fb7fe640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b65a0 0x7f44f00b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.389+0000 7f44fb7fe640 1 -- 172.21.15.29:0/2256959688 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f45000e75b0 con 0x7f4504103b60 2024-09-21T02:37:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.389+0000 7f4509966640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b65a0 0x7f44f00b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.389+0000 7f4509966640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b65a0 0x7f44f00b8ab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f44f40057c0 tx=0x7f44f4009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.392+0000 7f44fb7fe640 1 -- 172.21.15.29:0/2256959688 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f45000adaa0 con 0x7f4504103b60 2024-09-21T02:37:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:15 smithi066 ceph-mon[33436]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:15 smithi066 ceph-mon[33436]: pgmap v329: 129 pgs: 129 active+clean; 593 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.3 KiB/s wr, 18 op/s 2024-09-21T02:37:15.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.611+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f450410bc40 con 0x7f44f00b65a0 2024-09-21T02:37:15.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.618+0000 7f44fb7fe640 1 -- 172.21.15.29:0/2256959688 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f450410bc40 con 0x7f44f00b65a0 2024-09-21T02:37:15.619 INFO:teuthology.orchestra.run.smithi029.stdout:/volumes/qa/sv_0/2a51c03c-c7d2-43c6-8134-7298b740d3a8 2024-09-21T02:37:15.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b65a0 msgr2=0x7f44f00b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:15.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b65a0 0x7f44f00b8ab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f44f40057c0 tx=0x7f44f4009680 comp rx=0 tx=0).stop 2024-09-21T02:37:15.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4504103b60 msgr2=0x7f450419a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:15.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4504103b60 0x7f450419a3f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f450000d770 tx=0x7f450000dc30 comp rx=0 tx=0).stop 2024-09-21T02:37:15.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 shutdown_connections 2024-09-21T02:37:15.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b65a0 0x7f44f00b8ab0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4504109df0 0x7f450419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45041094a0 0x7f450419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 --2- 172.21.15.29:0/2256959688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4504103b60 0x7f450419a3f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:15.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 >> 172.21.15.29:0/2256959688 conn(0x7f450407b850 msgr2=0x7f45041075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:15.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 shutdown_connections 2024-09-21T02:37:15.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:15.620+0000 7f450c3cb640 1 -- 172.21.15.29:0/2256959688 wait complete. 2024-09-21T02:37:16.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:16 smithi029 ceph-mon[31006]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:16.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:16 smithi190 ceph-mon[33871]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:16.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-21T02:37:16.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:16 smithi066 ceph-mon[33436]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:16.642 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 -- 172.21.15.29:0/950934293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 msgr2=0x7f7964110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 --2- 172.21.15.29:0/950934293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 0x7f7964110300 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f795c009900 tx=0x7f795c02a410 comp rx=0 tx=0).stop 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 -- 172.21.15.29:0/950934293 shutdown_connections 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 --2- 172.21.15.29:0/950934293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 0x7f7964110300 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 --2- 172.21.15.29:0/950934293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7964105300 0x7f796410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 --2- 172.21.15.29:0/950934293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79641049b0 0x7f7964104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 -- 172.21.15.29:0/950934293 >> 172.21.15.29:0/950934293 conn(0x7f7964079730 msgr2=0x7f7964079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 -- 172.21.15.29:0/950934293 shutdown_connections 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.198+0000 7f7968d4f640 1 -- 172.21.15.29:0/950934293 wait complete. 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.199+0000 7f7968d4f640 1 Processor -- start 2024-09-21T02:37:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.199+0000 7f7968d4f640 1 -- start start 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.199+0000 7f7968d4f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79641049b0 0x7f796419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.199+0000 7f7968d4f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7964105300 0x7f796419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7968d4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 0x7f796419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7968d4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7964113940 con 0x7f796410de10 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7968d4f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f79641137c0 con 0x7f79641049b0 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7968d4f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7964113ac0 con 0x7f7964105300 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 0x7f796419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 0x7f796419fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42108/0 (socket says 172.21.15.29:42108) 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 -- 172.21.15.29:0/3530039165 learned_addr learned my addr 172.21.15.29:0/3530039165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f796259c640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79641049b0 0x7f796419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7961d9b640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7964105300 0x7f796419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 -- 172.21.15.29:0/3530039165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7964105300 msgr2=0x7f796419a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7964105300 0x7f796419a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 -- 172.21.15.29:0/3530039165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79641049b0 msgr2=0x7f796419a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79641049b0 0x7f796419a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 -- 172.21.15.29:0/3530039165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79641a02e0 con 0x7f796410de10 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f7962d9d640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 0x7f796419fbb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f795c005ab0 tx=0x7f795c0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.200+0000 7f79537fe640 1 -- 172.21.15.29:0/3530039165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c03f030 con 0x7f796410de10 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.201+0000 7f79537fe640 1 -- 172.21.15.29:0/3530039165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f795c003a80 con 0x7f796410de10 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.201+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79641a0570 con 0x7f796410de10 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.201+0000 7f79537fe640 1 -- 172.21.15.29:0/3530039165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c035070 con 0x7f796410de10 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.201+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79641a6d30 con 0x7f796410de10 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.202+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7964104dc0 con 0x7f796410de10 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.203+0000 7f79537fe640 1 -- 172.21.15.29:0/3530039165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f795c042030 con 0x7f796410de10 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.204+0000 7f79537fe640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f794c0b65a0 0x7f794c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.204+0000 7f796259c640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f794c0b65a0 0x7f794c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.204+0000 7f79537fe640 1 -- 172.21.15.29:0/3530039165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f795c107bf0 con 0x7f796410de10 2024-09-21T02:37:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.205+0000 7f796259c640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f794c0b65a0 0x7f794c0b8ab0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f796406bc60 tx=0x7f7954009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.209+0000 7f79537fe640 1 -- 172.21.15.29:0/3530039165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f795c0d2460 con 0x7f796410de10 2024-09-21T02:37:17.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:17 smithi190 ceph-mon[33871]: pgmap v330: 129 pgs: 129 active+clean; 593 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.0 KiB/s wr, 20 op/s 2024-09-21T02:37:17.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:17 smithi029 ceph-mon[31006]: pgmap v330: 129 pgs: 129 active+clean; 593 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.0 KiB/s wr, 20 op/s 2024-09-21T02:37:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.428+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7964108f70 con 0x7f794c0b65a0 2024-09-21T02:37:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.456+0000 7f79537fe640 1 -- 172.21.15.29:0/3530039165 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7964108f70 con 0x7f794c0b65a0 2024-09-21T02:37:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f794c0b65a0 msgr2=0x7f794c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f794c0b65a0 0x7f794c0b8ab0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f796406bc60 tx=0x7f7954009670 comp rx=0 tx=0).stop 2024-09-21T02:37:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 msgr2=0x7f796419fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 0x7f796419fbb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f795c005ab0 tx=0x7f795c0040e0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 shutdown_connections 2024-09-21T02:37:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f794c0b65a0 0x7f794c0b8ab0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796410de10 0x7f796419fbb0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7964105300 0x7f796419a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 --2- 172.21.15.29:0/3530039165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79641049b0 0x7f796419a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.458+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 >> 172.21.15.29:0/3530039165 conn(0x7f7964079730 msgr2=0x7f7964100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.459+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 shutdown_connections 2024-09-21T02:37:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:17.459+0000 7f7968d4f640 1 -- 172.21.15.29:0/3530039165 wait complete. 2024-09-21T02:37:17.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:17 smithi066 ceph-mon[33436]: pgmap v330: 129 pgs: 129 active+clean; 593 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.0 KiB/s wr, 20 op/s 2024-09-21T02:37:18.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-21T02:37:18.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:18 smithi190 ceph-mon[33871]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:18.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:18 smithi029 ceph-mon[31006]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:18.462 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:18.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:18 smithi066 ceph-mon[33436]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:19.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 -- 172.21.15.29:0/3630510150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 msgr2=0x7f62041087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:19.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 --2- 172.21.15.29:0/3630510150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 0x7f62041087d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f61f8009900 tx=0x7f61f802a410 comp rx=0 tx=0).stop 2024-09-21T02:37:19.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 -- 172.21.15.29:0/3630510150 shutdown_connections 2024-09-21T02:37:19.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 --2- 172.21.15.29:0/3630510150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6204108d10 0x7f620410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 --2- 172.21.15.29:0/3630510150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 0x7f62041087d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 --2- 172.21.15.29:0/3630510150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6204102a80 0x7f6204102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 -- 172.21.15.29:0/3630510150 >> 172.21.15.29:0/3630510150 conn(0x7f6204079740 msgr2=0x7f6204079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 -- 172.21.15.29:0/3630510150 shutdown_connections 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.052+0000 7f6208972640 1 -- 172.21.15.29:0/3630510150 wait complete. 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.053+0000 7f6208972640 1 Processor -- start 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.053+0000 7f6208972640 1 -- start start 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6208972640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6204102a80 0x7f620419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6208972640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 0x7f620419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6208972640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6204108d10 0x7f620419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6208972640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6204113950 con 0x7f62041083c0 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6208972640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f62041137d0 con 0x7f6204108d10 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6208972640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6204113ad0 con 0x7f6204102a80 2024-09-21T02:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 0x7f620419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 0x7f620419aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42130/0 (socket says 172.21.15.29:42130) 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 -- 172.21.15.29:0/668790551 learned_addr learned my addr 172.21.15.29:0/668790551 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f620259c640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6204102a80 0x7f620419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6202d9d640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6204108d10 0x7f620419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 -- 172.21.15.29:0/668790551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6204102a80 msgr2=0x7f620419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6204102a80 0x7f620419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 -- 172.21.15.29:0/668790551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6204108d10 msgr2=0x7f620419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6204108d10 0x7f620419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 -- 172.21.15.29:0/668790551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62041a0310 con 0x7f62041083c0 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.054+0000 7f6201d9b640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 0x7f620419aa00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f61f8005940 tx=0x7f61f80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.055+0000 7f61f37fe640 1 -- 172.21.15.29:0/668790551 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f803f030 con 0x7f62041083c0 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.055+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62041a05a0 con 0x7f62041083c0 2024-09-21T02:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.055+0000 7f61f37fe640 1 -- 172.21.15.29:0/668790551 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61f8003c90 con 0x7f62041083c0 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.055+0000 7f61f37fe640 1 -- 172.21.15.29:0/668790551 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f8035070 con 0x7f62041083c0 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.055+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62041a0ad0 con 0x7f62041083c0 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.056+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6204103d00 con 0x7f62041083c0 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.057+0000 7f61f37fe640 1 -- 172.21.15.29:0/668790551 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f61f8042030 con 0x7f62041083c0 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.058+0000 7f61f37fe640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61ec0b65d0 0x7f61ec0b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.058+0000 7f620259c640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61ec0b65d0 0x7f61ec0b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.058+0000 7f61f37fe640 1 -- 172.21.15.29:0/668790551 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f61f81080f0 con 0x7f62041083c0 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.059+0000 7f620259c640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61ec0b65d0 0x7f61ec0b8ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f61f400abe0 tx=0x7f61f400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.061+0000 7f61f37fe640 1 -- 172.21.15.29:0/668790551 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f62041a0ad0 con 0x7f62041083c0 2024-09-21T02:37:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.279+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f6204063800 con 0x7f61ec0b65d0 2024-09-21T02:37:19.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:19 smithi029 ceph-mon[31006]: pgmap v331: 129 pgs: 129 active+clean; 594 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.1 KiB/s wr, 20 op/s 2024-09-21T02:37:19.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.284+0000 7f61f37fe640 1 -- 172.21.15.29:0/668790551 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6204063800 con 0x7f61ec0b65d0 2024-09-21T02:37:19.285 INFO:teuthology.orchestra.run.smithi029.stdout:/volumes/qa/sv_1/04e3dc3f-b4d1-4607-a717-1a5449d2953b 2024-09-21T02:37:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61ec0b65d0 msgr2=0x7f61ec0b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61ec0b65d0 0x7f61ec0b8ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f61f400abe0 tx=0x7f61f400a2d0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 msgr2=0x7f620419aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 0x7f620419aa00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f61f8005940 tx=0x7f61f80042f0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 shutdown_connections 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61ec0b65d0 0x7f61ec0b8ae0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6204108d10 0x7f620419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62041083c0 0x7f620419aa00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 --2- 172.21.15.29:0/668790551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6204102a80 0x7f620419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 >> 172.21.15.29:0/668790551 conn(0x7f6204079740 msgr2=0x7f62041053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 shutdown_connections 2024-09-21T02:37:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:19.286+0000 7f6208972640 1 -- 172.21.15.29:0/668790551 wait complete. 2024-09-21T02:37:19.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:19 smithi190 ceph-mon[33871]: pgmap v331: 129 pgs: 129 active+clean; 594 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.1 KiB/s wr, 20 op/s 2024-09-21T02:37:19.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:19 smithi066 ceph-mon[33436]: pgmap v331: 129 pgs: 129 active+clean; 594 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.1 KiB/s wr, 20 op/s 2024-09-21T02:37:20.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:37:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:20.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:20 smithi190 ceph-mon[33871]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:20.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:20 smithi029 ceph-mon[31006]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:20.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:20 smithi066 ceph-mon[33436]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 -- 172.21.15.29:0/1848028826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c34102f30 msgr2=0x7f1c3410fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/1848028826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c34102f30 0x7f1c3410fc50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009900 tx=0x7f1c2802a480 comp rx=0 tx=0).stop 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 -- 172.21.15.29:0/1848028826 shutdown_connections 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/1848028826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34110190 0x7f1c341125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/1848028826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c34102f30 0x7f1c3410fc50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/1848028826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c341025e0 0x7f1c341029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 -- 172.21.15.29:0/1848028826 >> 172.21.15.29:0/1848028826 conn(0x7f1c34079710 msgr2=0x7f1c34079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 -- 172.21.15.29:0/1848028826 shutdown_connections 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.899+0000 7f1c39c8c640 1 -- 172.21.15.29:0/1848028826 wait complete. 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.900+0000 7f1c39c8c640 1 Processor -- start 2024-09-21T02:37:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.900+0000 7f1c39c8c640 1 -- start start 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c39c8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c341025e0 0x7f1c3419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c39c8c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c34102f30 0x7f1c3419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c39c8c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34110190 0x7f1c341a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c39c8c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34115b30 con 0x7f1c341025e0 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c39c8c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1c341159b0 con 0x7f1c34102f30 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c39c8c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34115cb0 con 0x7f1c34110190 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c341025e0 0x7f1c3419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c341025e0 0x7f1c3419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42162/0 (socket says 172.21.15.29:42162) 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 -- 172.21.15.29:0/4164799246 learned_addr learned my addr 172.21.15.29:0/4164799246 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c33fff640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34110190 0x7f1c341a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c32ffd640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c34102f30 0x7f1c3419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 -- 172.21.15.29:0/4164799246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34110190 msgr2=0x7f1c341a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34110190 0x7f1c341a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 -- 172.21.15.29:0/4164799246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c34102f30 msgr2=0x7f1c3419cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c34102f30 0x7f1c3419cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 -- 172.21.15.29:0/4164799246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c341a24d0 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c32ffd640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c34102f30 0x7f1c3419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c337fe640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c341025e0 0x7f1c3419c560 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400d950 tx=0x7f1c2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.901+0000 7f1c30ff9640 1 -- 172.21.15.29:0/4164799246 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24017070 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.902+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c341a27c0 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.902+0000 7f1c30ff9640 1 -- 172.21.15.29:0/4164799246 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c24009c30 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.902+0000 7f1c30ff9640 1 -- 172.21.15.29:0/4164799246 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400f620 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.902+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c341a8f20 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.902+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c3406c280 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.904+0000 7f1c30ff9640 1 -- 172.21.15.29:0/4164799246 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f1c24005030 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.905+0000 7f1c30ff9640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c1c0b65a0 0x7f1c1c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.905+0000 7f1c32ffd640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c1c0b65a0 0x7f1c1c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.905+0000 7f1c30ff9640 1 -- 172.21.15.29:0/4164799246 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f1c240e7300 con 0x7f1c341025e0 2024-09-21T02:37:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.906+0000 7f1c32ffd640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c1c0b65a0 0x7f1c1c0b8ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006010 tx=0x7f1c28000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:20.909+0000 7f1c30ff9640 1 -- 172.21.15.29:0/4164799246 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1c240ad7f0 con 0x7f1c341025e0 2024-09-21T02:37:21.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.135+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c341029f0 con 0x7f1c341025e0 2024-09-21T02:37:21.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.136+0000 7f1c30ff9640 1 -- 172.21.15.29:0/4164799246 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11971 (secure 0 0 0) 0x7f1c24009de0 con 0x7f1c341025e0 2024-09-21T02:37:21.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:21.137 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-09-21T02:37:10:039454+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:37:10.039429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:37:21.137 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-09-21T02:37:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c1c0b65a0 msgr2=0x7f1c1c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c1c0b65a0 0x7f1c1c0b8ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006010 tx=0x7f1c28000bc0 comp rx=0 tx=0).stop 2024-09-21T02:37:21.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c341025e0 msgr2=0x7f1c3419c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c341025e0 0x7f1c3419c560 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400d950 tx=0x7f1c2400de10 comp rx=0 tx=0).stop 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 shutdown_connections 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c1c0b65a0 0x7f1c1c0b8ab0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c34110190 0x7f1c341a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c34102f30 0x7f1c3419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 --2- 172.21.15.29:0/4164799246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c341025e0 0x7f1c3419c560 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 >> 172.21.15.29:0/4164799246 conn(0x7f1c34079710 msgr2=0x7f1c34109d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 shutdown_connections 2024-09-21T02:37:21.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:21.138+0000 7f1c39c8c640 1 -- 172.21.15.29:0/4164799246 wait complete. 2024-09-21T02:37:21.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:21 smithi190 ceph-mon[33871]: pgmap v332: 129 pgs: 129 active+clean; 630 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 KiB/s wr, 24 op/s 2024-09-21T02:37:21.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:21 smithi029 ceph-mon[31006]: pgmap v332: 129 pgs: 129 active+clean; 630 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 KiB/s wr, 24 op/s 2024-09-21T02:37:21.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:21 smithi066 ceph-mon[33436]: pgmap v332: 129 pgs: 129 active+clean; 630 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 KiB/s wr, 24 op/s 2024-09-21T02:37:21.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:37:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:22 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4164799246' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:37:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:22 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:37:22.151 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:22.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:22 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4164799246' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:37:22.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:22 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:37:22.483 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:22 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4164799246' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:37:22.483 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:22 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:37:23.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:23 smithi029 ceph-mon[31006]: pgmap v333: 129 pgs: 129 active+clean; 630 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-09-21T02:37:23.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:23 smithi190 ceph-mon[33871]: pgmap v333: 129 pgs: 129 active+clean; 630 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-09-21T02:37:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 -- 172.21.15.29:0/1944273837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10108cf0 msgr2=0x7f4c1010f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/1944273837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10108cf0 0x7f4c1010f520 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4c08009ca0 tx=0x7f4c0802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 -- 172.21.15.29:0/1944273837 shutdown_connections 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/1944273837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10108cf0 0x7f4c1010f520 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/1944273837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c101083a0 0x7f4c101087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/1944273837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c10102a60 0x7f4c10102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 -- 172.21.15.29:0/1944273837 >> 172.21.15.29:0/1944273837 conn(0x7f4c10079730 msgr2=0x7f4c10079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 -- 172.21.15.29:0/1944273837 shutdown_connections 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.400+0000 7f4c17f0f640 1 -- 172.21.15.29:0/1944273837 wait complete. 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.401+0000 7f4c17f0f640 1 Processor -- start 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.401+0000 7f4c17f0f640 1 -- start start 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.401+0000 7f4c17f0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10102a60 0x7f4c10199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.401+0000 7f4c17f0f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c101083a0 0x7f4c101997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.401+0000 7f4c17f0f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c10108cf0 0x7f4c1019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.401+0000 7f4c17f0f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4c10112860 con 0x7f4c10102a60 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.401+0000 7f4c17f0f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4c101126e0 con 0x7f4c10108cf0 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.401+0000 7f4c17f0f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4c101129e0 con 0x7f4c101083a0 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10102a60 0x7f4c10199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10102a60 0x7f4c10199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42176/0 (socket says 172.21.15.29:42176) 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 -- 172.21.15.29:0/994894833 learned_addr learned my addr 172.21.15.29:0/994894833 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c164ac640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c10108cf0 0x7f4c1019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c154aa640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c101083a0 0x7f4c101997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 -- 172.21.15.29:0/994894833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c101083a0 msgr2=0x7f4c101997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c101083a0 0x7f4c101997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 -- 172.21.15.29:0/994894833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c10108cf0 msgr2=0x7f4c1019ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c10108cf0 0x7f4c1019ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 -- 172.21.15.29:0/994894833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1019f180 con 0x7f4c10102a60 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c164ac640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c10108cf0 0x7f4c1019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c15cab640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10102a60 0x7f4c10199210 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00e9f0 tx=0x7f4c0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c06ffd640 1 -- 172.21.15.29:0/994894833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c00cd80 con 0x7f4c10102a60 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c06ffd640 1 -- 172.21.15.29:0/994894833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c0c0054b0 con 0x7f4c10102a60 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1019f470 con 0x7f4c10102a60 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c06ffd640 1 -- 172.21.15.29:0/994894833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c010430 con 0x7f4c10102a60 2024-09-21T02:37:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.402+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c101a5bd0 con 0x7f4c10102a60 2024-09-21T02:37:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.404+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c10102e70 con 0x7f4c10102a60 2024-09-21T02:37:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.405+0000 7f4c06ffd640 1 -- 172.21.15.29:0/994894833 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f4c0c0046c0 con 0x7f4c10102a60 2024-09-21T02:37:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.406+0000 7f4c06ffd640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bfc0b6620 0x7f4bfc0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.406+0000 7f4c154aa640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bfc0b6620 0x7f4bfc0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.406+0000 7f4c06ffd640 1 -- 172.21.15.29:0/994894833 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f4c0c0e7bc0 con 0x7f4c10102a60 2024-09-21T02:37:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.406+0000 7f4c154aa640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bfc0b6620 0x7f4bfc0b8b30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4c1006a260 tx=0x7f4c00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.410+0000 7f4c06ffd640 1 -- 172.21.15.29:0/994894833 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4c0c0ae0b0 con 0x7f4c10102a60 2024-09-21T02:37:23.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:23 smithi066 ceph-mon[33436]: pgmap v333: 129 pgs: 129 active+clean; 630 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-09-21T02:37:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.619+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c1019a340 con 0x7f4c10102a60 2024-09-21T02:37:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.620+0000 7f4c06ffd640 1 -- 172.21.15.29:0/994894833 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24199 (secure 0 0 0) 0x7f4c0c0b2490 con 0x7f4c10102a60 2024-09-21T02:37:23.622 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:23.622 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:36:35.844398+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:37:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bfc0b6620 msgr2=0x7f4bfc0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bfc0b6620 0x7f4bfc0b8b30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4c1006a260 tx=0x7f4c00005f90 comp rx=0 tx=0).stop 2024-09-21T02:37:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10102a60 msgr2=0x7f4c10199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10102a60 0x7f4c10199210 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00e9f0 tx=0x7f4c0c00edf0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 shutdown_connections 2024-09-21T02:37:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bfc0b6620 0x7f4bfc0b8b30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c10108cf0 0x7f4c1019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c101083a0 0x7f4c101997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 --2- 172.21.15.29:0/994894833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c10102a60 0x7f4c10199210 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 >> 172.21.15.29:0/994894833 conn(0x7f4c10079730 msgr2=0x7f4c101043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 shutdown_connections 2024-09-21T02:37:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:23.623+0000 7f4c17f0f640 1 -- 172.21.15.29:0/994894833 wait complete. 2024-09-21T02:37:24.370 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:24 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.371 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:24 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.371 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:24 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.371 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:24 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.371 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:24 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/994894833' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:24.399 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T02:37:24.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:37:24.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:24 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:24 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:24 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:24 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:24 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/994894833' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:24.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:24 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:24 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:24 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:24 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:24.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:24 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/994894833' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:24.683 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 -- 172.21.15.29:0/65435487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c105330 msgr2=0x7fe47c10d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 --2- 172.21.15.29:0/65435487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c105330 0x7fe47c10d880 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe470009900 tx=0x7fe47002a410 comp rx=0 tx=0).stop 2024-09-21T02:37:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 -- 172.21.15.29:0/65435487 shutdown_connections 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 --2- 172.21.15.29:0/65435487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe47c10de40 0x7fe47c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 --2- 172.21.15.29:0/65435487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c105330 0x7fe47c10d880 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 --2- 172.21.15.29:0/65435487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe47c1049e0 0x7fe47c104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 -- 172.21.15.29:0/65435487 >> 172.21.15.29:0/65435487 conn(0x7fe47c079750 msgr2=0x7fe47c079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 -- 172.21.15.29:0/65435487 shutdown_connections 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.206+0000 7fe482ad8640 1 -- 172.21.15.29:0/65435487 wait complete. 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe482ad8640 1 Processor -- start 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe482ad8640 1 -- start start 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe482ad8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c1049e0 0x7fe47c102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe482ad8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe47c105330 0x7fe47c102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe482ad8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe47c10de40 0x7fe47c1031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe482ad8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c113950 con 0x7fe47c1049e0 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe482ad8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c1137d0 con 0x7fe47c105330 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe482ad8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c113ad0 con 0x7fe47c10de40 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe480874640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c1049e0 0x7fe47c102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe480874640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c1049e0 0x7fe47c102720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45716/0 (socket says 172.21.15.29:45716) 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.207+0000 7fe480874640 1 -- 172.21.15.29:0/3086311991 learned_addr learned my addr 172.21.15.29:0/3086311991 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe47bfff640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe47c105330 0x7fe47c102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe481075640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe47c10de40 0x7fe47c1031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe480874640 1 -- 172.21.15.29:0/3086311991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe47c10de40 msgr2=0x7fe47c1031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe480874640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe47c10de40 0x7fe47c1031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe480874640 1 -- 172.21.15.29:0/3086311991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe47c105330 msgr2=0x7fe47c102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe480874640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe47c105330 0x7fe47c102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe480874640 1 -- 172.21.15.29:0/3086311991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47c1a86a0 con 0x7fe47c1049e0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe47bfff640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe47c105330 0x7fe47c102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe480874640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c1049e0 0x7fe47c102720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00d950 tx=0x7fe46c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe479ffb640 1 -- 172.21.15.29:0/3086311991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c017020 con 0x7fe47c1049e0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe479ffb640 1 -- 172.21.15.29:0/3086311991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe46c009c30 con 0x7fe47c1049e0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe479ffb640 1 -- 172.21.15.29:0/3086311991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c00f620 con 0x7fe47c1049e0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47c1a8990 con 0x7fe47c1049e0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.208+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe47c1a8ef0 con 0x7fe47c1049e0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.209+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe47c06c280 con 0x7fe47c1049e0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.211+0000 7fe479ffb640 1 -- 172.21.15.29:0/3086311991 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fe46c01a020 con 0x7fe47c1049e0 2024-09-21T02:37:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.211+0000 7fe479ffb640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4640b6620 0x7fe4640b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.212+0000 7fe47bfff640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4640b6620 0x7fe4640b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.212+0000 7fe479ffb640 1 -- 172.21.15.29:0/3086311991 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fe46c0e7690 con 0x7fe47c1049e0 2024-09-21T02:37:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.212+0000 7fe47bfff640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4640b6620 0x7fe4640b8b30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe4700057c0 tx=0x7fe470009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.216+0000 7fe479ffb640 1 -- 172.21.15.29:0/3086311991 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe46c021020 con 0x7fe47c1049e0 2024-09-21T02:37:25.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.401+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe47c063800 con 0x7fe4640b6620 2024-09-21T02:37:25.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.407+0000 7fe479ffb640 1 -- 172.21.15.29:0/3086311991 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340625 (secure 0 0 0) 0x7fe47c063800 con 0x7fe4640b6620 2024-09-21T02:37:25.412 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:25.414 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T02:37:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4640b6620 msgr2=0x7fe4640b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4640b6620 0x7fe4640b8b30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe4700057c0 tx=0x7fe470009680 comp rx=0 tx=0).stop 2024-09-21T02:37:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c1049e0 msgr2=0x7fe47c102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c1049e0 0x7fe47c102720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00d950 tx=0x7fe46c00de10 comp rx=0 tx=0).stop 2024-09-21T02:37:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 shutdown_connections 2024-09-21T02:37:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4640b6620 0x7fe4640b8b30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe47c10de40 0x7fe47c1031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe47c105330 0x7fe47c102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 --2- 172.21.15.29:0/3086311991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe47c1049e0 0x7fe47c102720 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:25.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 >> 172.21.15.29:0/3086311991 conn(0x7fe47c079750 msgr2=0x7fe47c100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:25.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 shutdown_connections 2024-09-21T02:37:25.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:25.411+0000 7fe482ad8640 1 -- 172.21.15.29:0/3086311991 wait complete. 2024-09-21T02:37:25.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:25 smithi029 ceph-mon[31006]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-09-21T02:37:25.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:25.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:25.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:37:25.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:37:25.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:25.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:25 smithi190 ceph-mon[33871]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-09-21T02:37:25.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:37:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:37:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:25 smithi066 ceph-mon[33436]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-09-21T02:37:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:37:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:37:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:37:26.250 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-21T02:37:23.942298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658106,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":238,"num_read_kb":222,"num_write":369,"num_write_kb":2290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364408,"kb_used_data":11000,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124577416,"statfs":{"total":1151940427776,"available":1151567273984,"internally_reserved":0,"allocated":11264000,"data_stored":7313936,"data_compressed":319842,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":58208,"num_objects":12,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":136,"num_read_kb":135,"num_write":56,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004000"},"pg_stats":[{"pgid":"2.1f","version":"81'13","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:22.067590+0000","last_change":"2024-09-21T02:36:23.729337+0000","last_active":"2024-09-21T02:37:22.067590+0000","last_peered":"2024-09-21T02:37:22.067590+0000","last_clean":"2024-09-21T02:37:22.067590+0000","last_became_active":"2024-09-21T02:36:23.728576+0000","last_became_peered":"2024-09-21T02:36:23.728576+0000","last_unstale":"2024-09-21T02:37:22.067590+0000","last_undegraded":"2024-09-21T02:37:22.067590+0000","last_fullsized":"2024-09-21T02:37:22.067590+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:31.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396463+0000","last_change":"2024-09-21T02:36:25.748925+0000","last_active":"2024-09-21T02:36:37.396463+0000","last_peered":"2024-09-21T02:36:37.396463+0000","last_clean":"2024-09-21T02:36:37.396463+0000","last_became_active":"2024-09-21T02:36:25.748333+0000","last_became_peered":"2024-09-21T02:36:25.748333+0000","last_unstale":"2024-09-21T02:36:37.396463+0000","last_undegraded":"2024-09-21T02:36:37.396463+0000","last_fullsized":"2024-09-21T02:36:37.396463+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:46.889497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397461+0000","last_change":"2024-09-21T02:36:32.807771+0000","last_active":"2024-09-21T02:36:37.397461+0000","last_peered":"2024-09-21T02:36:37.397461+0000","last_clean":"2024-09-21T02:36:37.397461+0000","last_became_active":"2024-09-21T02:36:32.807291+0000","last_became_peered":"2024-09-21T02:36:32.807291+0000","last_unstale":"2024-09-21T02:36:37.397461+0000","last_undegraded":"2024-09-21T02:36:37.397461+0000","last_fullsized":"2024-09-21T02:36:37.397461+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:41.468318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396711+0000","last_change":"2024-09-21T02:36:23.800779+0000","last_active":"2024-09-21T02:36:37.396711+0000","last_peered":"2024-09-21T02:36:37.396711+0000","last_clean":"2024-09-21T02:36:37.396711+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:36:37.396711+0000","last_undegraded":"2024-09-21T02:36:37.396711+0000","last_fullsized":"2024-09-21T02:36:37.396711+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:24.843233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991592+0000","last_change":"2024-09-21T02:36:25.745277+0000","last_active":"2024-09-21T02:36:36.991592+0000","last_peered":"2024-09-21T02:36:36.991592+0000","last_clean":"2024-09-21T02:36:36.991592+0000","last_became_active":"2024-09-21T02:36:25.744501+0000","last_became_peered":"2024-09-21T02:36:25.744501+0000","last_unstale":"2024-09-21T02:36:36.991592+0000","last_undegraded":"2024-09-21T02:36:36.991592+0000","last_fullsized":"2024-09-21T02:36:36.991592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:14.776250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396321+0000","last_change":"2024-09-21T02:36:32.801978+0000","last_active":"2024-09-21T02:36:37.396321+0000","last_peered":"2024-09-21T02:36:37.396321+0000","last_clean":"2024-09-21T02:36:37.396321+0000","last_became_active":"2024-09-21T02:36:32.801436+0000","last_became_peered":"2024-09-21T02:36:32.801436+0000","last_unstale":"2024-09-21T02:36:37.396321+0000","last_undegraded":"2024-09-21T02:36:37.396321+0000","last_fullsized":"2024-09-21T02:36:37.396321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:37.380417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:10.046781+0000","last_change":"2024-09-21T02:36:23.729336+0000","last_active":"2024-09-21T02:37:10.046781+0000","last_peered":"2024-09-21T02:37:10.046781+0000","last_clean":"2024-09-21T02:37:10.046781+0000","last_became_active":"2024-09-21T02:36:23.728724+0000","last_became_peered":"2024-09-21T02:36:23.728724+0000","last_unstale":"2024-09-21T02:37:10.046781+0000","last_undegraded":"2024-09-21T02:37:10.046781+0000","last_fullsized":"2024-09-21T02:37:10.046781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:31:52.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380640+0000","last_change":"2024-09-21T02:36:25.737577+0000","last_active":"2024-09-21T02:36:36.380640+0000","last_peered":"2024-09-21T02:36:36.380640+0000","last_clean":"2024-09-21T02:36:36.380640+0000","last_became_active":"2024-09-21T02:36:25.737221+0000","last_became_peered":"2024-09-21T02:36:25.737221+0000","last_unstale":"2024-09-21T02:36:36.380640+0000","last_undegraded":"2024-09-21T02:36:36.380640+0000","last_fullsized":"2024-09-21T02:36:36.380640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:42:54.786464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397059+0000","last_change":"2024-09-21T02:36:32.802507+0000","last_active":"2024-09-21T02:36:37.397059+0000","last_peered":"2024-09-21T02:36:37.397059+0000","last_clean":"2024-09-21T02:36:37.397059+0000","last_became_active":"2024-09-21T02:36:32.801879+0000","last_became_peered":"2024-09-21T02:36:32.801879+0000","last_unstale":"2024-09-21T02:36:37.397059+0000","last_undegraded":"2024-09-21T02:36:37.397059+0000","last_fullsized":"2024-09-21T02:36:37.397059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:09.873825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044202+0000","last_change":"2024-09-21T02:36:23.730889+0000","last_active":"2024-09-21T02:37:09.044202+0000","last_peered":"2024-09-21T02:37:09.044202+0000","last_clean":"2024-09-21T02:37:09.044202+0000","last_became_active":"2024-09-21T02:36:23.730551+0000","last_became_peered":"2024-09-21T02:36:23.730551+0000","last_unstale":"2024-09-21T02:37:09.044202+0000","last_undegraded":"2024-09-21T02:37:09.044202+0000","last_fullsized":"2024-09-21T02:37:09.044202+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:17.178848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.398583+0000","last_change":"2024-09-21T02:36:25.744531+0000","last_active":"2024-09-21T02:36:37.398583+0000","last_peered":"2024-09-21T02:36:37.398583+0000","last_clean":"2024-09-21T02:36:37.398583+0000","last_became_active":"2024-09-21T02:36:25.744110+0000","last_became_peered":"2024-09-21T02:36:25.744110+0000","last_unstale":"2024-09-21T02:36:37.398583+0000","last_undegraded":"2024-09-21T02:36:37.398583+0000","last_fullsized":"2024-09-21T02:36:37.398583+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:56:08.895972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396486+0000","last_change":"2024-09-21T02:36:32.800489+0000","last_active":"2024-09-21T02:36:37.396486+0000","last_peered":"2024-09-21T02:36:37.396486+0000","last_clean":"2024-09-21T02:36:37.396486+0000","last_became_active":"2024-09-21T02:36:32.799876+0000","last_became_peered":"2024-09-21T02:36:32.799876+0000","last_unstale":"2024-09-21T02:36:37.396486+0000","last_undegraded":"2024-09-21T02:36:37.396486+0000","last_fullsized":"2024-09-21T02:36:37.396486+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:02:56.711257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401371+0000","last_change":"2024-09-21T02:36:25.744561+0000","last_active":"2024-09-21T02:36:36.401371+0000","last_peered":"2024-09-21T02:36:36.401371+0000","last_clean":"2024-09-21T02:36:36.401371+0000","last_became_active":"2024-09-21T02:36:25.743757+0000","last_became_peered":"2024-09-21T02:36:25.743757+0000","last_unstale":"2024-09-21T02:36:36.401371+0000","last_undegraded":"2024-09-21T02:36:36.401371+0000","last_fullsized":"2024-09-21T02:36:36.401371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:30.235517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:06.018103+0000","last_change":"2024-09-21T02:36:23.729060+0000","last_active":"2024-09-21T02:37:06.018103+0000","last_peered":"2024-09-21T02:37:06.018103+0000","last_clean":"2024-09-21T02:37:06.018103+0000","last_became_active":"2024-09-21T02:36:23.728703+0000","last_became_peered":"2024-09-21T02:36:23.728703+0000","last_unstale":"2024-09-21T02:37:06.018103+0000","last_undegraded":"2024-09-21T02:37:06.018103+0000","last_fullsized":"2024-09-21T02:37:06.018103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:07.201729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396729+0000","last_change":"2024-09-21T02:36:33.003670+0000","last_active":"2024-09-21T02:36:37.396729+0000","last_peered":"2024-09-21T02:36:37.396729+0000","last_clean":"2024-09-21T02:36:37.396729+0000","last_became_active":"2024-09-21T02:36:33.003022+0000","last_became_peered":"2024-09-21T02:36:33.003022+0000","last_unstale":"2024-09-21T02:36:37.396729+0000","last_undegraded":"2024-09-21T02:36:37.396729+0000","last_fullsized":"2024-09-21T02:36:37.396729+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:12.753375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395962+0000","last_change":"2024-09-21T02:36:25.749404+0000","last_active":"2024-09-21T02:36:37.395962+0000","last_peered":"2024-09-21T02:36:37.395962+0000","last_clean":"2024-09-21T02:36:37.395962+0000","last_became_active":"2024-09-21T02:36:25.747854+0000","last_became_peered":"2024-09-21T02:36:25.747854+0000","last_unstale":"2024-09-21T02:36:37.395962+0000","last_undegraded":"2024-09-21T02:36:37.395962+0000","last_fullsized":"2024-09-21T02:36:37.395962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:41.224740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:08.034280+0000","last_change":"2024-09-21T02:36:23.726689+0000","last_active":"2024-09-21T02:37:08.034280+0000","last_peered":"2024-09-21T02:37:08.034280+0000","last_clean":"2024-09-21T02:37:08.034280+0000","last_became_active":"2024-09-21T02:36:23.726165+0000","last_became_peered":"2024-09-21T02:36:23.726165+0000","last_unstale":"2024-09-21T02:37:08.034280+0000","last_undegraded":"2024-09-21T02:37:08.034280+0000","last_fullsized":"2024-09-21T02:37:08.034280+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:58.166457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397094+0000","last_change":"2024-09-21T02:36:32.809594+0000","last_active":"2024-09-21T02:36:37.397094+0000","last_peered":"2024-09-21T02:36:37.397094+0000","last_clean":"2024-09-21T02:36:37.397094+0000","last_became_active":"2024-09-21T02:36:32.808893+0000","last_became_peered":"2024-09-21T02:36:32.808893+0000","last_unstale":"2024-09-21T02:36:37.397094+0000","last_undegraded":"2024-09-21T02:36:37.397094+0000","last_fullsized":"2024-09-21T02:36:37.397094+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:55.187241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899868+0000","last_change":"2024-09-21T02:36:25.755955+0000","last_active":"2024-09-21T02:36:35.899868+0000","last_peered":"2024-09-21T02:36:35.899868+0000","last_clean":"2024-09-21T02:36:35.899868+0000","last_became_active":"2024-09-21T02:36:25.755577+0000","last_became_peered":"2024-09-21T02:36:25.755577+0000","last_unstale":"2024-09-21T02:36:35.899868+0000","last_undegraded":"2024-09-21T02:36:35.899868+0000","last_fullsized":"2024-09-21T02:36:35.899868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:48.447758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044136+0000","last_change":"2024-09-21T02:36:23.730043+0000","last_active":"2024-09-21T02:37:09.044136+0000","last_peered":"2024-09-21T02:37:09.044136+0000","last_clean":"2024-09-21T02:37:09.044136+0000","last_became_active":"2024-09-21T02:36:23.729576+0000","last_became_peered":"2024-09-21T02:36:23.729576+0000","last_unstale":"2024-09-21T02:37:09.044136+0000","last_undegraded":"2024-09-21T02:37:09.044136+0000","last_fullsized":"2024-09-21T02:37:09.044136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:48:09.530913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.754143+0000","last_change":"2024-09-21T02:36:32.806923+0000","last_active":"2024-09-21T02:36:36.754143+0000","last_peered":"2024-09-21T02:36:36.754143+0000","last_clean":"2024-09-21T02:36:36.754143+0000","last_became_active":"2024-09-21T02:36:32.806176+0000","last_became_peered":"2024-09-21T02:36:32.806176+0000","last_unstale":"2024-09-21T02:36:36.754143+0000","last_undegraded":"2024-09-21T02:36:36.754143+0000","last_fullsized":"2024-09-21T02:36:36.754143+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.165638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396452+0000","last_change":"2024-09-21T02:36:25.747501+0000","last_active":"2024-09-21T02:36:37.396452+0000","last_peered":"2024-09-21T02:36:37.396452+0000","last_clean":"2024-09-21T02:36:37.396452+0000","last_became_active":"2024-09-21T02:36:25.747111+0000","last_became_peered":"2024-09-21T02:36:25.747111+0000","last_unstale":"2024-09-21T02:36:37.396452+0000","last_undegraded":"2024-09-21T02:36:37.396452+0000","last_fullsized":"2024-09-21T02:36:37.396452+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:03:04.061537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:06.019272+0000","last_change":"2024-09-21T02:36:23.800802+0000","last_active":"2024-09-21T02:37:06.019272+0000","last_peered":"2024-09-21T02:37:06.019272+0000","last_clean":"2024-09-21T02:37:06.019272+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:37:06.019272+0000","last_undegraded":"2024-09-21T02:37:06.019272+0000","last_fullsized":"2024-09-21T02:37:06.019272+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:44:34.165406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397014+0000","last_change":"2024-09-21T02:36:33.005535+0000","last_active":"2024-09-21T02:36:37.397014+0000","last_peered":"2024-09-21T02:36:37.397014+0000","last_clean":"2024-09-21T02:36:37.397014+0000","last_became_active":"2024-09-21T02:36:33.004912+0000","last_became_peered":"2024-09-21T02:36:33.004912+0000","last_unstale":"2024-09-21T02:36:37.397014+0000","last_undegraded":"2024-09-21T02:36:37.397014+0000","last_fullsized":"2024-09-21T02:36:37.397014+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:54:14.378790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991439+0000","last_change":"2024-09-21T02:36:25.746139+0000","last_active":"2024-09-21T02:36:36.991439+0000","last_peered":"2024-09-21T02:36:36.991439+0000","last_clean":"2024-09-21T02:36:36.991439+0000","last_became_active":"2024-09-21T02:36:25.744418+0000","last_became_peered":"2024-09-21T02:36:25.744418+0000","last_unstale":"2024-09-21T02:36:36.991439+0000","last_undegraded":"2024-09-21T02:36:36.991439+0000","last_fullsized":"2024-09-21T02:36:36.991439+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:21.706782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044967+0000","last_change":"2024-09-21T02:36:24.261945+0000","last_active":"2024-09-21T02:37:09.044967+0000","last_peered":"2024-09-21T02:37:09.044967+0000","last_clean":"2024-09-21T02:37:09.044967+0000","last_became_active":"2024-09-21T02:36:24.261486+0000","last_became_peered":"2024-09-21T02:36:24.261486+0000","last_unstale":"2024-09-21T02:37:09.044967+0000","last_undegraded":"2024-09-21T02:37:09.044967+0000","last_fullsized":"2024-09-21T02:37:09.044967+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:45:18.331035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397991+0000","last_change":"2024-09-21T02:36:32.799464+0000","last_active":"2024-09-21T02:36:37.397991+0000","last_peered":"2024-09-21T02:36:37.397991+0000","last_clean":"2024-09-21T02:36:37.397991+0000","last_became_active":"2024-09-21T02:36:32.799000+0000","last_became_peered":"2024-09-21T02:36:32.799000+0000","last_unstale":"2024-09-21T02:36:37.397991+0000","last_undegraded":"2024-09-21T02:36:37.397991+0000","last_fullsized":"2024-09-21T02:36:37.397991+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:56:36.271172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396612+0000","last_change":"2024-09-21T02:36:25.741151+0000","last_active":"2024-09-21T02:36:37.396612+0000","last_peered":"2024-09-21T02:36:37.396612+0000","last_clean":"2024-09-21T02:36:37.396612+0000","last_became_active":"2024-09-21T02:36:25.740803+0000","last_became_peered":"2024-09-21T02:36:25.740803+0000","last_unstale":"2024-09-21T02:36:37.396612+0000","last_undegraded":"2024-09-21T02:36:37.396612+0000","last_fullsized":"2024-09-21T02:36:37.396612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:44.585136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'14","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:19.749474+0000","last_change":"2024-09-21T02:36:23.725155+0000","last_active":"2024-09-21T02:37:19.749474+0000","last_peered":"2024-09-21T02:37:19.749474+0000","last_clean":"2024-09-21T02:37:19.749474+0000","last_became_active":"2024-09-21T02:36:23.724791+0000","last_became_peered":"2024-09-21T02:36:23.724791+0000","last_unstale":"2024-09-21T02:37:19.749474+0000","last_undegraded":"2024-09-21T02:37:19.749474+0000","last_fullsized":"2024-09-21T02:37:19.749474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:06.520465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395786+0000","last_change":"2024-09-21T02:36:33.003401+0000","last_active":"2024-09-21T02:36:37.395786+0000","last_peered":"2024-09-21T02:36:37.395786+0000","last_clean":"2024-09-21T02:36:37.395786+0000","last_became_active":"2024-09-21T02:36:33.002930+0000","last_became_peered":"2024-09-21T02:36:33.002930+0000","last_unstale":"2024-09-21T02:36:37.395786+0000","last_undegraded":"2024-09-21T02:36:37.395786+0000","last_fullsized":"2024-09-21T02:36:37.395786+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:40:23.629763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899410+0000","last_change":"2024-09-21T02:36:25.735501+0000","last_active":"2024-09-21T02:36:35.899410+0000","last_peered":"2024-09-21T02:36:35.899410+0000","last_clean":"2024-09-21T02:36:35.899410+0000","last_became_active":"2024-09-21T02:36:25.734453+0000","last_became_peered":"2024-09-21T02:36:25.734453+0000","last_unstale":"2024-09-21T02:36:35.899410+0000","last_undegraded":"2024-09-21T02:36:35.899410+0000","last_fullsized":"2024-09-21T02:36:35.899410+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:50.238529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.042607+0000","last_change":"2024-09-21T02:36:23.800338+0000","last_active":"2024-09-21T02:37:09.042607+0000","last_peered":"2024-09-21T02:37:09.042607+0000","last_clean":"2024-09-21T02:37:09.042607+0000","last_became_active":"2024-09-21T02:36:23.799796+0000","last_became_peered":"2024-09-21T02:36:23.799796+0000","last_unstale":"2024-09-21T02:37:09.042607+0000","last_undegraded":"2024-09-21T02:37:09.042607+0000","last_fullsized":"2024-09-21T02:37:09.042607+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:00:18.459515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899946+0000","last_change":"2024-09-21T02:36:32.799630+0000","last_active":"2024-09-21T02:36:35.899946+0000","last_peered":"2024-09-21T02:36:35.899946+0000","last_clean":"2024-09-21T02:36:35.899946+0000","last_became_active":"2024-09-21T02:36:32.798761+0000","last_became_peered":"2024-09-21T02:36:32.798761+0000","last_unstale":"2024-09-21T02:36:35.899946+0000","last_undegraded":"2024-09-21T02:36:35.899946+0000","last_fullsized":"2024-09-21T02:36:35.899946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:33.707902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900025+0000","last_change":"2024-09-21T02:36:25.745497+0000","last_active":"2024-09-21T02:36:35.900025+0000","last_peered":"2024-09-21T02:36:35.900025+0000","last_clean":"2024-09-21T02:36:35.900025+0000","last_became_active":"2024-09-21T02:36:25.745126+0000","last_became_peered":"2024-09-21T02:36:25.745126+0000","last_unstale":"2024-09-21T02:36:35.900025+0000","last_undegraded":"2024-09-21T02:36:35.900025+0000","last_fullsized":"2024-09-21T02:36:35.900025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:42:17.835784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.754164+0000","last_change":"2024-09-21T02:36:23.802440+0000","last_active":"2024-09-21T02:36:36.754164+0000","last_peered":"2024-09-21T02:36:36.754164+0000","last_clean":"2024-09-21T02:36:36.754164+0000","last_became_active":"2024-09-21T02:36:23.802065+0000","last_became_peered":"2024-09-21T02:36:23.802065+0000","last_unstale":"2024-09-21T02:36:36.754164+0000","last_undegraded":"2024-09-21T02:36:36.754164+0000","last_fullsized":"2024-09-21T02:36:36.754164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:22:20.647476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396426+0000","last_change":"2024-09-21T02:36:33.003465+0000","last_active":"2024-09-21T02:36:37.396426+0000","last_peered":"2024-09-21T02:36:37.396426+0000","last_clean":"2024-09-21T02:36:37.396426+0000","last_became_active":"2024-09-21T02:36:33.002874+0000","last_became_peered":"2024-09-21T02:36:33.002874+0000","last_unstale":"2024-09-21T02:36:37.396426+0000","last_undegraded":"2024-09-21T02:36:37.396426+0000","last_fullsized":"2024-09-21T02:36:37.396426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:02.388770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899847+0000","last_change":"2024-09-21T02:36:25.746092+0000","last_active":"2024-09-21T02:36:35.899847+0000","last_peered":"2024-09-21T02:36:35.899847+0000","last_clean":"2024-09-21T02:36:35.899847+0000","last_became_active":"2024-09-21T02:36:25.745730+0000","last_became_peered":"2024-09-21T02:36:25.745730+0000","last_unstale":"2024-09-21T02:36:35.899847+0000","last_undegraded":"2024-09-21T02:36:35.899847+0000","last_fullsized":"2024-09-21T02:36:35.899847+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:24:13.212450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:18.062868+0000","last_change":"2024-09-21T02:36:23.732090+0000","last_active":"2024-09-21T02:37:18.062868+0000","last_peered":"2024-09-21T02:37:18.062868+0000","last_clean":"2024-09-21T02:37:18.062868+0000","last_became_active":"2024-09-21T02:36:23.731653+0000","last_became_peered":"2024-09-21T02:36:23.731653+0000","last_unstale":"2024-09-21T02:37:18.062868+0000","last_undegraded":"2024-09-21T02:37:18.062868+0000","last_fullsized":"2024-09-21T02:37:18.062868+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:02:42.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899238+0000","last_change":"2024-09-21T02:36:32.794893+0000","last_active":"2024-09-21T02:36:35.899238+0000","last_peered":"2024-09-21T02:36:35.899238+0000","last_clean":"2024-09-21T02:36:35.899238+0000","last_became_active":"2024-09-21T02:36:32.794406+0000","last_became_peered":"2024-09-21T02:36:32.794406+0000","last_unstale":"2024-09-21T02:36:35.899238+0000","last_undegraded":"2024-09-21T02:36:35.899238+0000","last_fullsized":"2024-09-21T02:36:35.899238+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:40:39.621502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396958+0000","last_change":"2024-09-21T02:36:25.748220+0000","last_active":"2024-09-21T02:36:37.396958+0000","last_peered":"2024-09-21T02:36:37.396958+0000","last_clean":"2024-09-21T02:36:37.396958+0000","last_became_active":"2024-09-21T02:36:25.747553+0000","last_became_peered":"2024-09-21T02:36:25.747553+0000","last_unstale":"2024-09-21T02:36:37.396958+0000","last_undegraded":"2024-09-21T02:36:37.396958+0000","last_fullsized":"2024-09-21T02:36:37.396958+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:28.757981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:17.061382+0000","last_change":"2024-09-21T02:36:23.801087+0000","last_active":"2024-09-21T02:37:17.061382+0000","last_peered":"2024-09-21T02:37:17.061382+0000","last_clean":"2024-09-21T02:37:17.061382+0000","last_became_active":"2024-09-21T02:36:23.800077+0000","last_became_peered":"2024-09-21T02:36:23.800077+0000","last_unstale":"2024-09-21T02:37:17.061382+0000","last_undegraded":"2024-09-21T02:37:17.061382+0000","last_fullsized":"2024-09-21T02:37:17.061382+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:50.074650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899492+0000","last_change":"2024-09-21T02:36:33.003354+0000","last_active":"2024-09-21T02:36:35.899492+0000","last_peered":"2024-09-21T02:36:35.899492+0000","last_clean":"2024-09-21T02:36:35.899492+0000","last_became_active":"2024-09-21T02:36:33.002912+0000","last_became_peered":"2024-09-21T02:36:33.002912+0000","last_unstale":"2024-09-21T02:36:35.899492+0000","last_undegraded":"2024-09-21T02:36:35.899492+0000","last_fullsized":"2024-09-21T02:36:35.899492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:21.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.898724+0000","last_change":"2024-09-21T02:36:25.741701+0000","last_active":"2024-09-21T02:36:35.898724+0000","last_peered":"2024-09-21T02:36:35.898724+0000","last_clean":"2024-09-21T02:36:35.898724+0000","last_became_active":"2024-09-21T02:36:25.741222+0000","last_became_peered":"2024-09-21T02:36:25.741222+0000","last_unstale":"2024-09-21T02:36:35.898724+0000","last_undegraded":"2024-09-21T02:36:35.898724+0000","last_fullsized":"2024-09-21T02:36:35.898724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:35:39.896852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:07.024788+0000","last_change":"2024-09-21T02:36:23.728793+0000","last_active":"2024-09-21T02:37:07.024788+0000","last_peered":"2024-09-21T02:37:07.024788+0000","last_clean":"2024-09-21T02:37:07.024788+0000","last_became_active":"2024-09-21T02:36:23.728362+0000","last_became_peered":"2024-09-21T02:36:23.728362+0000","last_unstale":"2024-09-21T02:37:07.024788+0000","last_undegraded":"2024-09-21T02:37:07.024788+0000","last_fullsized":"2024-09-21T02:37:07.024788+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:28:15.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.398740+0000","last_change":"2024-09-21T02:36:33.005225+0000","last_active":"2024-09-21T02:36:37.398740+0000","last_peered":"2024-09-21T02:36:37.398740+0000","last_clean":"2024-09-21T02:36:37.398740+0000","last_became_active":"2024-09-21T02:36:33.004756+0000","last_became_peered":"2024-09-21T02:36:33.004756+0000","last_unstale":"2024-09-21T02:36:37.398740+0000","last_undegraded":"2024-09-21T02:36:37.398740+0000","last_fullsized":"2024-09-21T02:36:37.398740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:02:16.686284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396145+0000","last_change":"2024-09-21T02:36:25.745639+0000","last_active":"2024-09-21T02:36:37.396145+0000","last_peered":"2024-09-21T02:36:37.396145+0000","last_clean":"2024-09-21T02:36:37.396145+0000","last_became_active":"2024-09-21T02:36:25.745202+0000","last_became_peered":"2024-09-21T02:36:25.745202+0000","last_unstale":"2024-09-21T02:36:37.396145+0000","last_undegraded":"2024-09-21T02:36:37.396145+0000","last_fullsized":"2024-09-21T02:36:37.396145+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:08.712477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:10.046807+0000","last_change":"2024-09-21T02:36:23.730710+0000","last_active":"2024-09-21T02:37:10.046807+0000","last_peered":"2024-09-21T02:37:10.046807+0000","last_clean":"2024-09-21T02:37:10.046807+0000","last_became_active":"2024-09-21T02:36:23.730247+0000","last_became_peered":"2024-09-21T02:36:23.730247+0000","last_unstale":"2024-09-21T02:37:10.046807+0000","last_undegraded":"2024-09-21T02:37:10.046807+0000","last_fullsized":"2024-09-21T02:37:10.046807+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:43.863444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":104,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401483+0000","last_change":"2024-09-21T02:35:09.949841+0000","last_active":"2024-09-21T02:36:36.401483+0000","last_peered":"2024-09-21T02:36:36.401483+0000","last_clean":"2024-09-21T02:36:36.401483+0000","last_became_active":"2024-09-21T02:35:09.944224+0000","last_became_peered":"2024-09-21T02:35:09.944224+0000","last_unstale":"2024-09-21T02:36:36.401483+0000","last_undegraded":"2024-09-21T02:36:36.401483+0000","last_fullsized":"2024-09-21T02:36:36.401483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_clean_scrub_stamp":"2024-09-21T02:31:38.930125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:56.467576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401816+0000","last_change":"2024-09-21T02:36:32.805600+0000","last_active":"2024-09-21T02:36:36.401816+0000","last_peered":"2024-09-21T02:36:36.401816+0000","last_clean":"2024-09-21T02:36:36.401816+0000","last_became_active":"2024-09-21T02:36:32.805097+0000","last_became_peered":"2024-09-21T02:36:32.805097+0000","last_unstale":"2024-09-21T02:36:36.401816+0000","last_undegraded":"2024-09-21T02:36:36.401816+0000","last_fullsized":"2024-09-21T02:36:36.401816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:25.857807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401783+0000","last_change":"2024-09-21T02:36:25.741176+0000","last_active":"2024-09-21T02:36:36.401783+0000","last_peered":"2024-09-21T02:36:36.401783+0000","last_clean":"2024-09-21T02:36:36.401783+0000","last_became_active":"2024-09-21T02:36:25.740355+0000","last_became_peered":"2024-09-21T02:36:25.740355+0000","last_unstale":"2024-09-21T02:36:36.401783+0000","last_undegraded":"2024-09-21T02:36:36.401783+0000","last_fullsized":"2024-09-21T02:36:36.401783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:47:48.204768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:21.068113+0000","last_change":"2024-09-21T02:36:24.261500+0000","last_active":"2024-09-21T02:37:21.068113+0000","last_peered":"2024-09-21T02:37:21.068113+0000","last_clean":"2024-09-21T02:37:21.068113+0000","last_became_active":"2024-09-21T02:36:24.261089+0000","last_became_peered":"2024-09-21T02:36:24.261089+0000","last_unstale":"2024-09-21T02:37:21.068113+0000","last_undegraded":"2024-09-21T02:37:21.068113+0000","last_fullsized":"2024-09-21T02:37:21.068113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:33:33.158177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380535+0000","last_change":"2024-09-21T02:36:32.796980+0000","last_active":"2024-09-21T02:36:36.380535+0000","last_peered":"2024-09-21T02:36:36.380535+0000","last_clean":"2024-09-21T02:36:36.380535+0000","last_became_active":"2024-09-21T02:36:32.796517+0000","last_became_peered":"2024-09-21T02:36:32.796517+0000","last_unstale":"2024-09-21T02:36:36.380535+0000","last_undegraded":"2024-09-21T02:36:36.380535+0000","last_fullsized":"2024-09-21T02:36:36.380535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:07.230177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401654+0000","last_change":"2024-09-21T02:36:25.750931+0000","last_active":"2024-09-21T02:36:36.401654+0000","last_peered":"2024-09-21T02:36:36.401654+0000","last_clean":"2024-09-21T02:36:36.401654+0000","last_became_active":"2024-09-21T02:36:25.750604+0000","last_became_peered":"2024-09-21T02:36:25.750604+0000","last_unstale":"2024-09-21T02:36:36.401654+0000","last_undegraded":"2024-09-21T02:36:36.401654+0000","last_fullsized":"2024-09-21T02:36:36.401654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:21.325095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044460+0000","last_change":"2024-09-21T02:36:23.730700+0000","last_active":"2024-09-21T02:37:09.044460+0000","last_peered":"2024-09-21T02:37:09.044460+0000","last_clean":"2024-09-21T02:37:09.044460+0000","last_became_active":"2024-09-21T02:36:23.730367+0000","last_became_peered":"2024-09-21T02:36:23.730367+0000","last_unstale":"2024-09-21T02:37:09.044460+0000","last_undegraded":"2024-09-21T02:37:09.044460+0000","last_fullsized":"2024-09-21T02:37:09.044460+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:27:57.373077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991327+0000","last_change":"2024-09-21T02:36:32.805885+0000","last_active":"2024-09-21T02:36:36.991327+0000","last_peered":"2024-09-21T02:36:36.991327+0000","last_clean":"2024-09-21T02:36:36.991327+0000","last_became_active":"2024-09-21T02:36:32.805479+0000","last_became_peered":"2024-09-21T02:36:32.805479+0000","last_unstale":"2024-09-21T02:36:36.991327+0000","last_undegraded":"2024-09-21T02:36:36.991327+0000","last_fullsized":"2024-09-21T02:36:36.991327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:11:29.910642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401892+0000","last_change":"2024-09-21T02:36:25.742542+0000","last_active":"2024-09-21T02:36:36.401892+0000","last_peered":"2024-09-21T02:36:36.401892+0000","last_clean":"2024-09-21T02:36:36.401892+0000","last_became_active":"2024-09-21T02:36:25.742000+0000","last_became_peered":"2024-09-21T02:36:25.742000+0000","last_unstale":"2024-09-21T02:36:36.401892+0000","last_undegraded":"2024-09-21T02:36:36.401892+0000","last_fullsized":"2024-09-21T02:36:36.401892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:52.300212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.041107+0000","last_change":"2024-09-21T02:36:23.723896+0000","last_active":"2024-09-21T02:37:09.041107+0000","last_peered":"2024-09-21T02:37:09.041107+0000","last_clean":"2024-09-21T02:37:09.041107+0000","last_became_active":"2024-09-21T02:36:23.723518+0000","last_became_peered":"2024-09-21T02:36:23.723518+0000","last_unstale":"2024-09-21T02:37:09.041107+0000","last_undegraded":"2024-09-21T02:37:09.041107+0000","last_fullsized":"2024-09-21T02:37:09.041107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:37.250247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396493+0000","last_change":"2024-09-21T02:36:32.799924+0000","last_active":"2024-09-21T02:36:37.396493+0000","last_peered":"2024-09-21T02:36:37.396493+0000","last_clean":"2024-09-21T02:36:37.396493+0000","last_became_active":"2024-09-21T02:36:32.799437+0000","last_became_peered":"2024-09-21T02:36:32.799437+0000","last_unstale":"2024-09-21T02:36:37.396493+0000","last_undegraded":"2024-09-21T02:36:37.396493+0000","last_fullsized":"2024-09-21T02:36:37.396493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:19.258293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900030+0000","last_change":"2024-09-21T02:36:25.747321+0000","last_active":"2024-09-21T02:36:35.900030+0000","last_peered":"2024-09-21T02:36:35.900030+0000","last_clean":"2024-09-21T02:36:35.900030+0000","last_became_active":"2024-09-21T02:36:25.746955+0000","last_became_peered":"2024-09-21T02:36:25.746955+0000","last_unstale":"2024-09-21T02:36:35.900030+0000","last_undegraded":"2024-09-21T02:36:35.900030+0000","last_fullsized":"2024-09-21T02:36:35.900030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:25.148300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.019701+0000","last_change":"2024-09-21T02:36:23.725350+0000","last_active":"2024-09-21T02:37:09.019701+0000","last_peered":"2024-09-21T02:37:09.019701+0000","last_clean":"2024-09-21T02:37:09.019701+0000","last_became_active":"2024-09-21T02:36:23.725090+0000","last_became_peered":"2024-09-21T02:36:23.725090+0000","last_unstale":"2024-09-21T02:37:09.019701+0000","last_undegraded":"2024-09-21T02:37:09.019701+0000","last_fullsized":"2024-09-21T02:37:09.019701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:52.130345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397609+0000","last_change":"2024-09-21T02:36:32.801355+0000","last_active":"2024-09-21T02:36:37.397609+0000","last_peered":"2024-09-21T02:36:37.397609+0000","last_clean":"2024-09-21T02:36:37.397609+0000","last_became_active":"2024-09-21T02:36:32.800803+0000","last_became_peered":"2024-09-21T02:36:32.800803+0000","last_unstale":"2024-09-21T02:36:37.397609+0000","last_undegraded":"2024-09-21T02:36:37.397609+0000","last_fullsized":"2024-09-21T02:36:37.397609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:54.755733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397597+0000","last_change":"2024-09-21T02:36:25.743433+0000","last_active":"2024-09-21T02:36:37.397597+0000","last_peered":"2024-09-21T02:36:37.397597+0000","last_clean":"2024-09-21T02:36:37.397597+0000","last_became_active":"2024-09-21T02:36:25.742643+0000","last_became_peered":"2024-09-21T02:36:25.742643+0000","last_unstale":"2024-09-21T02:36:37.397597+0000","last_undegraded":"2024-09-21T02:36:37.397597+0000","last_fullsized":"2024-09-21T02:36:37.397597+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:20:09.849806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.042742+0000","last_change":"2024-09-21T02:36:23.800229+0000","last_active":"2024-09-21T02:37:09.042742+0000","last_peered":"2024-09-21T02:37:09.042742+0000","last_clean":"2024-09-21T02:37:09.042742+0000","last_became_active":"2024-09-21T02:36:23.799792+0000","last_became_peered":"2024-09-21T02:36:23.799792+0000","last_unstale":"2024-09-21T02:37:09.042742+0000","last_undegraded":"2024-09-21T02:37:09.042742+0000","last_fullsized":"2024-09-21T02:37:09.042742+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:27.446364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899481+0000","last_change":"2024-09-21T02:36:32.799728+0000","last_active":"2024-09-21T02:36:35.899481+0000","last_peered":"2024-09-21T02:36:35.899481+0000","last_clean":"2024-09-21T02:36:35.899481+0000","last_became_active":"2024-09-21T02:36:32.798419+0000","last_became_peered":"2024-09-21T02:36:32.798419+0000","last_unstale":"2024-09-21T02:36:35.899481+0000","last_undegraded":"2024-09-21T02:36:35.899481+0000","last_fullsized":"2024-09-21T02:36:35.899481+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:53.681473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401174+0000","last_change":"2024-09-21T02:36:25.747712+0000","last_active":"2024-09-21T02:36:36.401174+0000","last_peered":"2024-09-21T02:36:36.401174+0000","last_clean":"2024-09-21T02:36:36.401174+0000","last_became_active":"2024-09-21T02:36:25.747197+0000","last_became_peered":"2024-09-21T02:36:25.747197+0000","last_unstale":"2024-09-21T02:36:36.401174+0000","last_undegraded":"2024-09-21T02:36:36.401174+0000","last_fullsized":"2024-09-21T02:36:36.401174+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:03.900360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:19.064761+0000","last_change":"2024-09-21T02:36:23.732156+0000","last_active":"2024-09-21T02:37:19.064761+0000","last_peered":"2024-09-21T02:37:19.064761+0000","last_clean":"2024-09-21T02:37:19.064761+0000","last_became_active":"2024-09-21T02:36:23.730238+0000","last_became_peered":"2024-09-21T02:36:23.730238+0000","last_unstale":"2024-09-21T02:37:19.064761+0000","last_undegraded":"2024-09-21T02:37:19.064761+0000","last_fullsized":"2024-09-21T02:37:19.064761+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:18.053890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.754397+0000","last_change":"2024-09-21T02:36:32.806674+0000","last_active":"2024-09-21T02:36:36.754397+0000","last_peered":"2024-09-21T02:36:36.754397+0000","last_clean":"2024-09-21T02:36:36.754397+0000","last_became_active":"2024-09-21T02:36:32.805373+0000","last_became_peered":"2024-09-21T02:36:32.805373+0000","last_unstale":"2024-09-21T02:36:36.754397+0000","last_undegraded":"2024-09-21T02:36:36.754397+0000","last_fullsized":"2024-09-21T02:36:36.754397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:29.249243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401557+0000","last_change":"2024-09-21T02:36:25.748703+0000","last_active":"2024-09-21T02:36:36.401557+0000","last_peered":"2024-09-21T02:36:36.401557+0000","last_clean":"2024-09-21T02:36:36.401557+0000","last_became_active":"2024-09-21T02:36:25.748340+0000","last_became_peered":"2024-09-21T02:36:25.748340+0000","last_unstale":"2024-09-21T02:36:36.401557+0000","last_undegraded":"2024-09-21T02:36:36.401557+0000","last_fullsized":"2024-09-21T02:36:36.401557+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:07.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044175+0000","last_change":"2024-09-21T02:36:23.732332+0000","last_active":"2024-09-21T02:37:09.044175+0000","last_peered":"2024-09-21T02:37:09.044175+0000","last_clean":"2024-09-21T02:37:09.044175+0000","last_became_active":"2024-09-21T02:36:23.731974+0000","last_became_peered":"2024-09-21T02:36:23.731974+0000","last_unstale":"2024-09-21T02:37:09.044175+0000","last_undegraded":"2024-09-21T02:37:09.044175+0000","last_fullsized":"2024-09-21T02:37:09.044175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:52.284192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900088+0000","last_change":"2024-09-21T02:36:32.809480+0000","last_active":"2024-09-21T02:36:35.900088+0000","last_peered":"2024-09-21T02:36:35.900088+0000","last_clean":"2024-09-21T02:36:35.900088+0000","last_became_active":"2024-09-21T02:36:32.809024+0000","last_became_peered":"2024-09-21T02:36:32.809024+0000","last_unstale":"2024-09-21T02:36:35.900088+0000","last_undegraded":"2024-09-21T02:36:35.900088+0000","last_fullsized":"2024-09-21T02:36:35.900088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:10.608185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.898940+0000","last_change":"2024-09-21T02:36:25.751720+0000","last_active":"2024-09-21T02:36:35.898940+0000","last_peered":"2024-09-21T02:36:35.898940+0000","last_clean":"2024-09-21T02:36:35.898940+0000","last_became_active":"2024-09-21T02:36:25.751351+0000","last_became_peered":"2024-09-21T02:36:25.751351+0000","last_unstale":"2024-09-21T02:36:35.898940+0000","last_undegraded":"2024-09-21T02:36:35.898940+0000","last_fullsized":"2024-09-21T02:36:35.898940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:01.201453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:10.046596+0000","last_change":"2024-09-21T02:36:23.729981+0000","last_active":"2024-09-21T02:37:10.046596+0000","last_peered":"2024-09-21T02:37:10.046596+0000","last_clean":"2024-09-21T02:37:10.046596+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:37:10.046596+0000","last_undegraded":"2024-09-21T02:37:10.046596+0000","last_fullsized":"2024-09-21T02:37:10.046596+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:03.115712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.398637+0000","last_change":"2024-09-21T02:36:32.805286+0000","last_active":"2024-09-21T02:36:37.398637+0000","last_peered":"2024-09-21T02:36:37.398637+0000","last_clean":"2024-09-21T02:36:37.398637+0000","last_became_active":"2024-09-21T02:36:32.804770+0000","last_became_peered":"2024-09-21T02:36:32.804770+0000","last_unstale":"2024-09-21T02:36:37.398637+0000","last_undegraded":"2024-09-21T02:36:37.398637+0000","last_fullsized":"2024-09-21T02:36:37.398637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:18:34.155892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396592+0000","last_change":"2024-09-21T02:36:25.747812+0000","last_active":"2024-09-21T02:36:37.396592+0000","last_peered":"2024-09-21T02:36:37.396592+0000","last_clean":"2024-09-21T02:36:37.396592+0000","last_became_active":"2024-09-21T02:36:25.747187+0000","last_became_peered":"2024-09-21T02:36:25.747187+0000","last_unstale":"2024-09-21T02:36:37.396592+0000","last_undegraded":"2024-09-21T02:36:37.396592+0000","last_fullsized":"2024-09-21T02:36:37.396592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:51.395869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"81'31","reported_seq":95,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:22.070954+0000","last_change":"2024-09-21T02:36:23.735676+0000","last_active":"2024-09-21T02:37:22.070954+0000","last_peered":"2024-09-21T02:37:22.070954+0000","last_clean":"2024-09-21T02:37:22.070954+0000","last_became_active":"2024-09-21T02:36:23.735281+0000","last_became_peered":"2024-09-21T02:36:23.735281+0000","last_unstale":"2024-09-21T02:37:22.070954+0000","last_undegraded":"2024-09-21T02:37:22.070954+0000","last_fullsized":"2024-09-21T02:37:22.070954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:28:01.547245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60300,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":75,"num_write":33,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.753753+0000","last_change":"2024-09-21T02:36:32.806585+0000","last_active":"2024-09-21T02:36:36.753753+0000","last_peered":"2024-09-21T02:36:36.753753+0000","last_clean":"2024-09-21T02:36:36.753753+0000","last_became_active":"2024-09-21T02:36:32.805913+0000","last_became_peered":"2024-09-21T02:36:32.805913+0000","last_unstale":"2024-09-21T02:36:36.753753+0000","last_undegraded":"2024-09-21T02:36:36.753753+0000","last_fullsized":"2024-09-21T02:36:36.753753+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:59:31.888645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380722+0000","last_change":"2024-09-21T02:36:25.742208+0000","last_active":"2024-09-21T02:36:36.380722+0000","last_peered":"2024-09-21T02:36:36.380722+0000","last_clean":"2024-09-21T02:36:36.380722+0000","last_became_active":"2024-09-21T02:36:25.741849+0000","last_became_peered":"2024-09-21T02:36:25.741849+0000","last_unstale":"2024-09-21T02:36:36.380722+0000","last_undegraded":"2024-09-21T02:36:36.380722+0000","last_fullsized":"2024-09-21T02:36:36.380722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:21:43.720481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:08.032639+0000","last_change":"2024-09-21T02:36:23.800751+0000","last_active":"2024-09-21T02:37:08.032639+0000","last_peered":"2024-09-21T02:37:08.032639+0000","last_clean":"2024-09-21T02:37:08.032639+0000","last_became_active":"2024-09-21T02:36:23.800341+0000","last_became_peered":"2024-09-21T02:36:23.800341+0000","last_unstale":"2024-09-21T02:37:08.032639+0000","last_undegraded":"2024-09-21T02:37:08.032639+0000","last_fullsized":"2024-09-21T02:37:08.032639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:08:02.556549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396671+0000","last_change":"2024-09-21T02:36:32.802879+0000","last_active":"2024-09-21T02:36:37.396671+0000","last_peered":"2024-09-21T02:36:37.396671+0000","last_clean":"2024-09-21T02:36:37.396671+0000","last_became_active":"2024-09-21T02:36:32.802301+0000","last_became_peered":"2024-09-21T02:36:32.802301+0000","last_unstale":"2024-09-21T02:36:37.396671+0000","last_undegraded":"2024-09-21T02:36:37.396671+0000","last_fullsized":"2024-09-21T02:36:37.396671+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:08.135814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991675+0000","last_change":"2024-09-21T02:36:25.744647+0000","last_active":"2024-09-21T02:36:36.991675+0000","last_peered":"2024-09-21T02:36:36.991675+0000","last_clean":"2024-09-21T02:36:36.991675+0000","last_became_active":"2024-09-21T02:36:25.744049+0000","last_became_peered":"2024-09-21T02:36:25.744049+0000","last_unstale":"2024-09-21T02:36:36.991675+0000","last_undegraded":"2024-09-21T02:36:36.991675+0000","last_fullsized":"2024-09-21T02:36:36.991675+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:52:02.210380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.043661+0000","last_change":"2024-09-21T02:36:23.733590+0000","last_active":"2024-09-21T02:37:09.043661+0000","last_peered":"2024-09-21T02:37:09.043661+0000","last_clean":"2024-09-21T02:37:09.043661+0000","last_became_active":"2024-09-21T02:36:23.732922+0000","last_became_peered":"2024-09-21T02:36:23.732922+0000","last_unstale":"2024-09-21T02:37:09.043661+0000","last_undegraded":"2024-09-21T02:37:09.043661+0000","last_fullsized":"2024-09-21T02:37:09.043661+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.753659+0000","last_change":"2024-09-21T02:36:32.806540+0000","last_active":"2024-09-21T02:36:36.753659+0000","last_peered":"2024-09-21T02:36:36.753659+0000","last_clean":"2024-09-21T02:36:36.753659+0000","last_became_active":"2024-09-21T02:36:32.805612+0000","last_became_peered":"2024-09-21T02:36:32.805612+0000","last_unstale":"2024-09-21T02:36:36.753659+0000","last_undegraded":"2024-09-21T02:36:36.753659+0000","last_fullsized":"2024-09-21T02:36:36.753659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:23:24.686383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380907+0000","last_change":"2024-09-21T02:36:25.746229+0000","last_active":"2024-09-21T02:36:36.380907+0000","last_peered":"2024-09-21T02:36:36.380907+0000","last_clean":"2024-09-21T02:36:36.380907+0000","last_became_active":"2024-09-21T02:36:25.745892+0000","last_became_peered":"2024-09-21T02:36:25.745892+0000","last_unstale":"2024-09-21T02:36:36.380907+0000","last_undegraded":"2024-09-21T02:36:36.380907+0000","last_fullsized":"2024-09-21T02:36:36.380907+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:55.154499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:23.069956+0000","last_change":"2024-09-21T02:36:23.730987+0000","last_active":"2024-09-21T02:37:23.069956+0000","last_peered":"2024-09-21T02:37:23.069956+0000","last_clean":"2024-09-21T02:37:23.069956+0000","last_became_active":"2024-09-21T02:36:23.730566+0000","last_became_peered":"2024-09-21T02:36:23.730566+0000","last_unstale":"2024-09-21T02:37:23.069956+0000","last_undegraded":"2024-09-21T02:37:23.069956+0000","last_fullsized":"2024-09-21T02:37:23.069956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:18:12.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395905+0000","last_change":"2024-09-21T02:36:32.804608+0000","last_active":"2024-09-21T02:36:37.395905+0000","last_peered":"2024-09-21T02:36:37.395905+0000","last_clean":"2024-09-21T02:36:37.395905+0000","last_became_active":"2024-09-21T02:36:32.803567+0000","last_became_peered":"2024-09-21T02:36:32.803567+0000","last_unstale":"2024-09-21T02:36:37.395905+0000","last_undegraded":"2024-09-21T02:36:37.395905+0000","last_fullsized":"2024-09-21T02:36:37.395905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:41.692644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991291+0000","last_change":"2024-09-21T02:36:32.804687+0000","last_active":"2024-09-21T02:36:36.991291+0000","last_peered":"2024-09-21T02:36:36.991291+0000","last_clean":"2024-09-21T02:36:36.991291+0000","last_became_active":"2024-09-21T02:36:32.804232+0000","last_became_peered":"2024-09-21T02:36:32.804232+0000","last_unstale":"2024-09-21T02:36:36.991291+0000","last_undegraded":"2024-09-21T02:36:36.991291+0000","last_fullsized":"2024-09-21T02:36:36.991291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:43.899924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.753693+0000","last_change":"2024-09-21T02:36:25.749114+0000","last_active":"2024-09-21T02:36:36.753693+0000","last_peered":"2024-09-21T02:36:36.753693+0000","last_clean":"2024-09-21T02:36:36.753693+0000","last_became_active":"2024-09-21T02:36:25.748758+0000","last_became_peered":"2024-09-21T02:36:25.748758+0000","last_unstale":"2024-09-21T02:36:36.753693+0000","last_undegraded":"2024-09-21T02:36:36.753693+0000","last_fullsized":"2024-09-21T02:36:36.753693+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:57.958533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.043657+0000","last_change":"2024-09-21T02:36:23.731473+0000","last_active":"2024-09-21T02:37:09.043657+0000","last_peered":"2024-09-21T02:37:09.043657+0000","last_clean":"2024-09-21T02:37:09.043657+0000","last_became_active":"2024-09-21T02:36:23.731134+0000","last_became_peered":"2024-09-21T02:36:23.731134+0000","last_unstale":"2024-09-21T02:37:09.043657+0000","last_undegraded":"2024-09-21T02:37:09.043657+0000","last_fullsized":"2024-09-21T02:37:09.043657+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:05.368941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900104+0000","last_change":"2024-09-21T02:36:32.809630+0000","last_active":"2024-09-21T02:36:35.900104+0000","last_peered":"2024-09-21T02:36:35.900104+0000","last_clean":"2024-09-21T02:36:35.900104+0000","last_became_active":"2024-09-21T02:36:32.809030+0000","last_became_peered":"2024-09-21T02:36:32.809030+0000","last_unstale":"2024-09-21T02:36:35.900104+0000","last_undegraded":"2024-09-21T02:36:35.900104+0000","last_fullsized":"2024-09-21T02:36:35.900104+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:57:42.436590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395972+0000","last_change":"2024-09-21T02:36:25.742558+0000","last_active":"2024-09-21T02:36:37.395972+0000","last_peered":"2024-09-21T02:36:37.395972+0000","last_clean":"2024-09-21T02:36:37.395972+0000","last_became_active":"2024-09-21T02:36:25.742045+0000","last_became_peered":"2024-09-21T02:36:25.742045+0000","last_unstale":"2024-09-21T02:36:37.395972+0000","last_undegraded":"2024-09-21T02:36:37.395972+0000","last_fullsized":"2024-09-21T02:36:37.395972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:57:06.705561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:07.025569+0000","last_change":"2024-09-21T02:36:23.724345+0000","last_active":"2024-09-21T02:37:07.025569+0000","last_peered":"2024-09-21T02:37:07.025569+0000","last_clean":"2024-09-21T02:37:07.025569+0000","last_became_active":"2024-09-21T02:36:23.723635+0000","last_became_peered":"2024-09-21T02:36:23.723635+0000","last_unstale":"2024-09-21T02:37:07.025569+0000","last_undegraded":"2024-09-21T02:37:07.025569+0000","last_fullsized":"2024-09-21T02:37:07.025569+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:18.812176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:08.034637+0000","last_change":"2024-09-21T02:36:23.732897+0000","last_active":"2024-09-21T02:37:08.034637+0000","last_peered":"2024-09-21T02:37:08.034637+0000","last_clean":"2024-09-21T02:37:08.034637+0000","last_became_active":"2024-09-21T02:36:23.732509+0000","last_became_peered":"2024-09-21T02:36:23.732509+0000","last_unstale":"2024-09-21T02:37:08.034637+0000","last_undegraded":"2024-09-21T02:37:08.034637+0000","last_fullsized":"2024-09-21T02:37:08.034637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:12:27.117962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.398180+0000","last_change":"2024-09-21T02:36:25.748929+0000","last_active":"2024-09-21T02:36:37.398180+0000","last_peered":"2024-09-21T02:36:37.398180+0000","last_clean":"2024-09-21T02:36:37.398180+0000","last_became_active":"2024-09-21T02:36:25.748588+0000","last_became_peered":"2024-09-21T02:36:25.748588+0000","last_unstale":"2024-09-21T02:36:37.398180+0000","last_undegraded":"2024-09-21T02:36:37.398180+0000","last_fullsized":"2024-09-21T02:36:37.398180+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.086472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397254+0000","last_change":"2024-09-21T02:36:32.807655+0000","last_active":"2024-09-21T02:36:37.397254+0000","last_peered":"2024-09-21T02:36:37.397254+0000","last_clean":"2024-09-21T02:36:37.397254+0000","last_became_active":"2024-09-21T02:36:32.806995+0000","last_became_peered":"2024-09-21T02:36:32.806995+0000","last_unstale":"2024-09-21T02:36:37.397254+0000","last_undegraded":"2024-09-21T02:36:37.397254+0000","last_fullsized":"2024-09-21T02:36:37.397254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:28:20.420591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":52,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:20.066627+0000","last_change":"2024-09-21T02:36:23.729427+0000","last_active":"2024-09-21T02:37:20.066627+0000","last_peered":"2024-09-21T02:37:20.066627+0000","last_clean":"2024-09-21T02:37:20.066627+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:37:20.066627+0000","last_undegraded":"2024-09-21T02:37:20.066627+0000","last_fullsized":"2024-09-21T02:37:20.066627+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:10:41.251665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396720+0000","last_change":"2024-09-21T02:36:25.743212+0000","last_active":"2024-09-21T02:36:37.396720+0000","last_peered":"2024-09-21T02:36:37.396720+0000","last_clean":"2024-09-21T02:36:37.396720+0000","last_became_active":"2024-09-21T02:36:25.742777+0000","last_became_peered":"2024-09-21T02:36:25.742777+0000","last_unstale":"2024-09-21T02:36:37.396720+0000","last_undegraded":"2024-09-21T02:36:37.396720+0000","last_fullsized":"2024-09-21T02:36:37.396720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:12:19.917214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397806+0000","last_change":"2024-09-21T02:36:32.801880+0000","last_active":"2024-09-21T02:36:37.397806+0000","last_peered":"2024-09-21T02:36:37.397806+0000","last_clean":"2024-09-21T02:36:37.397806+0000","last_became_active":"2024-09-21T02:36:32.801389+0000","last_became_peered":"2024-09-21T02:36:32.801389+0000","last_unstale":"2024-09-21T02:36:37.397806+0000","last_undegraded":"2024-09-21T02:36:37.397806+0000","last_fullsized":"2024-09-21T02:36:37.397806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:11:44.877062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899699+0000","last_change":"2024-09-21T02:36:25.754014+0000","last_active":"2024-09-21T02:36:35.899699+0000","last_peered":"2024-09-21T02:36:35.899699+0000","last_clean":"2024-09-21T02:36:35.899699+0000","last_became_active":"2024-09-21T02:36:25.753548+0000","last_became_peered":"2024-09-21T02:36:25.753548+0000","last_unstale":"2024-09-21T02:36:35.899699+0000","last_undegraded":"2024-09-21T02:36:35.899699+0000","last_fullsized":"2024-09-21T02:36:35.899699+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:03.850625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397868+0000","last_change":"2024-09-21T02:36:25.751909+0000","last_active":"2024-09-21T02:36:37.397868+0000","last_peered":"2024-09-21T02:36:37.397868+0000","last_clean":"2024-09-21T02:36:37.397868+0000","last_became_active":"2024-09-21T02:36:25.751484+0000","last_became_peered":"2024-09-21T02:36:25.751484+0000","last_unstale":"2024-09-21T02:36:37.397868+0000","last_undegraded":"2024-09-21T02:36:37.397868+0000","last_fullsized":"2024-09-21T02:36:37.397868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:49:20.034294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396629+0000","last_change":"2024-09-21T02:36:25.752839+0000","last_active":"2024-09-21T02:36:37.396629+0000","last_peered":"2024-09-21T02:36:37.396629+0000","last_clean":"2024-09-21T02:36:37.396629+0000","last_became_active":"2024-09-21T02:36:25.750882+0000","last_became_peered":"2024-09-21T02:36:25.750882+0000","last_unstale":"2024-09-21T02:36:37.396629+0000","last_undegraded":"2024-09-21T02:36:37.396629+0000","last_fullsized":"2024-09-21T02:36:37.396629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:53.028611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991289+0000","last_change":"2024-09-21T02:36:25.744624+0000","last_active":"2024-09-21T02:36:36.991289+0000","last_peered":"2024-09-21T02:36:36.991289+0000","last_clean":"2024-09-21T02:36:36.991289+0000","last_became_active":"2024-09-21T02:36:25.744173+0000","last_became_peered":"2024-09-21T02:36:25.744173+0000","last_unstale":"2024-09-21T02:36:36.991289+0000","last_undegraded":"2024-09-21T02:36:36.991289+0000","last_fullsized":"2024-09-21T02:36:36.991289+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:55.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899306+0000","last_change":"2024-09-21T02:36:25.743669+0000","last_active":"2024-09-21T02:36:35.899306+0000","last_peered":"2024-09-21T02:36:35.899306+0000","last_clean":"2024-09-21T02:36:35.899306+0000","last_became_active":"2024-09-21T02:36:25.743332+0000","last_became_peered":"2024-09-21T02:36:25.743332+0000","last_unstale":"2024-09-21T02:36:35.899306+0000","last_undegraded":"2024-09-21T02:36:35.899306+0000","last_fullsized":"2024-09-21T02:36:35.899306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:25.189197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900404+0000","last_change":"2024-09-21T02:36:25.743476+0000","last_active":"2024-09-21T02:36:35.900404+0000","last_peered":"2024-09-21T02:36:35.900404+0000","last_clean":"2024-09-21T02:36:35.900404+0000","last_became_active":"2024-09-21T02:36:25.743117+0000","last_became_peered":"2024-09-21T02:36:25.743117+0000","last_unstale":"2024-09-21T02:36:35.900404+0000","last_undegraded":"2024-09-21T02:36:35.900404+0000","last_fullsized":"2024-09-21T02:36:35.900404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:08.084109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396952+0000","last_change":"2024-09-21T02:36:25.753014+0000","last_active":"2024-09-21T02:36:37.396952+0000","last_peered":"2024-09-21T02:36:37.396952+0000","last_clean":"2024-09-21T02:36:37.396952+0000","last_became_active":"2024-09-21T02:36:25.751232+0000","last_became_peered":"2024-09-21T02:36:25.751232+0000","last_unstale":"2024-09-21T02:36:37.396952+0000","last_undegraded":"2024-09-21T02:36:37.396952+0000","last_fullsized":"2024-09-21T02:36:37.396952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:03.052737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396567+0000","last_change":"2024-09-21T02:36:25.747296+0000","last_active":"2024-09-21T02:36:37.396567+0000","last_peered":"2024-09-21T02:36:37.396567+0000","last_clean":"2024-09-21T02:36:37.396567+0000","last_became_active":"2024-09-21T02:36:25.746832+0000","last_became_peered":"2024-09-21T02:36:25.746832+0000","last_unstale":"2024-09-21T02:36:37.396567+0000","last_undegraded":"2024-09-21T02:36:37.396567+0000","last_fullsized":"2024-09-21T02:36:37.396567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:21:55.099017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396829+0000","last_change":"2024-09-21T02:36:25.749605+0000","last_active":"2024-09-21T02:36:37.396829+0000","last_peered":"2024-09-21T02:36:37.396829+0000","last_clean":"2024-09-21T02:36:37.396829+0000","last_became_active":"2024-09-21T02:36:25.749165+0000","last_became_peered":"2024-09-21T02:36:25.749165+0000","last_unstale":"2024-09-21T02:36:37.396829+0000","last_undegraded":"2024-09-21T02:36:37.396829+0000","last_fullsized":"2024-09-21T02:36:37.396829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:15:51.412761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396144+0000","last_change":"2024-09-21T02:36:25.747494+0000","last_active":"2024-09-21T02:36:37.396144+0000","last_peered":"2024-09-21T02:36:37.396144+0000","last_clean":"2024-09-21T02:36:37.396144+0000","last_became_active":"2024-09-21T02:36:25.746965+0000","last_became_peered":"2024-09-21T02:36:25.746965+0000","last_unstale":"2024-09-21T02:36:37.396144+0000","last_undegraded":"2024-09-21T02:36:37.396144+0000","last_fullsized":"2024-09-21T02:36:37.396144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:51.309307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380672+0000","last_change":"2024-09-21T02:36:25.749517+0000","last_active":"2024-09-21T02:36:36.380672+0000","last_peered":"2024-09-21T02:36:36.380672+0000","last_clean":"2024-09-21T02:36:36.380672+0000","last_became_active":"2024-09-21T02:36:25.749117+0000","last_became_peered":"2024-09-21T02:36:25.749117+0000","last_unstale":"2024-09-21T02:36:36.380672+0000","last_undegraded":"2024-09-21T02:36:36.380672+0000","last_fullsized":"2024-09-21T02:36:36.380672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:38.758019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899175+0000","last_change":"2024-09-21T02:36:25.736067+0000","last_active":"2024-09-21T02:36:35.899175+0000","last_peered":"2024-09-21T02:36:35.899175+0000","last_clean":"2024-09-21T02:36:35.899175+0000","last_became_active":"2024-09-21T02:36:25.735710+0000","last_became_peered":"2024-09-21T02:36:25.735710+0000","last_unstale":"2024-09-21T02:36:35.899175+0000","last_undegraded":"2024-09-21T02:36:35.899175+0000","last_fullsized":"2024-09-21T02:36:35.899175+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.559448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899895+0000","last_change":"2024-09-21T02:36:25.746307+0000","last_active":"2024-09-21T02:36:35.899895+0000","last_peered":"2024-09-21T02:36:35.899895+0000","last_clean":"2024-09-21T02:36:35.899895+0000","last_became_active":"2024-09-21T02:36:25.745872+0000","last_became_peered":"2024-09-21T02:36:25.745872+0000","last_unstale":"2024-09-21T02:36:35.899895+0000","last_undegraded":"2024-09-21T02:36:35.899895+0000","last_fullsized":"2024-09-21T02:36:35.899895+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:58:53.074559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397917+0000","last_change":"2024-09-21T02:36:25.753393+0000","last_active":"2024-09-21T02:36:37.397917+0000","last_peered":"2024-09-21T02:36:37.397917+0000","last_clean":"2024-09-21T02:36:37.397917+0000","last_became_active":"2024-09-21T02:36:25.751871+0000","last_became_peered":"2024-09-21T02:36:25.751871+0000","last_unstale":"2024-09-21T02:36:37.397917+0000","last_undegraded":"2024-09-21T02:36:37.397917+0000","last_fullsized":"2024-09-21T02:36:37.397917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:28.199451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.754175+0000","last_change":"2024-09-21T02:36:25.744845+0000","last_active":"2024-09-21T02:36:36.754175+0000","last_peered":"2024-09-21T02:36:36.754175+0000","last_clean":"2024-09-21T02:36:36.754175+0000","last_became_active":"2024-09-21T02:36:25.744492+0000","last_became_peered":"2024-09-21T02:36:25.744492+0000","last_unstale":"2024-09-21T02:36:36.754175+0000","last_undegraded":"2024-09-21T02:36:36.754175+0000","last_fullsized":"2024-09-21T02:36:36.754175+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:08.524871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900318+0000","last_change":"2024-09-21T02:36:25.751187+0000","last_active":"2024-09-21T02:36:35.900318+0000","last_peered":"2024-09-21T02:36:35.900318+0000","last_clean":"2024-09-21T02:36:35.900318+0000","last_became_active":"2024-09-21T02:36:25.750813+0000","last_became_peered":"2024-09-21T02:36:25.750813+0000","last_unstale":"2024-09-21T02:36:35.900318+0000","last_undegraded":"2024-09-21T02:36:35.900318+0000","last_fullsized":"2024-09-21T02:36:35.900318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.590210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380272+0000","last_change":"2024-09-21T02:36:25.744046+0000","last_active":"2024-09-21T02:36:36.380272+0000","last_peered":"2024-09-21T02:36:36.380272+0000","last_clean":"2024-09-21T02:36:36.380272+0000","last_became_active":"2024-09-21T02:36:25.743695+0000","last_became_peered":"2024-09-21T02:36:25.743695+0000","last_unstale":"2024-09-21T02:36:36.380272+0000","last_undegraded":"2024-09-21T02:36:36.380272+0000","last_fullsized":"2024-09-21T02:36:36.380272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:12.092940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991020+0000","last_change":"2024-09-21T02:36:25.749113+0000","last_active":"2024-09-21T02:36:36.991020+0000","last_peered":"2024-09-21T02:36:36.991020+0000","last_clean":"2024-09-21T02:36:36.991020+0000","last_became_active":"2024-09-21T02:36:25.748759+0000","last_became_peered":"2024-09-21T02:36:25.748759+0000","last_unstale":"2024-09-21T02:36:36.991020+0000","last_undegraded":"2024-09-21T02:36:36.991020+0000","last_fullsized":"2024-09-21T02:36:36.991020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:22.785273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.990999+0000","last_change":"2024-09-21T02:36:25.745454+0000","last_active":"2024-09-21T02:36:36.990999+0000","last_peered":"2024-09-21T02:36:36.990999+0000","last_clean":"2024-09-21T02:36:36.990999+0000","last_became_active":"2024-09-21T02:36:25.745108+0000","last_became_peered":"2024-09-21T02:36:25.745108+0000","last_unstale":"2024-09-21T02:36:36.990999+0000","last_undegraded":"2024-09-21T02:36:36.990999+0000","last_fullsized":"2024-09-21T02:36:36.990999+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:36.710456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899061+0000","last_change":"2024-09-21T02:36:25.737213+0000","last_active":"2024-09-21T02:36:35.899061+0000","last_peered":"2024-09-21T02:36:35.899061+0000","last_clean":"2024-09-21T02:36:35.899061+0000","last_became_active":"2024-09-21T02:36:25.736641+0000","last_became_peered":"2024-09-21T02:36:25.736641+0000","last_unstale":"2024-09-21T02:36:35.899061+0000","last_undegraded":"2024-09-21T02:36:35.899061+0000","last_fullsized":"2024-09-21T02:36:35.899061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:47:50.258805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397271+0000","last_change":"2024-09-21T02:36:25.748230+0000","last_active":"2024-09-21T02:36:37.397271+0000","last_peered":"2024-09-21T02:36:37.397271+0000","last_clean":"2024-09-21T02:36:37.397271+0000","last_became_active":"2024-09-21T02:36:25.747859+0000","last_became_peered":"2024-09-21T02:36:25.747859+0000","last_unstale":"2024-09-21T02:36:37.397271+0000","last_undegraded":"2024-09-21T02:36:37.397271+0000","last_fullsized":"2024-09-21T02:36:37.397271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:15:21.244825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401717+0000","last_change":"2024-09-21T02:36:25.744379+0000","last_active":"2024-09-21T02:36:36.401717+0000","last_peered":"2024-09-21T02:36:36.401717+0000","last_clean":"2024-09-21T02:36:36.401717+0000","last_became_active":"2024-09-21T02:36:25.744003+0000","last_became_peered":"2024-09-21T02:36:25.744003+0000","last_unstale":"2024-09-21T02:36:36.401717+0000","last_undegraded":"2024-09-21T02:36:36.401717+0000","last_fullsized":"2024-09-21T02:36:36.401717+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:59:59.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900340+0000","last_change":"2024-09-21T02:36:25.755616+0000","last_active":"2024-09-21T02:36:35.900340+0000","last_peered":"2024-09-21T02:36:35.900340+0000","last_clean":"2024-09-21T02:36:35.900340+0000","last_became_active":"2024-09-21T02:36:25.755304+0000","last_became_peered":"2024-09-21T02:36:25.755304+0000","last_unstale":"2024-09-21T02:36:35.900340+0000","last_undegraded":"2024-09-21T02:36:35.900340+0000","last_fullsized":"2024-09-21T02:36:35.900340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:24.813780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380995+0000","last_change":"2024-09-21T02:36:25.739747+0000","last_active":"2024-09-21T02:36:36.380995+0000","last_peered":"2024-09-21T02:36:36.380995+0000","last_clean":"2024-09-21T02:36:36.380995+0000","last_became_active":"2024-09-21T02:36:25.739404+0000","last_became_peered":"2024-09-21T02:36:25.739404+0000","last_unstale":"2024-09-21T02:36:36.380995+0000","last_undegraded":"2024-09-21T02:36:36.380995+0000","last_fullsized":"2024-09-21T02:36:36.380995+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:17.975419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395850+0000","last_change":"2024-09-21T02:36:25.747723+0000","last_active":"2024-09-21T02:36:37.395850+0000","last_peered":"2024-09-21T02:36:37.395850+0000","last_clean":"2024-09-21T02:36:37.395850+0000","last_became_active":"2024-09-21T02:36:25.747368+0000","last_became_peered":"2024-09-21T02:36:25.747368+0000","last_unstale":"2024-09-21T02:36:37.395850+0000","last_undegraded":"2024-09-21T02:36:37.395850+0000","last_fullsized":"2024-09-21T02:36:37.395850+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.319756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401352+0000","last_change":"2024-09-21T02:36:25.743772+0000","last_active":"2024-09-21T02:36:36.401352+0000","last_peered":"2024-09-21T02:36:36.401352+0000","last_clean":"2024-09-21T02:36:36.401352+0000","last_became_active":"2024-09-21T02:36:25.743403+0000","last_became_peered":"2024-09-21T02:36:25.743403+0000","last_unstale":"2024-09-21T02:36:36.401352+0000","last_undegraded":"2024-09-21T02:36:36.401352+0000","last_fullsized":"2024-09-21T02:36:36.401352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:11:29.930248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899740+0000","last_change":"2024-09-21T02:36:25.746911+0000","last_active":"2024-09-21T02:36:35.899740+0000","last_peered":"2024-09-21T02:36:35.899740+0000","last_clean":"2024-09-21T02:36:35.899740+0000","last_became_active":"2024-09-21T02:36:25.746426+0000","last_became_peered":"2024-09-21T02:36:25.746426+0000","last_unstale":"2024-09-21T02:36:35.899740+0000","last_undegraded":"2024-09-21T02:36:35.899740+0000","last_fullsized":"2024-09-21T02:36:35.899740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:12.835851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397123+0000","last_change":"2024-09-21T02:36:25.751278+0000","last_active":"2024-09-21T02:36:37.397123+0000","last_peered":"2024-09-21T02:36:37.397123+0000","last_clean":"2024-09-21T02:36:37.397123+0000","last_became_active":"2024-09-21T02:36:25.750414+0000","last_became_peered":"2024-09-21T02:36:25.750414+0000","last_unstale":"2024-09-21T02:36:37.397123+0000","last_undegraded":"2024-09-21T02:36:37.397123+0000","last_fullsized":"2024-09-21T02:36:37.397123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:13:56.080122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397213+0000","last_change":"2024-09-21T02:36:25.752852+0000","last_active":"2024-09-21T02:36:37.397213+0000","last_peered":"2024-09-21T02:36:37.397213+0000","last_clean":"2024-09-21T02:36:37.397213+0000","last_became_active":"2024-09-21T02:36:25.752514+0000","last_became_peered":"2024-09-21T02:36:25.752514+0000","last_unstale":"2024-09-21T02:36:37.397213+0000","last_undegraded":"2024-09-21T02:36:37.397213+0000","last_fullsized":"2024-09-21T02:36:37.397213+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:55.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991017+0000","last_change":"2024-09-21T02:36:25.750081+0000","last_active":"2024-09-21T02:36:36.991017+0000","last_peered":"2024-09-21T02:36:36.991017+0000","last_clean":"2024-09-21T02:36:36.991017+0000","last_became_active":"2024-09-21T02:36:25.749602+0000","last_became_peered":"2024-09-21T02:36:25.749602+0000","last_unstale":"2024-09-21T02:36:36.991017+0000","last_undegraded":"2024-09-21T02:36:36.991017+0000","last_fullsized":"2024-09-21T02:36:36.991017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:06.785998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991322+0000","last_change":"2024-09-21T02:36:25.746897+0000","last_active":"2024-09-21T02:36:36.991322+0000","last_peered":"2024-09-21T02:36:36.991322+0000","last_clean":"2024-09-21T02:36:36.991322+0000","last_became_active":"2024-09-21T02:36:25.746477+0000","last_became_peered":"2024-09-21T02:36:25.746477+0000","last_unstale":"2024-09-21T02:36:36.991322+0000","last_undegraded":"2024-09-21T02:36:36.991322+0000","last_fullsized":"2024-09-21T02:36:36.991322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:34:00.739410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900270+0000","last_change":"2024-09-21T02:36:25.745233+0000","last_active":"2024-09-21T02:36:35.900270+0000","last_peered":"2024-09-21T02:36:35.900270+0000","last_clean":"2024-09-21T02:36:35.900270+0000","last_became_active":"2024-09-21T02:36:25.744637+0000","last_became_peered":"2024-09-21T02:36:25.744637+0000","last_unstale":"2024-09-21T02:36:35.900270+0000","last_undegraded":"2024-09-21T02:36:35.900270+0000","last_fullsized":"2024-09-21T02:36:35.900270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:34:59.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67504,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":162,"num_read_kb":158,"num_write":244,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":202512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249064,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":69,"seq":296352743447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":447975,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906973,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":1094700,"data_compressed":53859,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070499,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":451610,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266728,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":913408,"data_stored":447892,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462957,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":1044389,"data_compressed":53859,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659186,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":443018,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855416,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":1039971,"data_compressed":53859,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018940,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":901120,"data_stored":447092,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215169,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":905216,"data_stored":442726,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":505350,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":504348,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":917504,"data_stored":444865,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:37:26.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:37:26.515 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:26.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:26 smithi029 ceph-mon[31006]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:37:26.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:26 smithi190 ceph-mon[33871]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:37:27.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:26 smithi066 ceph-mon[33436]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:37:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 -- 172.21.15.29:0/280548126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c105270 msgr2=0x7f609c10d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 --2- 172.21.15.29:0/280548126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c105270 0x7f609c10d8b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6084009900 tx=0x7f608402a410 comp rx=0 tx=0).stop 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 -- 172.21.15.29:0/280548126 shutdown_connections 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 --2- 172.21.15.29:0/280548126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609c10ddf0 0x7f609c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 --2- 172.21.15.29:0/280548126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c105270 0x7f609c10d8b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 --2- 172.21.15.29:0/280548126 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f609c104840 0x7f609c104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 -- 172.21.15.29:0/280548126 >> 172.21.15.29:0/280548126 conn(0x7f609c079730 msgr2=0x7f609c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 -- 172.21.15.29:0/280548126 shutdown_connections 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.053+0000 7f60a0ced640 1 -- 172.21.15.29:0/280548126 wait complete. 2024-09-21T02:37:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f60a0ced640 1 Processor -- start 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f60a0ced640 1 -- start start 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f60a0ced640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f609c104840 0x7f609c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f60a0ced640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609c105270 0x7f609c19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f60a0ced640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c10ddf0 0x7f609c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f60a0ced640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f609c113980 con 0x7f609c10ddf0 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f60a0ced640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f609c113800 con 0x7f609c104840 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f60a0ced640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f609c113b00 con 0x7f609c105270 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f609ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c10ddf0 0x7f609c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f609ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c10ddf0 0x7f609c19fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45744/0 (socket says 172.21.15.29:45744) 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f609ad9d640 1 -- 172.21.15.29:0/3417591047 learned_addr learned my addr 172.21.15.29:0/3417591047 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.054+0000 7f609a59c640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f609c104840 0x7f609c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f6099d9b640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609c105270 0x7f609c19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f609ad9d640 1 -- 172.21.15.29:0/3417591047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609c105270 msgr2=0x7f609c19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f609ad9d640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609c105270 0x7f609c19aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f609ad9d640 1 -- 172.21.15.29:0/3417591047 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f609c104840 msgr2=0x7f609c19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f609ad9d640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f609c104840 0x7f609c19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f609ad9d640 1 -- 172.21.15.29:0/3417591047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609c1a03e0 con 0x7f609c10ddf0 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f609ad9d640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c10ddf0 0x7f609c19fcb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f609400d750 tx=0x7f609400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f60937fe640 1 -- 172.21.15.29:0/3417591047 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094017070 con 0x7f609c10ddf0 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f60937fe640 1 -- 172.21.15.29:0/3417591047 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60940054b0 con 0x7f609c10ddf0 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609c1a06d0 con 0x7f609c10ddf0 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f60937fe640 1 -- 172.21.15.29:0/3417591047 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094004b90 con 0x7f609c10ddf0 2024-09-21T02:37:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.055+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f609c1a6e30 con 0x7f609c10ddf0 2024-09-21T02:37:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.056+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f609c104c50 con 0x7f609c10ddf0 2024-09-21T02:37:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.058+0000 7f60937fe640 1 -- 172.21.15.29:0/3417591047 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f6094004d30 con 0x7f609c10ddf0 2024-09-21T02:37:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.059+0000 7f60937fe640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60880b6620 0x7f60880b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.059+0000 7f609a59c640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60880b6620 0x7f60880b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.059+0000 7f60937fe640 1 -- 172.21.15.29:0/3417591047 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f60940e7140 con 0x7f609c10ddf0 2024-09-21T02:37:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.060+0000 7f609a59c640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60880b6620 0x7f60880b8b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f609c06bc00 tx=0x7f608c00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.064+0000 7f60937fe640 1 -- 172.21.15.29:0/3417591047 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f60940ad5d0 con 0x7f609c10ddf0 2024-09-21T02:37:27.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.249+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f609c063800 con 0x7f60880b6620 2024-09-21T02:37:27.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.255+0000 7f60937fe640 1 -- 172.21.15.29:0/3417591047 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340623 (secure 0 0 0) 0x7f609c063800 con 0x7f60880b6620 2024-09-21T02:37:27.259 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:27.261 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60880b6620 msgr2=0x7f60880b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60880b6620 0x7f60880b8b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f609c06bc00 tx=0x7f608c00d040 comp rx=0 tx=0).stop 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c10ddf0 msgr2=0x7f609c19fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c10ddf0 0x7f609c19fcb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f609400d750 tx=0x7f609400dc10 comp rx=0 tx=0).stop 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 shutdown_connections 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60880b6620 0x7f60880b8b30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c10ddf0 0x7f609c19fcb0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609c105270 0x7f609c19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 --2- 172.21.15.29:0/3417591047 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f609c104840 0x7f609c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 >> 172.21.15.29:0/3417591047 conn(0x7f609c079730 msgr2=0x7f609c0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.258+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 shutdown_connections 2024-09-21T02:37:27.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:27.259+0000 7f60a0ced640 1 -- 172.21.15.29:0/3417591047 wait complete. 2024-09-21T02:37:27.745 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:27 smithi029 ceph-mon[31006]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-09-21T02:37:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:27 smithi190 ceph-mon[33871]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-09-21T02:37:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:27 smithi066 ceph-mon[33436]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-09-21T02:37:28.125 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-21T02:37:25.942939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658106,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":253,"num_read_kb":227,"num_write":369,"num_write_kb":2290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364408,"kb_used_data":11000,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124577416,"statfs":{"total":1151940427776,"available":1151567273984,"internally_reserved":0,"allocated":11264000,"data_stored":7313936,"data_compressed":319842,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":50505,"num_objects":5,"num_object_clones":0,"num_object_copies":17,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":106,"num_write":36,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004025"},"pg_stats":[{"pgid":"2.1f","version":"81'13","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:22.067590+0000","last_change":"2024-09-21T02:36:23.729337+0000","last_active":"2024-09-21T02:37:22.067590+0000","last_peered":"2024-09-21T02:37:22.067590+0000","last_clean":"2024-09-21T02:37:22.067590+0000","last_became_active":"2024-09-21T02:36:23.728576+0000","last_became_peered":"2024-09-21T02:36:23.728576+0000","last_unstale":"2024-09-21T02:37:22.067590+0000","last_undegraded":"2024-09-21T02:37:22.067590+0000","last_fullsized":"2024-09-21T02:37:22.067590+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:31.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396463+0000","last_change":"2024-09-21T02:36:25.748925+0000","last_active":"2024-09-21T02:36:37.396463+0000","last_peered":"2024-09-21T02:36:37.396463+0000","last_clean":"2024-09-21T02:36:37.396463+0000","last_became_active":"2024-09-21T02:36:25.748333+0000","last_became_peered":"2024-09-21T02:36:25.748333+0000","last_unstale":"2024-09-21T02:36:37.396463+0000","last_undegraded":"2024-09-21T02:36:37.396463+0000","last_fullsized":"2024-09-21T02:36:37.396463+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:46.889497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397461+0000","last_change":"2024-09-21T02:36:32.807771+0000","last_active":"2024-09-21T02:36:37.397461+0000","last_peered":"2024-09-21T02:36:37.397461+0000","last_clean":"2024-09-21T02:36:37.397461+0000","last_became_active":"2024-09-21T02:36:32.807291+0000","last_became_peered":"2024-09-21T02:36:32.807291+0000","last_unstale":"2024-09-21T02:36:37.397461+0000","last_undegraded":"2024-09-21T02:36:37.397461+0000","last_fullsized":"2024-09-21T02:36:37.397461+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:41.468318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396711+0000","last_change":"2024-09-21T02:36:23.800779+0000","last_active":"2024-09-21T02:36:37.396711+0000","last_peered":"2024-09-21T02:36:37.396711+0000","last_clean":"2024-09-21T02:36:37.396711+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:36:37.396711+0000","last_undegraded":"2024-09-21T02:36:37.396711+0000","last_fullsized":"2024-09-21T02:36:37.396711+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:24.843233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991592+0000","last_change":"2024-09-21T02:36:25.745277+0000","last_active":"2024-09-21T02:36:36.991592+0000","last_peered":"2024-09-21T02:36:36.991592+0000","last_clean":"2024-09-21T02:36:36.991592+0000","last_became_active":"2024-09-21T02:36:25.744501+0000","last_became_peered":"2024-09-21T02:36:25.744501+0000","last_unstale":"2024-09-21T02:36:36.991592+0000","last_undegraded":"2024-09-21T02:36:36.991592+0000","last_fullsized":"2024-09-21T02:36:36.991592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:14.776250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396321+0000","last_change":"2024-09-21T02:36:32.801978+0000","last_active":"2024-09-21T02:36:37.396321+0000","last_peered":"2024-09-21T02:36:37.396321+0000","last_clean":"2024-09-21T02:36:37.396321+0000","last_became_active":"2024-09-21T02:36:32.801436+0000","last_became_peered":"2024-09-21T02:36:32.801436+0000","last_unstale":"2024-09-21T02:36:37.396321+0000","last_undegraded":"2024-09-21T02:36:37.396321+0000","last_fullsized":"2024-09-21T02:36:37.396321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:37.380417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:10.046781+0000","last_change":"2024-09-21T02:36:23.729336+0000","last_active":"2024-09-21T02:37:10.046781+0000","last_peered":"2024-09-21T02:37:10.046781+0000","last_clean":"2024-09-21T02:37:10.046781+0000","last_became_active":"2024-09-21T02:36:23.728724+0000","last_became_peered":"2024-09-21T02:36:23.728724+0000","last_unstale":"2024-09-21T02:37:10.046781+0000","last_undegraded":"2024-09-21T02:37:10.046781+0000","last_fullsized":"2024-09-21T02:37:10.046781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:31:52.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380640+0000","last_change":"2024-09-21T02:36:25.737577+0000","last_active":"2024-09-21T02:36:36.380640+0000","last_peered":"2024-09-21T02:36:36.380640+0000","last_clean":"2024-09-21T02:36:36.380640+0000","last_became_active":"2024-09-21T02:36:25.737221+0000","last_became_peered":"2024-09-21T02:36:25.737221+0000","last_unstale":"2024-09-21T02:36:36.380640+0000","last_undegraded":"2024-09-21T02:36:36.380640+0000","last_fullsized":"2024-09-21T02:36:36.380640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:42:54.786464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397059+0000","last_change":"2024-09-21T02:36:32.802507+0000","last_active":"2024-09-21T02:36:37.397059+0000","last_peered":"2024-09-21T02:36:37.397059+0000","last_clean":"2024-09-21T02:36:37.397059+0000","last_became_active":"2024-09-21T02:36:32.801879+0000","last_became_peered":"2024-09-21T02:36:32.801879+0000","last_unstale":"2024-09-21T02:36:37.397059+0000","last_undegraded":"2024-09-21T02:36:37.397059+0000","last_fullsized":"2024-09-21T02:36:37.397059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:09.873825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044202+0000","last_change":"2024-09-21T02:36:23.730889+0000","last_active":"2024-09-21T02:37:09.044202+0000","last_peered":"2024-09-21T02:37:09.044202+0000","last_clean":"2024-09-21T02:37:09.044202+0000","last_became_active":"2024-09-21T02:36:23.730551+0000","last_became_peered":"2024-09-21T02:36:23.730551+0000","last_unstale":"2024-09-21T02:37:09.044202+0000","last_undegraded":"2024-09-21T02:37:09.044202+0000","last_fullsized":"2024-09-21T02:37:09.044202+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:17.178848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.398583+0000","last_change":"2024-09-21T02:36:25.744531+0000","last_active":"2024-09-21T02:36:37.398583+0000","last_peered":"2024-09-21T02:36:37.398583+0000","last_clean":"2024-09-21T02:36:37.398583+0000","last_became_active":"2024-09-21T02:36:25.744110+0000","last_became_peered":"2024-09-21T02:36:25.744110+0000","last_unstale":"2024-09-21T02:36:37.398583+0000","last_undegraded":"2024-09-21T02:36:37.398583+0000","last_fullsized":"2024-09-21T02:36:37.398583+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:56:08.895972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396486+0000","last_change":"2024-09-21T02:36:32.800489+0000","last_active":"2024-09-21T02:36:37.396486+0000","last_peered":"2024-09-21T02:36:37.396486+0000","last_clean":"2024-09-21T02:36:37.396486+0000","last_became_active":"2024-09-21T02:36:32.799876+0000","last_became_peered":"2024-09-21T02:36:32.799876+0000","last_unstale":"2024-09-21T02:36:37.396486+0000","last_undegraded":"2024-09-21T02:36:37.396486+0000","last_fullsized":"2024-09-21T02:36:37.396486+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:02:56.711257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401371+0000","last_change":"2024-09-21T02:36:25.744561+0000","last_active":"2024-09-21T02:36:36.401371+0000","last_peered":"2024-09-21T02:36:36.401371+0000","last_clean":"2024-09-21T02:36:36.401371+0000","last_became_active":"2024-09-21T02:36:25.743757+0000","last_became_peered":"2024-09-21T02:36:25.743757+0000","last_unstale":"2024-09-21T02:36:36.401371+0000","last_undegraded":"2024-09-21T02:36:36.401371+0000","last_fullsized":"2024-09-21T02:36:36.401371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:30.235517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:06.018103+0000","last_change":"2024-09-21T02:36:23.729060+0000","last_active":"2024-09-21T02:37:06.018103+0000","last_peered":"2024-09-21T02:37:06.018103+0000","last_clean":"2024-09-21T02:37:06.018103+0000","last_became_active":"2024-09-21T02:36:23.728703+0000","last_became_peered":"2024-09-21T02:36:23.728703+0000","last_unstale":"2024-09-21T02:37:06.018103+0000","last_undegraded":"2024-09-21T02:37:06.018103+0000","last_fullsized":"2024-09-21T02:37:06.018103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:07.201729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396729+0000","last_change":"2024-09-21T02:36:33.003670+0000","last_active":"2024-09-21T02:36:37.396729+0000","last_peered":"2024-09-21T02:36:37.396729+0000","last_clean":"2024-09-21T02:36:37.396729+0000","last_became_active":"2024-09-21T02:36:33.003022+0000","last_became_peered":"2024-09-21T02:36:33.003022+0000","last_unstale":"2024-09-21T02:36:37.396729+0000","last_undegraded":"2024-09-21T02:36:37.396729+0000","last_fullsized":"2024-09-21T02:36:37.396729+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:12.753375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395962+0000","last_change":"2024-09-21T02:36:25.749404+0000","last_active":"2024-09-21T02:36:37.395962+0000","last_peered":"2024-09-21T02:36:37.395962+0000","last_clean":"2024-09-21T02:36:37.395962+0000","last_became_active":"2024-09-21T02:36:25.747854+0000","last_became_peered":"2024-09-21T02:36:25.747854+0000","last_unstale":"2024-09-21T02:36:37.395962+0000","last_undegraded":"2024-09-21T02:36:37.395962+0000","last_fullsized":"2024-09-21T02:36:37.395962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:41.224740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:08.034280+0000","last_change":"2024-09-21T02:36:23.726689+0000","last_active":"2024-09-21T02:37:08.034280+0000","last_peered":"2024-09-21T02:37:08.034280+0000","last_clean":"2024-09-21T02:37:08.034280+0000","last_became_active":"2024-09-21T02:36:23.726165+0000","last_became_peered":"2024-09-21T02:36:23.726165+0000","last_unstale":"2024-09-21T02:37:08.034280+0000","last_undegraded":"2024-09-21T02:37:08.034280+0000","last_fullsized":"2024-09-21T02:37:08.034280+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:58.166457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397094+0000","last_change":"2024-09-21T02:36:32.809594+0000","last_active":"2024-09-21T02:36:37.397094+0000","last_peered":"2024-09-21T02:36:37.397094+0000","last_clean":"2024-09-21T02:36:37.397094+0000","last_became_active":"2024-09-21T02:36:32.808893+0000","last_became_peered":"2024-09-21T02:36:32.808893+0000","last_unstale":"2024-09-21T02:36:37.397094+0000","last_undegraded":"2024-09-21T02:36:37.397094+0000","last_fullsized":"2024-09-21T02:36:37.397094+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:55.187241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899868+0000","last_change":"2024-09-21T02:36:25.755955+0000","last_active":"2024-09-21T02:36:35.899868+0000","last_peered":"2024-09-21T02:36:35.899868+0000","last_clean":"2024-09-21T02:36:35.899868+0000","last_became_active":"2024-09-21T02:36:25.755577+0000","last_became_peered":"2024-09-21T02:36:25.755577+0000","last_unstale":"2024-09-21T02:36:35.899868+0000","last_undegraded":"2024-09-21T02:36:35.899868+0000","last_fullsized":"2024-09-21T02:36:35.899868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:48.447758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044136+0000","last_change":"2024-09-21T02:36:23.730043+0000","last_active":"2024-09-21T02:37:09.044136+0000","last_peered":"2024-09-21T02:37:09.044136+0000","last_clean":"2024-09-21T02:37:09.044136+0000","last_became_active":"2024-09-21T02:36:23.729576+0000","last_became_peered":"2024-09-21T02:36:23.729576+0000","last_unstale":"2024-09-21T02:37:09.044136+0000","last_undegraded":"2024-09-21T02:37:09.044136+0000","last_fullsized":"2024-09-21T02:37:09.044136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:48:09.530913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.754143+0000","last_change":"2024-09-21T02:36:32.806923+0000","last_active":"2024-09-21T02:36:36.754143+0000","last_peered":"2024-09-21T02:36:36.754143+0000","last_clean":"2024-09-21T02:36:36.754143+0000","last_became_active":"2024-09-21T02:36:32.806176+0000","last_became_peered":"2024-09-21T02:36:32.806176+0000","last_unstale":"2024-09-21T02:36:36.754143+0000","last_undegraded":"2024-09-21T02:36:36.754143+0000","last_fullsized":"2024-09-21T02:36:36.754143+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.165638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396452+0000","last_change":"2024-09-21T02:36:25.747501+0000","last_active":"2024-09-21T02:36:37.396452+0000","last_peered":"2024-09-21T02:36:37.396452+0000","last_clean":"2024-09-21T02:36:37.396452+0000","last_became_active":"2024-09-21T02:36:25.747111+0000","last_became_peered":"2024-09-21T02:36:25.747111+0000","last_unstale":"2024-09-21T02:36:37.396452+0000","last_undegraded":"2024-09-21T02:36:37.396452+0000","last_fullsized":"2024-09-21T02:36:37.396452+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:03:04.061537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:06.019272+0000","last_change":"2024-09-21T02:36:23.800802+0000","last_active":"2024-09-21T02:37:06.019272+0000","last_peered":"2024-09-21T02:37:06.019272+0000","last_clean":"2024-09-21T02:37:06.019272+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:37:06.019272+0000","last_undegraded":"2024-09-21T02:37:06.019272+0000","last_fullsized":"2024-09-21T02:37:06.019272+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:44:34.165406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397014+0000","last_change":"2024-09-21T02:36:33.005535+0000","last_active":"2024-09-21T02:36:37.397014+0000","last_peered":"2024-09-21T02:36:37.397014+0000","last_clean":"2024-09-21T02:36:37.397014+0000","last_became_active":"2024-09-21T02:36:33.004912+0000","last_became_peered":"2024-09-21T02:36:33.004912+0000","last_unstale":"2024-09-21T02:36:37.397014+0000","last_undegraded":"2024-09-21T02:36:37.397014+0000","last_fullsized":"2024-09-21T02:36:37.397014+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:54:14.378790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991439+0000","last_change":"2024-09-21T02:36:25.746139+0000","last_active":"2024-09-21T02:36:36.991439+0000","last_peered":"2024-09-21T02:36:36.991439+0000","last_clean":"2024-09-21T02:36:36.991439+0000","last_became_active":"2024-09-21T02:36:25.744418+0000","last_became_peered":"2024-09-21T02:36:25.744418+0000","last_unstale":"2024-09-21T02:36:36.991439+0000","last_undegraded":"2024-09-21T02:36:36.991439+0000","last_fullsized":"2024-09-21T02:36:36.991439+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:21.706782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044967+0000","last_change":"2024-09-21T02:36:24.261945+0000","last_active":"2024-09-21T02:37:09.044967+0000","last_peered":"2024-09-21T02:37:09.044967+0000","last_clean":"2024-09-21T02:37:09.044967+0000","last_became_active":"2024-09-21T02:36:24.261486+0000","last_became_peered":"2024-09-21T02:36:24.261486+0000","last_unstale":"2024-09-21T02:37:09.044967+0000","last_undegraded":"2024-09-21T02:37:09.044967+0000","last_fullsized":"2024-09-21T02:37:09.044967+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:45:18.331035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397991+0000","last_change":"2024-09-21T02:36:32.799464+0000","last_active":"2024-09-21T02:36:37.397991+0000","last_peered":"2024-09-21T02:36:37.397991+0000","last_clean":"2024-09-21T02:36:37.397991+0000","last_became_active":"2024-09-21T02:36:32.799000+0000","last_became_peered":"2024-09-21T02:36:32.799000+0000","last_unstale":"2024-09-21T02:36:37.397991+0000","last_undegraded":"2024-09-21T02:36:37.397991+0000","last_fullsized":"2024-09-21T02:36:37.397991+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:56:36.271172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396612+0000","last_change":"2024-09-21T02:36:25.741151+0000","last_active":"2024-09-21T02:36:37.396612+0000","last_peered":"2024-09-21T02:36:37.396612+0000","last_clean":"2024-09-21T02:36:37.396612+0000","last_became_active":"2024-09-21T02:36:25.740803+0000","last_became_peered":"2024-09-21T02:36:25.740803+0000","last_unstale":"2024-09-21T02:36:37.396612+0000","last_undegraded":"2024-09-21T02:36:37.396612+0000","last_fullsized":"2024-09-21T02:36:37.396612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:44.585136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'14","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:19.749474+0000","last_change":"2024-09-21T02:36:23.725155+0000","last_active":"2024-09-21T02:37:19.749474+0000","last_peered":"2024-09-21T02:37:19.749474+0000","last_clean":"2024-09-21T02:37:19.749474+0000","last_became_active":"2024-09-21T02:36:23.724791+0000","last_became_peered":"2024-09-21T02:36:23.724791+0000","last_unstale":"2024-09-21T02:37:19.749474+0000","last_undegraded":"2024-09-21T02:37:19.749474+0000","last_fullsized":"2024-09-21T02:37:19.749474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:06.520465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395786+0000","last_change":"2024-09-21T02:36:33.003401+0000","last_active":"2024-09-21T02:36:37.395786+0000","last_peered":"2024-09-21T02:36:37.395786+0000","last_clean":"2024-09-21T02:36:37.395786+0000","last_became_active":"2024-09-21T02:36:33.002930+0000","last_became_peered":"2024-09-21T02:36:33.002930+0000","last_unstale":"2024-09-21T02:36:37.395786+0000","last_undegraded":"2024-09-21T02:36:37.395786+0000","last_fullsized":"2024-09-21T02:36:37.395786+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:40:23.629763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899410+0000","last_change":"2024-09-21T02:36:25.735501+0000","last_active":"2024-09-21T02:36:35.899410+0000","last_peered":"2024-09-21T02:36:35.899410+0000","last_clean":"2024-09-21T02:36:35.899410+0000","last_became_active":"2024-09-21T02:36:25.734453+0000","last_became_peered":"2024-09-21T02:36:25.734453+0000","last_unstale":"2024-09-21T02:36:35.899410+0000","last_undegraded":"2024-09-21T02:36:35.899410+0000","last_fullsized":"2024-09-21T02:36:35.899410+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:50.238529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.042607+0000","last_change":"2024-09-21T02:36:23.800338+0000","last_active":"2024-09-21T02:37:09.042607+0000","last_peered":"2024-09-21T02:37:09.042607+0000","last_clean":"2024-09-21T02:37:09.042607+0000","last_became_active":"2024-09-21T02:36:23.799796+0000","last_became_peered":"2024-09-21T02:36:23.799796+0000","last_unstale":"2024-09-21T02:37:09.042607+0000","last_undegraded":"2024-09-21T02:37:09.042607+0000","last_fullsized":"2024-09-21T02:37:09.042607+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:00:18.459515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899946+0000","last_change":"2024-09-21T02:36:32.799630+0000","last_active":"2024-09-21T02:36:35.899946+0000","last_peered":"2024-09-21T02:36:35.899946+0000","last_clean":"2024-09-21T02:36:35.899946+0000","last_became_active":"2024-09-21T02:36:32.798761+0000","last_became_peered":"2024-09-21T02:36:32.798761+0000","last_unstale":"2024-09-21T02:36:35.899946+0000","last_undegraded":"2024-09-21T02:36:35.899946+0000","last_fullsized":"2024-09-21T02:36:35.899946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:33.707902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900025+0000","last_change":"2024-09-21T02:36:25.745497+0000","last_active":"2024-09-21T02:36:35.900025+0000","last_peered":"2024-09-21T02:36:35.900025+0000","last_clean":"2024-09-21T02:36:35.900025+0000","last_became_active":"2024-09-21T02:36:25.745126+0000","last_became_peered":"2024-09-21T02:36:25.745126+0000","last_unstale":"2024-09-21T02:36:35.900025+0000","last_undegraded":"2024-09-21T02:36:35.900025+0000","last_fullsized":"2024-09-21T02:36:35.900025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:42:17.835784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.754164+0000","last_change":"2024-09-21T02:36:23.802440+0000","last_active":"2024-09-21T02:36:36.754164+0000","last_peered":"2024-09-21T02:36:36.754164+0000","last_clean":"2024-09-21T02:36:36.754164+0000","last_became_active":"2024-09-21T02:36:23.802065+0000","last_became_peered":"2024-09-21T02:36:23.802065+0000","last_unstale":"2024-09-21T02:36:36.754164+0000","last_undegraded":"2024-09-21T02:36:36.754164+0000","last_fullsized":"2024-09-21T02:36:36.754164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:22:20.647476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396426+0000","last_change":"2024-09-21T02:36:33.003465+0000","last_active":"2024-09-21T02:36:37.396426+0000","last_peered":"2024-09-21T02:36:37.396426+0000","last_clean":"2024-09-21T02:36:37.396426+0000","last_became_active":"2024-09-21T02:36:33.002874+0000","last_became_peered":"2024-09-21T02:36:33.002874+0000","last_unstale":"2024-09-21T02:36:37.396426+0000","last_undegraded":"2024-09-21T02:36:37.396426+0000","last_fullsized":"2024-09-21T02:36:37.396426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:02.388770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899847+0000","last_change":"2024-09-21T02:36:25.746092+0000","last_active":"2024-09-21T02:36:35.899847+0000","last_peered":"2024-09-21T02:36:35.899847+0000","last_clean":"2024-09-21T02:36:35.899847+0000","last_became_active":"2024-09-21T02:36:25.745730+0000","last_became_peered":"2024-09-21T02:36:25.745730+0000","last_unstale":"2024-09-21T02:36:35.899847+0000","last_undegraded":"2024-09-21T02:36:35.899847+0000","last_fullsized":"2024-09-21T02:36:35.899847+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:24:13.212450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":52,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:23.069442+0000","last_change":"2024-09-21T02:36:23.732090+0000","last_active":"2024-09-21T02:37:23.069442+0000","last_peered":"2024-09-21T02:37:23.069442+0000","last_clean":"2024-09-21T02:37:23.069442+0000","last_became_active":"2024-09-21T02:36:23.731653+0000","last_became_peered":"2024-09-21T02:36:23.731653+0000","last_unstale":"2024-09-21T02:37:23.069442+0000","last_undegraded":"2024-09-21T02:37:23.069442+0000","last_fullsized":"2024-09-21T02:37:23.069442+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:02:42.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899238+0000","last_change":"2024-09-21T02:36:32.794893+0000","last_active":"2024-09-21T02:36:35.899238+0000","last_peered":"2024-09-21T02:36:35.899238+0000","last_clean":"2024-09-21T02:36:35.899238+0000","last_became_active":"2024-09-21T02:36:32.794406+0000","last_became_peered":"2024-09-21T02:36:32.794406+0000","last_unstale":"2024-09-21T02:36:35.899238+0000","last_undegraded":"2024-09-21T02:36:35.899238+0000","last_fullsized":"2024-09-21T02:36:35.899238+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:40:39.621502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396958+0000","last_change":"2024-09-21T02:36:25.748220+0000","last_active":"2024-09-21T02:36:37.396958+0000","last_peered":"2024-09-21T02:36:37.396958+0000","last_clean":"2024-09-21T02:36:37.396958+0000","last_became_active":"2024-09-21T02:36:25.747553+0000","last_became_peered":"2024-09-21T02:36:25.747553+0000","last_unstale":"2024-09-21T02:36:37.396958+0000","last_undegraded":"2024-09-21T02:36:37.396958+0000","last_fullsized":"2024-09-21T02:36:37.396958+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:28.757981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:22.068225+0000","last_change":"2024-09-21T02:36:23.801087+0000","last_active":"2024-09-21T02:37:22.068225+0000","last_peered":"2024-09-21T02:37:22.068225+0000","last_clean":"2024-09-21T02:37:22.068225+0000","last_became_active":"2024-09-21T02:36:23.800077+0000","last_became_peered":"2024-09-21T02:36:23.800077+0000","last_unstale":"2024-09-21T02:37:22.068225+0000","last_undegraded":"2024-09-21T02:37:22.068225+0000","last_fullsized":"2024-09-21T02:37:22.068225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:50.074650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899492+0000","last_change":"2024-09-21T02:36:33.003354+0000","last_active":"2024-09-21T02:36:35.899492+0000","last_peered":"2024-09-21T02:36:35.899492+0000","last_clean":"2024-09-21T02:36:35.899492+0000","last_became_active":"2024-09-21T02:36:33.002912+0000","last_became_peered":"2024-09-21T02:36:33.002912+0000","last_unstale":"2024-09-21T02:36:35.899492+0000","last_undegraded":"2024-09-21T02:36:35.899492+0000","last_fullsized":"2024-09-21T02:36:35.899492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:21.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.898724+0000","last_change":"2024-09-21T02:36:25.741701+0000","last_active":"2024-09-21T02:36:35.898724+0000","last_peered":"2024-09-21T02:36:35.898724+0000","last_clean":"2024-09-21T02:36:35.898724+0000","last_became_active":"2024-09-21T02:36:25.741222+0000","last_became_peered":"2024-09-21T02:36:25.741222+0000","last_unstale":"2024-09-21T02:36:35.898724+0000","last_undegraded":"2024-09-21T02:36:35.898724+0000","last_fullsized":"2024-09-21T02:36:35.898724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:35:39.896852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:07.024788+0000","last_change":"2024-09-21T02:36:23.728793+0000","last_active":"2024-09-21T02:37:07.024788+0000","last_peered":"2024-09-21T02:37:07.024788+0000","last_clean":"2024-09-21T02:37:07.024788+0000","last_became_active":"2024-09-21T02:36:23.728362+0000","last_became_peered":"2024-09-21T02:36:23.728362+0000","last_unstale":"2024-09-21T02:37:07.024788+0000","last_undegraded":"2024-09-21T02:37:07.024788+0000","last_fullsized":"2024-09-21T02:37:07.024788+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:28:15.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.398740+0000","last_change":"2024-09-21T02:36:33.005225+0000","last_active":"2024-09-21T02:36:37.398740+0000","last_peered":"2024-09-21T02:36:37.398740+0000","last_clean":"2024-09-21T02:36:37.398740+0000","last_became_active":"2024-09-21T02:36:33.004756+0000","last_became_peered":"2024-09-21T02:36:33.004756+0000","last_unstale":"2024-09-21T02:36:37.398740+0000","last_undegraded":"2024-09-21T02:36:37.398740+0000","last_fullsized":"2024-09-21T02:36:37.398740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:02:16.686284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396145+0000","last_change":"2024-09-21T02:36:25.745639+0000","last_active":"2024-09-21T02:36:37.396145+0000","last_peered":"2024-09-21T02:36:37.396145+0000","last_clean":"2024-09-21T02:36:37.396145+0000","last_became_active":"2024-09-21T02:36:25.745202+0000","last_became_peered":"2024-09-21T02:36:25.745202+0000","last_unstale":"2024-09-21T02:36:37.396145+0000","last_undegraded":"2024-09-21T02:36:37.396145+0000","last_fullsized":"2024-09-21T02:36:37.396145+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:08.712477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:10.046807+0000","last_change":"2024-09-21T02:36:23.730710+0000","last_active":"2024-09-21T02:37:10.046807+0000","last_peered":"2024-09-21T02:37:10.046807+0000","last_clean":"2024-09-21T02:37:10.046807+0000","last_became_active":"2024-09-21T02:36:23.730247+0000","last_became_peered":"2024-09-21T02:36:23.730247+0000","last_unstale":"2024-09-21T02:37:10.046807+0000","last_undegraded":"2024-09-21T02:37:10.046807+0000","last_fullsized":"2024-09-21T02:37:10.046807+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:43.863444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":104,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401483+0000","last_change":"2024-09-21T02:35:09.949841+0000","last_active":"2024-09-21T02:36:36.401483+0000","last_peered":"2024-09-21T02:36:36.401483+0000","last_clean":"2024-09-21T02:36:36.401483+0000","last_became_active":"2024-09-21T02:35:09.944224+0000","last_became_peered":"2024-09-21T02:35:09.944224+0000","last_unstale":"2024-09-21T02:36:36.401483+0000","last_undegraded":"2024-09-21T02:36:36.401483+0000","last_fullsized":"2024-09-21T02:36:36.401483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_clean_scrub_stamp":"2024-09-21T02:31:38.930125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:56.467576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401816+0000","last_change":"2024-09-21T02:36:32.805600+0000","last_active":"2024-09-21T02:36:36.401816+0000","last_peered":"2024-09-21T02:36:36.401816+0000","last_clean":"2024-09-21T02:36:36.401816+0000","last_became_active":"2024-09-21T02:36:32.805097+0000","last_became_peered":"2024-09-21T02:36:32.805097+0000","last_unstale":"2024-09-21T02:36:36.401816+0000","last_undegraded":"2024-09-21T02:36:36.401816+0000","last_fullsized":"2024-09-21T02:36:36.401816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:25.857807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401783+0000","last_change":"2024-09-21T02:36:25.741176+0000","last_active":"2024-09-21T02:36:36.401783+0000","last_peered":"2024-09-21T02:36:36.401783+0000","last_clean":"2024-09-21T02:36:36.401783+0000","last_became_active":"2024-09-21T02:36:25.740355+0000","last_became_peered":"2024-09-21T02:36:25.740355+0000","last_unstale":"2024-09-21T02:36:36.401783+0000","last_undegraded":"2024-09-21T02:36:36.401783+0000","last_fullsized":"2024-09-21T02:36:36.401783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:47:48.204768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:21.068113+0000","last_change":"2024-09-21T02:36:24.261500+0000","last_active":"2024-09-21T02:37:21.068113+0000","last_peered":"2024-09-21T02:37:21.068113+0000","last_clean":"2024-09-21T02:37:21.068113+0000","last_became_active":"2024-09-21T02:36:24.261089+0000","last_became_peered":"2024-09-21T02:36:24.261089+0000","last_unstale":"2024-09-21T02:37:21.068113+0000","last_undegraded":"2024-09-21T02:37:21.068113+0000","last_fullsized":"2024-09-21T02:37:21.068113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:33:33.158177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380535+0000","last_change":"2024-09-21T02:36:32.796980+0000","last_active":"2024-09-21T02:36:36.380535+0000","last_peered":"2024-09-21T02:36:36.380535+0000","last_clean":"2024-09-21T02:36:36.380535+0000","last_became_active":"2024-09-21T02:36:32.796517+0000","last_became_peered":"2024-09-21T02:36:32.796517+0000","last_unstale":"2024-09-21T02:36:36.380535+0000","last_undegraded":"2024-09-21T02:36:36.380535+0000","last_fullsized":"2024-09-21T02:36:36.380535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:07.230177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401654+0000","last_change":"2024-09-21T02:36:25.750931+0000","last_active":"2024-09-21T02:36:36.401654+0000","last_peered":"2024-09-21T02:36:36.401654+0000","last_clean":"2024-09-21T02:36:36.401654+0000","last_became_active":"2024-09-21T02:36:25.750604+0000","last_became_peered":"2024-09-21T02:36:25.750604+0000","last_unstale":"2024-09-21T02:36:36.401654+0000","last_undegraded":"2024-09-21T02:36:36.401654+0000","last_fullsized":"2024-09-21T02:36:36.401654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:21.325095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044460+0000","last_change":"2024-09-21T02:36:23.730700+0000","last_active":"2024-09-21T02:37:09.044460+0000","last_peered":"2024-09-21T02:37:09.044460+0000","last_clean":"2024-09-21T02:37:09.044460+0000","last_became_active":"2024-09-21T02:36:23.730367+0000","last_became_peered":"2024-09-21T02:36:23.730367+0000","last_unstale":"2024-09-21T02:37:09.044460+0000","last_undegraded":"2024-09-21T02:37:09.044460+0000","last_fullsized":"2024-09-21T02:37:09.044460+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:27:57.373077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991327+0000","last_change":"2024-09-21T02:36:32.805885+0000","last_active":"2024-09-21T02:36:36.991327+0000","last_peered":"2024-09-21T02:36:36.991327+0000","last_clean":"2024-09-21T02:36:36.991327+0000","last_became_active":"2024-09-21T02:36:32.805479+0000","last_became_peered":"2024-09-21T02:36:32.805479+0000","last_unstale":"2024-09-21T02:36:36.991327+0000","last_undegraded":"2024-09-21T02:36:36.991327+0000","last_fullsized":"2024-09-21T02:36:36.991327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:11:29.910642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401892+0000","last_change":"2024-09-21T02:36:25.742542+0000","last_active":"2024-09-21T02:36:36.401892+0000","last_peered":"2024-09-21T02:36:36.401892+0000","last_clean":"2024-09-21T02:36:36.401892+0000","last_became_active":"2024-09-21T02:36:25.742000+0000","last_became_peered":"2024-09-21T02:36:25.742000+0000","last_unstale":"2024-09-21T02:36:36.401892+0000","last_undegraded":"2024-09-21T02:36:36.401892+0000","last_fullsized":"2024-09-21T02:36:36.401892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:52.300212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.041107+0000","last_change":"2024-09-21T02:36:23.723896+0000","last_active":"2024-09-21T02:37:09.041107+0000","last_peered":"2024-09-21T02:37:09.041107+0000","last_clean":"2024-09-21T02:37:09.041107+0000","last_became_active":"2024-09-21T02:36:23.723518+0000","last_became_peered":"2024-09-21T02:36:23.723518+0000","last_unstale":"2024-09-21T02:37:09.041107+0000","last_undegraded":"2024-09-21T02:37:09.041107+0000","last_fullsized":"2024-09-21T02:37:09.041107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:37.250247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396493+0000","last_change":"2024-09-21T02:36:32.799924+0000","last_active":"2024-09-21T02:36:37.396493+0000","last_peered":"2024-09-21T02:36:37.396493+0000","last_clean":"2024-09-21T02:36:37.396493+0000","last_became_active":"2024-09-21T02:36:32.799437+0000","last_became_peered":"2024-09-21T02:36:32.799437+0000","last_unstale":"2024-09-21T02:36:37.396493+0000","last_undegraded":"2024-09-21T02:36:37.396493+0000","last_fullsized":"2024-09-21T02:36:37.396493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:19.258293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900030+0000","last_change":"2024-09-21T02:36:25.747321+0000","last_active":"2024-09-21T02:36:35.900030+0000","last_peered":"2024-09-21T02:36:35.900030+0000","last_clean":"2024-09-21T02:36:35.900030+0000","last_became_active":"2024-09-21T02:36:25.746955+0000","last_became_peered":"2024-09-21T02:36:25.746955+0000","last_unstale":"2024-09-21T02:36:35.900030+0000","last_undegraded":"2024-09-21T02:36:35.900030+0000","last_fullsized":"2024-09-21T02:36:35.900030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:25.148300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.019701+0000","last_change":"2024-09-21T02:36:23.725350+0000","last_active":"2024-09-21T02:37:09.019701+0000","last_peered":"2024-09-21T02:37:09.019701+0000","last_clean":"2024-09-21T02:37:09.019701+0000","last_became_active":"2024-09-21T02:36:23.725090+0000","last_became_peered":"2024-09-21T02:36:23.725090+0000","last_unstale":"2024-09-21T02:37:09.019701+0000","last_undegraded":"2024-09-21T02:37:09.019701+0000","last_fullsized":"2024-09-21T02:37:09.019701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:52.130345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397609+0000","last_change":"2024-09-21T02:36:32.801355+0000","last_active":"2024-09-21T02:36:37.397609+0000","last_peered":"2024-09-21T02:36:37.397609+0000","last_clean":"2024-09-21T02:36:37.397609+0000","last_became_active":"2024-09-21T02:36:32.800803+0000","last_became_peered":"2024-09-21T02:36:32.800803+0000","last_unstale":"2024-09-21T02:36:37.397609+0000","last_undegraded":"2024-09-21T02:36:37.397609+0000","last_fullsized":"2024-09-21T02:36:37.397609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:54.755733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397597+0000","last_change":"2024-09-21T02:36:25.743433+0000","last_active":"2024-09-21T02:36:37.397597+0000","last_peered":"2024-09-21T02:36:37.397597+0000","last_clean":"2024-09-21T02:36:37.397597+0000","last_became_active":"2024-09-21T02:36:25.742643+0000","last_became_peered":"2024-09-21T02:36:25.742643+0000","last_unstale":"2024-09-21T02:36:37.397597+0000","last_undegraded":"2024-09-21T02:36:37.397597+0000","last_fullsized":"2024-09-21T02:36:37.397597+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:20:09.849806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.042742+0000","last_change":"2024-09-21T02:36:23.800229+0000","last_active":"2024-09-21T02:37:09.042742+0000","last_peered":"2024-09-21T02:37:09.042742+0000","last_clean":"2024-09-21T02:37:09.042742+0000","last_became_active":"2024-09-21T02:36:23.799792+0000","last_became_peered":"2024-09-21T02:36:23.799792+0000","last_unstale":"2024-09-21T02:37:09.042742+0000","last_undegraded":"2024-09-21T02:37:09.042742+0000","last_fullsized":"2024-09-21T02:37:09.042742+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:27.446364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899481+0000","last_change":"2024-09-21T02:36:32.799728+0000","last_active":"2024-09-21T02:36:35.899481+0000","last_peered":"2024-09-21T02:36:35.899481+0000","last_clean":"2024-09-21T02:36:35.899481+0000","last_became_active":"2024-09-21T02:36:32.798419+0000","last_became_peered":"2024-09-21T02:36:32.798419+0000","last_unstale":"2024-09-21T02:36:35.899481+0000","last_undegraded":"2024-09-21T02:36:35.899481+0000","last_fullsized":"2024-09-21T02:36:35.899481+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:53.681473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401174+0000","last_change":"2024-09-21T02:36:25.747712+0000","last_active":"2024-09-21T02:36:36.401174+0000","last_peered":"2024-09-21T02:36:36.401174+0000","last_clean":"2024-09-21T02:36:36.401174+0000","last_became_active":"2024-09-21T02:36:25.747197+0000","last_became_peered":"2024-09-21T02:36:25.747197+0000","last_unstale":"2024-09-21T02:36:36.401174+0000","last_undegraded":"2024-09-21T02:36:36.401174+0000","last_fullsized":"2024-09-21T02:36:36.401174+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:03.900360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:24.070997+0000","last_change":"2024-09-21T02:36:23.732156+0000","last_active":"2024-09-21T02:37:24.070997+0000","last_peered":"2024-09-21T02:37:24.070997+0000","last_clean":"2024-09-21T02:37:24.070997+0000","last_became_active":"2024-09-21T02:36:23.730238+0000","last_became_peered":"2024-09-21T02:36:23.730238+0000","last_unstale":"2024-09-21T02:37:24.070997+0000","last_undegraded":"2024-09-21T02:37:24.070997+0000","last_fullsized":"2024-09-21T02:37:24.070997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:18.053890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.754397+0000","last_change":"2024-09-21T02:36:32.806674+0000","last_active":"2024-09-21T02:36:36.754397+0000","last_peered":"2024-09-21T02:36:36.754397+0000","last_clean":"2024-09-21T02:36:36.754397+0000","last_became_active":"2024-09-21T02:36:32.805373+0000","last_became_peered":"2024-09-21T02:36:32.805373+0000","last_unstale":"2024-09-21T02:36:36.754397+0000","last_undegraded":"2024-09-21T02:36:36.754397+0000","last_fullsized":"2024-09-21T02:36:36.754397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:29.249243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401557+0000","last_change":"2024-09-21T02:36:25.748703+0000","last_active":"2024-09-21T02:36:36.401557+0000","last_peered":"2024-09-21T02:36:36.401557+0000","last_clean":"2024-09-21T02:36:36.401557+0000","last_became_active":"2024-09-21T02:36:25.748340+0000","last_became_peered":"2024-09-21T02:36:25.748340+0000","last_unstale":"2024-09-21T02:36:36.401557+0000","last_undegraded":"2024-09-21T02:36:36.401557+0000","last_fullsized":"2024-09-21T02:36:36.401557+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:07.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.044175+0000","last_change":"2024-09-21T02:36:23.732332+0000","last_active":"2024-09-21T02:37:09.044175+0000","last_peered":"2024-09-21T02:37:09.044175+0000","last_clean":"2024-09-21T02:37:09.044175+0000","last_became_active":"2024-09-21T02:36:23.731974+0000","last_became_peered":"2024-09-21T02:36:23.731974+0000","last_unstale":"2024-09-21T02:37:09.044175+0000","last_undegraded":"2024-09-21T02:37:09.044175+0000","last_fullsized":"2024-09-21T02:37:09.044175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:52.284192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900088+0000","last_change":"2024-09-21T02:36:32.809480+0000","last_active":"2024-09-21T02:36:35.900088+0000","last_peered":"2024-09-21T02:36:35.900088+0000","last_clean":"2024-09-21T02:36:35.900088+0000","last_became_active":"2024-09-21T02:36:32.809024+0000","last_became_peered":"2024-09-21T02:36:32.809024+0000","last_unstale":"2024-09-21T02:36:35.900088+0000","last_undegraded":"2024-09-21T02:36:35.900088+0000","last_fullsized":"2024-09-21T02:36:35.900088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:10.608185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.898940+0000","last_change":"2024-09-21T02:36:25.751720+0000","last_active":"2024-09-21T02:36:35.898940+0000","last_peered":"2024-09-21T02:36:35.898940+0000","last_clean":"2024-09-21T02:36:35.898940+0000","last_became_active":"2024-09-21T02:36:25.751351+0000","last_became_peered":"2024-09-21T02:36:25.751351+0000","last_unstale":"2024-09-21T02:36:35.898940+0000","last_undegraded":"2024-09-21T02:36:35.898940+0000","last_fullsized":"2024-09-21T02:36:35.898940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:01.201453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:10.046596+0000","last_change":"2024-09-21T02:36:23.729981+0000","last_active":"2024-09-21T02:37:10.046596+0000","last_peered":"2024-09-21T02:37:10.046596+0000","last_clean":"2024-09-21T02:37:10.046596+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:37:10.046596+0000","last_undegraded":"2024-09-21T02:37:10.046596+0000","last_fullsized":"2024-09-21T02:37:10.046596+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:03.115712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.398637+0000","last_change":"2024-09-21T02:36:32.805286+0000","last_active":"2024-09-21T02:36:37.398637+0000","last_peered":"2024-09-21T02:36:37.398637+0000","last_clean":"2024-09-21T02:36:37.398637+0000","last_became_active":"2024-09-21T02:36:32.804770+0000","last_became_peered":"2024-09-21T02:36:32.804770+0000","last_unstale":"2024-09-21T02:36:37.398637+0000","last_undegraded":"2024-09-21T02:36:37.398637+0000","last_fullsized":"2024-09-21T02:36:37.398637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:18:34.155892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396592+0000","last_change":"2024-09-21T02:36:25.747812+0000","last_active":"2024-09-21T02:36:37.396592+0000","last_peered":"2024-09-21T02:36:37.396592+0000","last_clean":"2024-09-21T02:36:37.396592+0000","last_became_active":"2024-09-21T02:36:25.747187+0000","last_became_peered":"2024-09-21T02:36:25.747187+0000","last_unstale":"2024-09-21T02:36:37.396592+0000","last_undegraded":"2024-09-21T02:36:37.396592+0000","last_fullsized":"2024-09-21T02:36:37.396592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:51.395869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"81'31","reported_seq":95,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:22.070954+0000","last_change":"2024-09-21T02:36:23.735676+0000","last_active":"2024-09-21T02:37:22.070954+0000","last_peered":"2024-09-21T02:37:22.070954+0000","last_clean":"2024-09-21T02:37:22.070954+0000","last_became_active":"2024-09-21T02:36:23.735281+0000","last_became_peered":"2024-09-21T02:36:23.735281+0000","last_unstale":"2024-09-21T02:37:22.070954+0000","last_undegraded":"2024-09-21T02:37:22.070954+0000","last_fullsized":"2024-09-21T02:37:22.070954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:28:01.547245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60300,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":75,"num_write":33,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.753753+0000","last_change":"2024-09-21T02:36:32.806585+0000","last_active":"2024-09-21T02:36:36.753753+0000","last_peered":"2024-09-21T02:36:36.753753+0000","last_clean":"2024-09-21T02:36:36.753753+0000","last_became_active":"2024-09-21T02:36:32.805913+0000","last_became_peered":"2024-09-21T02:36:32.805913+0000","last_unstale":"2024-09-21T02:36:36.753753+0000","last_undegraded":"2024-09-21T02:36:36.753753+0000","last_fullsized":"2024-09-21T02:36:36.753753+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:59:31.888645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380722+0000","last_change":"2024-09-21T02:36:25.742208+0000","last_active":"2024-09-21T02:36:36.380722+0000","last_peered":"2024-09-21T02:36:36.380722+0000","last_clean":"2024-09-21T02:36:36.380722+0000","last_became_active":"2024-09-21T02:36:25.741849+0000","last_became_peered":"2024-09-21T02:36:25.741849+0000","last_unstale":"2024-09-21T02:36:36.380722+0000","last_undegraded":"2024-09-21T02:36:36.380722+0000","last_fullsized":"2024-09-21T02:36:36.380722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:21:43.720481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:08.032639+0000","last_change":"2024-09-21T02:36:23.800751+0000","last_active":"2024-09-21T02:37:08.032639+0000","last_peered":"2024-09-21T02:37:08.032639+0000","last_clean":"2024-09-21T02:37:08.032639+0000","last_became_active":"2024-09-21T02:36:23.800341+0000","last_became_peered":"2024-09-21T02:36:23.800341+0000","last_unstale":"2024-09-21T02:37:08.032639+0000","last_undegraded":"2024-09-21T02:37:08.032639+0000","last_fullsized":"2024-09-21T02:37:08.032639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:08:02.556549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396671+0000","last_change":"2024-09-21T02:36:32.802879+0000","last_active":"2024-09-21T02:36:37.396671+0000","last_peered":"2024-09-21T02:36:37.396671+0000","last_clean":"2024-09-21T02:36:37.396671+0000","last_became_active":"2024-09-21T02:36:32.802301+0000","last_became_peered":"2024-09-21T02:36:32.802301+0000","last_unstale":"2024-09-21T02:36:37.396671+0000","last_undegraded":"2024-09-21T02:36:37.396671+0000","last_fullsized":"2024-09-21T02:36:37.396671+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:08.135814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991675+0000","last_change":"2024-09-21T02:36:25.744647+0000","last_active":"2024-09-21T02:36:36.991675+0000","last_peered":"2024-09-21T02:36:36.991675+0000","last_clean":"2024-09-21T02:36:36.991675+0000","last_became_active":"2024-09-21T02:36:25.744049+0000","last_became_peered":"2024-09-21T02:36:25.744049+0000","last_unstale":"2024-09-21T02:36:36.991675+0000","last_undegraded":"2024-09-21T02:36:36.991675+0000","last_fullsized":"2024-09-21T02:36:36.991675+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:52:02.210380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.043661+0000","last_change":"2024-09-21T02:36:23.733590+0000","last_active":"2024-09-21T02:37:09.043661+0000","last_peered":"2024-09-21T02:37:09.043661+0000","last_clean":"2024-09-21T02:37:09.043661+0000","last_became_active":"2024-09-21T02:36:23.732922+0000","last_became_peered":"2024-09-21T02:36:23.732922+0000","last_unstale":"2024-09-21T02:37:09.043661+0000","last_undegraded":"2024-09-21T02:37:09.043661+0000","last_fullsized":"2024-09-21T02:37:09.043661+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.753659+0000","last_change":"2024-09-21T02:36:32.806540+0000","last_active":"2024-09-21T02:36:36.753659+0000","last_peered":"2024-09-21T02:36:36.753659+0000","last_clean":"2024-09-21T02:36:36.753659+0000","last_became_active":"2024-09-21T02:36:32.805612+0000","last_became_peered":"2024-09-21T02:36:32.805612+0000","last_unstale":"2024-09-21T02:36:36.753659+0000","last_undegraded":"2024-09-21T02:36:36.753659+0000","last_fullsized":"2024-09-21T02:36:36.753659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:23:24.686383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380907+0000","last_change":"2024-09-21T02:36:25.746229+0000","last_active":"2024-09-21T02:36:36.380907+0000","last_peered":"2024-09-21T02:36:36.380907+0000","last_clean":"2024-09-21T02:36:36.380907+0000","last_became_active":"2024-09-21T02:36:25.745892+0000","last_became_peered":"2024-09-21T02:36:25.745892+0000","last_unstale":"2024-09-21T02:36:36.380907+0000","last_undegraded":"2024-09-21T02:36:36.380907+0000","last_fullsized":"2024-09-21T02:36:36.380907+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:55.154499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:23.069956+0000","last_change":"2024-09-21T02:36:23.730987+0000","last_active":"2024-09-21T02:37:23.069956+0000","last_peered":"2024-09-21T02:37:23.069956+0000","last_clean":"2024-09-21T02:37:23.069956+0000","last_became_active":"2024-09-21T02:36:23.730566+0000","last_became_peered":"2024-09-21T02:36:23.730566+0000","last_unstale":"2024-09-21T02:37:23.069956+0000","last_undegraded":"2024-09-21T02:37:23.069956+0000","last_fullsized":"2024-09-21T02:37:23.069956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:18:12.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395905+0000","last_change":"2024-09-21T02:36:32.804608+0000","last_active":"2024-09-21T02:36:37.395905+0000","last_peered":"2024-09-21T02:36:37.395905+0000","last_clean":"2024-09-21T02:36:37.395905+0000","last_became_active":"2024-09-21T02:36:32.803567+0000","last_became_peered":"2024-09-21T02:36:32.803567+0000","last_unstale":"2024-09-21T02:36:37.395905+0000","last_undegraded":"2024-09-21T02:36:37.395905+0000","last_fullsized":"2024-09-21T02:36:37.395905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:41.692644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991291+0000","last_change":"2024-09-21T02:36:32.804687+0000","last_active":"2024-09-21T02:36:36.991291+0000","last_peered":"2024-09-21T02:36:36.991291+0000","last_clean":"2024-09-21T02:36:36.991291+0000","last_became_active":"2024-09-21T02:36:32.804232+0000","last_became_peered":"2024-09-21T02:36:32.804232+0000","last_unstale":"2024-09-21T02:36:36.991291+0000","last_undegraded":"2024-09-21T02:36:36.991291+0000","last_fullsized":"2024-09-21T02:36:36.991291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:43.899924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.753693+0000","last_change":"2024-09-21T02:36:25.749114+0000","last_active":"2024-09-21T02:36:36.753693+0000","last_peered":"2024-09-21T02:36:36.753693+0000","last_clean":"2024-09-21T02:36:36.753693+0000","last_became_active":"2024-09-21T02:36:25.748758+0000","last_became_peered":"2024-09-21T02:36:25.748758+0000","last_unstale":"2024-09-21T02:36:36.753693+0000","last_undegraded":"2024-09-21T02:36:36.753693+0000","last_fullsized":"2024-09-21T02:36:36.753693+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:57.958533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:09.043657+0000","last_change":"2024-09-21T02:36:23.731473+0000","last_active":"2024-09-21T02:37:09.043657+0000","last_peered":"2024-09-21T02:37:09.043657+0000","last_clean":"2024-09-21T02:37:09.043657+0000","last_became_active":"2024-09-21T02:36:23.731134+0000","last_became_peered":"2024-09-21T02:36:23.731134+0000","last_unstale":"2024-09-21T02:37:09.043657+0000","last_undegraded":"2024-09-21T02:37:09.043657+0000","last_fullsized":"2024-09-21T02:37:09.043657+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:05.368941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900104+0000","last_change":"2024-09-21T02:36:32.809630+0000","last_active":"2024-09-21T02:36:35.900104+0000","last_peered":"2024-09-21T02:36:35.900104+0000","last_clean":"2024-09-21T02:36:35.900104+0000","last_became_active":"2024-09-21T02:36:32.809030+0000","last_became_peered":"2024-09-21T02:36:32.809030+0000","last_unstale":"2024-09-21T02:36:35.900104+0000","last_undegraded":"2024-09-21T02:36:35.900104+0000","last_fullsized":"2024-09-21T02:36:35.900104+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:57:42.436590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395972+0000","last_change":"2024-09-21T02:36:25.742558+0000","last_active":"2024-09-21T02:36:37.395972+0000","last_peered":"2024-09-21T02:36:37.395972+0000","last_clean":"2024-09-21T02:36:37.395972+0000","last_became_active":"2024-09-21T02:36:25.742045+0000","last_became_peered":"2024-09-21T02:36:25.742045+0000","last_unstale":"2024-09-21T02:36:37.395972+0000","last_undegraded":"2024-09-21T02:36:37.395972+0000","last_fullsized":"2024-09-21T02:36:37.395972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:57:06.705561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:07.025569+0000","last_change":"2024-09-21T02:36:23.724345+0000","last_active":"2024-09-21T02:37:07.025569+0000","last_peered":"2024-09-21T02:37:07.025569+0000","last_clean":"2024-09-21T02:37:07.025569+0000","last_became_active":"2024-09-21T02:36:23.723635+0000","last_became_peered":"2024-09-21T02:36:23.723635+0000","last_unstale":"2024-09-21T02:37:07.025569+0000","last_undegraded":"2024-09-21T02:37:07.025569+0000","last_fullsized":"2024-09-21T02:37:07.025569+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:18.812176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:08.034637+0000","last_change":"2024-09-21T02:36:23.732897+0000","last_active":"2024-09-21T02:37:08.034637+0000","last_peered":"2024-09-21T02:37:08.034637+0000","last_clean":"2024-09-21T02:37:08.034637+0000","last_became_active":"2024-09-21T02:36:23.732509+0000","last_became_peered":"2024-09-21T02:36:23.732509+0000","last_unstale":"2024-09-21T02:37:08.034637+0000","last_undegraded":"2024-09-21T02:37:08.034637+0000","last_fullsized":"2024-09-21T02:37:08.034637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:12:27.117962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.398180+0000","last_change":"2024-09-21T02:36:25.748929+0000","last_active":"2024-09-21T02:36:37.398180+0000","last_peered":"2024-09-21T02:36:37.398180+0000","last_clean":"2024-09-21T02:36:37.398180+0000","last_became_active":"2024-09-21T02:36:25.748588+0000","last_became_peered":"2024-09-21T02:36:25.748588+0000","last_unstale":"2024-09-21T02:36:37.398180+0000","last_undegraded":"2024-09-21T02:36:37.398180+0000","last_fullsized":"2024-09-21T02:36:37.398180+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.086472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397254+0000","last_change":"2024-09-21T02:36:32.807655+0000","last_active":"2024-09-21T02:36:37.397254+0000","last_peered":"2024-09-21T02:36:37.397254+0000","last_clean":"2024-09-21T02:36:37.397254+0000","last_became_active":"2024-09-21T02:36:32.806995+0000","last_became_peered":"2024-09-21T02:36:32.806995+0000","last_unstale":"2024-09-21T02:36:37.397254+0000","last_undegraded":"2024-09-21T02:36:37.397254+0000","last_fullsized":"2024-09-21T02:36:37.397254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:28:20.420591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":52,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:37:20.066627+0000","last_change":"2024-09-21T02:36:23.729427+0000","last_active":"2024-09-21T02:37:20.066627+0000","last_peered":"2024-09-21T02:37:20.066627+0000","last_clean":"2024-09-21T02:37:20.066627+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:37:20.066627+0000","last_undegraded":"2024-09-21T02:37:20.066627+0000","last_fullsized":"2024-09-21T02:37:20.066627+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:10:41.251665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396720+0000","last_change":"2024-09-21T02:36:25.743212+0000","last_active":"2024-09-21T02:36:37.396720+0000","last_peered":"2024-09-21T02:36:37.396720+0000","last_clean":"2024-09-21T02:36:37.396720+0000","last_became_active":"2024-09-21T02:36:25.742777+0000","last_became_peered":"2024-09-21T02:36:25.742777+0000","last_unstale":"2024-09-21T02:36:37.396720+0000","last_undegraded":"2024-09-21T02:36:37.396720+0000","last_fullsized":"2024-09-21T02:36:37.396720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:12:19.917214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397806+0000","last_change":"2024-09-21T02:36:32.801880+0000","last_active":"2024-09-21T02:36:37.397806+0000","last_peered":"2024-09-21T02:36:37.397806+0000","last_clean":"2024-09-21T02:36:37.397806+0000","last_became_active":"2024-09-21T02:36:32.801389+0000","last_became_peered":"2024-09-21T02:36:32.801389+0000","last_unstale":"2024-09-21T02:36:37.397806+0000","last_undegraded":"2024-09-21T02:36:37.397806+0000","last_fullsized":"2024-09-21T02:36:37.397806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:11:44.877062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899699+0000","last_change":"2024-09-21T02:36:25.754014+0000","last_active":"2024-09-21T02:36:35.899699+0000","last_peered":"2024-09-21T02:36:35.899699+0000","last_clean":"2024-09-21T02:36:35.899699+0000","last_became_active":"2024-09-21T02:36:25.753548+0000","last_became_peered":"2024-09-21T02:36:25.753548+0000","last_unstale":"2024-09-21T02:36:35.899699+0000","last_undegraded":"2024-09-21T02:36:35.899699+0000","last_fullsized":"2024-09-21T02:36:35.899699+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:03.850625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397868+0000","last_change":"2024-09-21T02:36:25.751909+0000","last_active":"2024-09-21T02:36:37.397868+0000","last_peered":"2024-09-21T02:36:37.397868+0000","last_clean":"2024-09-21T02:36:37.397868+0000","last_became_active":"2024-09-21T02:36:25.751484+0000","last_became_peered":"2024-09-21T02:36:25.751484+0000","last_unstale":"2024-09-21T02:36:37.397868+0000","last_undegraded":"2024-09-21T02:36:37.397868+0000","last_fullsized":"2024-09-21T02:36:37.397868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:49:20.034294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396629+0000","last_change":"2024-09-21T02:36:25.752839+0000","last_active":"2024-09-21T02:36:37.396629+0000","last_peered":"2024-09-21T02:36:37.396629+0000","last_clean":"2024-09-21T02:36:37.396629+0000","last_became_active":"2024-09-21T02:36:25.750882+0000","last_became_peered":"2024-09-21T02:36:25.750882+0000","last_unstale":"2024-09-21T02:36:37.396629+0000","last_undegraded":"2024-09-21T02:36:37.396629+0000","last_fullsized":"2024-09-21T02:36:37.396629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:53.028611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991289+0000","last_change":"2024-09-21T02:36:25.744624+0000","last_active":"2024-09-21T02:36:36.991289+0000","last_peered":"2024-09-21T02:36:36.991289+0000","last_clean":"2024-09-21T02:36:36.991289+0000","last_became_active":"2024-09-21T02:36:25.744173+0000","last_became_peered":"2024-09-21T02:36:25.744173+0000","last_unstale":"2024-09-21T02:36:36.991289+0000","last_undegraded":"2024-09-21T02:36:36.991289+0000","last_fullsized":"2024-09-21T02:36:36.991289+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:55.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899306+0000","last_change":"2024-09-21T02:36:25.743669+0000","last_active":"2024-09-21T02:36:35.899306+0000","last_peered":"2024-09-21T02:36:35.899306+0000","last_clean":"2024-09-21T02:36:35.899306+0000","last_became_active":"2024-09-21T02:36:25.743332+0000","last_became_peered":"2024-09-21T02:36:25.743332+0000","last_unstale":"2024-09-21T02:36:35.899306+0000","last_undegraded":"2024-09-21T02:36:35.899306+0000","last_fullsized":"2024-09-21T02:36:35.899306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:25.189197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900404+0000","last_change":"2024-09-21T02:36:25.743476+0000","last_active":"2024-09-21T02:36:35.900404+0000","last_peered":"2024-09-21T02:36:35.900404+0000","last_clean":"2024-09-21T02:36:35.900404+0000","last_became_active":"2024-09-21T02:36:25.743117+0000","last_became_peered":"2024-09-21T02:36:25.743117+0000","last_unstale":"2024-09-21T02:36:35.900404+0000","last_undegraded":"2024-09-21T02:36:35.900404+0000","last_fullsized":"2024-09-21T02:36:35.900404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:08.084109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396952+0000","last_change":"2024-09-21T02:36:25.753014+0000","last_active":"2024-09-21T02:36:37.396952+0000","last_peered":"2024-09-21T02:36:37.396952+0000","last_clean":"2024-09-21T02:36:37.396952+0000","last_became_active":"2024-09-21T02:36:25.751232+0000","last_became_peered":"2024-09-21T02:36:25.751232+0000","last_unstale":"2024-09-21T02:36:37.396952+0000","last_undegraded":"2024-09-21T02:36:37.396952+0000","last_fullsized":"2024-09-21T02:36:37.396952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:03.052737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396567+0000","last_change":"2024-09-21T02:36:25.747296+0000","last_active":"2024-09-21T02:36:37.396567+0000","last_peered":"2024-09-21T02:36:37.396567+0000","last_clean":"2024-09-21T02:36:37.396567+0000","last_became_active":"2024-09-21T02:36:25.746832+0000","last_became_peered":"2024-09-21T02:36:25.746832+0000","last_unstale":"2024-09-21T02:36:37.396567+0000","last_undegraded":"2024-09-21T02:36:37.396567+0000","last_fullsized":"2024-09-21T02:36:37.396567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:21:55.099017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396829+0000","last_change":"2024-09-21T02:36:25.749605+0000","last_active":"2024-09-21T02:36:37.396829+0000","last_peered":"2024-09-21T02:36:37.396829+0000","last_clean":"2024-09-21T02:36:37.396829+0000","last_became_active":"2024-09-21T02:36:25.749165+0000","last_became_peered":"2024-09-21T02:36:25.749165+0000","last_unstale":"2024-09-21T02:36:37.396829+0000","last_undegraded":"2024-09-21T02:36:37.396829+0000","last_fullsized":"2024-09-21T02:36:37.396829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:15:51.412761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.396144+0000","last_change":"2024-09-21T02:36:25.747494+0000","last_active":"2024-09-21T02:36:37.396144+0000","last_peered":"2024-09-21T02:36:37.396144+0000","last_clean":"2024-09-21T02:36:37.396144+0000","last_became_active":"2024-09-21T02:36:25.746965+0000","last_became_peered":"2024-09-21T02:36:25.746965+0000","last_unstale":"2024-09-21T02:36:37.396144+0000","last_undegraded":"2024-09-21T02:36:37.396144+0000","last_fullsized":"2024-09-21T02:36:37.396144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:51.309307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380672+0000","last_change":"2024-09-21T02:36:25.749517+0000","last_active":"2024-09-21T02:36:36.380672+0000","last_peered":"2024-09-21T02:36:36.380672+0000","last_clean":"2024-09-21T02:36:36.380672+0000","last_became_active":"2024-09-21T02:36:25.749117+0000","last_became_peered":"2024-09-21T02:36:25.749117+0000","last_unstale":"2024-09-21T02:36:36.380672+0000","last_undegraded":"2024-09-21T02:36:36.380672+0000","last_fullsized":"2024-09-21T02:36:36.380672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:38.758019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899175+0000","last_change":"2024-09-21T02:36:25.736067+0000","last_active":"2024-09-21T02:36:35.899175+0000","last_peered":"2024-09-21T02:36:35.899175+0000","last_clean":"2024-09-21T02:36:35.899175+0000","last_became_active":"2024-09-21T02:36:25.735710+0000","last_became_peered":"2024-09-21T02:36:25.735710+0000","last_unstale":"2024-09-21T02:36:35.899175+0000","last_undegraded":"2024-09-21T02:36:35.899175+0000","last_fullsized":"2024-09-21T02:36:35.899175+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.559448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899895+0000","last_change":"2024-09-21T02:36:25.746307+0000","last_active":"2024-09-21T02:36:35.899895+0000","last_peered":"2024-09-21T02:36:35.899895+0000","last_clean":"2024-09-21T02:36:35.899895+0000","last_became_active":"2024-09-21T02:36:25.745872+0000","last_became_peered":"2024-09-21T02:36:25.745872+0000","last_unstale":"2024-09-21T02:36:35.899895+0000","last_undegraded":"2024-09-21T02:36:35.899895+0000","last_fullsized":"2024-09-21T02:36:35.899895+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:58:53.074559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397917+0000","last_change":"2024-09-21T02:36:25.753393+0000","last_active":"2024-09-21T02:36:37.397917+0000","last_peered":"2024-09-21T02:36:37.397917+0000","last_clean":"2024-09-21T02:36:37.397917+0000","last_became_active":"2024-09-21T02:36:25.751871+0000","last_became_peered":"2024-09-21T02:36:25.751871+0000","last_unstale":"2024-09-21T02:36:37.397917+0000","last_undegraded":"2024-09-21T02:36:37.397917+0000","last_fullsized":"2024-09-21T02:36:37.397917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:28.199451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.754175+0000","last_change":"2024-09-21T02:36:25.744845+0000","last_active":"2024-09-21T02:36:36.754175+0000","last_peered":"2024-09-21T02:36:36.754175+0000","last_clean":"2024-09-21T02:36:36.754175+0000","last_became_active":"2024-09-21T02:36:25.744492+0000","last_became_peered":"2024-09-21T02:36:25.744492+0000","last_unstale":"2024-09-21T02:36:36.754175+0000","last_undegraded":"2024-09-21T02:36:36.754175+0000","last_fullsized":"2024-09-21T02:36:36.754175+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:08.524871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900318+0000","last_change":"2024-09-21T02:36:25.751187+0000","last_active":"2024-09-21T02:36:35.900318+0000","last_peered":"2024-09-21T02:36:35.900318+0000","last_clean":"2024-09-21T02:36:35.900318+0000","last_became_active":"2024-09-21T02:36:25.750813+0000","last_became_peered":"2024-09-21T02:36:25.750813+0000","last_unstale":"2024-09-21T02:36:35.900318+0000","last_undegraded":"2024-09-21T02:36:35.900318+0000","last_fullsized":"2024-09-21T02:36:35.900318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.590210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380272+0000","last_change":"2024-09-21T02:36:25.744046+0000","last_active":"2024-09-21T02:36:36.380272+0000","last_peered":"2024-09-21T02:36:36.380272+0000","last_clean":"2024-09-21T02:36:36.380272+0000","last_became_active":"2024-09-21T02:36:25.743695+0000","last_became_peered":"2024-09-21T02:36:25.743695+0000","last_unstale":"2024-09-21T02:36:36.380272+0000","last_undegraded":"2024-09-21T02:36:36.380272+0000","last_fullsized":"2024-09-21T02:36:36.380272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:12.092940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991020+0000","last_change":"2024-09-21T02:36:25.749113+0000","last_active":"2024-09-21T02:36:36.991020+0000","last_peered":"2024-09-21T02:36:36.991020+0000","last_clean":"2024-09-21T02:36:36.991020+0000","last_became_active":"2024-09-21T02:36:25.748759+0000","last_became_peered":"2024-09-21T02:36:25.748759+0000","last_unstale":"2024-09-21T02:36:36.991020+0000","last_undegraded":"2024-09-21T02:36:36.991020+0000","last_fullsized":"2024-09-21T02:36:36.991020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:22.785273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.990999+0000","last_change":"2024-09-21T02:36:25.745454+0000","last_active":"2024-09-21T02:36:36.990999+0000","last_peered":"2024-09-21T02:36:36.990999+0000","last_clean":"2024-09-21T02:36:36.990999+0000","last_became_active":"2024-09-21T02:36:25.745108+0000","last_became_peered":"2024-09-21T02:36:25.745108+0000","last_unstale":"2024-09-21T02:36:36.990999+0000","last_undegraded":"2024-09-21T02:36:36.990999+0000","last_fullsized":"2024-09-21T02:36:36.990999+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:36.710456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899061+0000","last_change":"2024-09-21T02:36:25.737213+0000","last_active":"2024-09-21T02:36:35.899061+0000","last_peered":"2024-09-21T02:36:35.899061+0000","last_clean":"2024-09-21T02:36:35.899061+0000","last_became_active":"2024-09-21T02:36:25.736641+0000","last_became_peered":"2024-09-21T02:36:25.736641+0000","last_unstale":"2024-09-21T02:36:35.899061+0000","last_undegraded":"2024-09-21T02:36:35.899061+0000","last_fullsized":"2024-09-21T02:36:35.899061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:47:50.258805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397271+0000","last_change":"2024-09-21T02:36:25.748230+0000","last_active":"2024-09-21T02:36:37.397271+0000","last_peered":"2024-09-21T02:36:37.397271+0000","last_clean":"2024-09-21T02:36:37.397271+0000","last_became_active":"2024-09-21T02:36:25.747859+0000","last_became_peered":"2024-09-21T02:36:25.747859+0000","last_unstale":"2024-09-21T02:36:37.397271+0000","last_undegraded":"2024-09-21T02:36:37.397271+0000","last_fullsized":"2024-09-21T02:36:37.397271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:15:21.244825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401717+0000","last_change":"2024-09-21T02:36:25.744379+0000","last_active":"2024-09-21T02:36:36.401717+0000","last_peered":"2024-09-21T02:36:36.401717+0000","last_clean":"2024-09-21T02:36:36.401717+0000","last_became_active":"2024-09-21T02:36:25.744003+0000","last_became_peered":"2024-09-21T02:36:25.744003+0000","last_unstale":"2024-09-21T02:36:36.401717+0000","last_undegraded":"2024-09-21T02:36:36.401717+0000","last_fullsized":"2024-09-21T02:36:36.401717+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:59:59.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900340+0000","last_change":"2024-09-21T02:36:25.755616+0000","last_active":"2024-09-21T02:36:35.900340+0000","last_peered":"2024-09-21T02:36:35.900340+0000","last_clean":"2024-09-21T02:36:35.900340+0000","last_became_active":"2024-09-21T02:36:25.755304+0000","last_became_peered":"2024-09-21T02:36:25.755304+0000","last_unstale":"2024-09-21T02:36:35.900340+0000","last_undegraded":"2024-09-21T02:36:35.900340+0000","last_fullsized":"2024-09-21T02:36:35.900340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:24.813780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.380995+0000","last_change":"2024-09-21T02:36:25.739747+0000","last_active":"2024-09-21T02:36:36.380995+0000","last_peered":"2024-09-21T02:36:36.380995+0000","last_clean":"2024-09-21T02:36:36.380995+0000","last_became_active":"2024-09-21T02:36:25.739404+0000","last_became_peered":"2024-09-21T02:36:25.739404+0000","last_unstale":"2024-09-21T02:36:36.380995+0000","last_undegraded":"2024-09-21T02:36:36.380995+0000","last_fullsized":"2024-09-21T02:36:36.380995+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:17.975419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.395850+0000","last_change":"2024-09-21T02:36:25.747723+0000","last_active":"2024-09-21T02:36:37.395850+0000","last_peered":"2024-09-21T02:36:37.395850+0000","last_clean":"2024-09-21T02:36:37.395850+0000","last_became_active":"2024-09-21T02:36:25.747368+0000","last_became_peered":"2024-09-21T02:36:25.747368+0000","last_unstale":"2024-09-21T02:36:37.395850+0000","last_undegraded":"2024-09-21T02:36:37.395850+0000","last_fullsized":"2024-09-21T02:36:37.395850+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.319756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.401352+0000","last_change":"2024-09-21T02:36:25.743772+0000","last_active":"2024-09-21T02:36:36.401352+0000","last_peered":"2024-09-21T02:36:36.401352+0000","last_clean":"2024-09-21T02:36:36.401352+0000","last_became_active":"2024-09-21T02:36:25.743403+0000","last_became_peered":"2024-09-21T02:36:25.743403+0000","last_unstale":"2024-09-21T02:36:36.401352+0000","last_undegraded":"2024-09-21T02:36:36.401352+0000","last_fullsized":"2024-09-21T02:36:36.401352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:11:29.930248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.899740+0000","last_change":"2024-09-21T02:36:25.746911+0000","last_active":"2024-09-21T02:36:35.899740+0000","last_peered":"2024-09-21T02:36:35.899740+0000","last_clean":"2024-09-21T02:36:35.899740+0000","last_became_active":"2024-09-21T02:36:25.746426+0000","last_became_peered":"2024-09-21T02:36:25.746426+0000","last_unstale":"2024-09-21T02:36:35.899740+0000","last_undegraded":"2024-09-21T02:36:35.899740+0000","last_fullsized":"2024-09-21T02:36:35.899740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:12.835851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397123+0000","last_change":"2024-09-21T02:36:25.751278+0000","last_active":"2024-09-21T02:36:37.397123+0000","last_peered":"2024-09-21T02:36:37.397123+0000","last_clean":"2024-09-21T02:36:37.397123+0000","last_became_active":"2024-09-21T02:36:25.750414+0000","last_became_peered":"2024-09-21T02:36:25.750414+0000","last_unstale":"2024-09-21T02:36:37.397123+0000","last_undegraded":"2024-09-21T02:36:37.397123+0000","last_fullsized":"2024-09-21T02:36:37.397123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:13:56.080122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:37.397213+0000","last_change":"2024-09-21T02:36:25.752852+0000","last_active":"2024-09-21T02:36:37.397213+0000","last_peered":"2024-09-21T02:36:37.397213+0000","last_clean":"2024-09-21T02:36:37.397213+0000","last_became_active":"2024-09-21T02:36:25.752514+0000","last_became_peered":"2024-09-21T02:36:25.752514+0000","last_unstale":"2024-09-21T02:36:37.397213+0000","last_undegraded":"2024-09-21T02:36:37.397213+0000","last_fullsized":"2024-09-21T02:36:37.397213+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:55.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991017+0000","last_change":"2024-09-21T02:36:25.750081+0000","last_active":"2024-09-21T02:36:36.991017+0000","last_peered":"2024-09-21T02:36:36.991017+0000","last_clean":"2024-09-21T02:36:36.991017+0000","last_became_active":"2024-09-21T02:36:25.749602+0000","last_became_peered":"2024-09-21T02:36:25.749602+0000","last_unstale":"2024-09-21T02:36:36.991017+0000","last_undegraded":"2024-09-21T02:36:36.991017+0000","last_fullsized":"2024-09-21T02:36:36.991017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:06.785998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:36.991322+0000","last_change":"2024-09-21T02:36:25.746897+0000","last_active":"2024-09-21T02:36:36.991322+0000","last_peered":"2024-09-21T02:36:36.991322+0000","last_clean":"2024-09-21T02:36:36.991322+0000","last_became_active":"2024-09-21T02:36:25.746477+0000","last_became_peered":"2024-09-21T02:36:25.746477+0000","last_unstale":"2024-09-21T02:36:36.991322+0000","last_undegraded":"2024-09-21T02:36:36.991322+0000","last_fullsized":"2024-09-21T02:36:36.991322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:34:00.739410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T02:36:35.900270+0000","last_change":"2024-09-21T02:36:25.745233+0000","last_active":"2024-09-21T02:36:35.900270+0000","last_peered":"2024-09-21T02:36:35.900270+0000","last_clean":"2024-09-21T02:36:35.900270+0000","last_became_active":"2024-09-21T02:36:25.744637+0000","last_became_peered":"2024-09-21T02:36:25.744637+0000","last_unstale":"2024-09-21T02:36:35.900270+0000","last_undegraded":"2024-09-21T02:36:35.900270+0000","last_fullsized":"2024-09-21T02:36:35.900270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:34:59.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67504,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":177,"num_read_kb":163,"num_write":244,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":202512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249064,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":69,"seq":296352743448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":447975,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906973,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":1094700,"data_compressed":53859,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070499,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":451610,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266728,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":913408,"data_stored":447892,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462958,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":1044389,"data_compressed":53859,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659187,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":443018,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855416,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":1039971,"data_compressed":53859,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018941,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":901120,"data_stored":447092,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215169,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":905216,"data_stored":442726,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":505350,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":504348,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":917504,"data_stored":444865,"data_compressed":17585,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:37:28.131 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T02:37:28.131 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T02:37:28.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T02:37:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:28.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:28 smithi029 ceph-mon[31006]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:37:28.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:28 smithi190 ceph-mon[33871]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:37:28.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 -- 172.21.15.29:0/3455225839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0102640 msgr2=0x7f1eb0102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/3455225839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0102640 0x7f1eb0102ab0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009900 tx=0x7f1eac02a480 comp rx=0 tx=0).stop 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 -- 172.21.15.29:0/3455225839 shutdown_connections 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/3455225839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb0104070 0x7f1eb010aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/3455225839 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1eb01036d0 0x7f1eb0103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/3455225839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0102640 0x7f1eb0102ab0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 -- 172.21.15.29:0/3455225839 >> 172.21.15.29:0/3455225839 conn(0x7f1eb00fdfb0 msgr2=0x7f1eb0100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 -- 172.21.15.29:0/3455225839 shutdown_connections 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.909+0000 7f1eb72c8640 1 -- 172.21.15.29:0/3455225839 wait complete. 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb72c8640 1 Processor -- start 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb72c8640 1 -- start start 2024-09-21T02:37:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb72c8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb0102640 0x7f1eb019c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb72c8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1eb01036d0 0x7f1eb019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb72c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0104070 0x7f1eb019d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb72c8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb010e110 con 0x7f1eb0104070 2024-09-21T02:37:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb72c8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb010df90 con 0x7f1eb01036d0 2024-09-21T02:37:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb72c8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb010e290 con 0x7f1eb0102640 2024-09-21T02:37:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb5865640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0104070 0x7f1eb019d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb5865640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0104070 0x7f1eb019d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45754/0 (socket says 172.21.15.29:45754) 2024-09-21T02:37:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb5865640 1 -- 172.21.15.29:0/2244038270 learned_addr learned my addr 172.21.15.29:0/2244038270 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.910+0000 7f1eb4863640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1eb01036d0 0x7f1eb019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb5064640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb0102640 0x7f1eb019c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb5865640 1 -- 172.21.15.29:0/2244038270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb0102640 msgr2=0x7f1eb019c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb5865640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb0102640 0x7f1eb019c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb5865640 1 -- 172.21.15.29:0/2244038270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1eb01036d0 msgr2=0x7f1eb019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb5865640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1eb01036d0 0x7f1eb019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb5865640 1 -- 172.21.15.29:0/2244038270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb019da90 con 0x7f1eb0104070 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb5865640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0104070 0x7f1eb019d300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800cce0 tx=0x7f1ea800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1ea67fc640 1 -- 172.21.15.29:0/2244038270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea800f450 con 0x7f1eb0104070 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1ea67fc640 1 -- 172.21.15.29:0/2244038270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ea800fa80 con 0x7f1eb0104070 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1ea67fc640 1 -- 172.21.15.29:0/2244038270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8018ad0 con 0x7f1eb0104070 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb019dd80 con 0x7f1eb0104070 2024-09-21T02:37:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.911+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb01a4340 con 0x7f1eb0104070 2024-09-21T02:37:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.912+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb01a45d0 con 0x7f1eb0104070 2024-09-21T02:37:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.914+0000 7f1ea67fc640 1 -- 172.21.15.29:0/2244038270 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f1ea800f5f0 con 0x7f1eb0104070 2024-09-21T02:37:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.915+0000 7f1ea67fc640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1e9c0b65a0 0x7f1e9c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.915+0000 7f1eb5064640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1e9c0b65a0 0x7f1e9c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.915+0000 7f1ea67fc640 1 -- 172.21.15.29:0/2244038270 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f1ea80e7f50 con 0x7f1eb0104070 2024-09-21T02:37:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.916+0000 7f1eb5064640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1e9c0b65a0 0x7f1e9c0b8ab0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1eac000c00 tx=0x7f1eac005720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:28.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:28.918+0000 7f1ea67fc640 1 -- 172.21.15.29:0/2244038270 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1ea8014030 con 0x7f1eb0104070 2024-09-21T02:37:29.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:28 smithi066 ceph-mon[33436]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:37:29.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.178+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f1eb01a4840 con 0x7f1eb0104070 2024-09-21T02:37:29.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.181+0000 7f1ea67fc640 1 -- 172.21.15.29:0/2244038270 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f1ea80ae4d0 con 0x7f1eb0104070 2024-09-21T02:37:29.182 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-09-21T02:37:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBpMe5mCISzChAAEbLXFFakCPO2qlqb0SKb2Q== 2024-09-21T02:37:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1e9c0b65a0 msgr2=0x7f1e9c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1e9c0b65a0 0x7f1e9c0b8ab0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1eac000c00 tx=0x7f1eac005720 comp rx=0 tx=0).stop 2024-09-21T02:37:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0104070 msgr2=0x7f1eb019d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0104070 0x7f1eb019d300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800cce0 tx=0x7f1ea800a620 comp rx=0 tx=0).stop 2024-09-21T02:37:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 shutdown_connections 2024-09-21T02:37:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1e9c0b65a0 0x7f1e9c0b8ab0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1eb0104070 0x7f1eb019d300 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1eb01036d0 0x7f1eb019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 --2- 172.21.15.29:0/2244038270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb0102640 0x7f1eb019c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.183+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 >> 172.21.15.29:0/2244038270 conn(0x7f1eb00fdfb0 msgr2=0x7f1eb00ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.184+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 shutdown_connections 2024-09-21T02:37:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:29.184+0000 7f1eb72c8640 1 -- 172.21.15.29:0/2244038270 wait complete. 2024-09-21T02:37:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:29 smithi190 ceph-mon[33871]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-09-21T02:37:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:29 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2244038270' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T02:37:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:29 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2244038270' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T02:37:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:29 smithi029 ceph-mon[31006]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-09-21T02:37:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:29 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2244038270' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T02:37:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:29 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2244038270' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T02:37:30.000 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:37:30.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T02:37:30.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T02:37:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:29 smithi066 ceph-mon[33436]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-09-21T02:37:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:29 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2244038270' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T02:37:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:29 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2244038270' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T02:37:30.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T02:37:30.342 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 -- 172.21.15.29:0/607393541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 msgr2=0x7fba941087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 --2- 172.21.15.29:0/607393541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 0x7fba941087d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009900 tx=0x7fba7c02a3f0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 -- 172.21.15.29:0/607393541 shutdown_connections 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 --2- 172.21.15.29:0/607393541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba94108d10 0x7fba9410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 --2- 172.21.15.29:0/607393541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 0x7fba941087d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 --2- 172.21.15.29:0/607393541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba94102a80 0x7fba94102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 -- 172.21.15.29:0/607393541 >> 172.21.15.29:0/607393541 conn(0x7fba94079740 msgr2=0x7fba94079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 -- 172.21.15.29:0/607393541 shutdown_connections 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.975+0000 7fba9a31f640 1 -- 172.21.15.29:0/607393541 wait complete. 2024-09-21T02:37:31.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba9a31f640 1 Processor -- start 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba9a31f640 1 -- start start 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba9a31f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba94102a80 0x7fba9419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba9a31f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 0x7fba9419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba9a31f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba94108d10 0x7fba9419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba9a31f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fba94112830 con 0x7fba941083c0 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba9a31f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fba941126b0 con 0x7fba94102a80 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba9a31f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fba941129b0 con 0x7fba94108d10 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 0x7fba9419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 0x7fba9419aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45770/0 (socket says 172.21.15.29:45770) 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.976+0000 7fba8bfff640 1 -- 172.21.15.29:0/1903462718 learned_addr learned my addr 172.21.15.29:0/1903462718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba988bc640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba94108d10 0x7fba9419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba93fff640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba94102a80 0x7fba9419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba8bfff640 1 -- 172.21.15.29:0/1903462718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba94108d10 msgr2=0x7fba9419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba8bfff640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba94108d10 0x7fba9419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba8bfff640 1 -- 172.21.15.29:0/1903462718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba94102a80 msgr2=0x7fba9419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba8bfff640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba94102a80 0x7fba9419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba8bfff640 1 -- 172.21.15.29:0/1903462718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba941a0360 con 0x7fba941083c0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba8bfff640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 0x7fba9419aa50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fba7c005940 tx=0x7fba7c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba91ffb640 1 -- 172.21.15.29:0/1903462718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c03f030 con 0x7fba941083c0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba91ffb640 1 -- 172.21.15.29:0/1903462718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba7c003c90 con 0x7fba941083c0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba941a05f0 con 0x7fba941083c0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba91ffb640 1 -- 172.21.15.29:0/1903462718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c035070 con 0x7fba941083c0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.977+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba941a0ad0 con 0x7fba941083c0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.979+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba94103d00 con 0x7fba941083c0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.980+0000 7fba91ffb640 1 -- 172.21.15.29:0/1903462718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fba7c042030 con 0x7fba941083c0 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.981+0000 7fba91ffb640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fba800b65d0 0x7fba800b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.981+0000 7fba93fff640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fba800b65d0 0x7fba800b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:31.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.981+0000 7fba91ffb640 1 -- 172.21.15.29:0/1903462718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fba7c107b90 con 0x7fba941083c0 2024-09-21T02:37:31.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.981+0000 7fba93fff640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fba800b65d0 0x7fba800b8ae0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fba84005fd0 tx=0x7fba8400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:31.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:30.985+0000 7fba91ffb640 1 -- 172.21.15.29:0/1903462718 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fba7c0d2460 con 0x7fba941083c0 2024-09-21T02:37:31.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.179+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fba9419b5e0 con 0x7fba941083c0 2024-09-21T02:37:31.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.180+0000 7fba91ffb640 1 -- 172.21.15.29:0/1903462718 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v50) ==== 86+0+133 (secure 0 0 0) 0x7fba9419b5e0 con 0x7fba941083c0 2024-09-21T02:37:31.181 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:31.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fba800b65d0 msgr2=0x7fba800b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fba800b65d0 0x7fba800b8ae0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fba84005fd0 tx=0x7fba8400b390 comp rx=0 tx=0).stop 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 msgr2=0x7fba9419aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 0x7fba9419aa50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fba7c005940 tx=0x7fba7c0042f0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 shutdown_connections 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fba800b65d0 0x7fba800b8ae0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba94108d10 0x7fba9419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba941083c0 0x7fba9419aa50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 --2- 172.21.15.29:0/1903462718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba94102a80 0x7fba9419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.182+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 >> 172.21.15.29:0/1903462718 conn(0x7fba94079740 msgr2=0x7fba94101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.183+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 shutdown_connections 2024-09-21T02:37:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:31.183+0000 7fba9a31f640 1 -- 172.21.15.29:0/1903462718 wait complete. 2024-09-21T02:37:31.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:31 smithi190 ceph-mon[33871]: pgmap v337: 129 pgs: 129 active+clean; 647 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 14 op/s 2024-09-21T02:37:31.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:31 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1903462718' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T02:37:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:31 smithi029 ceph-mon[31006]: pgmap v337: 129 pgs: 129 active+clean; 647 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 14 op/s 2024-09-21T02:37:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:31 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1903462718' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T02:37:31.929 INFO:teuthology.orchestra.run.smithi029.stdout:[{"version":50,"timestamp":"2024-09-21T02:36:15.762867+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-09-21T02:37:31.930 INFO:tasks.ceph_manager:config epoch is 50 2024-09-21T02:37:31.930 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T02:37:31.930 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T02:37:31.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T02:37:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:31 smithi066 ceph-mon[33436]: pgmap v337: 129 pgs: 129 active+clean; 647 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 14 op/s 2024-09-21T02:37:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:31 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1903462718' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T02:37:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 -- 172.21.15.29:0/2599788590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 msgr2=0x7f1b94110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 --2- 172.21.15.29:0/2599788590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 0x7f1b94110620 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009900 tx=0x7f1b8c02a410 comp rx=0 tx=0).stop 2024-09-21T02:37:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 -- 172.21.15.29:0/2599788590 shutdown_connections 2024-09-21T02:37:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 --2- 172.21.15.29:0/2599788590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 0x7f1b94110620 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 --2- 172.21.15.29:0/2599788590 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b941094a0 0x7f1b941098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 --2- 172.21.15.29:0/2599788590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b94103b60 0x7f1b94103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 -- 172.21.15.29:0/2599788590 >> 172.21.15.29:0/2599788590 conn(0x7f1b9407b850 msgr2=0x7f1b9407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 -- 172.21.15.29:0/2599788590 shutdown_connections 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.797+0000 7f1b98b01640 1 -- 172.21.15.29:0/2599788590 wait complete. 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b98b01640 1 Processor -- start 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b98b01640 1 -- start start 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b98b01640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b94103b60 0x7f1b9419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b98b01640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b941094a0 0x7f1b9419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b98b01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 0x7f1b9419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b98b01640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94113960 con 0x7f1b94109df0 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b98b01640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941137e0 con 0x7f1b94103b60 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b98b01640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94113ae0 con 0x7f1b941094a0 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b92d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 0x7f1b9419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b92d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 0x7f1b9419fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45792/0 (socket says 172.21.15.29:45792) 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.798+0000 7f1b92d9d640 1 -- 172.21.15.29:0/3765344663 learned_addr learned my addr 172.21.15.29:0/3765344663 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b9259c640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b94103b60 0x7f1b9419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b91d9b640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b941094a0 0x7f1b9419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b92d9d640 1 -- 172.21.15.29:0/3765344663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b941094a0 msgr2=0x7f1b9419a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b92d9d640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b941094a0 0x7f1b9419a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b92d9d640 1 -- 172.21.15.29:0/3765344663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b94103b60 msgr2=0x7f1b9419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b92d9d640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b94103b60 0x7f1b9419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b92d9d640 1 -- 172.21.15.29:0/3765344663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b941a02a0 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b91d9b640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b941094a0 0x7f1b9419a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b92d9d640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 0x7f1b9419fb70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c004770 tx=0x7f1b8c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b837fe640 1 -- 172.21.15.29:0/3765344663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c03f030 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b941a0530 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b837fe640 1 -- 172.21.15.29:0/3765344663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b8c004440 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b837fe640 1 -- 172.21.15.29:0/3765344663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c035070 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.799+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b941a6cf0 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.800+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b94104de0 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.802+0000 7f1b837fe640 1 -- 172.21.15.29:0/3765344663 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f1b8c042030 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.803+0000 7f1b837fe640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b65a0 0x7f1b7c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.803+0000 7f1b9259c640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b65a0 0x7f1b7c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.803+0000 7f1b837fe640 1 -- 172.21.15.29:0/3765344663 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f1b8c107ee0 con 0x7f1b94109df0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.803+0000 7f1b9259c640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b65a0 0x7f1b7c0b8ab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1b84005fd0 tx=0x7f1b84009760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:32.806+0000 7f1b837fe640 1 -- 172.21.15.29:0/3765344663 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1b8c0ce3d0 con 0x7f1b94109df0 2024-09-21T02:37:33.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.051+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f1b9410bbe0 con 0x7f1b94109df0 2024-09-21T02:37:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.058+0000 7f1b837fe640 1 -- 172.21.15.29:0/3765344663 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+296435 (secure 0 0 0) 0x7f1b8c0d27b0 con 0x7f1b94109df0 2024-09-21T02:37:33.063 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b65a0 msgr2=0x7f1b7c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b65a0 0x7f1b7c0b8ab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1b84005fd0 tx=0x7f1b84009760 comp rx=0 tx=0).stop 2024-09-21T02:37:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 msgr2=0x7f1b9419fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 0x7f1b9419fb70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c004770 tx=0x7f1b8c004040 comp rx=0 tx=0).stop 2024-09-21T02:37:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 shutdown_connections 2024-09-21T02:37:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b65a0 0x7f1b7c0b8ab0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109df0 0x7f1b9419fb70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b941094a0 0x7f1b9419a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 --2- 172.21.15.29:0/3765344663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b94103b60 0x7f1b9419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 >> 172.21.15.29:0/3765344663 conn(0x7f1b9407b850 msgr2=0x7f1b941074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 shutdown_connections 2024-09-21T02:37:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:33.062+0000 7f1b98b01640 1 -- 172.21.15.29:0/3765344663 wait complete. 2024-09-21T02:37:33.768 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:33 smithi029 ceph-mon[31006]: pgmap v338: 129 pgs: 129 active+clean; 647 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-09-21T02:37:33.768 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3765344663' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T02:37:33.799 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":257624116},{"type":"v1","addr":"172.21.15.29:6801","nonce":257624116}]},"active_addr":"172.21.15.29:6801/257624116","active_change":"2024-09-21T02:28:37.826720+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24118,"name":"z","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24128,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.29:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2606589568}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1819393002}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2180096491}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2895165666}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2849189209}]}]} 2024-09-21T02:37:33.805 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T02:37:33.806 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T02:37:33.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:37:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:33 smithi190 ceph-mon[33871]: pgmap v338: 129 pgs: 129 active+clean; 647 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-09-21T02:37:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:33 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3765344663' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T02:37:34.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:33 smithi066 ceph-mon[33436]: pgmap v338: 129 pgs: 129 active+clean; 647 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-09-21T02:37:34.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:33 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3765344663' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T02:37:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 -- 172.21.15.29:0/1943636497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72641094b0 msgr2=0x7f72641098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 --2- 172.21.15.29:0/1943636497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72641094b0 0x7f72641098c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7254009900 tx=0x7f725402a3f0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 -- 172.21.15.29:0/1943636497 shutdown_connections 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 --2- 172.21.15.29:0/1943636497 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7264109e00 0x7f7264110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 --2- 172.21.15.29:0/1943636497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72641094b0 0x7f72641098c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 --2- 172.21.15.29:0/1943636497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7264101f70 0x7f7264102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 -- 172.21.15.29:0/1943636497 >> 172.21.15.29:0/1943636497 conn(0x7f7264077c60 msgr2=0x7f72640780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 -- 172.21.15.29:0/1943636497 shutdown_connections 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.580+0000 7f726bb11640 1 -- 172.21.15.29:0/1943636497 wait complete. 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726bb11640 1 Processor -- start 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726bb11640 1 -- start start 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726bb11640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7264101f70 0x7f726419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726bb11640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72641094b0 0x7f726419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726bb11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7264109e00 0x7f726419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726bb11640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f72641139c0 con 0x7f7264109e00 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726bb11640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7264113840 con 0x7f72641094b0 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726bb11640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7264113b40 con 0x7f7264101f70 2024-09-21T02:37:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726a0ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7264109e00 0x7f726419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726a0ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7264109e00 0x7f726419fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55152/0 (socket says 172.21.15.29:55152) 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f726a0ae640 1 -- 172.21.15.29:0/550772599 learned_addr learned my addr 172.21.15.29:0/550772599 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f72690ac640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72641094b0 0x7f726419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.581+0000 7f72698ad640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7264101f70 0x7f726419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f726a0ae640 1 -- 172.21.15.29:0/550772599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7264101f70 msgr2=0x7f726419a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f726a0ae640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7264101f70 0x7f726419a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f726a0ae640 1 -- 172.21.15.29:0/550772599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72641094b0 msgr2=0x7f726419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f726a0ae640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72641094b0 0x7f726419a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f726a0ae640 1 -- 172.21.15.29:0/550772599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72641a0260 con 0x7f7264109e00 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f726a0ae640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7264109e00 0x7f726419fbf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f725c00cce0 tx=0x7f725c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f725affd640 1 -- 172.21.15.29:0/550772599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c00f450 con 0x7f7264109e00 2024-09-21T02:37:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f725affd640 1 -- 172.21.15.29:0/550772599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f725c00fa80 con 0x7f7264109e00 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72641a0550 con 0x7f7264109e00 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f725affd640 1 -- 172.21.15.29:0/550772599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c0189c0 con 0x7f7264109e00 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.582+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72641a0a10 con 0x7f7264109e00 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.583+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7264102380 con 0x7f7264109e00 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.585+0000 7f725affd640 1 -- 172.21.15.29:0/550772599 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f725c00fc30 con 0x7f7264109e00 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.586+0000 7f725affd640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72500b65d0 0x7f72500b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.586+0000 7f72698ad640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72500b65d0 0x7f72500b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.586+0000 7f725affd640 1 -- 172.21.15.29:0/550772599 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f725c0e7ac0 con 0x7f7264109e00 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.587+0000 7f72698ad640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72500b65d0 0x7f72500b8ae0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f726000abe0 tx=0x7f726000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.589+0000 7f725affd640 1 -- 172.21.15.29:0/550772599 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f725c014030 con 0x7f7264109e00 2024-09-21T02:37:34.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.779+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f726419b4e0 con 0x7f7264109e00 2024-09-21T02:37:34.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.780+0000 7f725affd640 1 -- 172.21.15.29:0/550772599 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24199 (secure 0 0 0) 0x7f726419b4e0 con 0x7f7264109e00 2024-09-21T02:37:34.782 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:34.783 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:36:35.844398+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:37:34.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72500b65d0 msgr2=0x7f72500b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:34.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72500b65d0 0x7f72500b8ae0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f726000abe0 tx=0x7f726000a2d0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7264109e00 msgr2=0x7f726419fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7264109e00 0x7f726419fbf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f725c00cce0 tx=0x7f725c00a620 comp rx=0 tx=0).stop 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 shutdown_connections 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72500b65d0 0x7f72500b8ae0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7264109e00 0x7f726419fbf0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72641094b0 0x7f726419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 --2- 172.21.15.29:0/550772599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7264101f70 0x7f726419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 >> 172.21.15.29:0/550772599 conn(0x7f7264077c60 msgr2=0x7f7264105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 shutdown_connections 2024-09-21T02:37:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:34.783+0000 7f726bb11640 1 -- 172.21.15.29:0/550772599 wait complete. 2024-09-21T02:37:35.538 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T02:37:35.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:37:35.799 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:35 smithi190 ceph-mon[33871]: pgmap v339: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-21T02:37:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:35 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/550772599' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:35 smithi029 ceph-mon[31006]: pgmap v339: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-21T02:37:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:35 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/550772599' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:36.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:35 smithi066 ceph-mon[33436]: pgmap v339: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-21T02:37:36.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:35 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/550772599' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 -- 172.21.15.29:0/3676746892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f84109df0 msgr2=0x7f8f84110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 --2- 172.21.15.29:0/3676746892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f84109df0 0x7f8f84110620 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009ca0 tx=0x7f8f7c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 -- 172.21.15.29:0/3676746892 shutdown_connections 2024-09-21T02:37:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 --2- 172.21.15.29:0/3676746892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f84109df0 0x7f8f84110620 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 --2- 172.21.15.29:0/3676746892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f841094a0 0x7f8f841098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 --2- 172.21.15.29:0/3676746892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8f84103b60 0x7f8f84103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 -- 172.21.15.29:0/3676746892 >> 172.21.15.29:0/3676746892 conn(0x7f8f8407b850 msgr2=0x7f8f8407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 -- 172.21.15.29:0/3676746892 shutdown_connections 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 -- 172.21.15.29:0/3676746892 wait complete. 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.340+0000 7f8f89b22640 1 Processor -- start 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f89b22640 1 -- start start 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f89b22640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8f84103b60 0x7f8f84075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f89b22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841094a0 0x7f8f84075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f89b22640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f84109df0 0x7f8f84076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f89b22640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8f841139b0 con 0x7f8f841094a0 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f89b22640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8f84113830 con 0x7f8f84103b60 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f89b22640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8f84113b30 con 0x7f8f84109df0 2024-09-21T02:37:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841094a0 0x7f8f84075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841094a0 0x7f8f84075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55168/0 (socket says 172.21.15.29:55168) 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f82ffd640 1 -- 172.21.15.29:0/1963668636 learned_addr learned my addr 172.21.15.29:0/1963668636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f837fe640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8f84103b60 0x7f8f84075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f83fff640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f84109df0 0x7f8f84076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f82ffd640 1 -- 172.21.15.29:0/1963668636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f84109df0 msgr2=0x7f8f84076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f82ffd640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f84109df0 0x7f8f84076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f82ffd640 1 -- 172.21.15.29:0/1963668636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8f84103b60 msgr2=0x7f8f84075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f82ffd640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8f84103b60 0x7f8f84075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f82ffd640 1 -- 172.21.15.29:0/1963668636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f841a85b0 con 0x7f8f841094a0 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.341+0000 7f8f83fff640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f84109df0 0x7f8f84076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.342+0000 7f8f82ffd640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841094a0 0x7f8f84075c50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800c8c0 tx=0x7f8f7800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.342+0000 7f8f80ff9640 1 -- 172.21.15.29:0/1963668636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78007bb0 con 0x7f8f841094a0 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.342+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f841a88a0 con 0x7f8f841094a0 2024-09-21T02:37:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.342+0000 7f8f80ff9640 1 -- 172.21.15.29:0/1963668636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8f78007d50 con 0x7f8f841094a0 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.342+0000 7f8f80ff9640 1 -- 172.21.15.29:0/1963668636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78018660 con 0x7f8f841094a0 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.342+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f841a8e30 con 0x7f8f841094a0 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.343+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f84103f70 con 0x7f8f841094a0 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.344+0000 7f8f80ff9640 1 -- 172.21.15.29:0/1963668636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f8f7800fa00 con 0x7f8f841094a0 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.345+0000 7f8f80ff9640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8f6c0b65a0 0x7f8f6c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.345+0000 7f8f837fe640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8f6c0b65a0 0x7f8f6c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.345+0000 7f8f80ff9640 1 -- 172.21.15.29:0/1963668636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f8f780e7720 con 0x7f8f841094a0 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.346+0000 7f8f837fe640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8f6c0b65a0 0x7f8f6c0b8ab0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8f84077160 tx=0x7f8f7400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.350+0000 7f8f80ff9640 1 -- 172.21.15.29:0/1963668636 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8f78014030 con 0x7f8f841094a0 2024-09-21T02:37:36.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.538+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f8402e640 con 0x7f8f841094a0 2024-09-21T02:37:36.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.539+0000 7f8f80ff9640 1 -- 172.21.15.29:0/1963668636 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24199 (secure 0 0 0) 0x7f8f780adc10 con 0x7f8f841094a0 2024-09-21T02:37:36.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:37:36.541 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:36:35.844398+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:37:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8f6c0b65a0 msgr2=0x7f8f6c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8f6c0b65a0 0x7f8f6c0b8ab0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8f84077160 tx=0x7f8f7400b390 comp rx=0 tx=0).stop 2024-09-21T02:37:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841094a0 msgr2=0x7f8f84075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841094a0 0x7f8f84075c50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800c8c0 tx=0x7f8f7800cd80 comp rx=0 tx=0).stop 2024-09-21T02:37:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 shutdown_connections 2024-09-21T02:37:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8f6c0b65a0 0x7f8f6c0b8ab0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f84109df0 0x7f8f84076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841094a0 0x7f8f84075c50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 --2- 172.21.15.29:0/1963668636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8f84103b60 0x7f8f84075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 >> 172.21.15.29:0/1963668636 conn(0x7f8f8407b850 msgr2=0x7f8f841054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 shutdown_connections 2024-09-21T02:37:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:36.542+0000 7f8f89b22640 1 -- 172.21.15.29:0/1963668636 wait complete. 2024-09-21T02:37:36.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:36 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1963668636' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:36.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:36 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1963668636' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:37.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:36 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1963668636' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:37:37.314 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T02:37:37.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T02:37:37.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T02:37:37.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T02:37:37.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T02:37:37.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T02:37:37.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T02:37:37.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T02:37:37.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.8 flush_pg_stats 2024-09-21T02:37:37.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.9 flush_pg_stats 2024-09-21T02:37:37.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.10 flush_pg_stats 2024-09-21T02:37:37.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell osd.11 flush_pg_stats 2024-09-21T02:37:37.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:37 smithi190 ceph-mon[33871]: pgmap v340: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:37:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:37 smithi029 ceph-mon[31006]: pgmap v340: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:37:38.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:37 smithi066 ceph-mon[33436]: pgmap v340: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:37:38.026 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:38.137 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:38.144 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:38.147 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 -- 172.21.15.29:0/549534520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c1083d0 msgr2=0x7fc32c1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 --2- 172.21.15.29:0/549534520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c1083d0 0x7fc32c1087e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc320009900 tx=0x7fc32002a410 comp rx=0 tx=0).stop 2024-09-21T02:37:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 -- 172.21.15.29:0/549534520 shutdown_connections 2024-09-21T02:37:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 --2- 172.21.15.29:0/549534520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc32c108d20 0x7fc32c10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 --2- 172.21.15.29:0/549534520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c1083d0 0x7fc32c1087e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 --2- 172.21.15.29:0/549534520 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc32c102a90 0x7fc32c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 -- 172.21.15.29:0/549534520 >> 172.21.15.29:0/549534520 conn(0x7fc32c079750 msgr2=0x7fc32c079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 -- 172.21.15.29:0/549534520 shutdown_connections 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.687+0000 7fc334687640 1 -- 172.21.15.29:0/549534520 wait complete. 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc334687640 1 Processor -- start 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc334687640 1 -- start start 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc334687640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc32c102a90 0x7fc32c19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc334687640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc32c1083d0 0x7fc32c19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc334687640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c108d20 0x7fc32c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc334687640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c112840 con 0x7fc32c108d20 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc334687640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1126c0 con 0x7fc32c102a90 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc334687640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1129c0 con 0x7fc32c1083d0 2024-09-21T02:37:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.688+0000 7fc332c24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c108d20 0x7fc32c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332c24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c108d20 0x7fc32c19fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55176/0 (socket says 172.21.15.29:55176) 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332c24640 1 -- 172.21.15.29:0/3238965836 learned_addr learned my addr 172.21.15.29:0/3238965836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332423640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc32c102a90 0x7fc32c19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc331c22640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc32c1083d0 0x7fc32c19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332c24640 1 -- 172.21.15.29:0/3238965836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc32c1083d0 msgr2=0x7fc32c19a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332c24640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc32c1083d0 0x7fc32c19a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332c24640 1 -- 172.21.15.29:0/3238965836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc32c102a90 msgr2=0x7fc32c19a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332c24640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc32c102a90 0x7fc32c19a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332c24640 1 -- 172.21.15.29:0/3238965836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c1a0330 con 0x7fc32c108d20 2024-09-21T02:37:38.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc332c24640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c108d20 0x7fc32c19fc00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc32400e970 tx=0x7fc32400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:38.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32400ccc0 con 0x7fc32c108d20 2024-09-21T02:37:38.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc32400ce60 con 0x7fc32c108d20 2024-09-21T02:37:38.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c1a0620 con 0x7fc32c108d20 2024-09-21T02:37:38.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324010430 con 0x7fc32c108d20 2024-09-21T02:37:38.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.689+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c1a6d80 con 0x7fc32c108d20 2024-09-21T02:37:38.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.690+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc32c063b50 con 0x7fc32c108d20 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.692+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fc324004ef0 con 0x7fc32c108d20 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.693+0000 7fc32b7fe640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3140b65a0 0x7fc3140b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.693+0000 7fc332423640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3140b65a0 0x7fc3140b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.693+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9092+0+0 (secure 0 0 0) 0x7fc3240e8120 con 0x7fc32c108d20 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.693+0000 7fc32b7fe640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] conn(0x7fc3140c16b0 0x7fc3140c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.693+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 --> [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc324011e50 con 0x7fc3140c16b0 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.693+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fc3240e84d0 con 0x7fc32c108d20 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.693+0000 7fc332423640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3140b65a0 0x7fc3140b8ab0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc31c005fd0 tx=0x7fc31c009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.694+0000 7fc331c22640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] conn(0x7fc3140c16b0 0x7fc3140c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.694+0000 7fc331c22640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] conn(0x7fc3140c16b0 0x7fc3140c3ba0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:38.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.695+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 <== osd.2 v2:172.21.15.190:6800/334233995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc324011e50 con 0x7fc3140c16b0 2024-09-21T02:37:38.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.711+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 --> [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc32c063fd0 con 0x7fc3140c16b0 2024-09-21T02:37:38.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.712+0000 7fc32b7fe640 1 -- 172.21.15.29:0/3238965836 <== osd.2 v2:172.21.15.190:6800/334233995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc32c063fd0 con 0x7fc3140c16b0 2024-09-21T02:37:38.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 >> [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] conn(0x7fc3140c16b0 msgr2=0x7fc3140c3ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] conn(0x7fc3140c16b0 0x7fc3140c3ba0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3140b65a0 msgr2=0x7fc3140b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3140b65a0 0x7fc3140b8ab0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc31c005fd0 tx=0x7fc31c009520 comp rx=0 tx=0).stop 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c108d20 msgr2=0x7fc32c19fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 --2- 172.21.15.29:0/3238965836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc32c108d20 0x7fc32c19fc00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc32400e970 tx=0x7fc32400ed70 comp rx=0 tx=0).stop 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc332c24640 1 -- 172.21.15.29:0/3238965836 reap_dead start 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 shutdown_connections 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 >> 172.21.15.29:0/3238965836 conn(0x7fc32c079750 msgr2=0x7fc32c06bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 shutdown_connections 2024-09-21T02:37:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:38.713+0000 7fc334687640 1 -- 172.21.15.29:0/3238965836 wait complete. 2024-09-21T02:37:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]} : dispatch 2024-09-21T02:37:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]} : dispatch 2024-09-21T02:37:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]} : dispatch 2024-09-21T02:37:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]} : dispatch 2024-09-21T02:37:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]} : dispatch 2024-09-21T02:37:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]} : dispatch 2024-09-21T02:37:38.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 8]} : dispatch 2024-09-21T02:37:38.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]} : dispatch 2024-09-21T02:37:38.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]} : dispatch 2024-09-21T02:37:38.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]} : dispatch 2024-09-21T02:37:38.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]} : dispatch 2024-09-21T02:37:38.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]} : dispatch 2024-09-21T02:37:38.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]} : dispatch 2024-09-21T02:37:38.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]} : dispatch 2024-09-21T02:37:38.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]} : dispatch 2024-09-21T02:37:38.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]} : dispatch 2024-09-21T02:37:38.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 8]} : dispatch 2024-09-21T02:37:38.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]} : dispatch 2024-09-21T02:37:38.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]} : dispatch 2024-09-21T02:37:38.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 8]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]} : dispatch 2024-09-21T02:37:39.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]} : dispatch 2024-09-21T02:37:39.682 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:39.682 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 -- 172.21.15.29:0/393752456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 msgr2=0x7f344010f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 --2- 172.21.15.29:0/393752456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 0x7f344010f590 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3438009ca0 tx=0x7f343802a8e0 comp rx=0 tx=0).stop 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 -- 172.21.15.29:0/393752456 shutdown_connections 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 --2- 172.21.15.29:0/393752456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 0x7f344010f590 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 --2- 172.21.15.29:0/393752456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3440108410 0x7f3440108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 --2- 172.21.15.29:0/393752456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3440102b20 0x7f3440102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 -- 172.21.15.29:0/393752456 >> 172.21.15.29:0/393752456 conn(0x7f3440079700 msgr2=0x7f3440079b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 -- 172.21.15.29:0/393752456 shutdown_connections 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.776+0000 7f34464cd640 1 -- 172.21.15.29:0/393752456 wait complete. 2024-09-21T02:37:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f34464cd640 1 Processor -- start 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f34464cd640 1 -- start start 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f34464cd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3440102b20 0x7f344019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f34464cd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3440108410 0x7f344019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f34464cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 0x7f344019bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f34464cd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3440112930 con 0x7f3440108d60 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f34464cd640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f34401127b0 con 0x7f3440108410 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f34464cd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3440112ab0 con 0x7f3440102b20 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.777+0000 7f3444a6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 0x7f344019bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f3444a6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 0x7f344019bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55198/0 (socket says 172.21.15.29:55198) 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f3444a6a640 1 -- 172.21.15.29:0/3967118946 learned_addr learned my addr 172.21.15.29:0/3967118946 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f343ffff640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3440102b20 0x7f344019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f343f7fe640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3440108410 0x7f344019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f3444a6a640 1 -- 172.21.15.29:0/3967118946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3440102b20 msgr2=0x7f344019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f3444a6a640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3440102b20 0x7f344019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f3444a6a640 1 -- 172.21.15.29:0/3967118946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3440108410 msgr2=0x7f344019b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f3444a6a640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3440108410 0x7f344019b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f3444a6a640 1 -- 172.21.15.29:0/3967118946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34401a65d0 con 0x7f3440108d60 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f3444a6a640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 0x7f344019bdc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3438006010 tx=0x7f3438003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3438004240 con 0x7f3440108d60 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3438031cc0 con 0x7f3440108d60 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3438030be0 con 0x7f3440108d60 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34401a6860 con 0x7f3440108d60 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.778+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34401a6da0 con 0x7f3440108d60 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.779+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3440063ad0 con 0x7f3440108d60 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.781+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f3438030d80 con 0x7f3440108d60 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.782+0000 7f343d7fa640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34280b65d0 0x7f34280b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.782+0000 7f343ffff640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34280b65d0 0x7f34280b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.782+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f3438108bc0 con 0x7f3440108d60 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.782+0000 7f343d7fa640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] conn(0x7f34280c16e0 0x7f34280c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.782+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 --> [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34280c42a0 con 0x7f34280c16e0 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.782+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f3438109010 con 0x7f3440108d60 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.782+0000 7f343f7fe640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] conn(0x7f34280c16e0 0x7f34280c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.782+0000 7f343ffff640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34280b65d0 0x7f34280b8ae0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f344006c6c0 tx=0x7f3430005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.783+0000 7f343f7fe640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] conn(0x7f34280c16e0 0x7f34280c3bd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.783+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 <== osd.9 v2:172.21.15.29:6826/1846064827 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f34280c42a0 con 0x7f34280c16e0 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.797+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 --> [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3440063ed0 con 0x7f34280c16e0 2024-09-21T02:37:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.797+0000 7f343d7fa640 1 -- 172.21.15.29:0/3967118946 <== osd.9 v2:172.21.15.29:6826/1846064827 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3440063ed0 con 0x7f34280c16e0 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] conn(0x7f34280c16e0 msgr2=0x7f34280c3bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] conn(0x7f34280c16e0 0x7f34280c3bd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34280b65d0 msgr2=0x7f34280b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34280b65d0 0x7f34280b8ae0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f344006c6c0 tx=0x7f3430005f90 comp rx=0 tx=0).stop 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 msgr2=0x7f344019bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 --2- 172.21.15.29:0/3967118946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3440108d60 0x7f344019bdc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3438006010 tx=0x7f3438003c20 comp rx=0 tx=0).stop 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f3444a6a640 1 -- 172.21.15.29:0/3967118946 reap_dead start 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 shutdown_connections 2024-09-21T02:37:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 >> 172.21.15.29:0/3967118946 conn(0x7f3440079700 msgr2=0x7f34401044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:39.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 shutdown_connections 2024-09-21T02:37:39.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:39.798+0000 7f34464cd640 1 -- 172.21.15.29:0/3967118946 wait complete. 2024-09-21T02:37:39.812 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:39.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: pgmap v341: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 8]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-21T02:37:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:39 smithi190 ceph-mon[33871]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: pgmap v341: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 8]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-21T02:37:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:39 smithi029 ceph-mon[31006]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: pgmap v341: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 8]}]': finished 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-09-21T02:37:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-09-21T02:37:40.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-21T02:37:40.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:39 smithi066 ceph-mon[33436]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T02:37:40.532 INFO:teuthology.orchestra.run.smithi029.stdout:253403070503 2024-09-21T02:37:40.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.9 2024-09-21T02:37:40.758 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:40 smithi029 ceph-mon[31006]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T02:37:40.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:40 smithi190 ceph-mon[33871]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T02:37:41.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:40 smithi066 ceph-mon[33436]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T02:37:41.313 INFO:teuthology.orchestra.run.smithi029.stdout:77309411404 2024-09-21T02:37:41.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T02:37:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:41 smithi190 ceph-mon[33871]: pgmap v344: 129 pgs: 1 activating+degraded, 5 peering, 123 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 639 B/s wr, 11 op/s; 1/308 objects degraded (0.325%) 2024-09-21T02:37:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:41 smithi190 ceph-mon[33871]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-21T02:37:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:41 smithi190 ceph-mon[33871]: Health check failed: Degraded data redundancy: 1/308 objects degraded (0.325%), 1 pg degraded (PG_DEGRADED) 2024-09-21T02:37:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:41 smithi029 ceph-mon[31006]: pgmap v344: 129 pgs: 1 activating+degraded, 5 peering, 123 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 639 B/s wr, 11 op/s; 1/308 objects degraded (0.325%) 2024-09-21T02:37:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:41 smithi029 ceph-mon[31006]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-21T02:37:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:41 smithi029 ceph-mon[31006]: Health check failed: Degraded data redundancy: 1/308 objects degraded (0.325%), 1 pg degraded (PG_DEGRADED) 2024-09-21T02:37:42.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:41 smithi066 ceph-mon[33436]: pgmap v344: 129 pgs: 1 activating+degraded, 5 peering, 123 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 639 B/s wr, 11 op/s; 1/308 objects degraded (0.325%) 2024-09-21T02:37:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:41 smithi066 ceph-mon[33436]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-21T02:37:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:41 smithi066 ceph-mon[33436]: Health check failed: Degraded data redundancy: 1/308 objects degraded (0.325%), 1 pg degraded (PG_DEGRADED) 2024-09-21T02:37:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 -- 172.21.15.29:0/4132519332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 msgr2=0x7f4d8c10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 --2- 172.21.15.29:0/4132519332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 0x7f4d8c10f520 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009ca0 tx=0x7f4d8402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 -- 172.21.15.29:0/4132519332 shutdown_connections 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 --2- 172.21.15.29:0/4132519332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 0x7f4d8c10f520 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 --2- 172.21.15.29:0/4132519332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8c1083a0 0x7f4d8c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 --2- 172.21.15.29:0/4132519332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d8c102a60 0x7f4d8c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 -- 172.21.15.29:0/4132519332 >> 172.21.15.29:0/4132519332 conn(0x7f4d8c079730 msgr2=0x7f4d8c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 -- 172.21.15.29:0/4132519332 shutdown_connections 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.053+0000 7f4d91e8c640 1 -- 172.21.15.29:0/4132519332 wait complete. 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.054+0000 7f4d91e8c640 1 Processor -- start 2024-09-21T02:37:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.054+0000 7f4d91e8c640 1 -- start start 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d91e8c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8c102a60 0x7f4d8c199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d91e8c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d8c1083a0 0x7f4d8c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d91e8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 0x7f4d8c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d91e8c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c112860 con 0x7f4d8c108cf0 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d91e8c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c1126e0 con 0x7f4d8c1083a0 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d91e8c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c1129e0 con 0x7f4d8c102a60 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 0x7f4d8c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 0x7f4d8c19ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55254/0 (socket says 172.21.15.29:55254) 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8bfff640 1 -- 172.21.15.29:0/570800337 learned_addr learned my addr 172.21.15.29:0/570800337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8b7fe640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8c102a60 0x7f4d8c199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8affd640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d8c1083a0 0x7f4d8c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8bfff640 1 -- 172.21.15.29:0/570800337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8c102a60 msgr2=0x7f4d8c199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8bfff640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8c102a60 0x7f4d8c199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8bfff640 1 -- 172.21.15.29:0/570800337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d8c1083a0 msgr2=0x7f4d8c199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8bfff640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d8c1083a0 0x7f4d8c199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.055+0000 7f4d8bfff640 1 -- 172.21.15.29:0/570800337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8c19f200 con 0x7f4d8c108cf0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.056+0000 7f4d8bfff640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 0x7f4d8c19ead0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4d84005cc0 tx=0x7f4d84003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.056+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84032c10 con 0x7f4d8c108cf0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.056+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d84032db0 con 0x7f4d8c108cf0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.056+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8c19f490 con 0x7f4d8c108cf0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.056+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84030a60 con 0x7f4d8c108cf0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.056+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8c06ae60 con 0x7f4d8c108cf0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.057+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d8c06b3d0 con 0x7f4d8c108cf0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.058+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f4d84030c00 con 0x7f4d8c108cf0 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.059+0000 7f4d88ff9640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4d740b6620 0x7f4d740b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.059+0000 7f4d8b7fe640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4d740b6620 0x7f4d740b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.059+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f4d84108ab0 con 0x7f4d8c108cf0 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.059+0000 7f4d88ff9640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] conn(0x7f4d740c1730 0x7f4d740c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.059+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 --> [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d8403ee50 con 0x7f4d740c1730 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.059+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f4d84108f00 con 0x7f4d8c108cf0 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.060+0000 7f4d8b7fe640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4d740b6620 0x7f4d740b8b30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009bd0 tx=0x7f4d7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.060+0000 7f4d8affd640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] conn(0x7f4d740c1730 0x7f4d740c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.060+0000 7f4d8affd640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] conn(0x7f4d740c1730 0x7f4d740c3c20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.061+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 <== osd.4 v2:172.21.15.66:6808/3255192229 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d8403ee50 con 0x7f4d740c1730 2024-09-21T02:37:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.075+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 --> [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d8c06b7d0 con 0x7f4d740c1730 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.075+0000 7f4d88ff9640 1 -- 172.21.15.29:0/570800337 <== osd.4 v2:172.21.15.66:6808/3255192229 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4d8c06b7d0 con 0x7f4d740c1730 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 >> [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] conn(0x7f4d740c1730 msgr2=0x7f4d740c3c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] conn(0x7f4d740c1730 0x7f4d740c3c20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4d740b6620 msgr2=0x7f4d740b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4d740b6620 0x7f4d740b8b30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009bd0 tx=0x7f4d7c005f90 comp rx=0 tx=0).stop 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 msgr2=0x7f4d8c19ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 --2- 172.21.15.29:0/570800337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d8c108cf0 0x7f4d8c19ead0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4d84005cc0 tx=0x7f4d84003f90 comp rx=0 tx=0).stop 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d8bfff640 1 -- 172.21.15.29:0/570800337 reap_dead start 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 shutdown_connections 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 >> 172.21.15.29:0/570800337 conn(0x7f4d8c079730 msgr2=0x7f4d8c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 shutdown_connections 2024-09-21T02:37:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:42.076+0000 7f4d91e8c640 1 -- 172.21.15.29:0/570800337 wait complete. 2024-09-21T02:37:43.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 -- 172.21.15.29:0/3501768235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 msgr2=0x7ff51410d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 --2- 172.21.15.29:0/3501768235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 0x7ff51410d930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff504009900 tx=0x7ff50402a410 comp rx=0 tx=0).stop 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 -- 172.21.15.29:0/3501768235 shutdown_connections 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 --2- 172.21.15.29:0/3501768235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51410de70 0x7ff514110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 --2- 172.21.15.29:0/3501768235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 0x7ff51410d930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 --2- 172.21.15.29:0/3501768235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5141048c0 0x7ff514104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 -- 172.21.15.29:0/3501768235 >> 172.21.15.29:0/3501768235 conn(0x7ff514079760 msgr2=0x7ff514079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 -- 172.21.15.29:0/3501768235 shutdown_connections 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.028+0000 7ff51c165640 1 -- 172.21.15.29:0/3501768235 wait complete. 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.029+0000 7ff51c165640 1 Processor -- start 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.029+0000 7ff51c165640 1 -- start start 2024-09-21T02:37:43.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff51c165640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5141048c0 0x7ff5141027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:43.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff51c165640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 0x7ff514102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:43.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff51c165640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51410de70 0x7ff514103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:43.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff51c165640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff514113960 con 0x7ff5141052a0 2024-09-21T02:37:43.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff51c165640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff5141137e0 con 0x7ff5141048c0 2024-09-21T02:37:43.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff51c165640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff514113ae0 con 0x7ff51410de70 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519700640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 0x7ff514102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519700640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 0x7ff514102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55276/0 (socket says 172.21.15.29:55276) 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519700640 1 -- 172.21.15.29:0/2416838234 learned_addr learned my addr 172.21.15.29:0/2416838234 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff51a702640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51410de70 0x7ff514103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519f01640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5141048c0 0x7ff5141027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519700640 1 -- 172.21.15.29:0/2416838234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51410de70 msgr2=0x7ff514103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519700640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51410de70 0x7ff514103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519700640 1 -- 172.21.15.29:0/2416838234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5141048c0 msgr2=0x7ff5141027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519700640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5141048c0 0x7ff5141027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.030+0000 7ff519700640 1 -- 172.21.15.29:0/2416838234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5141a8690 con 0x7ff5141052a0 2024-09-21T02:37:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.031+0000 7ff519700640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 0x7ff514102d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff504005940 tx=0x7ff5040040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:43.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.031+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50403f030 con 0x7ff5141052a0 2024-09-21T02:37:43.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.031+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff504003a80 con 0x7ff5141052a0 2024-09-21T02:37:43.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.031+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5141a8920 con 0x7ff5141052a0 2024-09-21T02:37:43.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.031+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504035070 con 0x7ff5141052a0 2024-09-21T02:37:43.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.031+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5141a8e50 con 0x7ff5141052a0 2024-09-21T02:37:43.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.032+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff514063b50 con 0x7ff5141052a0 2024-09-21T02:37:43.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.034+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7ff504042030 con 0x7ff5141052a0 2024-09-21T02:37:43.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.034+0000 7ff50affd640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5000b6620 0x7ff5000b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.035+0000 7ff519f01640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5000b6620 0x7ff5000b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.035+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7ff504108060 con 0x7ff5141052a0 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.035+0000 7ff50affd640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] conn(0x7ff5000c1730 0x7ff5000c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.035+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 --> [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5000c42f0 con 0x7ff5000c1730 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.035+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7ff5041084b0 con 0x7ff5141052a0 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.035+0000 7ff519f01640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5000b6620 0x7ff5000b8b30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff514104220 tx=0x7ff51000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.035+0000 7ff51a702640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] conn(0x7ff5000c1730 0x7ff5000c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.036+0000 7ff51a702640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] conn(0x7ff5000c1730 0x7ff5000c3c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:43.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.037+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 <== osd.1 v2:172.21.15.66:6800/3928353915 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5000c42f0 con 0x7ff5000c1730 2024-09-21T02:37:43.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.050+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 --> [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff514063f50 con 0x7ff5000c1730 2024-09-21T02:37:43.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff50affd640 1 -- 172.21.15.29:0/2416838234 <== osd.1 v2:172.21.15.66:6800/3928353915 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff514063f50 con 0x7ff5000c1730 2024-09-21T02:37:43.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 >> [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] conn(0x7ff5000c1730 msgr2=0x7ff5000c3c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:43.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51c165640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] conn(0x7ff5000c1730 0x7ff5000c3c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:43.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5000b6620 msgr2=0x7ff5000b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:43.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51c165640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5000b6620 0x7ff5000b8b30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff514104220 tx=0x7ff51000b390 comp rx=0 tx=0).stop 2024-09-21T02:37:43.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 msgr2=0x7ff514102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:43.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51c165640 1 --2- 172.21.15.29:0/2416838234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5141052a0 0x7ff514102d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff504005940 tx=0x7ff5040040e0 comp rx=0 tx=0).stop 2024-09-21T02:37:43.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51a702640 1 -- 172.21.15.29:0/2416838234 reap_dead start 2024-09-21T02:37:43.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 shutdown_connections 2024-09-21T02:37:43.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.051+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 >> 172.21.15.29:0/2416838234 conn(0x7ff514079760 msgr2=0x7ff5141009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:43.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.052+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 shutdown_connections 2024-09-21T02:37:43.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:43.052+0000 7ff51c165640 1 -- 172.21.15.29:0/2416838234 wait complete. 2024-09-21T02:37:43.814 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:43 smithi029 ceph-mon[31006]: pgmap v345: 129 pgs: 1 activating+degraded, 5 peering, 123 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 1/308 objects degraded (0.325%); 0 B/s, 0 objects/s recovering 2024-09-21T02:37:43.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:43 smithi190 ceph-mon[33871]: pgmap v345: 129 pgs: 1 activating+degraded, 5 peering, 123 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 1/308 objects degraded (0.325%); 0 B/s, 0 objects/s recovering 2024-09-21T02:37:44.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:43 smithi066 ceph-mon[33436]: pgmap v345: 129 pgs: 1 activating+degraded, 5 peering, 123 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 1/308 objects degraded (0.325%); 0 B/s, 0 objects/s recovering 2024-09-21T02:37:44.837 INFO:teuthology.orchestra.run.smithi029.stdout:55834574929 2024-09-21T02:37:44.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T02:37:45.267 INFO:teuthology.orchestra.run.smithi029.stdout:128849018945 2024-09-21T02:37:45.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T02:37:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 -- 172.21.15.29:0/3838154156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e60108d00 msgr2=0x7f4e6010f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 --2- 172.21.15.29:0/3838154156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e60108d00 0x7f4e6010f530 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009ca0 tx=0x7f4e5802a8e0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 -- 172.21.15.29:0/3838154156 shutdown_connections 2024-09-21T02:37:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 --2- 172.21.15.29:0/3838154156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e60108d00 0x7f4e6010f530 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 --2- 172.21.15.29:0/3838154156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e601083b0 0x7f4e601087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 --2- 172.21.15.29:0/3838154156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e60103140 0x7f4e60103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 -- 172.21.15.29:0/3838154156 >> 172.21.15.29:0/3838154156 conn(0x7f4e60077c60 msgr2=0x7f4e600780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 -- 172.21.15.29:0/3838154156 shutdown_connections 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.925+0000 7f4e657f7640 1 -- 172.21.15.29:0/3838154156 wait complete. 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.926+0000 7f4e657f7640 1 Processor -- start 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.926+0000 7f4e657f7640 1 -- start start 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.926+0000 7f4e657f7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e60103140 0x7f4e6019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.926+0000 7f4e657f7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e601083b0 0x7f4e6019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.926+0000 7f4e657f7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e60108d00 0x7f4e6019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.926+0000 7f4e657f7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4e601139b0 con 0x7f4e601083b0 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.926+0000 7f4e657f7640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4e60113830 con 0x7f4e60108d00 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.926+0000 7f4e657f7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4e60113b30 con 0x7f4e60103140 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e601083b0 0x7f4e6019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e601083b0 0x7f4e6019a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55230/0 (socket says 172.21.15.29:55230) 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 -- 172.21.15.29:0/787970338 learned_addr learned my addr 172.21.15.29:0/787970338 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5effd640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e60103140 0x7f4e6019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5f7fe640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e60108d00 0x7f4e6019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 -- 172.21.15.29:0/787970338 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e60103140 msgr2=0x7f4e6019a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e60103140 0x7f4e6019a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 -- 172.21.15.29:0/787970338 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e60108d00 msgr2=0x7f4e6019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e60108d00 0x7f4e6019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 -- 172.21.15.29:0/787970338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e601a02f0 con 0x7f4e601083b0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e5e7fc640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e601083b0 0x7f4e6019a920 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4e54006fd0 tx=0x7f4e5400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e54011830 con 0x7f4e601083b0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e601a05e0 con 0x7f4e601083b0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e54016070 con 0x7f4e601083b0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e54010d30 con 0x7f4e601083b0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.927+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e601a6d40 con 0x7f4e601083b0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.928+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4e60063b10 con 0x7f4e601083b0 2024-09-21T02:37:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.930+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f4e540119d0 con 0x7f4e601083b0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.931+0000 7f4e47fff640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4e480b65a0 0x7f4e480b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.931+0000 7f4e5effd640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4e480b65a0 0x7f4e480b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.931+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f4e540e78a0 con 0x7f4e601083b0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.931+0000 7f4e47fff640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] conn(0x7f4e480c16b0 0x7f4e480c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.931+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 --> [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e54022e50 con 0x7f4e480c16b0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.931+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f4e540e7c50 con 0x7f4e601083b0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.931+0000 7f4e5f7fe640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] conn(0x7f4e480c16b0 0x7f4e480c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.932+0000 7f4e5effd640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4e480b65a0 0x7f4e480b8ab0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4e500058c0 tx=0x7f4e5000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.932+0000 7f4e5f7fe640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] conn(0x7f4e480c16b0 0x7f4e480c3ba0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.933+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 <== osd.0 v2:172.21.15.29:6802/3066733118 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4e54022e50 con 0x7f4e480c16b0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.946+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 --> [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4e60063f90 con 0x7f4e480c16b0 2024-09-21T02:37:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e47fff640 1 -- 172.21.15.29:0/787970338 <== osd.0 v2:172.21.15.29:6802/3066733118 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4e60063f90 con 0x7f4e480c16b0 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] conn(0x7f4e480c16b0 msgr2=0x7f4e480c3ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] conn(0x7f4e480c16b0 0x7f4e480c3ba0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4e480b65a0 msgr2=0x7f4e480b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4e480b65a0 0x7f4e480b8ab0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4e500058c0 tx=0x7f4e5000a2d0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e601083b0 msgr2=0x7f4e6019a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 --2- 172.21.15.29:0/787970338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e601083b0 0x7f4e6019a920 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4e54006fd0 tx=0x7f4e5400b8e0 comp rx=0 tx=0).stop 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e5f7fe640 1 -- 172.21.15.29:0/787970338 reap_dead start 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 shutdown_connections 2024-09-21T02:37:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 >> 172.21.15.29:0/787970338 conn(0x7f4e60077c60 msgr2=0x7f4e60106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:45.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 shutdown_connections 2024-09-21T02:37:45.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:41.947+0000 7f4e657f7640 1 -- 172.21.15.29:0/787970338 wait complete. 2024-09-21T02:37:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:45 smithi190 ceph-mon[33871]: pgmap v346: 129 pgs: 1 activating+degraded, 4 peering, 124 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 1/308 objects degraded (0.325%); 0 B/s, 0 objects/s recovering 2024-09-21T02:37:45.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:45 smithi029 ceph-mon[31006]: pgmap v346: 129 pgs: 1 activating+degraded, 4 peering, 124 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 1/308 objects degraded (0.325%); 0 B/s, 0 objects/s recovering 2024-09-21T02:37:46.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:45 smithi066 ceph-mon[33436]: pgmap v346: 129 pgs: 1 activating+degraded, 4 peering, 124 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 1/308 objects degraded (0.325%); 0 B/s, 0 objects/s recovering 2024-09-21T02:37:46.098 INFO:teuthology.orchestra.run.smithi029.stdout:34359738455 2024-09-21T02:37:46.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T02:37:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.031+0000 7fcca6f12640 1 -- 172.21.15.29:0/1572002755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca0103b60 msgr2=0x7fcca0103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.031+0000 7fcca6f12640 1 --2- 172.21.15.29:0/1572002755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca0103b60 0x7fcca0103f70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcc94009900 tx=0x7fcc9402a410 comp rx=0 tx=0).stop 2024-09-21T02:37:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.031+0000 7fcca6f12640 1 -- 172.21.15.29:0/1572002755 shutdown_connections 2024-09-21T02:37:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.031+0000 7fcca6f12640 1 --2- 172.21.15.29:0/1572002755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcca0109df0 0x7fcca0110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.031+0000 7fcca6f12640 1 --2- 172.21.15.29:0/1572002755 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcca01094a0 0x7fcca01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.031+0000 7fcca6f12640 1 --2- 172.21.15.29:0/1572002755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca0103b60 0x7fcca0103f70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.031+0000 7fcca6f12640 1 -- 172.21.15.29:0/1572002755 >> 172.21.15.29:0/1572002755 conn(0x7fcca007b850 msgr2=0x7fcca007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.032+0000 7fcca6f12640 1 -- 172.21.15.29:0/1572002755 shutdown_connections 2024-09-21T02:37:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.032+0000 7fcca6f12640 1 -- 172.21.15.29:0/1572002755 wait complete. 2024-09-21T02:37:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.032+0000 7fcca6f12640 1 Processor -- start 2024-09-21T02:37:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.032+0000 7fcca6f12640 1 -- start start 2024-09-21T02:37:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.032+0000 7fcca6f12640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcca0103b60 0x7fcca019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.032+0000 7fcca6f12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca01094a0 0x7fcca019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca6f12640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcca0109df0 0x7fcca019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca6f12640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcca0113960 con 0x7fcca01094a0 2024-09-21T02:37:46.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca6f12640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcca01137e0 con 0x7fcca0109df0 2024-09-21T02:37:46.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca6f12640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcca0113ae0 con 0x7fcca0103b60 2024-09-21T02:37:46.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca01094a0 0x7fcca019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca01094a0 0x7fcca019a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55296/0 (socket says 172.21.15.29:55296) 2024-09-21T02:37:46.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 -- 172.21.15.29:0/1395054666 learned_addr learned my addr 172.21.15.29:0/1395054666 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:46.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca4cae640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcca0103b60 0x7fcca019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca54af640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcca0109df0 0x7fcca019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 -- 172.21.15.29:0/1395054666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcca0103b60 msgr2=0x7fcca019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcca0103b60 0x7fcca019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 -- 172.21.15.29:0/1395054666 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcca0109df0 msgr2=0x7fcca019fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcca0109df0 0x7fcca019fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 -- 172.21.15.29:0/1395054666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca01a0380 con 0x7fcca01094a0 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca4cae640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcca0103b60 0x7fcca019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca54af640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcca0109df0 0x7fcca019fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9ffff640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca01094a0 0x7fcca019a9b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00c970 tx=0x7fcc8c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c004220 con 0x7fcca01094a0 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca01a0670 con 0x7fcca01094a0 2024-09-21T02:37:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc8c007e40 con 0x7fcca01094a0 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.033+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c0047a0 con 0x7fcca01094a0 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.034+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca01a6dd0 con 0x7fcca01094a0 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.035+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcca0063b10 con 0x7fcca01094a0 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.036+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fcc8c0079b0 con 0x7fcca01094a0 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.037+0000 7fcc9dffb640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcc900b65a0 0x7fcc900b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.037+0000 7fcca4cae640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcc900b65a0 0x7fcc900b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.037+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fcc8c0e79d0 con 0x7fcca01094a0 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.037+0000 7fcc9dffb640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] conn(0x7fcc900c16b0 0x7fcc900c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.037+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 --> [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc900c4270 con 0x7fcc900c16b0 2024-09-21T02:37:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.037+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fcc8c0e7d80 con 0x7fcca01094a0 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.037+0000 7fcca54af640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] conn(0x7fcc900c16b0 0x7fcc900c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.038+0000 7fcca4cae640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcc900b65a0 0x7fcc900b8ab0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcc94006010 tx=0x7fcc94000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.038+0000 7fcca54af640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] conn(0x7fcc900c16b0 0x7fcc900c3ba0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.039+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 <== osd.6 v2:172.21.15.29:6818/2439227427 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc900c4270 con 0x7fcc900c16b0 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.052+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 --> [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcca0063f90 con 0x7fcc900c16b0 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcc9dffb640 1 -- 172.21.15.29:0/1395054666 <== osd.6 v2:172.21.15.29:6818/2439227427 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcca0063f90 con 0x7fcc900c16b0 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] conn(0x7fcc900c16b0 msgr2=0x7fcc900c3ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] conn(0x7fcc900c16b0 0x7fcc900c3ba0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcc900b65a0 msgr2=0x7fcc900b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcc900b65a0 0x7fcc900b8ab0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcc94006010 tx=0x7fcc94000bc0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca01094a0 msgr2=0x7fcca019a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 --2- 172.21.15.29:0/1395054666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcca01094a0 0x7fcca019a9b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00c970 tx=0x7fcc8c00ce30 comp rx=0 tx=0).stop 2024-09-21T02:37:46.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca54af640 1 -- 172.21.15.29:0/1395054666 reap_dead start 2024-09-21T02:37:46.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 shutdown_connections 2024-09-21T02:37:46.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 >> 172.21.15.29:0/1395054666 conn(0x7fcca007b850 msgr2=0x7fcca01075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:46.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 shutdown_connections 2024-09-21T02:37:46.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.053+0000 7fcca6f12640 1 -- 172.21.15.29:0/1395054666 wait complete. 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 -- 172.21.15.29:0/1990722321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c41094a0 msgr2=0x7f16c41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 --2- 172.21.15.29:0/1990722321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c41094a0 0x7f16c41098b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009ca0 tx=0x7f16b802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 -- 172.21.15.29:0/1990722321 shutdown_connections 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 --2- 172.21.15.29:0/1990722321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16c4109df0 0x7f16c4110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 --2- 172.21.15.29:0/1990722321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c41094a0 0x7f16c41098b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 --2- 172.21.15.29:0/1990722321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c4103b60 0x7f16c4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 -- 172.21.15.29:0/1990722321 >> 172.21.15.29:0/1990722321 conn(0x7f16c407b850 msgr2=0x7f16c407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 -- 172.21.15.29:0/1990722321 shutdown_connections 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.602+0000 7f16c8d8b640 1 -- 172.21.15.29:0/1990722321 wait complete. 2024-09-21T02:37:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.603+0000 7f16c8d8b640 1 Processor -- start 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.603+0000 7f16c8d8b640 1 -- start start 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.603+0000 7f16c8d8b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c4103b60 0x7f16c419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.603+0000 7f16c8d8b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16c41094a0 0x7f16c419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.603+0000 7f16c8d8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c4109df0 0x7f16c419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.603+0000 7f16c8d8b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f16c41139b0 con 0x7f16c4109df0 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.603+0000 7f16c8d8b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4113830 con 0x7f16c4103b60 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.603+0000 7f16c8d8b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4113b30 con 0x7f16c41094a0 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c4109df0 0x7f16c419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c4109df0 0x7f16c419fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39020/0 (socket says 172.21.15.29:39020) 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 -- 172.21.15.29:0/438078377 learned_addr learned my addr 172.21.15.29:0/438078377 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c259c640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c4103b60 0x7f16c419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c1d9b640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16c41094a0 0x7f16c419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 -- 172.21.15.29:0/438078377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16c41094a0 msgr2=0x7f16c419a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16c41094a0 0x7f16c419a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 -- 172.21.15.29:0/438078377 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c4103b60 msgr2=0x7f16c419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c4103b60 0x7f16c419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 -- 172.21.15.29:0/438078377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c41a0270 con 0x7f16c4109df0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c2d9d640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c4109df0 0x7f16c419fc00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00d750 tx=0x7f16bc00db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc017070 con 0x7f16c4109df0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c41a0560 con 0x7f16c4109df0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16bc0097c0 con 0x7f16c4109df0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc004b20 con 0x7f16c4109df0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.604+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c41a6d80 con 0x7f16c4109df0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.605+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f16c4063b50 con 0x7f16c4109df0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.607+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f16bc004cc0 con 0x7f16c4109df0 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.608+0000 7f16b37fe640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ac0b65d0 0x7f16ac0b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.608+0000 7f16c259c640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ac0b65d0 0x7f16ac0b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.608+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f16bc0e7550 con 0x7f16c4109df0 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.608+0000 7f16b37fe640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] conn(0x7f16ac0c16e0 0x7f16ac0c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.608+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 --> [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16bc021e50 con 0x7f16ac0c16e0 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.608+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f16bc0e79a0 con 0x7f16c4109df0 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.608+0000 7f16c259c640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ac0b65d0 0x7f16ac0b8ae0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f16b400abe0 tx=0x7f16b400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.609+0000 7f16c1d9b640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] conn(0x7f16ac0c16e0 0x7f16ac0c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.609+0000 7f16c1d9b640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] conn(0x7f16ac0c16e0 0x7f16ac0c3bd0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.610+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 <== osd.8 v2:172.21.15.190:6816/2392824828 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16bc021e50 con 0x7f16ac0c16e0 2024-09-21T02:37:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.626+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 --> [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f16c4063f50 con 0x7f16ac0c16e0 2024-09-21T02:37:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.627+0000 7f16b37fe640 1 -- 172.21.15.29:0/438078377 <== osd.8 v2:172.21.15.190:6816/2392824828 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f16c4063f50 con 0x7f16ac0c16e0 2024-09-21T02:37:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.627+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 >> [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] conn(0x7f16ac0c16e0 msgr2=0x7f16ac0c3bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.627+0000 7f16c8d8b640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] conn(0x7f16ac0c16e0 0x7f16ac0c3bd0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.627+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ac0b65d0 msgr2=0x7f16ac0b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.627+0000 7f16c8d8b640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ac0b65d0 0x7f16ac0b8ae0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f16b400abe0 tx=0x7f16b400a2d0 comp rx=0 tx=0).stop 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.628+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c4109df0 msgr2=0x7f16c419fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.628+0000 7f16c8d8b640 1 --2- 172.21.15.29:0/438078377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16c4109df0 0x7f16c419fc00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00d750 tx=0x7f16bc00db50 comp rx=0 tx=0).stop 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.628+0000 7f16c2d9d640 1 -- 172.21.15.29:0/438078377 reap_dead start 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.628+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 shutdown_connections 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.628+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 >> 172.21.15.29:0/438078377 conn(0x7f16c407b850 msgr2=0x7f16c4107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.628+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 shutdown_connections 2024-09-21T02:37:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:46.628+0000 7f16c8d8b640 1 -- 172.21.15.29:0/438078377 wait complete. 2024-09-21T02:37:46.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:46 smithi029 ceph-mon[31006]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T02:37:46.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:46 smithi029 ceph-mon[31006]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/308 objects degraded (0.325%), 1 pg degraded) 2024-09-21T02:37:46.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:46 smithi029 ceph-mon[31006]: Cluster is now healthy 2024-09-21T02:37:47.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:46 smithi066 ceph-mon[33436]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T02:37:47.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:46 smithi066 ceph-mon[33436]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/308 objects degraded (0.325%), 1 pg degraded) 2024-09-21T02:37:47.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:46 smithi066 ceph-mon[33436]: Cluster is now healthy 2024-09-21T02:37:47.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:46 smithi190 ceph-mon[33871]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T02:37:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:46 smithi190 ceph-mon[33871]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/308 objects degraded (0.325%), 1 pg degraded) 2024-09-21T02:37:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:46 smithi190 ceph-mon[33871]: Cluster is now healthy 2024-09-21T02:37:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:47.394 INFO:teuthology.orchestra.run.smithi029.stdout:176093659192 2024-09-21T02:37:47.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T02:37:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:47 smithi029 ceph-mon[31006]: pgmap v347: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s; 28 B/s, 1 objects/s recovering 2024-09-21T02:37:48.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:47 smithi066 ceph-mon[33436]: pgmap v347: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s; 28 B/s, 1 objects/s recovering 2024-09-21T02:37:48.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.152+0000 7f88e2e08640 1 -- 172.21.15.29:0/2545585907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc105b70 msgr2=0x7f88dc10f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:48.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.152+0000 7f88e2e08640 1 --2- 172.21.15.29:0/2545585907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc105b70 0x7f88dc10f600 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f88d4009ca0 tx=0x7f88d402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:48.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 -- 172.21.15.29:0/2545585907 shutdown_connections 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 --2- 172.21.15.29:0/2545585907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc105b70 0x7f88dc10f600 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 --2- 172.21.15.29:0/2545585907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc105220 0x7f88dc105630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 --2- 172.21.15.29:0/2545585907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88dc103250 0x7f88dc103660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 -- 172.21.15.29:0/2545585907 >> 172.21.15.29:0/2545585907 conn(0x7f88dc079ea0 msgr2=0x7f88dc07a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 -- 172.21.15.29:0/2545585907 shutdown_connections 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 -- 172.21.15.29:0/2545585907 wait complete. 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 Processor -- start 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.153+0000 7f88e2e08640 1 -- start start 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e2e08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc103250 0x7f88dc199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e2e08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc105220 0x7f88dc1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e2e08640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88dc105b70 0x7f88dc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e2e08640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc112910 con 0x7f88dc103250 2024-09-21T02:37:48.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e2e08640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc112790 con 0x7f88dc105b70 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e2e08640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc112a90 con 0x7f88dc105220 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc103250 0x7f88dc199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc103250 0x7f88dc199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38998/0 (socket says 172.21.15.29:38998) 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 -- 172.21.15.29:0/1885099094 learned_addr learned my addr 172.21.15.29:0/1885099094 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88dbfff640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc105220 0x7f88dc1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e13a5640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88dc105b70 0x7f88dc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 -- 172.21.15.29:0/1885099094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc105220 msgr2=0x7f88dc1998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc105220 0x7f88dc1998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 -- 172.21.15.29:0/1885099094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88dc105b70 msgr2=0x7f88dc19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88dc105b70 0x7f88dc19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 -- 172.21.15.29:0/1885099094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc19f290 con 0x7f88dc103250 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.154+0000 7f88e0ba4640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc103250 0x7f88dc199320 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f88d000e9c0 tx=0x7f88d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:48.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.155+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d000cd80 con 0x7f88dc103250 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.155+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88dc19f580 con 0x7f88dc103250 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.155+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f88d00044b0 con 0x7f88dc103250 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.155+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0010430 con 0x7f88dc103250 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.155+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88dc1a5ce0 con 0x7f88dc103250 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.156+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88dc063ad0 con 0x7f88dc103250 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.157+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f88d0003f00 con 0x7f88dc103250 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.158+0000 7f88d9ffb640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88cc0b65a0 0x7f88cc0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.158+0000 7f88dbfff640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88cc0b65a0 0x7f88cc0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:48.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.158+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f88d00e7f00 con 0x7f88dc103250 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.158+0000 7f88d9ffb640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] conn(0x7f88cc0c16b0 0x7f88cc0c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.158+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 --> [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88cc0c4270 con 0x7f88cc0c16b0 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.158+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f88d00e8350 con 0x7f88dc103250 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.159+0000 7f88e13a5640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] conn(0x7f88cc0c16b0 0x7f88cc0c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.159+0000 7f88dbfff640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88cc0b65a0 0x7f88cc0b8ab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f88dc109390 tx=0x7f88c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.159+0000 7f88e13a5640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] conn(0x7f88cc0c16b0 0x7f88cc0c3ba0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.160+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 <== osd.3 v2:172.21.15.29:6810/1263922693 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88cc0c4270 con 0x7f88cc0c16b0 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.173+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 --> [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f88dc063ed0 con 0x7f88cc0c16b0 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88d9ffb640 1 -- 172.21.15.29:0/1885099094 <== osd.3 v2:172.21.15.29:6810/1263922693 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f88dc063ed0 con 0x7f88cc0c16b0 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] conn(0x7f88cc0c16b0 msgr2=0x7f88cc0c3ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] conn(0x7f88cc0c16b0 0x7f88cc0c3ba0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88cc0b65a0 msgr2=0x7f88cc0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:48.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88cc0b65a0 0x7f88cc0b8ab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f88dc109390 tx=0x7f88c8005f90 comp rx=0 tx=0).stop 2024-09-21T02:37:48.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc103250 msgr2=0x7f88dc199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:48.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 --2- 172.21.15.29:0/1885099094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88dc103250 0x7f88dc199320 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f88d000e9c0 tx=0x7f88d000ee80 comp rx=0 tx=0).stop 2024-09-21T02:37:48.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e13a5640 1 -- 172.21.15.29:0/1885099094 reap_dead start 2024-09-21T02:37:48.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 shutdown_connections 2024-09-21T02:37:48.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 >> 172.21.15.29:0/1885099094 conn(0x7f88dc079ea0 msgr2=0x7f88dc106470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:48.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 shutdown_connections 2024-09-21T02:37:48.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:44.174+0000 7f88e2e08640 1 -- 172.21.15.29:0/1885099094 wait complete. 2024-09-21T02:37:48.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:47 smithi190 ceph-mon[33871]: pgmap v347: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s; 28 B/s, 1 objects/s recovering 2024-09-21T02:37:48.966 INFO:teuthology.orchestra.run.smithi029.stdout:103079215174 2024-09-21T02:37:48.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.299+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3252353269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 msgr2=0x7f15b810f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.299+0000 7f15bfa7c640 1 --2- 172.21.15.29:0/3252353269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 0x7f15b810f550 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009ca0 tx=0x7f15b002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3252353269 shutdown_connections 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 --2- 172.21.15.29:0/3252353269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 0x7f15b810f550 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 --2- 172.21.15.29:0/3252353269 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b81083d0 0x7f15b81087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 --2- 172.21.15.29:0/3252353269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15b8102ae0 0x7f15b8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3252353269 >> 172.21.15.29:0/3252353269 conn(0x7f15b8079760 msgr2=0x7f15b8079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3252353269 shutdown_connections 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3252353269 wait complete. 2024-09-21T02:37:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 Processor -- start 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.300+0000 7f15bfa7c640 1 -- start start 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bfa7c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15b8102ae0 0x7f15b819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bfa7c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b81083d0 0x7f15b819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bfa7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 0x7f15b819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bfa7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f15b8112890 con 0x7f15b8108d20 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bfa7c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f15b8112710 con 0x7f15b81083d0 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bfa7c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f15b8112a10 con 0x7f15b8102ae0 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 0x7f15b819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 0x7f15b819fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39036/0 (socket says 172.21.15.29:39036) 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 -- 172.21.15.29:0/3831247142 learned_addr learned my addr 172.21.15.29:0/3831247142 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bd017640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b81083d0 0x7f15b819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bd818640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15b8102ae0 0x7f15b819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 -- 172.21.15.29:0/3831247142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15b8102ae0 msgr2=0x7f15b819a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15b8102ae0 0x7f15b819a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 -- 172.21.15.29:0/3831247142 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b81083d0 msgr2=0x7f15b819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b81083d0 0x7f15b819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 -- 172.21.15.29:0/3831247142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b81a0220 con 0x7f15b8108d20 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15bd017640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b81083d0 0x7f15b819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.301+0000 7f15be019640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 0x7f15b819fbb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f15b0005cc0 tx=0x7f15b0032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.302+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0032e00 con 0x7f15b8108d20 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.302+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15b00044d0 con 0x7f15b8108d20 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.302+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b81a04b0 con 0x7f15b8108d20 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.302+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0030b70 con 0x7f15b8108d20 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.302+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b81a09f0 con 0x7f15b8108d20 2024-09-21T02:37:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.302+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f15b8063b50 con 0x7f15b8108d20 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.304+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f15b0030d10 con 0x7f15b8108d20 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.305+0000 7f15aeffd640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15a40b65d0 0x7f15a40b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.305+0000 7f15bd818640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15a40b65d0 0x7f15a40b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.305+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f15b0108f10 con 0x7f15b8108d20 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.305+0000 7f15aeffd640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] conn(0x7f15a40c16e0 0x7f15a40c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.305+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 --> [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15b0004320 con 0x7f15a40c16e0 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.305+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f15b01092c0 con 0x7f15b8108d20 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.306+0000 7f15bd818640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15a40b65d0 0x7f15a40b8ae0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f15b4005e10 tx=0x7f15b4005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.306+0000 7f15bd017640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] conn(0x7f15a40c16e0 0x7f15a40c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.306+0000 7f15bd017640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] conn(0x7f15a40c16e0 0x7f15a40c3bd0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.307+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 <== osd.11 v2:172.21.15.190:6824/2545559018 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15b0004320 con 0x7f15a40c16e0 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.321+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 --> [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f15b8063f50 con 0x7f15a40c16e0 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.321+0000 7f15aeffd640 1 -- 172.21.15.29:0/3831247142 <== osd.11 v2:172.21.15.190:6824/2545559018 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f15b8063f50 con 0x7f15a40c16e0 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 >> [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] conn(0x7f15a40c16e0 msgr2=0x7f15a40c3bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] conn(0x7f15a40c16e0 0x7f15a40c3bd0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15a40b65d0 msgr2=0x7f15a40b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15a40b65d0 0x7f15a40b8ae0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f15b4005e10 tx=0x7f15b4005da0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 msgr2=0x7f15b819fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 --2- 172.21.15.29:0/3831247142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b8108d20 0x7f15b819fbb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f15b0005cc0 tx=0x7f15b0032a10 comp rx=0 tx=0).stop 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15be019640 1 -- 172.21.15.29:0/3831247142 reap_dead start 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 shutdown_connections 2024-09-21T02:37:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 >> 172.21.15.29:0/3831247142 conn(0x7f15b8079760 msgr2=0x7f15b8101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 shutdown_connections 2024-09-21T02:37:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:48.322+0000 7f15bfa7c640 1 -- 172.21.15.29:0/3831247142 wait complete. 2024-09-21T02:37:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 -- 172.21.15.29:0/3857685474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c1083d0 msgr2=0x7f4f0c1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 --2- 172.21.15.29:0/3857685474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c1083d0 0x7f4f0c1087e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009900 tx=0x7f4f0002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 -- 172.21.15.29:0/3857685474 shutdown_connections 2024-09-21T02:37:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 --2- 172.21.15.29:0/3857685474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f0c108d20 0x7f4f0c10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 --2- 172.21.15.29:0/3857685474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c1083d0 0x7f4f0c1087e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 --2- 172.21.15.29:0/3857685474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f0c102ae0 0x7f4f0c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 -- 172.21.15.29:0/3857685474 >> 172.21.15.29:0/3857685474 conn(0x7f4f0c079760 msgr2=0x7f4f0c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 -- 172.21.15.29:0/3857685474 shutdown_connections 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.516+0000 7f4f13872640 1 -- 172.21.15.29:0/3857685474 wait complete. 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f13872640 1 Processor -- start 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f13872640 1 -- start start 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f13872640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c102ae0 0x7f4f0c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f13872640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f0c1083d0 0x7f4f0c19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f13872640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f0c108d20 0x7f4f0c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f13872640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c113960 con 0x7f4f0c102ae0 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f13872640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c1137e0 con 0x7f4f0c108d20 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f13872640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c113ae0 con 0x7f4f0c1083d0 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f1160e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c102ae0 0x7f4f0c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f1160e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c102ae0 0x7f4f0c19a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39062/0 (socket says 172.21.15.29:39062) 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.517+0000 7f4f1160e640 1 -- 172.21.15.29:0/847886040 learned_addr learned my addr 172.21.15.29:0/847886040 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f11e0f640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f0c108d20 0x7f4f0c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f10e0d640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f0c1083d0 0x7f4f0c19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f1160e640 1 -- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f0c1083d0 msgr2=0x7f4f0c19a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f1160e640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f0c1083d0 0x7f4f0c19a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f1160e640 1 -- 172.21.15.29:0/847886040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f0c108d20 msgr2=0x7f4f0c19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f1160e640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f0c108d20 0x7f4f0c19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f1160e640 1 -- 172.21.15.29:0/847886040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f0c1a0260 con 0x7f4f0c102ae0 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f11e0f640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f0c108d20 0x7f4f0c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f10e0d640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f0c1083d0 0x7f4f0c19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f1160e640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c102ae0 0x7f4f0c19a3b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00cce0 tx=0x7f4efc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc004b20 con 0x7f4f0c102ae0 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f0c1a0550 con 0x7f4f0c102ae0 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4efc004cc0 con 0x7f4f0c102ae0 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc00f6b0 con 0x7f4f0c102ae0 2024-09-21T02:37:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.518+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f0c1a6d70 con 0x7f4f0c102ae0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.519+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4f0c063b50 con 0x7f4f0c102ae0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.521+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f4efc00f850 con 0x7f4f0c102ae0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.521+0000 7f4f0a7fc640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ef40b65d0 0x7f4ef40b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.522+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f4efc0e7d10 con 0x7f4f0c102ae0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.522+0000 7f4f10e0d640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ef40b65d0 0x7f4ef40b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.522+0000 7f4f0a7fc640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] conn(0x7f4ef40c16e0 0x7f4ef40c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.522+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 --> [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4efc021e50 con 0x7f4ef40c16e0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.522+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f4efc0e8160 con 0x7f4f0c102ae0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.522+0000 7f4f11e0f640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] conn(0x7f4ef40c16e0 0x7f4ef40c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.522+0000 7f4f10e0d640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ef40b65d0 0x7f4ef40b8ae0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4f00006010 tx=0x7f4f00000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.523+0000 7f4f11e0f640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] conn(0x7f4ef40c16e0 0x7f4ef40c3bd0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.523+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 <== osd.5 v2:172.21.15.190:6808/4042925750 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4efc021e50 con 0x7f4ef40c16e0 2024-09-21T02:37:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.537+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 --> [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4f0c063f50 con 0x7f4ef40c16e0 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f0a7fc640 1 -- 172.21.15.29:0/847886040 <== osd.5 v2:172.21.15.190:6808/4042925750 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4f0c063f50 con 0x7f4ef40c16e0 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] conn(0x7f4ef40c16e0 msgr2=0x7f4ef40c3bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] conn(0x7f4ef40c16e0 0x7f4ef40c3bd0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ef40b65d0 msgr2=0x7f4ef40b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ef40b65d0 0x7f4ef40b8ae0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4f00006010 tx=0x7f4f00000bc0 comp rx=0 tx=0).stop 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c102ae0 msgr2=0x7f4f0c19a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 --2- 172.21.15.29:0/847886040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f0c102ae0 0x7f4f0c19a3b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00cce0 tx=0x7f4efc00a620 comp rx=0 tx=0).stop 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f11e0f640 1 -- 172.21.15.29:0/847886040 reap_dead start 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 shutdown_connections 2024-09-21T02:37:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 >> 172.21.15.29:0/847886040 conn(0x7f4f0c079760 msgr2=0x7f4f0c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:49.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.538+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 shutdown_connections 2024-09-21T02:37:49.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:49.539+0000 7f4f13872640 1 -- 172.21.15.29:0/847886040 wait complete. 2024-09-21T02:37:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:49 smithi029 ceph-mon[31006]: pgmap v348: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s; 24 B/s, 1 objects/s recovering 2024-09-21T02:37:50.000 INFO:teuthology.orchestra.run.smithi029.stdout:227633266734 2024-09-21T02:37:50.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.8 2024-09-21T02:37:50.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:49 smithi066 ceph-mon[33436]: pgmap v348: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s; 24 B/s, 1 objects/s recovering 2024-09-21T02:37:50.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:49 smithi190 ceph-mon[33871]: pgmap v348: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s; 24 B/s, 1 objects/s recovering 2024-09-21T02:37:50.496 INFO:teuthology.orchestra.run.smithi029.stdout:296352743453 2024-09-21T02:37:50.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.11 2024-09-21T02:37:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 -- 172.21.15.29:0/2631340701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb50103b60 msgr2=0x7fdb50103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 --2- 172.21.15.29:0/2631340701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb50103b60 0x7fdb50103f70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009ca0 tx=0x7fdb4002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 -- 172.21.15.29:0/2631340701 shutdown_connections 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 --2- 172.21.15.29:0/2631340701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb50109df0 0x7fdb50110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 --2- 172.21.15.29:0/2631340701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb501094a0 0x7fdb501098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 --2- 172.21.15.29:0/2631340701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb50103b60 0x7fdb50103f70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 -- 172.21.15.29:0/2631340701 >> 172.21.15.29:0/2631340701 conn(0x7fdb5007b850 msgr2=0x7fdb5007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 -- 172.21.15.29:0/2631340701 shutdown_connections 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.966+0000 7fdb55f1d640 1 -- 172.21.15.29:0/2631340701 wait complete. 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb55f1d640 1 Processor -- start 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb55f1d640 1 -- start start 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb55f1d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb50103b60 0x7fdb5019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb55f1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb501094a0 0x7fdb5019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb55f1d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb50109df0 0x7fdb5019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb55f1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdb501139d0 con 0x7fdb501094a0 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb55f1d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdb50113850 con 0x7fdb50109df0 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb55f1d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb50113b50 con 0x7fdb50103b60 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb4effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb501094a0 0x7fdb5019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb4effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb501094a0 0x7fdb5019a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39088/0 (socket says 172.21.15.29:39088) 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb4effd640 1 -- 172.21.15.29:0/3087426483 learned_addr learned my addr 172.21.15.29:0/3087426483 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.967+0000 7fdb4f7fe640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb50103b60 0x7fdb5019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4ffff640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb50109df0 0x7fdb5019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4effd640 1 -- 172.21.15.29:0/3087426483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb50103b60 msgr2=0x7fdb5019a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4effd640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb50103b60 0x7fdb5019a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4effd640 1 -- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb50109df0 msgr2=0x7fdb5019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4effd640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb50109df0 0x7fdb5019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4effd640 1 -- 172.21.15.29:0/3087426483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb501a0350 con 0x7fdb501094a0 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4ffff640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb50109df0 0x7fdb5019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4effd640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb501094a0 0x7fdb5019a980 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400c970 tx=0x7fdb4400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44004220 con 0x7fdb501094a0 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb501a0640 con 0x7fdb501094a0 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb44007e40 con 0x7fdb501094a0 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb440047a0 con 0x7fdb501094a0 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.968+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb501a6da0 con 0x7fdb501094a0 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.969+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb50063b10 con 0x7fdb501094a0 2024-09-21T02:37:50.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.971+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fdb440079b0 con 0x7fdb501094a0 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.972+0000 7fdb4cff9640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb380b65a0 0x7fdb380b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.972+0000 7fdb4f7fe640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb380b65a0 0x7fdb380b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.972+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fdb440e78b0 con 0x7fdb501094a0 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.972+0000 7fdb4cff9640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] conn(0x7fdb380c16b0 0x7fdb380c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.972+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 --> [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb380c4270 con 0x7fdb380c16b0 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.972+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fdb440e7cf0 con 0x7fdb501094a0 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.973+0000 7fdb4ffff640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] conn(0x7fdb380c16b0 0x7fdb380c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.973+0000 7fdb4f7fe640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb380b65a0 0x7fdb380b8ab0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdb400098d0 tx=0x7fdb40031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.973+0000 7fdb4ffff640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] conn(0x7fdb380c16b0 0x7fdb380c3ba0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:50.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.974+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 <== osd.10 v2:172.21.15.66:6824/2015526133 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb380c4270 con 0x7fdb380c16b0 2024-09-21T02:37:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.988+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 --> [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb50063f90 con 0x7fdb380c16b0 2024-09-21T02:37:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.988+0000 7fdb4cff9640 1 -- 172.21.15.29:0/3087426483 <== osd.10 v2:172.21.15.66:6824/2015526133 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdb50063f90 con 0x7fdb380c16b0 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] conn(0x7fdb380c16b0 msgr2=0x7fdb380c3ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] conn(0x7fdb380c16b0 0x7fdb380c3ba0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb380b65a0 msgr2=0x7fdb380b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb380b65a0 0x7fdb380b8ab0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdb400098d0 tx=0x7fdb40031000 comp rx=0 tx=0).stop 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb501094a0 msgr2=0x7fdb5019a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 --2- 172.21.15.29:0/3087426483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb501094a0 0x7fdb5019a980 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400c970 tx=0x7fdb4400ce30 comp rx=0 tx=0).stop 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb4ffff640 1 -- 172.21.15.29:0/3087426483 reap_dead start 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 shutdown_connections 2024-09-21T02:37:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 >> 172.21.15.29:0/3087426483 conn(0x7fdb5007b850 msgr2=0x7fdb50107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:50.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 shutdown_connections 2024-09-21T02:37:50.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:50.989+0000 7fdb55f1d640 1 -- 172.21.15.29:0/3087426483 wait complete. 2024-09-21T02:37:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:51 smithi029 ceph-mon[31006]: pgmap v349: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s; 22 B/s, 1 objects/s recovering 2024-09-21T02:37:51.940 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:51 smithi190 ceph-mon[33871]: pgmap v349: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s; 22 B/s, 1 objects/s recovering 2024-09-21T02:37:52.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:51 smithi066 ceph-mon[33436]: pgmap v349: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s; 22 B/s, 1 objects/s recovering 2024-09-21T02:37:52.095 INFO:teuthology.orchestra.run.smithi029.stdout:150323855422 2024-09-21T02:37:52.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.273+0000 7f99a8110640 1 -- 172.21.15.29:0/3851461986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 msgr2=0x7f99a0102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.273+0000 7f99a8110640 1 --2- 172.21.15.29:0/3851461986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 0x7f99a0102e70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f999c009900 tx=0x7f999c02a410 comp rx=0 tx=0).stop 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.273+0000 7f99a8110640 1 -- 172.21.15.29:0/3851461986 shutdown_connections 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.273+0000 7f99a8110640 1 --2- 172.21.15.29:0/3851461986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a0108cf0 0x7f99a010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.273+0000 7f99a8110640 1 --2- 172.21.15.29:0/3851461986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99a01083a0 0x7f99a01087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.273+0000 7f99a8110640 1 --2- 172.21.15.29:0/3851461986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 0x7f99a0102e70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.273+0000 7f99a8110640 1 -- 172.21.15.29:0/3851461986 >> 172.21.15.29:0/3851461986 conn(0x7f99a0079730 msgr2=0x7f99a0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.274+0000 7f99a8110640 1 -- 172.21.15.29:0/3851461986 shutdown_connections 2024-09-21T02:37:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.274+0000 7f99a8110640 1 -- 172.21.15.29:0/3851461986 wait complete. 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.274+0000 7f99a8110640 1 Processor -- start 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a8110640 1 -- start start 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a8110640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 0x7f99a01992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a8110640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a01083a0 0x7f99a0199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a8110640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99a0108cf0 0x7f99a019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a8110640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99a0112810 con 0x7f99a0102a60 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a8110640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f99a0112690 con 0x7f99a0108cf0 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a8110640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f99a0112990 con 0x7f99a01083a0 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a5eac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 0x7f99a01992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a5eac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 0x7f99a01992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39098/0 (socket says 172.21.15.29:39098) 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a5eac640 1 -- 172.21.15.29:0/1259792485 learned_addr learned my addr 172.21.15.29:0/1259792485 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a56ab640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a01083a0 0x7f99a0199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a66ad640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99a0108cf0 0x7f99a019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a5eac640 1 -- 172.21.15.29:0/1259792485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a01083a0 msgr2=0x7f99a0199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a5eac640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a01083a0 0x7f99a0199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a5eac640 1 -- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99a0108cf0 msgr2=0x7f99a019eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a5eac640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99a0108cf0 0x7f99a019eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a5eac640 1 -- 172.21.15.29:0/1259792485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a019f210 con 0x7f99a0102a60 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.275+0000 7f99a66ad640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99a0108cf0 0x7f99a019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.276+0000 7f99a5eac640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 0x7f99a01992a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f999c009230 tx=0x7f999c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.276+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c03f030 con 0x7f99a0102a60 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.276+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f999c003a80 con 0x7f99a0102a60 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.276+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c035070 con 0x7f99a0102a60 2024-09-21T02:37:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.276+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a019f4a0 con 0x7f99a0102a60 2024-09-21T02:37:52.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.276+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a006ae60 con 0x7f99a0102a60 2024-09-21T02:37:52.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.277+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f99a006b3d0 con 0x7f99a0102a60 2024-09-21T02:37:52.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.279+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f999c042030 con 0x7f99a0102a60 2024-09-21T02:37:52.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.279+0000 7f9996ffd640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f998c0b65a0 0x7f998c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:52.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.279+0000 7f99a56ab640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f998c0b65a0 0x7f998c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:52.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.280+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f999c108340 con 0x7f99a0102a60 2024-09-21T02:37:52.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.280+0000 7f9996ffd640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] conn(0x7f998c0c16b0 0x7f998c0c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:52.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.280+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 --> [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f999c03ce50 con 0x7f998c0c16b0 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.280+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f999c108820 con 0x7f99a0102a60 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.280+0000 7f99a56ab640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f998c0b65a0 0x7f998c0b8ab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f99a0100cc0 tx=0x7f999000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.280+0000 7f99a66ad640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] conn(0x7f998c0c16b0 0x7f998c0c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.281+0000 7f99a66ad640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] conn(0x7f998c0c16b0 0x7f998c0c3ba0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.282+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 <== osd.7 v2:172.21.15.66:6816/3852321903 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f999c03ce50 con 0x7f998c0c16b0 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.296+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 --> [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f99a006b850 con 0x7f998c0c16b0 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.296+0000 7f9996ffd640 1 -- 172.21.15.29:0/1259792485 <== osd.7 v2:172.21.15.66:6816/3852321903 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f99a006b850 con 0x7f998c0c16b0 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] conn(0x7f998c0c16b0 msgr2=0x7f998c0c3ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] conn(0x7f998c0c16b0 0x7f998c0c3ba0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f998c0b65a0 msgr2=0x7f998c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f998c0b65a0 0x7f998c0b8ab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f99a0100cc0 tx=0x7f999000b410 comp rx=0 tx=0).stop 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 msgr2=0x7f99a01992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 --2- 172.21.15.29:0/1259792485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a0102a60 0x7f99a01992a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f999c009230 tx=0x7f999c004270 comp rx=0 tx=0).stop 2024-09-21T02:37:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a66ad640 1 -- 172.21.15.29:0/1259792485 reap_dead start 2024-09-21T02:37:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 shutdown_connections 2024-09-21T02:37:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 >> 172.21.15.29:0/1259792485 conn(0x7f99a0079730 msgr2=0x7f99a0101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 shutdown_connections 2024-09-21T02:37:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:52.297+0000 7f99a8110640 1 -- 172.21.15.29:0/1259792485 wait complete. 2024-09-21T02:37:53.272 INFO:teuthology.orchestra.run.smithi029.stdout:274877906979 2024-09-21T02:37:53.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.10 2024-09-21T02:37:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:53 smithi029 ceph-mon[31006]: pgmap v350: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:53.883 INFO:teuthology.orchestra.run.smithi029.stdout:201863462964 2024-09-21T02:37:53.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T02:37:53.948 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:53.950 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:53.961 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:53 smithi066 ceph-mon[33436]: pgmap v350: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:54.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:53 smithi190 ceph-mon[33871]: pgmap v350: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 -- 172.21.15.29:0/3879883736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106330 msgr2=0x7efd5c106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 --2- 172.21.15.29:0/3879883736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106330 0x7efd5c106740 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009900 tx=0x7efd4c02a410 comp rx=0 tx=0).stop 2024-09-21T02:37:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 -- 172.21.15.29:0/3879883736 shutdown_connections 2024-09-21T02:37:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 --2- 172.21.15.29:0/3879883736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c106c80 0x7efd5c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 --2- 172.21.15.29:0/3879883736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106330 0x7efd5c106740 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 --2- 172.21.15.29:0/3879883736 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c1020e0 0x7efd5c1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 -- 172.21.15.29:0/3879883736 >> 172.21.15.29:0/3879883736 conn(0x7efd5c079ec0 msgr2=0x7efd5c07a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 -- 172.21.15.29:0/3879883736 shutdown_connections 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 -- 172.21.15.29:0/3879883736 wait complete. 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.533+0000 7efd63fa0640 1 Processor -- start 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd63fa0640 1 -- start start 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd63fa0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c1020e0 0x7efd5c103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd63fa0640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c106330 0x7efd5c1035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd63fa0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106c80 0x7efd5c103b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd63fa0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd5c113a00 con 0x7efd5c106c80 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd63fa0640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7efd5c113880 con 0x7efd5c106330 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd63fa0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efd5c113b80 con 0x7efd5c1020e0 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106c80 0x7efd5c103b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106c80 0x7efd5c103b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47928/0 (socket says 172.21.15.29:47928) 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 -- 172.21.15.29:0/279842853 learned_addr learned my addr 172.21.15.29:0/279842853 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd61d3c640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c1020e0 0x7efd5c103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6153b640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c106330 0x7efd5c1035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 -- 172.21.15.29:0/279842853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c1020e0 msgr2=0x7efd5c103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c1020e0 0x7efd5c103080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 -- 172.21.15.29:0/279842853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c106330 msgr2=0x7efd5c1035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c106330 0x7efd5c1035c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 -- 172.21.15.29:0/279842853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5c1a8640 con 0x7efd5c106c80 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6153b640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c106330 0x7efd5c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.534+0000 7efd6253d640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106c80 0x7efd5c103b00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efd5400cce0 tx=0x7efd5400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.535+0000 7efd52ffd640 1 -- 172.21.15.29:0/279842853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5400f450 con 0x7efd5c106c80 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.535+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd5c1a8930 con 0x7efd5c106c80 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.535+0000 7efd52ffd640 1 -- 172.21.15.29:0/279842853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd5400fa80 con 0x7efd5c106c80 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.535+0000 7efd52ffd640 1 -- 172.21.15.29:0/279842853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd54018ad0 con 0x7efd5c106c80 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.535+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd5c1a8e70 con 0x7efd5c106c80 2024-09-21T02:37:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.536+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd5c1024f0 con 0x7efd5c106c80 2024-09-21T02:37:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.537+0000 7efd52ffd640 1 -- 172.21.15.29:0/279842853 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7efd5400f5f0 con 0x7efd5c106c80 2024-09-21T02:37:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.538+0000 7efd52ffd640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd480b65a0 0x7efd480b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.538+0000 7efd52ffd640 1 -- 172.21.15.29:0/279842853 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7efd540e8080 con 0x7efd5c106c80 2024-09-21T02:37:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.538+0000 7efd61d3c640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd480b65a0 0x7efd480b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.539+0000 7efd61d3c640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd480b65a0 0x7efd480b8ab0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efd5c104ad0 tx=0x7efd58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.542+0000 7efd52ffd640 1 -- 172.21.15.29:0/279842853 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7efd54014030 con 0x7efd5c106c80 2024-09-21T02:37:54.635 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.643 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.644 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.645 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.648 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.648 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.652 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.662 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.770+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7efd5c02e640 con 0x7efd5c106c80 2024-09-21T02:37:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.770+0000 7efd52ffd640 1 -- 172.21.15.29:0/279842853 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efd540ae450 con 0x7efd5c106c80 2024-09-21T02:37:54.772 INFO:teuthology.orchestra.run.smithi029.stdout:253403070506 2024-09-21T02:37:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd480b65a0 msgr2=0x7efd480b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd480b65a0 0x7efd480b8ab0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efd5c104ad0 tx=0x7efd58005f90 comp rx=0 tx=0).stop 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106c80 msgr2=0x7efd5c103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106c80 0x7efd5c103b00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efd5400cce0 tx=0x7efd5400a620 comp rx=0 tx=0).stop 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 shutdown_connections 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efd480b65a0 0x7efd480b8ab0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd5c106c80 0x7efd5c103b00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd5c106330 0x7efd5c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 --2- 172.21.15.29:0/279842853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd5c1020e0 0x7efd5c103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 >> 172.21.15.29:0/279842853 conn(0x7efd5c079ec0 msgr2=0x7efd5c107620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 shutdown_connections 2024-09-21T02:37:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:54.772+0000 7efd63fa0640 1 -- 172.21.15.29:0/279842853 wait complete. 2024-09-21T02:37:55.660 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070503 got 253403070506 for osd.9 2024-09-21T02:37:55.660 DEBUG:teuthology.parallel:result is None 2024-09-21T02:37:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:55 smithi029 ceph-mon[31006]: pgmap v351: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:55 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/279842853' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T02:37:56.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:55 smithi066 ceph-mon[33436]: pgmap v351: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:55 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/279842853' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T02:37:56.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:55 smithi190 ceph-mon[33871]: pgmap v351: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:55 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/279842853' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T02:37:56.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 -- 172.21.15.29:0/3841827861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b41083d0 msgr2=0x7fd2b41087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:56.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 --2- 172.21.15.29:0/3841827861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b41083d0 0x7fd2b41087e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009900 tx=0x7fd2a802a410 comp rx=0 tx=0).stop 2024-09-21T02:37:56.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 -- 172.21.15.29:0/3841827861 shutdown_connections 2024-09-21T02:37:56.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 --2- 172.21.15.29:0/3841827861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2b4108d20 0x7fd2b410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 --2- 172.21.15.29:0/3841827861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b41083d0 0x7fd2b41087e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 --2- 172.21.15.29:0/3841827861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2b4102ae0 0x7fd2b4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 -- 172.21.15.29:0/3841827861 >> 172.21.15.29:0/3841827861 conn(0x7fd2b4079760 msgr2=0x7fd2b4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 -- 172.21.15.29:0/3841827861 shutdown_connections 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 -- 172.21.15.29:0/3841827861 wait complete. 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.648+0000 7fd2baded640 1 Processor -- start 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2baded640 1 -- start start 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2baded640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b4102ae0 0x7fd2b41992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2baded640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2b41083d0 0x7fd2b4199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2baded640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2b4108d20 0x7fd2b419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2baded640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b4112840 con 0x7fd2b4102ae0 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2baded640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41126c0 con 0x7fd2b41083d0 2024-09-21T02:37:56.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2baded640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41129c0 con 0x7fd2b4108d20 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b4102ae0 0x7fd2b41992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b4102ae0 0x7fd2b41992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47952/0 (socket says 172.21.15.29:47952) 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 -- 172.21.15.29:0/895719263 learned_addr learned my addr 172.21.15.29:0/895719263 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b3fff640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2b41083d0 0x7fd2b4199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b938a640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2b4108d20 0x7fd2b419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 -- 172.21.15.29:0/895719263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2b4108d20 msgr2=0x7fd2b419eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2b4108d20 0x7fd2b419eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 -- 172.21.15.29:0/895719263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2b41083d0 msgr2=0x7fd2b4199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2b41083d0 0x7fd2b4199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 -- 172.21.15.29:0/895719263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b419f220 con 0x7fd2b4102ae0 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b938a640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2b4108d20 0x7fd2b419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.649+0000 7fd2b8b89640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b4102ae0 0x7fd2b41992b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400d950 tx=0x7fd2a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:56.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.650+0000 7fd2b1ffb640 1 -- 172.21.15.29:0/895719263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4017070 con 0x7fd2b4102ae0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.650+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b419f510 con 0x7fd2b4102ae0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.650+0000 7fd2b1ffb640 1 -- 172.21.15.29:0/895719263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2a4009c30 con 0x7fd2b4102ae0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.650+0000 7fd2b1ffb640 1 -- 172.21.15.29:0/895719263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a400f620 con 0x7fd2b4102ae0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.650+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b41a5c70 con 0x7fd2b4102ae0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.651+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2b4103d10 con 0x7fd2b4102ae0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.652+0000 7fd2b1ffb640 1 -- 172.21.15.29:0/895719263 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fd2a4010460 con 0x7fd2b4102ae0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.653+0000 7fd2b1ffb640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd29c0b65a0 0x7fd29c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.654+0000 7fd2b3fff640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd29c0b65a0 0x7fd29c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.654+0000 7fd2b1ffb640 1 -- 172.21.15.29:0/895719263 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fd2a40e7420 con 0x7fd2b4102ae0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.654+0000 7fd2b3fff640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd29c0b65a0 0x7fd29c0b8ab0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8004770 tx=0x7fd2a80058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.658+0000 7fd2b1ffb640 1 -- 172.21.15.29:0/895719263 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd2a40ad7f0 con 0x7fd2b4102ae0 2024-09-21T02:37:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.850+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd2b4102ef0 con 0x7fd2b4102ae0 2024-09-21T02:37:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.850+0000 7fd2b1ffb640 1 -- 172.21.15.29:0/895719263 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd2a4009de0 con 0x7fd2b4102ae0 2024-09-21T02:37:56.852 INFO:teuthology.orchestra.run.smithi029.stdout:55834574931 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.852+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd29c0b65a0 msgr2=0x7fd29c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.852+0000 7fd2baded640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd29c0b65a0 0x7fd29c0b8ab0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8004770 tx=0x7fd2a80058d0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.852+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b4102ae0 msgr2=0x7fd2b41992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.852+0000 7fd2baded640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b4102ae0 0x7fd2b41992b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400d950 tx=0x7fd2a400de10 comp rx=0 tx=0).stop 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.853+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 shutdown_connections 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.853+0000 7fd2baded640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd29c0b65a0 0x7fd29c0b8ab0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.853+0000 7fd2baded640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2b4108d20 0x7fd2b419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.853+0000 7fd2baded640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2b41083d0 0x7fd2b4199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.853+0000 7fd2baded640 1 --2- 172.21.15.29:0/895719263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b4102ae0 0x7fd2b41992b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.853+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 >> 172.21.15.29:0/895719263 conn(0x7fd2b4079760 msgr2=0x7fd2b41044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.853+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 shutdown_connections 2024-09-21T02:37:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:56.853+0000 7fd2baded640 1 -- 172.21.15.29:0/895719263 wait complete. 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 -- 172.21.15.29:0/1943310401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 msgr2=0x7ff7601098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 --2- 172.21.15.29:0/1943310401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 0x7ff7601098b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff750009900 tx=0x7ff75002a410 comp rx=0 tx=0).stop 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 -- 172.21.15.29:0/1943310401 shutdown_connections 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 --2- 172.21.15.29:0/1943310401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760109df0 0x7ff760110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 --2- 172.21.15.29:0/1943310401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 0x7ff7601098b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 --2- 172.21.15.29:0/1943310401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760103b60 0x7ff760103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 -- 172.21.15.29:0/1943310401 >> 172.21.15.29:0/1943310401 conn(0x7ff76007b850 msgr2=0x7ff76007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 -- 172.21.15.29:0/1943310401 shutdown_connections 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 -- 172.21.15.29:0/1943310401 wait complete. 2024-09-21T02:37:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.602+0000 7ff7677cf640 1 Processor -- start 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff7677cf640 1 -- start start 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff7677cf640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760103b60 0x7ff760075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff7677cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 0x7ff760075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff7677cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760109df0 0x7ff760076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff7677cf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff7601139b0 con 0x7ff7601094a0 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff7677cf640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff760113830 con 0x7ff760103b60 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff7677cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff760113b30 con 0x7ff760109df0 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff764d6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 0x7ff760075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff764d6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 0x7ff760075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47968/0 (socket says 172.21.15.29:47968) 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff764d6a640 1 -- 172.21.15.29:0/346504172 learned_addr learned my addr 172.21.15.29:0/346504172 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff76556b640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760103b60 0x7ff760075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff765d6c640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760109df0 0x7ff760076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff764d6a640 1 -- 172.21.15.29:0/346504172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760109df0 msgr2=0x7ff760076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff764d6a640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760109df0 0x7ff760076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff764d6a640 1 -- 172.21.15.29:0/346504172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760103b60 msgr2=0x7ff760075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff764d6a640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760103b60 0x7ff760075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff764d6a640 1 -- 172.21.15.29:0/346504172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7601a86a0 con 0x7ff7601094a0 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff765d6c640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760109df0 0x7ff760076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.603+0000 7ff76556b640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760103b60 0x7ff760075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.604+0000 7ff764d6a640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 0x7ff760075c50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff75002a930 tx=0x7ff750003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.604+0000 7ff7567fc640 1 -- 172.21.15.29:0/346504172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75003f070 con 0x7ff7601094a0 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.604+0000 7ff7567fc640 1 -- 172.21.15.29:0/346504172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff75003baf0 con 0x7ff7601094a0 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.604+0000 7ff7567fc640 1 -- 172.21.15.29:0/346504172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff750035030 con 0x7ff7601094a0 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.604+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7601a8930 con 0x7ff7601094a0 2024-09-21T02:37:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.604+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7601a8e60 con 0x7ff7601094a0 2024-09-21T02:37:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.605+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff760103f70 con 0x7ff7601094a0 2024-09-21T02:37:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.607+0000 7ff7567fc640 1 -- 172.21.15.29:0/346504172 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7ff750042070 con 0x7ff7601094a0 2024-09-21T02:37:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.608+0000 7ff7567fc640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff74c0b6620 0x7ff74c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.608+0000 7ff76556b640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff74c0b6620 0x7ff74c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.608+0000 7ff7567fc640 1 -- 172.21.15.29:0/346504172 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7ff7501082d0 con 0x7ff7601094a0 2024-09-21T02:37:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.608+0000 7ff76556b640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff74c0b6620 0x7ff74c0b8b30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff760077160 tx=0x7ff75c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.610+0000 7ff7567fc640 1 -- 172.21.15.29:0/346504172 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff7500ce6a0 con 0x7ff7601094a0 2024-09-21T02:37:57.637 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:37:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.811+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff76002e640 con 0x7ff7601094a0 2024-09-21T02:37:57.812 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:57 smithi029 ceph-mon[31006]: pgmap v352: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:57.812 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:57 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/895719263' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T02:37:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.811+0000 7ff7567fc640 1 -- 172.21.15.29:0/346504172 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff7500d2a80 con 0x7ff7601094a0 2024-09-21T02:37:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:77309411407 2024-09-21T02:37:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff74c0b6620 msgr2=0x7ff74c0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff74c0b6620 0x7ff74c0b8b30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff760077160 tx=0x7ff75c005f90 comp rx=0 tx=0).stop 2024-09-21T02:37:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 msgr2=0x7ff760075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 0x7ff760075c50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff75002a930 tx=0x7ff750003b90 comp rx=0 tx=0).stop 2024-09-21T02:37:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 shutdown_connections 2024-09-21T02:37:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff74c0b6620 0x7ff74c0b8b30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760109df0 0x7ff760076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7601094a0 0x7ff760075c50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 --2- 172.21.15.29:0/346504172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760103b60 0x7ff760075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 >> 172.21.15.29:0/346504172 conn(0x7ff76007b850 msgr2=0x7ff760107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 shutdown_connections 2024-09-21T02:37:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.814+0000 7ff7677cf640 1 -- 172.21.15.29:0/346504172 wait complete. 2024-09-21T02:37:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.850+0000 7f8980b62640 1 -- 172.21.15.29:0/3179107146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c1049b0 msgr2=0x7f897c104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.850+0000 7f8980b62640 1 --2- 172.21.15.29:0/3179107146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c1049b0 0x7f897c104dc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f896c009900 tx=0x7f896c02a410 comp rx=0 tx=0).stop 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.851+0000 7f8980b62640 1 -- 172.21.15.29:0/3179107146 shutdown_connections 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.851+0000 7f8980b62640 1 --2- 172.21.15.29:0/3179107146 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f897c10de10 0x7f897c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.851+0000 7f8980b62640 1 --2- 172.21.15.29:0/3179107146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c105300 0x7f897c10d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.851+0000 7f8980b62640 1 --2- 172.21.15.29:0/3179107146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c1049b0 0x7f897c104dc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.851+0000 7f8980b62640 1 -- 172.21.15.29:0/3179107146 >> 172.21.15.29:0/3179107146 conn(0x7f897c079730 msgr2=0x7f897c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.851+0000 7f8980b62640 1 -- 172.21.15.29:0/3179107146 shutdown_connections 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.851+0000 7f8980b62640 1 -- 172.21.15.29:0/3179107146 wait complete. 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8980b62640 1 Processor -- start 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8980b62640 1 -- start start 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8980b62640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f897c1049b0 0x7f897c19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8980b62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c105300 0x7f897c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8980b62640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c10de10 0x7f897c19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8980b62640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f897c113920 con 0x7f897c105300 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8980b62640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f897c1137a0 con 0x7f897c1049b0 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8980b62640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f897c113aa0 con 0x7f897c10de10 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8979d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c105300 0x7f897c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8979d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c105300 0x7f897c19a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47986/0 (socket says 172.21.15.29:47986) 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8979d9b640 1 -- 172.21.15.29:0/169751221 learned_addr learned my addr 172.21.15.29:0/169751221 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f897a59c640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f897c1049b0 0x7f897c19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f897ad9d640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c10de10 0x7f897c19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8979d9b640 1 -- 172.21.15.29:0/169751221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c10de10 msgr2=0x7f897c19fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8979d9b640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c10de10 0x7f897c19fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8979d9b640 1 -- 172.21.15.29:0/169751221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f897c1049b0 msgr2=0x7f897c19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8979d9b640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f897c1049b0 0x7f897c19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.852+0000 7f8979d9b640 1 -- 172.21.15.29:0/169751221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c1a03c0 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.853+0000 7f8979d9b640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c105300 0x7f897c19a9f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f897000d740 tx=0x7f897000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.853+0000 7f896b7fe640 1 -- 172.21.15.29:0/169751221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8970017070 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.853+0000 7f896b7fe640 1 -- 172.21.15.29:0/169751221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f897000de50 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.853+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897c1a06b0 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.853+0000 7f896b7fe640 1 -- 172.21.15.29:0/169751221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897000f620 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.853+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897c1a6e10 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.854+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f897c104dc0 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.856+0000 7f896b7fe640 1 -- 172.21.15.29:0/169751221 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f897001a070 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.856+0000 7f896b7fe640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89640b6620 0x7f89640b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.856+0000 7f896b7fe640 1 -- 172.21.15.29:0/169751221 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f89700e7170 con 0x7f897c105300 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.857+0000 7f897a59c640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89640b6620 0x7f89640b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.857+0000 7f897a59c640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89640b6620 0x7f89640b8b30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f896c004770 tx=0x7f896c0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:57.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:57.860+0000 7f896b7fe640 1 -- 172.21.15.29:0/169751221 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f89700ad5c0 con 0x7f897c105300 2024-09-21T02:37:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:57 smithi066 ceph-mon[33436]: pgmap v352: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:57 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/895719263' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T02:37:58.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.055+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f897c19b580 con 0x7f897c105300 2024-09-21T02:37:58.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.055+0000 7f896b7fe640 1 -- 172.21.15.29:0/169751221 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f89700b19a0 con 0x7f897c105300 2024-09-21T02:37:58.056 INFO:teuthology.orchestra.run.smithi029.stdout:103079215176 2024-09-21T02:37:58.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89640b6620 msgr2=0x7f89640b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89640b6620 0x7f89640b8b30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f896c004770 tx=0x7f896c0058d0 comp rx=0 tx=0).stop 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c105300 msgr2=0x7f897c19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c105300 0x7f897c19a9f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f897000d740 tx=0x7f897000dc00 comp rx=0 tx=0).stop 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 shutdown_connections 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89640b6620 0x7f89640b8b30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f897c10de10 0x7f897c19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f897c105300 0x7f897c19a9f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 --2- 172.21.15.29:0/169751221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f897c1049b0 0x7f897c19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 >> 172.21.15.29:0/169751221 conn(0x7f897c079730 msgr2=0x7f897c0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 shutdown_connections 2024-09-21T02:37:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:58.057+0000 7f8980b62640 1 -- 172.21.15.29:0/169751221 wait complete. 2024-09-21T02:37:58.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:57 smithi190 ceph-mon[33871]: pgmap v352: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 19 B/s, 0 objects/s recovering 2024-09-21T02:37:58.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:57 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/895719263' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T02:37:58.811 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411404 got 77309411407 for osd.2 2024-09-21T02:37:58.812 DEBUG:teuthology.parallel:result is None 2024-09-21T02:37:59.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:58 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/346504172' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T02:37:59.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:58 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/169751221' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T02:37:59.019 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:58 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/346504172' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T02:37:59.019 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:58 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/169751221' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T02:37:59.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:58 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/346504172' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T02:37:59.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:58 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/169751221' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T02:37:59.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 -- 172.21.15.29:0/3432406991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c108d10 msgr2=0x7f8c9c10f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:59.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 --2- 172.21.15.29:0/3432406991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c108d10 0x7f8c9c10f540 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8c94009ca0 tx=0x7f8c9402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 -- 172.21.15.29:0/3432406991 shutdown_connections 2024-09-21T02:37:59.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 --2- 172.21.15.29:0/3432406991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c108d10 0x7f8c9c10f540 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 --2- 172.21.15.29:0/3432406991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c9c1083c0 0x7f8c9c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 --2- 172.21.15.29:0/3432406991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c9c102a80 0x7f8c9c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 -- 172.21.15.29:0/3432406991 >> 172.21.15.29:0/3432406991 conn(0x7f8c9c079740 msgr2=0x7f8c9c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 -- 172.21.15.29:0/3432406991 shutdown_connections 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 -- 172.21.15.29:0/3432406991 wait complete. 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.520+0000 7f8ca2045640 1 Processor -- start 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8ca2045640 1 -- start start 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8ca2045640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c102a80 0x7f8c9c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8ca2045640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c9c1083c0 0x7f8c9c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8ca2045640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c9c108d10 0x7f8c9c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:59.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8ca2045640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c113980 con 0x7f8c9c102a80 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8ca2045640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c113800 con 0x7f8c9c108d10 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8ca2045640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c113b00 con 0x7f8c9c1083c0 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c102a80 0x7f8c9c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c102a80 0x7f8c9c19a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48022/0 (socket says 172.21.15.29:48022) 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9b7fe640 1 -- 172.21.15.29:0/2254433199 learned_addr learned my addr 172.21.15.29:0/2254433199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9bfff640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c9c108d10 0x7f8c9c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9affd640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c9c1083c0 0x7f8c9c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9b7fe640 1 -- 172.21.15.29:0/2254433199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c9c1083c0 msgr2=0x7f8c9c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9b7fe640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c9c1083c0 0x7f8c9c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9b7fe640 1 -- 172.21.15.29:0/2254433199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c9c108d10 msgr2=0x7f8c9c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9b7fe640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c9c108d10 0x7f8c9c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.521+0000 7f8c9b7fe640 1 -- 172.21.15.29:0/2254433199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9c1a02e0 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.522+0000 7f8c9b7fe640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c102a80 0x7f8c9c19a370 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00d950 tx=0x7f8c8c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.522+0000 7f8c98ff9640 1 -- 172.21.15.29:0/2254433199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c017030 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.522+0000 7f8c98ff9640 1 -- 172.21.15.29:0/2254433199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c8c004020 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.522+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c9c1a05d0 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.522+0000 7f8c98ff9640 1 -- 172.21.15.29:0/2254433199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0055d0 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.522+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c9c1a6d30 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.523+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c9c103d00 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.525+0000 7f8c98ff9640 1 -- 172.21.15.29:0/2254433199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f8c8c009a30 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.525+0000 7f8c98ff9640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c840b6620 0x7f8c840b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.525+0000 7f8c98ff9640 1 -- 172.21.15.29:0/2254433199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f8c8c0e7850 con 0x7f8c9c102a80 2024-09-21T02:37:59.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.525+0000 7f8c9affd640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c840b6620 0x7f8c840b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:37:59.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.526+0000 7f8c9affd640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c840b6620 0x7f8c840b8b30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c1079b0 tx=0x7f8c90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:37:59.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.529+0000 7f8c98ff9640 1 -- 172.21.15.29:0/2254433199 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8c8c0adca0 con 0x7f8c9c102a80 2024-09-21T02:37:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.722+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8c9c102e90 con 0x7f8c9c102a80 2024-09-21T02:37:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.722+0000 7f8c98ff9640 1 -- 172.21.15.29:0/2254433199 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8c8c0b2080 con 0x7f8c9c102a80 2024-09-21T02:37:59.724 INFO:teuthology.orchestra.run.smithi029.stdout:128849018948 2024-09-21T02:37:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c840b6620 msgr2=0x7f8c840b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c840b6620 0x7f8c840b8b30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c1079b0 tx=0x7f8c90005f90 comp rx=0 tx=0).stop 2024-09-21T02:37:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c102a80 msgr2=0x7f8c9c19a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:37:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c102a80 0x7f8c9c19a370 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00d950 tx=0x7f8c8c00de10 comp rx=0 tx=0).stop 2024-09-21T02:37:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 shutdown_connections 2024-09-21T02:37:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8c840b6620 0x7f8c840b8b30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c9c108d10 0x7f8c9c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c9c1083c0 0x7f8c9c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 --2- 172.21.15.29:0/2254433199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c9c102a80 0x7f8c9c19a370 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:37:59.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 >> 172.21.15.29:0/2254433199 conn(0x7f8c9c079740 msgr2=0x7f8c9c1052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:37:59.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 shutdown_connections 2024-09-21T02:37:59.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.725+0000 7f8ca2045640 1 -- 172.21.15.29:0/2254433199 wait complete. 2024-09-21T02:37:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:59 smithi029 ceph-mon[31006]: pgmap v353: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:37:59.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:37:59 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3962936170' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T02:38:00.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:59 smithi066 ceph-mon[33436]: pgmap v353: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:00.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:37:59 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3962936170' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T02:38:00.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 -- 172.21.15.29:0/2488308755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf0103b60 msgr2=0x7fbaf0103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 --2- 172.21.15.29:0/2488308755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf0103b60 0x7fbaf0103f70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009ca0 tx=0x7fbae002a8e0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 -- 172.21.15.29:0/2488308755 shutdown_connections 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 --2- 172.21.15.29:0/2488308755 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaf0109df0 0x7fbaf0110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 --2- 172.21.15.29:0/2488308755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf01094a0 0x7fbaf01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 --2- 172.21.15.29:0/2488308755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf0103b60 0x7fbaf0103f70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 -- 172.21.15.29:0/2488308755 >> 172.21.15.29:0/2488308755 conn(0x7fbaf007b850 msgr2=0x7fbaf007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 -- 172.21.15.29:0/2488308755 shutdown_connections 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.119+0000 7fbaf728f640 1 -- 172.21.15.29:0/2488308755 wait complete. 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf728f640 1 Processor -- start 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf728f640 1 -- start start 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf728f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf0103b60 0x7fbaf0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:00.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf728f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf01094a0 0x7fbaf0075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf728f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaf0109df0 0x7fbaf0076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf728f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf01139a0 con 0x7fbaf01094a0 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf728f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf0113820 con 0x7fbaf0109df0 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf728f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf0113b20 con 0x7fbaf0103b60 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf482a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf01094a0 0x7fbaf0075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf482a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf01094a0 0x7fbaf0075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48006/0 (socket says 172.21.15.29:48006) 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf482a640 1 -- 172.21.15.29:0/3962936170 learned_addr learned my addr 172.21.15.29:0/3962936170 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.120+0000 7fbaf582c640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaf0109df0 0x7fbaf0076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf502b640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf0103b60 0x7fbaf0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf482a640 1 -- 172.21.15.29:0/3962936170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf0103b60 msgr2=0x7fbaf0075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf482a640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf0103b60 0x7fbaf0075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf482a640 1 -- 172.21.15.29:0/3962936170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaf0109df0 msgr2=0x7fbaf0076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf482a640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaf0109df0 0x7fbaf0076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf482a640 1 -- 172.21.15.29:0/3962936170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf01a8660 con 0x7fbaf01094a0 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf582c640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaf0109df0 0x7fbaf0076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf502b640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf0103b60 0x7fbaf0075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf482a640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf01094a0 0x7fbaf0075c50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006fd0 tx=0x7fbae400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaee7fc640 1 -- 172.21.15.29:0/3962936170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4011830 con 0x7fbaf01094a0 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stdout:150323855424 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf01a8950 con 0x7fbaf01094a0 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaee7fc640 1 -- 172.21.15.29:0/3962936170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbae4016070 con 0x7fbaf01094a0 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaee7fc640 1 -- 172.21.15.29:0/3962936170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4010cf0 con 0x7fbaf01094a0 2024-09-21T02:38:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.121+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf01a8f10 con 0x7fbaf01094a0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.122+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf0103f70 con 0x7fbaf01094a0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.124+0000 7fbaee7fc640 1 -- 172.21.15.29:0/3962936170 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fbae40119d0 con 0x7fbaf01094a0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.125+0000 7fbaee7fc640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbad80b65a0 0x7fbad80b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.125+0000 7fbaf502b640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbad80b65a0 0x7fbad80b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.125+0000 7fbaee7fc640 1 -- 172.21.15.29:0/3962936170 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fbae40e75a0 con 0x7fbaf01094a0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.125+0000 7fbaf502b640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbad80b65a0 0x7fbad80b8ab0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbae0031fd0 tx=0x7fbae0009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.128+0000 7fbaee7fc640 1 -- 172.21.15.29:0/3962936170 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbae40ad910 con 0x7fbaf01094a0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.326+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbaf0076a10 con 0x7fbaf01094a0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.326+0000 7fbaee7fc640 1 -- 172.21.15.29:0/3962936170 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbae40b1cf0 con 0x7fbaf01094a0 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbad80b65a0 msgr2=0x7fbad80b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbad80b65a0 0x7fbad80b8ab0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbae0031fd0 tx=0x7fbae0009870 comp rx=0 tx=0).stop 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf01094a0 msgr2=0x7fbaf0075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf01094a0 0x7fbaf0075c50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006fd0 tx=0x7fbae400b8e0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 shutdown_connections 2024-09-21T02:38:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbad80b65a0 0x7fbad80b8ab0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaf0109df0 0x7fbaf0076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf01094a0 0x7fbaf0075c50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 --2- 172.21.15.29:0/3962936170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf0103b60 0x7fbaf0075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 >> 172.21.15.29:0/3962936170 conn(0x7fbaf007b850 msgr2=0x7fbaf01075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 shutdown_connections 2024-09-21T02:38:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.329+0000 7fbaf728f640 1 -- 172.21.15.29:0/3962936170 wait complete. 2024-09-21T02:38:00.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:59 smithi190 ceph-mon[33871]: pgmap v353: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:00.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:37:59 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3962936170' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T02:38:01.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:00 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2254433199' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T02:38:01.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:00 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3056398737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T02:38:01.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:00 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2254433199' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T02:38:01.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:00 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3056398737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T02:38:01.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:00 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2254433199' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T02:38:01.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:00 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3056398737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T02:38:01.712 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018945 got 128849018948 for osd.4 2024-09-21T02:38:01.713 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 -- 172.21.15.29:0/2661140117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 msgr2=0x7f5acc110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 --2- 172.21.15.29:0/2661140117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 0x7f5acc110300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009900 tx=0x7f5ac402a410 comp rx=0 tx=0).stop 2024-09-21T02:38:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 -- 172.21.15.29:0/2661140117 shutdown_connections 2024-09-21T02:38:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 --2- 172.21.15.29:0/2661140117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 0x7f5acc110300 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 --2- 172.21.15.29:0/2661140117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc105270 0x7f5acc10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 --2- 172.21.15.29:0/2661140117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5acc104840 0x7f5acc104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 -- 172.21.15.29:0/2661140117 >> 172.21.15.29:0/2661140117 conn(0x7f5acc079730 msgr2=0x7f5acc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 -- 172.21.15.29:0/2661140117 shutdown_connections 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.546+0000 7f5ad1490640 1 -- 172.21.15.29:0/2661140117 wait complete. 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.547+0000 7f5ad1490640 1 Processor -- start 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.547+0000 7f5ad1490640 1 -- start start 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.547+0000 7f5ad1490640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc104840 0x7f5acc102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.547+0000 7f5ad1490640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5acc105270 0x7f5acc102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.547+0000 7f5ad1490640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 0x7f5acc1031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.547+0000 7f5ad1490640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc113940 con 0x7f5acc10ddf0 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.547+0000 7f5ad1490640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc1137c0 con 0x7f5acc105270 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.547+0000 7f5ad1490640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc113ac0 con 0x7f5acc104840 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 0x7f5acc1031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:02.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 0x7f5acc1031d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48072/0 (socket says 172.21.15.29:48072) 2024-09-21T02:38:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 -- 172.21.15.29:0/4248327156 learned_addr learned my addr 172.21.15.29:0/4248327156 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:02.008 INFO:teuthology.orchestra.run.smithi029.stdout:201863462966 2024-09-21T02:38:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acaffd640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc104840 0x7f5acc102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5aca7fc640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5acc105270 0x7f5acc102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:02.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 -- 172.21.15.29:0/4248327156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc104840 msgr2=0x7f5acc102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc104840 0x7f5acc102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 -- 172.21.15.29:0/4248327156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5acc105270 msgr2=0x7f5acc102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5acc105270 0x7f5acc102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 -- 172.21.15.29:0/4248327156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5acc1a8690 con 0x7f5acc10ddf0 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5aca7fc640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5acc105270 0x7f5acc102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5acb7fe640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 0x7f5acc1031d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4005ab0 tx=0x7f5ac40040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5ac3fff640 1 -- 172.21.15.29:0/4248327156 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac403f030 con 0x7f5acc10ddf0 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5ac3fff640 1 -- 172.21.15.29:0/4248327156 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ac4003a80 con 0x7f5acc10ddf0 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5acc1a8920 con 0x7f5acc10ddf0 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.548+0000 7f5ac3fff640 1 -- 172.21.15.29:0/4248327156 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4035070 con 0x7f5acc10ddf0 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.549+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5acc1a8e00 con 0x7f5acc10ddf0 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.550+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5acc104c50 con 0x7f5acc10ddf0 2024-09-21T02:38:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.551+0000 7f5ac3fff640 1 -- 172.21.15.29:0/4248327156 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f5ac4042030 con 0x7f5acc10ddf0 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.552+0000 7f5ac3fff640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ab80b65a0 0x7f5ab80b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.553+0000 7f5acaffd640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ab80b65a0 0x7f5ab80b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.553+0000 7f5ac3fff640 1 -- 172.21.15.29:0/4248327156 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f5ac4108160 con 0x7f5acc10ddf0 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.553+0000 7f5acaffd640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ab80b65a0 0x7f5ab80b8ab0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5acc1041a0 tx=0x7f5abc009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.556+0000 7f5ac3fff640 1 -- 172.21.15.29:0/4248327156 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5ac40ce530 con 0x7f5acc10ddf0 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.753+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5acc02e640 con 0x7f5acc10ddf0 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.753+0000 7f5ac3fff640 1 -- 172.21.15.29:0/4248327156 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5ac40d2910 con 0x7f5acc10ddf0 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ab80b65a0 msgr2=0x7f5ab80b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ab80b65a0 0x7f5ab80b8ab0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5acc1041a0 tx=0x7f5abc009670 comp rx=0 tx=0).stop 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 msgr2=0x7f5acc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 0x7f5acc1031d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4005ab0 tx=0x7f5ac40040e0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 shutdown_connections 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ab80b65a0 0x7f5ab80b8ab0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc10ddf0 0x7f5acc1031d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5acc105270 0x7f5acc102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 --2- 172.21.15.29:0/4248327156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc104840 0x7f5acc102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.755+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 >> 172.21.15.29:0/4248327156 conn(0x7f5acc079730 msgr2=0x7f5acc100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.756+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 shutdown_connections 2024-09-21T02:38:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:01.756+0000 7f5ad1490640 1 -- 172.21.15.29:0/4248327156 wait complete. 2024-09-21T02:38:02.011 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:01 smithi029 ceph-mon[31006]: pgmap v354: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:02.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:01 smithi066 ceph-mon[33436]: pgmap v354: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:02.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:01 smithi190 ceph-mon[33871]: pgmap v354: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:02.827 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462964 got 201863462966 for osd.7 2024-09-21T02:38:02.827 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:03.004 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:02 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4248327156' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T02:38:03.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:02 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4248327156' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3896537841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 msgr2=0x7fddb81087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3896537841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 0x7fddb81087d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009900 tx=0x7fdda002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3896537841 shutdown_connections 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3896537841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fddb8108d10 0x7fddb810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3896537841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 0x7fddb81087d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3896537841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddb8102a80 0x7fddb8102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3896537841 >> 172.21.15.29:0/3896537841 conn(0x7fddb8079740 msgr2=0x7fddb8079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3896537841 shutdown_connections 2024-09-21T02:38:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.961+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3896537841 wait complete. 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddbcf9e640 1 Processor -- start 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddbcf9e640 1 -- start start 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddbcf9e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddb8102a80 0x7fddb8199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddbcf9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 0x7fddb8199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddbcf9e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fddb8108d10 0x7fddb819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddbcf9e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fddb8112830 con 0x7fddb81083c0 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddbcf9e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fddb81126b0 con 0x7fddb8108d10 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddbcf9e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fddb81129b0 con 0x7fddb8102a80 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 0x7fddb8199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 0x7fddb8199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48048/0 (socket says 172.21.15.29:48048) 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddaffff640 1 -- 172.21.15.29:0/3056398737 learned_addr learned my addr 172.21.15.29:0/3056398737 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddb659c640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddb8102a80 0x7fddb8199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.963+0000 7fddb6d9d640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fddb8108d10 0x7fddb819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:03.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaffff640 1 -- 172.21.15.29:0/3056398737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddb8102a80 msgr2=0x7fddb8199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaffff640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddb8102a80 0x7fddb8199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaffff640 1 -- 172.21.15.29:0/3056398737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fddb8108d10 msgr2=0x7fddb819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaffff640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fddb8108d10 0x7fddb819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaffff640 1 -- 172.21.15.29:0/3056398737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb819f110 con 0x7fddb81083c0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaffff640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 0x7fddb8199800 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdda0005940 tx=0x7fdda0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaf7fe640 1 -- 172.21.15.29:0/3056398737 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0046070 con 0x7fddb81083c0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaf7fe640 1 -- 172.21.15.29:0/3056398737 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdda0003a80 con 0x7fddb81083c0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddaf7fe640 1 -- 172.21.15.29:0/3056398737 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0035050 con 0x7fddb81083c0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb819f3a0 con 0x7fddb81083c0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.964+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb819f880 con 0x7fddb81083c0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.965+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddb8103d00 con 0x7fddb81083c0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.967+0000 7fddaf7fe640 1 -- 172.21.15.29:0/3056398737 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fdda0049070 con 0x7fddb81083c0 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.968+0000 7fddaf7fe640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdda40b65d0 0x7fdda40b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:03.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.968+0000 7fddb659c640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdda40b65d0 0x7fdda40b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:03.127 INFO:teuthology.orchestra.run.smithi029.stdout:227633266736 2024-09-21T02:38:03.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.968+0000 7fddaf7fe640 1 -- 172.21.15.29:0/3056398737 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fdda0109950 con 0x7fddb81083c0 2024-09-21T02:38:03.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.969+0000 7fddb659c640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdda40b65d0 0x7fdda40b8ae0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdda800abe0 tx=0x7fdda800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:03.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:37:59.972+0000 7fddaf7fe640 1 -- 172.21.15.29:0/3056398737 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdda0050020 con 0x7fddb81083c0 2024-09-21T02:38:03.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.163+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fddb8102e90 con 0x7fddb81083c0 2024-09-21T02:38:03.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.163+0000 7fddaf7fe640 1 -- 172.21.15.29:0/3056398737 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdda00cfd20 con 0x7fddb81083c0 2024-09-21T02:38:03.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdda40b65d0 msgr2=0x7fdda40b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:03.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdda40b65d0 0x7fdda40b8ae0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdda800abe0 tx=0x7fdda800a2d0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 msgr2=0x7fddb8199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 0x7fddb8199800 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdda0005940 tx=0x7fdda0004270 comp rx=0 tx=0).stop 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 shutdown_connections 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdda40b65d0 0x7fdda40b8ae0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fddb8108d10 0x7fddb819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb81083c0 0x7fddb8199800 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 --2- 172.21.15.29:0/3056398737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddb8102a80 0x7fddb8199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 >> 172.21.15.29:0/3056398737 conn(0x7fddb8079740 msgr2=0x7fddb8104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 shutdown_connections 2024-09-21T02:38:03.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:00.166+0000 7fddbcf9e640 1 -- 172.21.15.29:0/3056398737 wait complete. 2024-09-21T02:38:03.129 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:02 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4248327156' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T02:38:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:03 smithi029 ceph-mon[31006]: pgmap v355: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-21T02:38:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:03 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2018078030' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T02:38:04.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:03 smithi066 ceph-mon[33436]: pgmap v355: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-21T02:38:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:03 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2018078030' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T02:38:04.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 -- 172.21.15.29:0/1697879711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38102a80 msgr2=0x7f7e38102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:04.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 --2- 172.21.15.29:0/1697879711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38102a80 0x7f7e38102e90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009900 tx=0x7f7e3002a410 comp rx=0 tx=0).stop 2024-09-21T02:38:04.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 -- 172.21.15.29:0/1697879711 shutdown_connections 2024-09-21T02:38:04.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 --2- 172.21.15.29:0/1697879711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e38108d10 0x7f7e3810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 --2- 172.21.15.29:0/1697879711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e381083c0 0x7f7e381087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 --2- 172.21.15.29:0/1697879711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38102a80 0x7f7e38102e90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 -- 172.21.15.29:0/1697879711 >> 172.21.15.29:0/1697879711 conn(0x7f7e38079740 msgr2=0x7f7e38079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 -- 172.21.15.29:0/1697879711 shutdown_connections 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.091+0000 7f7e402b7640 1 -- 172.21.15.29:0/1697879711 wait complete. 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e402b7640 1 Processor -- start 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e402b7640 1 -- start start 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e402b7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e38102a80 0x7f7e3819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e402b7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e381083c0 0x7f7e3819b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e402b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38108d10 0x7f7e3819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e402b7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7e38112830 con 0x7f7e38108d10 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e402b7640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7e381126b0 con 0x7f7e38102a80 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e402b7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e381129b0 con 0x7f7e381083c0 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.092+0000 7f7e3e854640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38108d10 0x7f7e3819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e854640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38108d10 0x7f7e3819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48098/0 (socket says 172.21.15.29:48098) 2024-09-21T02:38:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e854640 1 -- 172.21.15.29:0/3269568937 learned_addr learned my addr 172.21.15.29:0/3269568937 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e053640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e38102a80 0x7f7e3819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3d852640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e381083c0 0x7f7e3819b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e854640 1 -- 172.21.15.29:0/3269568937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e381083c0 msgr2=0x7f7e3819b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e854640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e381083c0 0x7f7e3819b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e854640 1 -- 172.21.15.29:0/3269568937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e38102a80 msgr2=0x7f7e3819b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e854640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e38102a80 0x7f7e3819b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e854640 1 -- 172.21.15.29:0/3269568937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e381a6450 con 0x7f7e38108d10 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e053640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e38102a80 0x7f7e3819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e3e854640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38108d10 0x7f7e3819bc40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800eb10 tx=0x7f7e2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e377fe640 1 -- 172.21.15.29:0/3269568937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28004b20 con 0x7f7e38108d10 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e381a6740 con 0x7f7e38108d10 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e377fe640 1 -- 172.21.15.29:0/3269568937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e28004cc0 con 0x7f7e38108d10 2024-09-21T02:38:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.093+0000 7f7e377fe640 1 -- 172.21.15.29:0/3269568937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28010690 con 0x7f7e38108d10 2024-09-21T02:38:04.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.094+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e381a6c80 con 0x7f7e38108d10 2024-09-21T02:38:04.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.095+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e38102e90 con 0x7f7e38108d10 2024-09-21T02:38:04.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.097+0000 7f7e377fe640 1 -- 172.21.15.29:0/3269568937 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f7e28010830 con 0x7f7e38108d10 2024-09-21T02:38:04.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.097+0000 7f7e377fe640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e200b65d0 0x7f7e200b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:04.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.098+0000 7f7e3e053640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e200b65d0 0x7f7e200b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:04.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.098+0000 7f7e377fe640 1 -- 172.21.15.29:0/3269568937 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f7e280e80a0 con 0x7f7e38108d10 2024-09-21T02:38:04.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.098+0000 7f7e3e053640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e200b65d0 0x7f7e200b8ae0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e30006010 tx=0x7f7e30005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:04.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.100+0000 7f7e377fe640 1 -- 172.21.15.29:0/3269568937 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7e28056710 con 0x7f7e38108d10 2024-09-21T02:38:04.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:03 smithi190 ceph-mon[33871]: pgmap v355: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-21T02:38:04.126 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:03 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2018078030' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T02:38:04.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.294+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f7e3806bc30 con 0x7f7e38108d10 2024-09-21T02:38:04.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.294+0000 7f7e377fe640 1 -- 172.21.15.29:0/3269568937 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7e3806bc30 con 0x7f7e38108d10 2024-09-21T02:38:04.296 INFO:teuthology.orchestra.run.smithi029.stdout:274877906981 2024-09-21T02:38:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e200b65d0 msgr2=0x7f7e200b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e200b65d0 0x7f7e200b8ae0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e30006010 tx=0x7f7e30005a20 comp rx=0 tx=0).stop 2024-09-21T02:38:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38108d10 msgr2=0x7f7e3819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38108d10 0x7f7e3819bc40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800eb10 tx=0x7f7e2800c2d0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 shutdown_connections 2024-09-21T02:38:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e200b65d0 0x7f7e200b8ae0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e38108d10 0x7f7e3819bc40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e381083c0 0x7f7e3819b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 --2- 172.21.15.29:0/3269568937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e38102a80 0x7f7e3819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 >> 172.21.15.29:0/3269568937 conn(0x7f7e38079740 msgr2=0x7f7e381014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 shutdown_connections 2024-09-21T02:38:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:04.297+0000 7f7e402b7640 1 -- 172.21.15.29:0/3269568937 wait complete. 2024-09-21T02:38:05.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:04 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3269568937' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T02:38:05.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:04 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3269568937' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T02:38:05.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:04 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3269568937' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T02:38:05.157 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266734 got 227633266736 for osd.8 2024-09-21T02:38:05.158 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:05.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 -- 172.21.15.29:0/1495099931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e8102a60 msgr2=0x7f94e8102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:05.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 --2- 172.21.15.29:0/1495099931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e8102a60 0x7f94e8102e70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009900 tx=0x7f94e002a410 comp rx=0 tx=0).stop 2024-09-21T02:38:05.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 -- 172.21.15.29:0/1495099931 shutdown_connections 2024-09-21T02:38:05.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 --2- 172.21.15.29:0/1495099931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e8108cf0 0x7f94e810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 --2- 172.21.15.29:0/1495099931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e81083a0 0x7f94e81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 --2- 172.21.15.29:0/1495099931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e8102a60 0x7f94e8102e70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 -- 172.21.15.29:0/1495099931 >> 172.21.15.29:0/1495099931 conn(0x7f94e8079730 msgr2=0x7f94e8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 -- 172.21.15.29:0/1495099931 shutdown_connections 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.457+0000 7f94eff9b640 1 -- 172.21.15.29:0/1495099931 wait complete. 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.458+0000 7f94eff9b640 1 Processor -- start 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.458+0000 7f94eff9b640 1 -- start start 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.458+0000 7f94eff9b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e8102a60 0x7f94e819a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.458+0000 7f94eff9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e81083a0 0x7f94e819a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94eff9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e8108cf0 0x7f94e819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94eff9b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94e8112810 con 0x7f94e81083a0 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94eff9b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f94e8112690 con 0x7f94e8102a60 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94eff9b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f94e8112990 con 0x7f94e8108cf0 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e81083a0 0x7f94e819a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e81083a0 0x7f94e819a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44924/0 (socket says 172.21.15.29:44924) 2024-09-21T02:38:05.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 -- 172.21.15.29:0/3363651841 learned_addr learned my addr 172.21.15.29:0/3363651841 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ee538640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e8108cf0 0x7f94e819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94edd37640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e8102a60 0x7f94e819a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 -- 172.21.15.29:0/3363651841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e8108cf0 msgr2=0x7f94e819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e8108cf0 0x7f94e819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 -- 172.21.15.29:0/3363651841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e8102a60 msgr2=0x7f94e819a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e8102a60 0x7f94e819a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 -- 172.21.15.29:0/3363651841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e81a0340 con 0x7f94e81083a0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94ed536640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e81083a0 0x7f94e819a970 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00c970 tx=0x7f94dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94e6ffd640 1 -- 172.21.15.29:0/3363651841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc007bb0 con 0x7f94e81083a0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.459+0000 7f94e6ffd640 1 -- 172.21.15.29:0/3363651841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94dc007d50 con 0x7f94e81083a0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.460+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e81a0630 con 0x7f94e81083a0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.460+0000 7f94e6ffd640 1 -- 172.21.15.29:0/3363651841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc018660 con 0x7f94e81083a0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.460+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e81a6d90 con 0x7f94e81083a0 2024-09-21T02:38:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.461+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e8102e70 con 0x7f94e81083a0 2024-09-21T02:38:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.462+0000 7f94e6ffd640 1 -- 172.21.15.29:0/3363651841 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f94dc00f970 con 0x7f94e81083a0 2024-09-21T02:38:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.463+0000 7f94e6ffd640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6620 0x7f94d00b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.463+0000 7f94e6ffd640 1 -- 172.21.15.29:0/3363651841 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f94dc0e7bb0 con 0x7f94e81083a0 2024-09-21T02:38:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.463+0000 7f94edd37640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6620 0x7f94d00b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.464+0000 7f94edd37640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6620 0x7f94d00b8b30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f94e0000c00 tx=0x7f94e0009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.466+0000 7f94e6ffd640 1 -- 172.21.15.29:0/3363651841 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f94dc014030 con 0x7f94e81083a0 2024-09-21T02:38:05.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.659+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f94e81a70e0 con 0x7f94e81083a0 2024-09-21T02:38:05.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.659+0000 7f94e6ffd640 1 -- 172.21.15.29:0/3363651841 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f94dc0ae000 con 0x7f94e81083a0 2024-09-21T02:38:05.660 INFO:teuthology.orchestra.run.smithi029.stdout:176093659195 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6620 msgr2=0x7f94d00b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6620 0x7f94d00b8b30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f94e0000c00 tx=0x7f94e0009680 comp rx=0 tx=0).stop 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e81083a0 msgr2=0x7f94e819a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e81083a0 0x7f94e819a970 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00c970 tx=0x7f94dc00ce30 comp rx=0 tx=0).stop 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 shutdown_connections 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6620 0x7f94d00b8b30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e8108cf0 0x7f94e819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e81083a0 0x7f94e819a970 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 --2- 172.21.15.29:0/3363651841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e8102a60 0x7f94e819a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 >> 172.21.15.29:0/3363651841 conn(0x7f94e8079730 msgr2=0x7f94e8101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:05.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 shutdown_connections 2024-09-21T02:38:05.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:05.662+0000 7f94eff9b640 1 -- 172.21.15.29:0/3363651841 wait complete. 2024-09-21T02:38:06.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:05 smithi066 ceph-mon[33436]: pgmap v356: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:38:06.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:05 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3363651841' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T02:38:06.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:05 smithi190 ceph-mon[33871]: pgmap v356: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:38:06.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:05 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3363651841' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T02:38:06.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:05 smithi029 ceph-mon[31006]: pgmap v356: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:38:06.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:05 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3363651841' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T02:38:06.211 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215174 got 103079215176 for osd.3 2024-09-21T02:38:06.211 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:06.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 -- 172.21.15.29:0/2395501798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1701083d0 msgr2=0x7fe1701087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:06.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2395501798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1701083d0 0x7fe1701087e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe164009900 tx=0x7fe16402a410 comp rx=0 tx=0).stop 2024-09-21T02:38:06.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 -- 172.21.15.29:0/2395501798 shutdown_connections 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2395501798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe170108d20 0x7fe17010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2395501798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1701083d0 0x7fe1701087e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2395501798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe170102ae0 0x7fe170102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 -- 172.21.15.29:0/2395501798 >> 172.21.15.29:0/2395501798 conn(0x7fe170079760 msgr2=0x7fe170079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 -- 172.21.15.29:0/2395501798 shutdown_connections 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 -- 172.21.15.29:0/2395501798 wait complete. 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.111+0000 7fe1765f8640 1 Processor -- start 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe1765f8640 1 -- start start 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe1765f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe170102ae0 0x7fe170199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe1765f8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe1701083d0 0x7fe1701998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe1765f8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe170108d20 0x7fe17019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe1765f8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe170112840 con 0x7fe170102ae0 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe1765f8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701126c0 con 0x7fe1701083d0 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe1765f8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701129c0 con 0x7fe170108d20 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe170102ae0 0x7fe170199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe170102ae0 0x7fe170199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48084/0 (socket says 172.21.15.29:48084) 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe16ffff640 1 -- 172.21.15.29:0/2018078030 learned_addr learned my addr 172.21.15.29:0/2018078030 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe16f7fe640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe1701083d0 0x7fe1701998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.112+0000 7fe174b95640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe170108d20 0x7fe17019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16ffff640 1 -- 172.21.15.29:0/2018078030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe170108d20 msgr2=0x7fe17019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16ffff640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe170108d20 0x7fe17019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16ffff640 1 -- 172.21.15.29:0/2018078030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe1701083d0 msgr2=0x7fe1701998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16ffff640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe1701083d0 0x7fe1701998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16ffff640 1 -- 172.21.15.29:0/2018078030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17019f2a0 con 0x7fe170102ae0 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16ffff640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe170102ae0 0x7fe170199330 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe16000d950 tx=0x7fe16000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16d7fa640 1 -- 172.21.15.29:0/2018078030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160017070 con 0x7fe170102ae0 2024-09-21T02:38:06.553 INFO:teuthology.orchestra.run.smithi029.stdout:34359738459 2024-09-21T02:38:06.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17019f590 con 0x7fe170102ae0 2024-09-21T02:38:06.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16d7fa640 1 -- 172.21.15.29:0/2018078030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe160009c30 con 0x7fe170102ae0 2024-09-21T02:38:06.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe16d7fa640 1 -- 172.21.15.29:0/2018078030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16000f620 con 0x7fe170102ae0 2024-09-21T02:38:06.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.113+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1701a5cf0 con 0x7fe170102ae0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.114+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe170103d10 con 0x7fe170102ae0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.116+0000 7fe16d7fa640 1 -- 172.21.15.29:0/2018078030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fe160005030 con 0x7fe170102ae0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.117+0000 7fe16d7fa640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1580b65a0 0x7fe1580b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.117+0000 7fe16f7fe640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1580b65a0 0x7fe1580b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.117+0000 7fe16d7fa640 1 -- 172.21.15.29:0/2018078030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fe1600e7370 con 0x7fe170102ae0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.117+0000 7fe16f7fe640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1580b65a0 0x7fe1580b8ab0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe164004770 tx=0x7fe1640058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.120+0000 7fe16d7fa640 1 -- 172.21.15.29:0/2018078030 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe160009de0 con 0x7fe170102ae0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.312+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe170102ef0 con 0x7fe170102ae0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.312+0000 7fe16d7fa640 1 -- 172.21.15.29:0/2018078030 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe1600b1b20 con 0x7fe170102ae0 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1580b65a0 msgr2=0x7fe1580b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1580b65a0 0x7fe1580b8ab0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe164004770 tx=0x7fe1640058d0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe170102ae0 msgr2=0x7fe170199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe170102ae0 0x7fe170199330 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe16000d950 tx=0x7fe16000de10 comp rx=0 tx=0).stop 2024-09-21T02:38:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 shutdown_connections 2024-09-21T02:38:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1580b65a0 0x7fe1580b8ab0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe170108d20 0x7fe17019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe1701083d0 0x7fe1701998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 --2- 172.21.15.29:0/2018078030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe170102ae0 0x7fe170199330 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 >> 172.21.15.29:0/2018078030 conn(0x7fe170079760 msgr2=0x7fe1701044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 shutdown_connections 2024-09-21T02:38:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:03.314+0000 7fe1765f8640 1 -- 172.21.15.29:0/2018078030 wait complete. 2024-09-21T02:38:07.392 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738455 got 34359738459 for osd.0 2024-09-21T02:38:07.392 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:07.871 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906979 got 274877906981 for osd.10 2024-09-21T02:38:07.872 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:08.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:07 smithi066 ceph-mon[33436]: pgmap v357: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T02:38:08.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:07 smithi029 ceph-mon[31006]: pgmap v357: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 -- 172.21.15.29:0/2917660216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2141049b0 msgr2=0x7fa214104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 --2- 172.21.15.29:0/2917660216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2141049b0 0x7fa214104dc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa210009900 tx=0x7fa21002a410 comp rx=0 tx=0).stop 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 -- 172.21.15.29:0/2917660216 shutdown_connections 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 --2- 172.21.15.29:0/2917660216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa21410de10 0x7fa214110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 --2- 172.21.15.29:0/2917660216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa214105300 0x7fa21410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 --2- 172.21.15.29:0/2917660216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2141049b0 0x7fa214104dc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 -- 172.21.15.29:0/2917660216 >> 172.21.15.29:0/2917660216 conn(0x7fa214079730 msgr2=0x7fa214079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 -- 172.21.15.29:0/2917660216 shutdown_connections 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.056+0000 7fa21bea8640 1 -- 172.21.15.29:0/2917660216 wait complete. 2024-09-21T02:38:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.057+0000 7fa21bea8640 1 Processor -- start 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.057+0000 7fa21bea8640 1 -- start start 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.057+0000 7fa21bea8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2141049b0 0x7fa2141026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.057+0000 7fa21bea8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa214105300 0x7fa214102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.057+0000 7fa21bea8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21410de10 0x7fa214103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.057+0000 7fa21bea8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa214113980 con 0x7fa21410de10 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.057+0000 7fa21bea8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa214113800 con 0x7fa214105300 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.057+0000 7fa21bea8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa214113b00 con 0x7fa2141049b0 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21410de10 0x7fa214103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21410de10 0x7fa214103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44946/0 (socket says 172.21.15.29:44946) 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 -- 172.21.15.29:0/4268844737 learned_addr learned my addr 172.21.15.29:0/4268844737 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa219c44640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2141049b0 0x7fa2141026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa219443640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa214105300 0x7fa214102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 -- 172.21.15.29:0/4268844737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2141049b0 msgr2=0x7fa2141026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2141049b0 0x7fa2141026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 -- 172.21.15.29:0/4268844737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa214105300 msgr2=0x7fa214102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa214105300 0x7fa214102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 -- 172.21.15.29:0/4268844737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2141a86a0 con 0x7fa21410de10 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa219443640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa214105300 0x7fa214102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21a445640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21410de10 0x7fa214103170 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa20c00cce0 tx=0x7fa20c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa20affd640 1 -- 172.21.15.29:0/4268844737 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c00f450 con 0x7fa21410de10 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2141a8990 con 0x7fa21410de10 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa20affd640 1 -- 172.21.15.29:0/4268844737 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa20c00fa80 con 0x7fa21410de10 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.058+0000 7fa20affd640 1 -- 172.21.15.29:0/4268844737 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c018ad0 con 0x7fa21410de10 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.059+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2141a8e80 con 0x7fa21410de10 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.059+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa214104dc0 con 0x7fa21410de10 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.061+0000 7fa20affd640 1 -- 172.21.15.29:0/4268844737 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fa20c00f5f0 con 0x7fa21410de10 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.062+0000 7fa20affd640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2000b65a0 0x7fa2000b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.062+0000 7fa219c44640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2000b65a0 0x7fa2000b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:08.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.062+0000 7fa20affd640 1 -- 172.21.15.29:0/4268844737 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fa20c0e8130 con 0x7fa21410de10 2024-09-21T02:38:08.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.063+0000 7fa219c44640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2000b65a0 0x7fa2000b8ab0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa2100057c0 tx=0x7fa210009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:08.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.065+0000 7fa20affd640 1 -- 172.21.15.29:0/4268844737 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa20c014030 con 0x7fa21410de10 2024-09-21T02:38:08.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:07 smithi190 ceph-mon[33871]: pgmap v357: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T02:38:08.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.262+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fa2141039f0 con 0x7fa21410de10 2024-09-21T02:38:08.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.262+0000 7fa20affd640 1 -- 172.21.15.29:0/4268844737 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa2141039f0 con 0x7fa21410de10 2024-09-21T02:38:08.263 INFO:teuthology.orchestra.run.smithi029.stdout:296352743457 2024-09-21T02:38:08.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2000b65a0 msgr2=0x7fa2000b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:08.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2000b65a0 0x7fa2000b8ab0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa2100057c0 tx=0x7fa210009680 comp rx=0 tx=0).stop 2024-09-21T02:38:08.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21410de10 msgr2=0x7fa214103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:08.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21410de10 0x7fa214103170 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa20c00cce0 tx=0x7fa20c00a620 comp rx=0 tx=0).stop 2024-09-21T02:38:08.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 shutdown_connections 2024-09-21T02:38:08.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa2000b65a0 0x7fa2000b8ab0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21410de10 0x7fa214103170 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa214105300 0x7fa214102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.264+0000 7fa21bea8640 1 --2- 172.21.15.29:0/4268844737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2141049b0 0x7fa2141026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:08.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.265+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 >> 172.21.15.29:0/4268844737 conn(0x7fa214079730 msgr2=0x7fa214100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:08.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.265+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 shutdown_connections 2024-09-21T02:38:08.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:08.265+0000 7fa21bea8640 1 -- 172.21.15.29:0/4268844737 wait complete. 2024-09-21T02:38:08.704 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574929 got 55834574931 for osd.1 2024-09-21T02:38:08.705 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:08.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:08 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4268844737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T02:38:09.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:08 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4268844737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T02:38:09.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:08 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4268844737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T02:38:09.599 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855422 got 150323855424 for osd.5 2024-09-21T02:38:09.599 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:09 smithi029 ceph-mon[31006]: pgmap v358: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-21T02:38:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:09 smithi066 ceph-mon[33436]: pgmap v358: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-21T02:38:10.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:09 smithi190 ceph-mon[33871]: pgmap v358: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-21T02:38:10.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743453 got 296352743457 for osd.11 2024-09-21T02:38:10.162 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:10.635 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659192 got 176093659195 for osd.6 2024-09-21T02:38:10.636 DEBUG:teuthology.parallel:result is None 2024-09-21T02:38:10.636 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T02:38:10.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:38:10.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:10 smithi029 ceph-mon[31006]: pgmap v359: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:11.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:10 smithi066 ceph-mon[33436]: pgmap v359: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:11.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:10 smithi190 ceph-mon[33871]: pgmap v359: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 -- 172.21.15.29:0/1603667351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 msgr2=0x7f6410103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 --2- 172.21.15.29:0/1603667351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 0x7f6410103f70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6400009900 tx=0x7f640002a410 comp rx=0 tx=0).stop 2024-09-21T02:38:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 -- 172.21.15.29:0/1603667351 shutdown_connections 2024-09-21T02:38:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 --2- 172.21.15.29:0/1603667351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6410109df0 0x7f6410110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 --2- 172.21.15.29:0/1603667351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64101094a0 0x7f64101098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 --2- 172.21.15.29:0/1603667351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 0x7f6410103f70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 -- 172.21.15.29:0/1603667351 >> 172.21.15.29:0/1603667351 conn(0x7f641007b850 msgr2=0x7f641007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 -- 172.21.15.29:0/1603667351 shutdown_connections 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 -- 172.21.15.29:0/1603667351 wait complete. 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.415+0000 7f641535d640 1 Processor -- start 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f641535d640 1 -- start start 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f641535d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 0x7f641019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f641535d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64101094a0 0x7f641019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f641535d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6410109df0 0x7f641019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f641535d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6410113960 con 0x7f6410103b60 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f641535d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f64101137e0 con 0x7f6410109df0 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f641535d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6410113ae0 con 0x7f64101094a0 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 0x7f641019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 0x7f641019a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44964/0 (socket says 172.21.15.29:44964) 2024-09-21T02:38:11.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640effd640 1 -- 172.21.15.29:0/2436002162 learned_addr learned my addr 172.21.15.29:0/2436002162 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640f7fe640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6410109df0 0x7f641019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640e7fc640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64101094a0 0x7f641019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640effd640 1 -- 172.21.15.29:0/2436002162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64101094a0 msgr2=0x7f641019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640effd640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64101094a0 0x7f641019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640effd640 1 -- 172.21.15.29:0/2436002162 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6410109df0 msgr2=0x7f641019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640effd640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6410109df0 0x7f641019fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.416+0000 7f640effd640 1 -- 172.21.15.29:0/2436002162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64101a0320 con 0x7f6410103b60 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.417+0000 7f640effd640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 0x7f641019a3b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6400009230 tx=0x7f6400004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.417+0000 7f6407fff640 1 -- 172.21.15.29:0/2436002162 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640003f030 con 0x7f6410103b60 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.417+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64101a05b0 con 0x7f6410103b60 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.417+0000 7f6407fff640 1 -- 172.21.15.29:0/2436002162 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6400003a80 con 0x7f6410103b60 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.417+0000 7f6407fff640 1 -- 172.21.15.29:0/2436002162 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6400035070 con 0x7f6410103b60 2024-09-21T02:38:11.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.417+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64101a6d70 con 0x7f6410103b60 2024-09-21T02:38:11.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.418+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6410103f70 con 0x7f6410103b60 2024-09-21T02:38:11.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.421+0000 7f6407fff640 1 -- 172.21.15.29:0/2436002162 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f6400042030 con 0x7f6410103b60 2024-09-21T02:38:11.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.421+0000 7f6407fff640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f63fc0b65a0 0x7f63fc0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:11.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.422+0000 7f640e7fc640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f63fc0b65a0 0x7f63fc0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:11.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.422+0000 7f6407fff640 1 -- 172.21.15.29:0/2436002162 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f6400108380 con 0x7f6410103b60 2024-09-21T02:38:11.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.422+0000 7f640e7fc640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f63fc0b65a0 0x7f63fc0b8ab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005fd0 tx=0x7f63f8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:11.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.425+0000 7f6407fff640 1 -- 172.21.15.29:0/2436002162 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f64000ce750 con 0x7f6410103b60 2024-09-21T02:38:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.608+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f641010bbf0 con 0x7f63fc0b65a0 2024-09-21T02:38:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.615+0000 7f6407fff640 1 -- 172.21.15.29:0/2436002162 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340650 (secure 0 0 0) 0x7f641010bbf0 con 0x7f63fc0b65a0 2024-09-21T02:38:11.619 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:11.620 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T02:38:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f63fc0b65a0 msgr2=0x7f63fc0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f63fc0b65a0 0x7f63fc0b8ab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005fd0 tx=0x7f63f8005850 comp rx=0 tx=0).stop 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 msgr2=0x7f641019a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 0x7f641019a3b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6400009230 tx=0x7f6400004270 comp rx=0 tx=0).stop 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 shutdown_connections 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f63fc0b65a0 0x7f63fc0b8ab0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6410109df0 0x7f641019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64101094a0 0x7f641019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 --2- 172.21.15.29:0/2436002162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6410103b60 0x7f641019a3b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 >> 172.21.15.29:0/2436002162 conn(0x7f641007b850 msgr2=0x7f64101075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.618+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 shutdown_connections 2024-09-21T02:38:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:11.619+0000 7f641535d640 1 -- 172.21.15.29:0/2436002162 wait complete. 2024-09-21T02:38:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:11 smithi029 ceph-mon[31006]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:38:12.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:11 smithi066 ceph-mon[33436]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:38:12.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:11 smithi190 ceph-mon[33871]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:38:12.532 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-21T02:38:09.957512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665426,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":696,"num_read_kb":456,"num_write":377,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":2952069,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364796,"kb_used_data":11312,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124577028,"statfs":{"total":1151940427776,"available":1151566876672,"internally_reserved":0,"allocated":11583488,"data_stored":7563152,"data_compressed":391470,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":74,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003614"},"pg_stats":[{"pgid":"2.1f","version":"81'13","reported_seq":107,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:06.129508+0000","last_change":"2024-09-21T02:36:23.729337+0000","last_active":"2024-09-21T02:38:06.129508+0000","last_peered":"2024-09-21T02:38:06.129508+0000","last_clean":"2024-09-21T02:38:06.129508+0000","last_became_active":"2024-09-21T02:36:23.728576+0000","last_became_peered":"2024-09-21T02:36:23.728576+0000","last_unstale":"2024-09-21T02:38:06.129508+0000","last_undegraded":"2024-09-21T02:38:06.129508+0000","last_fullsized":"2024-09-21T02:38:06.129508+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:31.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":59,"num_read_kb":59,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866296+0000","last_change":"2024-09-21T02:36:25.748925+0000","last_active":"2024-09-21T02:37:39.866296+0000","last_peered":"2024-09-21T02:37:39.866296+0000","last_clean":"2024-09-21T02:37:39.866296+0000","last_became_active":"2024-09-21T02:36:25.748333+0000","last_became_peered":"2024-09-21T02:36:25.748333+0000","last_unstale":"2024-09-21T02:37:39.866296+0000","last_undegraded":"2024-09-21T02:37:39.866296+0000","last_fullsized":"2024-09-21T02:37:39.866296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:46.889497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602004+0000","last_change":"2024-09-21T02:36:32.807771+0000","last_active":"2024-09-21T02:37:39.602004+0000","last_peered":"2024-09-21T02:37:39.602004+0000","last_clean":"2024-09-21T02:37:39.602004+0000","last_became_active":"2024-09-21T02:36:32.807291+0000","last_became_peered":"2024-09-21T02:36:32.807291+0000","last_unstale":"2024-09-21T02:37:39.602004+0000","last_undegraded":"2024-09-21T02:37:39.602004+0000","last_fullsized":"2024-09-21T02:37:39.602004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:41.468318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866613+0000","last_change":"2024-09-21T02:36:23.800779+0000","last_active":"2024-09-21T02:37:39.866613+0000","last_peered":"2024-09-21T02:37:39.866613+0000","last_clean":"2024-09-21T02:37:39.866613+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:37:39.866613+0000","last_undegraded":"2024-09-21T02:37:39.866613+0000","last_fullsized":"2024-09-21T02:37:39.866613+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:24.843233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608328+0000","last_change":"2024-09-21T02:36:25.745277+0000","last_active":"2024-09-21T02:37:39.608328+0000","last_peered":"2024-09-21T02:37:39.608328+0000","last_clean":"2024-09-21T02:37:39.608328+0000","last_became_active":"2024-09-21T02:36:25.744501+0000","last_became_peered":"2024-09-21T02:36:25.744501+0000","last_unstale":"2024-09-21T02:37:39.608328+0000","last_undegraded":"2024-09-21T02:37:39.608328+0000","last_fullsized":"2024-09-21T02:37:39.608328+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:14.776250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603390+0000","last_change":"2024-09-21T02:36:32.801978+0000","last_active":"2024-09-21T02:37:39.603390+0000","last_peered":"2024-09-21T02:37:39.603390+0000","last_clean":"2024-09-21T02:37:39.603390+0000","last_became_active":"2024-09-21T02:36:32.801436+0000","last_became_peered":"2024-09-21T02:36:32.801436+0000","last_unstale":"2024-09-21T02:37:39.603390+0000","last_undegraded":"2024-09-21T02:37:39.603390+0000","last_fullsized":"2024-09-21T02:37:39.603390+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:37.380417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606474+0000","last_change":"2024-09-21T02:36:23.729336+0000","last_active":"2024-09-21T02:37:39.606474+0000","last_peered":"2024-09-21T02:37:39.606474+0000","last_clean":"2024-09-21T02:37:39.606474+0000","last_became_active":"2024-09-21T02:36:23.728724+0000","last_became_peered":"2024-09-21T02:36:23.728724+0000","last_unstale":"2024-09-21T02:37:39.606474+0000","last_undegraded":"2024-09-21T02:37:39.606474+0000","last_fullsized":"2024-09-21T02:37:39.606474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:31:52.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867454+0000","last_change":"2024-09-21T02:36:25.737577+0000","last_active":"2024-09-21T02:37:39.867454+0000","last_peered":"2024-09-21T02:37:39.867454+0000","last_clean":"2024-09-21T02:37:39.867454+0000","last_became_active":"2024-09-21T02:36:25.737221+0000","last_became_peered":"2024-09-21T02:36:25.737221+0000","last_unstale":"2024-09-21T02:37:39.867454+0000","last_undegraded":"2024-09-21T02:37:39.867454+0000","last_fullsized":"2024-09-21T02:37:39.867454+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:42:54.786464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604191+0000","last_change":"2024-09-21T02:36:32.802507+0000","last_active":"2024-09-21T02:37:39.604191+0000","last_peered":"2024-09-21T02:37:39.604191+0000","last_clean":"2024-09-21T02:37:39.604191+0000","last_became_active":"2024-09-21T02:36:32.801879+0000","last_became_peered":"2024-09-21T02:36:32.801879+0000","last_unstale":"2024-09-21T02:37:39.604191+0000","last_undegraded":"2024-09-21T02:37:39.604191+0000","last_fullsized":"2024-09-21T02:37:39.604191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:09.873825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605829+0000","last_change":"2024-09-21T02:36:23.730889+0000","last_active":"2024-09-21T02:37:39.605829+0000","last_peered":"2024-09-21T02:37:39.605829+0000","last_clean":"2024-09-21T02:37:39.605829+0000","last_became_active":"2024-09-21T02:36:23.730551+0000","last_became_peered":"2024-09-21T02:36:23.730551+0000","last_unstale":"2024-09-21T02:37:39.605829+0000","last_undegraded":"2024-09-21T02:37:39.605829+0000","last_fullsized":"2024-09-21T02:37:39.605829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:17.178848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.869371+0000","last_change":"2024-09-21T02:37:39.869272+0000","last_active":"2024-09-21T02:37:39.869371+0000","last_peered":"2024-09-21T02:37:39.869371+0000","last_clean":"2024-09-21T02:37:39.869371+0000","last_became_active":"2024-09-21T02:37:39.868824+0000","last_became_peered":"2024-09-21T02:37:39.868824+0000","last_unstale":"2024-09-21T02:37:39.869371+0000","last_undegraded":"2024-09-21T02:37:39.869371+0000","last_fullsized":"2024-09-21T02:37:39.869371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:05:31.503359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603327+0000","last_change":"2024-09-21T02:36:32.800489+0000","last_active":"2024-09-21T02:37:39.603327+0000","last_peered":"2024-09-21T02:37:39.603327+0000","last_clean":"2024-09-21T02:37:39.603327+0000","last_became_active":"2024-09-21T02:36:32.799876+0000","last_became_peered":"2024-09-21T02:36:32.799876+0000","last_unstale":"2024-09-21T02:37:39.603327+0000","last_undegraded":"2024-09-21T02:37:39.603327+0000","last_fullsized":"2024-09-21T02:37:39.603327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:02:56.711257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604876+0000","last_change":"2024-09-21T02:36:25.744561+0000","last_active":"2024-09-21T02:37:39.604876+0000","last_peered":"2024-09-21T02:37:39.604876+0000","last_clean":"2024-09-21T02:37:39.604876+0000","last_became_active":"2024-09-21T02:36:25.743757+0000","last_became_peered":"2024-09-21T02:36:25.743757+0000","last_unstale":"2024-09-21T02:37:39.604876+0000","last_undegraded":"2024-09-21T02:37:39.604876+0000","last_fullsized":"2024-09-21T02:37:39.604876+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:30.235517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.616747+0000","last_change":"2024-09-21T02:37:39.616628+0000","last_active":"2024-09-21T02:37:39.616747+0000","last_peered":"2024-09-21T02:37:39.616747+0000","last_clean":"2024-09-21T02:37:39.616747+0000","last_became_active":"2024-09-21T02:37:39.609108+0000","last_became_peered":"2024-09-21T02:37:39.609108+0000","last_unstale":"2024-09-21T02:37:39.616747+0000","last_undegraded":"2024-09-21T02:37:39.616747+0000","last_fullsized":"2024-09-21T02:37:39.616747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:11:28.944244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603713+0000","last_change":"2024-09-21T02:36:33.003670+0000","last_active":"2024-09-21T02:37:39.603713+0000","last_peered":"2024-09-21T02:37:39.603713+0000","last_clean":"2024-09-21T02:37:39.603713+0000","last_became_active":"2024-09-21T02:36:33.003022+0000","last_became_peered":"2024-09-21T02:36:33.003022+0000","last_unstale":"2024-09-21T02:37:39.603713+0000","last_undegraded":"2024-09-21T02:37:39.603713+0000","last_fullsized":"2024-09-21T02:37:39.603713+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:12.753375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866246+0000","last_change":"2024-09-21T02:36:25.749404+0000","last_active":"2024-09-21T02:37:39.866246+0000","last_peered":"2024-09-21T02:37:39.866246+0000","last_clean":"2024-09-21T02:37:39.866246+0000","last_became_active":"2024-09-21T02:36:25.747854+0000","last_became_peered":"2024-09-21T02:36:25.747854+0000","last_unstale":"2024-09-21T02:37:39.866246+0000","last_undegraded":"2024-09-21T02:37:39.866246+0000","last_fullsized":"2024-09-21T02:37:39.866246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:41.224740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602302+0000","last_change":"2024-09-21T02:36:23.726689+0000","last_active":"2024-09-21T02:37:39.602302+0000","last_peered":"2024-09-21T02:37:39.602302+0000","last_clean":"2024-09-21T02:37:39.602302+0000","last_became_active":"2024-09-21T02:36:23.726165+0000","last_became_peered":"2024-09-21T02:36:23.726165+0000","last_unstale":"2024-09-21T02:37:39.602302+0000","last_undegraded":"2024-09-21T02:37:39.602302+0000","last_fullsized":"2024-09-21T02:37:39.602302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:58.166457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604112+0000","last_change":"2024-09-21T02:36:32.809594+0000","last_active":"2024-09-21T02:37:39.604112+0000","last_peered":"2024-09-21T02:37:39.604112+0000","last_clean":"2024-09-21T02:37:39.604112+0000","last_became_active":"2024-09-21T02:36:32.808893+0000","last_became_peered":"2024-09-21T02:36:32.808893+0000","last_unstale":"2024-09-21T02:37:39.604112+0000","last_undegraded":"2024-09-21T02:37:39.604112+0000","last_fullsized":"2024-09-21T02:37:39.604112+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:55.187241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605126+0000","last_change":"2024-09-21T02:36:25.755955+0000","last_active":"2024-09-21T02:37:39.605126+0000","last_peered":"2024-09-21T02:37:39.605126+0000","last_clean":"2024-09-21T02:37:39.605126+0000","last_became_active":"2024-09-21T02:36:25.755577+0000","last_became_peered":"2024-09-21T02:36:25.755577+0000","last_unstale":"2024-09-21T02:37:39.605126+0000","last_undegraded":"2024-09-21T02:37:39.605126+0000","last_fullsized":"2024-09-21T02:37:39.605126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:48.447758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866328+0000","last_change":"2024-09-21T02:36:23.730043+0000","last_active":"2024-09-21T02:37:39.866328+0000","last_peered":"2024-09-21T02:37:39.866328+0000","last_clean":"2024-09-21T02:37:39.866328+0000","last_became_active":"2024-09-21T02:36:23.729576+0000","last_became_peered":"2024-09-21T02:36:23.729576+0000","last_unstale":"2024-09-21T02:37:39.866328+0000","last_undegraded":"2024-09-21T02:37:39.866328+0000","last_fullsized":"2024-09-21T02:37:39.866328+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:48:09.530913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601976+0000","last_change":"2024-09-21T02:36:32.806923+0000","last_active":"2024-09-21T02:37:39.601976+0000","last_peered":"2024-09-21T02:37:39.601976+0000","last_clean":"2024-09-21T02:37:39.601976+0000","last_became_active":"2024-09-21T02:36:32.806176+0000","last_became_peered":"2024-09-21T02:36:32.806176+0000","last_unstale":"2024-09-21T02:37:39.601976+0000","last_undegraded":"2024-09-21T02:37:39.601976+0000","last_fullsized":"2024-09-21T02:37:39.601976+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.165638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605188+0000","last_change":"2024-09-21T02:36:25.747501+0000","last_active":"2024-09-21T02:37:39.605188+0000","last_peered":"2024-09-21T02:37:39.605188+0000","last_clean":"2024-09-21T02:37:39.605188+0000","last_became_active":"2024-09-21T02:36:25.747111+0000","last_became_peered":"2024-09-21T02:36:25.747111+0000","last_unstale":"2024-09-21T02:37:39.605188+0000","last_undegraded":"2024-09-21T02:37:39.605188+0000","last_fullsized":"2024-09-21T02:37:39.605188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:03:04.061537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866583+0000","last_change":"2024-09-21T02:36:23.800802+0000","last_active":"2024-09-21T02:37:39.866583+0000","last_peered":"2024-09-21T02:37:39.866583+0000","last_clean":"2024-09-21T02:37:39.866583+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:37:39.866583+0000","last_undegraded":"2024-09-21T02:37:39.866583+0000","last_fullsized":"2024-09-21T02:37:39.866583+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:44:34.165406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.870161+0000","last_change":"2024-09-21T02:37:39.870041+0000","last_active":"2024-09-21T02:37:39.870161+0000","last_peered":"2024-09-21T02:37:39.870161+0000","last_clean":"2024-09-21T02:37:39.870161+0000","last_became_active":"2024-09-21T02:37:39.869518+0000","last_became_peered":"2024-09-21T02:37:39.869518+0000","last_unstale":"2024-09-21T02:37:39.870161+0000","last_undegraded":"2024-09-21T02:37:39.870161+0000","last_fullsized":"2024-09-21T02:37:39.870161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:20.276471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608088+0000","last_change":"2024-09-21T02:36:25.746139+0000","last_active":"2024-09-21T02:37:39.608088+0000","last_peered":"2024-09-21T02:37:39.608088+0000","last_clean":"2024-09-21T02:37:39.608088+0000","last_became_active":"2024-09-21T02:36:25.744418+0000","last_became_peered":"2024-09-21T02:36:25.744418+0000","last_unstale":"2024-09-21T02:37:39.608088+0000","last_undegraded":"2024-09-21T02:37:39.608088+0000","last_fullsized":"2024-09-21T02:37:39.608088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:21.706782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603890+0000","last_change":"2024-09-21T02:36:24.261945+0000","last_active":"2024-09-21T02:37:39.603890+0000","last_peered":"2024-09-21T02:37:39.603890+0000","last_clean":"2024-09-21T02:37:39.603890+0000","last_became_active":"2024-09-21T02:36:24.261486+0000","last_became_peered":"2024-09-21T02:36:24.261486+0000","last_unstale":"2024-09-21T02:37:39.603890+0000","last_undegraded":"2024-09-21T02:37:39.603890+0000","last_fullsized":"2024-09-21T02:37:39.603890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:45:18.331035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602209+0000","last_change":"2024-09-21T02:36:32.799464+0000","last_active":"2024-09-21T02:37:39.602209+0000","last_peered":"2024-09-21T02:37:39.602209+0000","last_clean":"2024-09-21T02:37:39.602209+0000","last_became_active":"2024-09-21T02:36:32.799000+0000","last_became_peered":"2024-09-21T02:36:32.799000+0000","last_unstale":"2024-09-21T02:37:39.602209+0000","last_undegraded":"2024-09-21T02:37:39.602209+0000","last_fullsized":"2024-09-21T02:37:39.602209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:56:36.271172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866824+0000","last_change":"2024-09-21T02:36:25.741151+0000","last_active":"2024-09-21T02:37:39.866824+0000","last_peered":"2024-09-21T02:37:39.866824+0000","last_clean":"2024-09-21T02:37:39.866824+0000","last_became_active":"2024-09-21T02:36:25.740803+0000","last_became_peered":"2024-09-21T02:36:25.740803+0000","last_unstale":"2024-09-21T02:37:39.866824+0000","last_undegraded":"2024-09-21T02:37:39.866824+0000","last_fullsized":"2024-09-21T02:37:39.866824+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:44.585136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'16","reported_seq":51,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601919+0000","last_change":"2024-09-21T02:36:23.725155+0000","last_active":"2024-09-21T02:37:39.601919+0000","last_peered":"2024-09-21T02:37:39.601919+0000","last_clean":"2024-09-21T02:37:39.601919+0000","last_became_active":"2024-09-21T02:36:23.724791+0000","last_became_peered":"2024-09-21T02:36:23.724791+0000","last_unstale":"2024-09-21T02:37:39.601919+0000","last_undegraded":"2024-09-21T02:37:39.601919+0000","last_fullsized":"2024-09-21T02:37:39.601919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:06.520465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.865963+0000","last_change":"2024-09-21T02:36:33.003401+0000","last_active":"2024-09-21T02:37:39.865963+0000","last_peered":"2024-09-21T02:37:39.865963+0000","last_clean":"2024-09-21T02:37:39.865963+0000","last_became_active":"2024-09-21T02:36:33.002930+0000","last_became_peered":"2024-09-21T02:36:33.002930+0000","last_unstale":"2024-09-21T02:37:39.865963+0000","last_undegraded":"2024-09-21T02:37:39.865963+0000","last_fullsized":"2024-09-21T02:37:39.865963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:40:23.629763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605001+0000","last_change":"2024-09-21T02:36:25.735501+0000","last_active":"2024-09-21T02:37:39.605001+0000","last_peered":"2024-09-21T02:37:39.605001+0000","last_clean":"2024-09-21T02:37:39.605001+0000","last_became_active":"2024-09-21T02:36:25.734453+0000","last_became_peered":"2024-09-21T02:36:25.734453+0000","last_unstale":"2024-09-21T02:37:39.605001+0000","last_undegraded":"2024-09-21T02:37:39.605001+0000","last_fullsized":"2024-09-21T02:37:39.605001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:50.238529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606569+0000","last_change":"2024-09-21T02:36:23.800338+0000","last_active":"2024-09-21T02:37:39.606569+0000","last_peered":"2024-09-21T02:37:39.606569+0000","last_clean":"2024-09-21T02:37:39.606569+0000","last_became_active":"2024-09-21T02:36:23.799796+0000","last_became_peered":"2024-09-21T02:36:23.799796+0000","last_unstale":"2024-09-21T02:37:39.606569+0000","last_undegraded":"2024-09-21T02:37:39.606569+0000","last_fullsized":"2024-09-21T02:37:39.606569+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:00:18.459515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.611641+0000","last_change":"2024-09-21T02:37:39.611513+0000","last_active":"2024-09-21T02:37:39.611641+0000","last_peered":"2024-09-21T02:37:39.611641+0000","last_clean":"2024-09-21T02:37:39.611641+0000","last_became_active":"2024-09-21T02:37:39.610843+0000","last_became_peered":"2024-09-21T02:37:39.610843+0000","last_unstale":"2024-09-21T02:37:39.611641+0000","last_undegraded":"2024-09-21T02:37:39.611641+0000","last_fullsized":"2024-09-21T02:37:39.611641+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:51.911830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.868107+0000","last_change":"2024-09-21T02:36:25.745497+0000","last_active":"2024-09-21T02:37:39.868107+0000","last_peered":"2024-09-21T02:37:39.868107+0000","last_clean":"2024-09-21T02:37:39.868107+0000","last_became_active":"2024-09-21T02:36:25.745126+0000","last_became_peered":"2024-09-21T02:36:25.745126+0000","last_unstale":"2024-09-21T02:37:39.868107+0000","last_undegraded":"2024-09-21T02:37:39.868107+0000","last_fullsized":"2024-09-21T02:37:39.868107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:42:17.835784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601889+0000","last_change":"2024-09-21T02:36:23.802440+0000","last_active":"2024-09-21T02:37:39.601889+0000","last_peered":"2024-09-21T02:37:39.601889+0000","last_clean":"2024-09-21T02:37:39.601889+0000","last_became_active":"2024-09-21T02:36:23.802065+0000","last_became_peered":"2024-09-21T02:36:23.802065+0000","last_unstale":"2024-09-21T02:37:39.601889+0000","last_undegraded":"2024-09-21T02:37:39.601889+0000","last_fullsized":"2024-09-21T02:37:39.601889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:22:20.647476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605448+0000","last_change":"2024-09-21T02:36:33.003465+0000","last_active":"2024-09-21T02:37:39.605448+0000","last_peered":"2024-09-21T02:37:39.605448+0000","last_clean":"2024-09-21T02:37:39.605448+0000","last_became_active":"2024-09-21T02:36:33.002874+0000","last_became_peered":"2024-09-21T02:36:33.002874+0000","last_unstale":"2024-09-21T02:37:39.605448+0000","last_undegraded":"2024-09-21T02:37:39.605448+0000","last_fullsized":"2024-09-21T02:37:39.605448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:02.388770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867451+0000","last_change":"2024-09-21T02:36:25.746092+0000","last_active":"2024-09-21T02:37:39.867451+0000","last_peered":"2024-09-21T02:37:39.867451+0000","last_clean":"2024-09-21T02:37:39.867451+0000","last_became_active":"2024-09-21T02:36:25.745730+0000","last_became_peered":"2024-09-21T02:36:25.745730+0000","last_unstale":"2024-09-21T02:37:39.867451+0000","last_undegraded":"2024-09-21T02:37:39.867451+0000","last_fullsized":"2024-09-21T02:37:39.867451+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:24:13.212450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":101,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:08.133112+0000","last_change":"2024-09-21T02:36:23.732090+0000","last_active":"2024-09-21T02:38:08.133112+0000","last_peered":"2024-09-21T02:38:08.133112+0000","last_clean":"2024-09-21T02:38:08.133112+0000","last_became_active":"2024-09-21T02:36:23.731653+0000","last_became_peered":"2024-09-21T02:36:23.731653+0000","last_unstale":"2024-09-21T02:38:08.133112+0000","last_undegraded":"2024-09-21T02:38:08.133112+0000","last_fullsized":"2024-09-21T02:38:08.133112+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:02:42.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604825+0000","last_change":"2024-09-21T02:36:32.794893+0000","last_active":"2024-09-21T02:37:39.604825+0000","last_peered":"2024-09-21T02:37:39.604825+0000","last_clean":"2024-09-21T02:37:39.604825+0000","last_became_active":"2024-09-21T02:36:32.794406+0000","last_became_peered":"2024-09-21T02:36:32.794406+0000","last_unstale":"2024-09-21T02:37:39.604825+0000","last_undegraded":"2024-09-21T02:37:39.604825+0000","last_fullsized":"2024-09-21T02:37:39.604825+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:40:39.621502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608789+0000","last_change":"2024-09-21T02:36:25.748220+0000","last_active":"2024-09-21T02:37:39.608789+0000","last_peered":"2024-09-21T02:37:39.608789+0000","last_clean":"2024-09-21T02:37:39.608789+0000","last_became_active":"2024-09-21T02:36:25.747553+0000","last_became_peered":"2024-09-21T02:36:25.747553+0000","last_unstale":"2024-09-21T02:37:39.608789+0000","last_undegraded":"2024-09-21T02:37:39.608789+0000","last_fullsized":"2024-09-21T02:37:39.608789+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:28.757981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":100,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:07.131874+0000","last_change":"2024-09-21T02:36:23.801087+0000","last_active":"2024-09-21T02:38:07.131874+0000","last_peered":"2024-09-21T02:38:07.131874+0000","last_clean":"2024-09-21T02:38:07.131874+0000","last_became_active":"2024-09-21T02:36:23.800077+0000","last_became_peered":"2024-09-21T02:36:23.800077+0000","last_unstale":"2024-09-21T02:38:07.131874+0000","last_undegraded":"2024-09-21T02:38:07.131874+0000","last_fullsized":"2024-09-21T02:38:07.131874+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:50.074650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":60,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605077+0000","last_change":"2024-09-21T02:36:33.003354+0000","last_active":"2024-09-21T02:37:39.605077+0000","last_peered":"2024-09-21T02:37:39.605077+0000","last_clean":"2024-09-21T02:37:39.605077+0000","last_became_active":"2024-09-21T02:36:33.002912+0000","last_became_peered":"2024-09-21T02:36:33.002912+0000","last_unstale":"2024-09-21T02:37:39.605077+0000","last_undegraded":"2024-09-21T02:37:39.605077+0000","last_fullsized":"2024-09-21T02:37:39.605077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:21.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604307+0000","last_change":"2024-09-21T02:36:25.741701+0000","last_active":"2024-09-21T02:37:39.604307+0000","last_peered":"2024-09-21T02:37:39.604307+0000","last_clean":"2024-09-21T02:37:39.604307+0000","last_became_active":"2024-09-21T02:36:25.741222+0000","last_became_peered":"2024-09-21T02:36:25.741222+0000","last_unstale":"2024-09-21T02:37:39.604307+0000","last_undegraded":"2024-09-21T02:37:39.604307+0000","last_fullsized":"2024-09-21T02:37:39.604307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:35:39.896852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604926+0000","last_change":"2024-09-21T02:36:23.728793+0000","last_active":"2024-09-21T02:37:39.604926+0000","last_peered":"2024-09-21T02:37:39.604926+0000","last_clean":"2024-09-21T02:37:39.604926+0000","last_became_active":"2024-09-21T02:36:23.728362+0000","last_became_peered":"2024-09-21T02:36:23.728362+0000","last_unstale":"2024-09-21T02:37:39.604926+0000","last_undegraded":"2024-09-21T02:37:39.604926+0000","last_fullsized":"2024-09-21T02:37:39.604926+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:28:15.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610665+0000","last_change":"2024-09-21T02:36:33.005225+0000","last_active":"2024-09-21T02:37:39.610665+0000","last_peered":"2024-09-21T02:37:39.610665+0000","last_clean":"2024-09-21T02:37:39.610665+0000","last_became_active":"2024-09-21T02:36:33.004756+0000","last_became_peered":"2024-09-21T02:36:33.004756+0000","last_unstale":"2024-09-21T02:37:39.610665+0000","last_undegraded":"2024-09-21T02:37:39.610665+0000","last_fullsized":"2024-09-21T02:37:39.610665+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:02:16.686284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866648+0000","last_change":"2024-09-21T02:36:25.745639+0000","last_active":"2024-09-21T02:37:39.866648+0000","last_peered":"2024-09-21T02:37:39.866648+0000","last_clean":"2024-09-21T02:37:39.866648+0000","last_became_active":"2024-09-21T02:36:25.745202+0000","last_became_peered":"2024-09-21T02:36:25.745202+0000","last_unstale":"2024-09-21T02:37:39.866648+0000","last_undegraded":"2024-09-21T02:37:39.866648+0000","last_fullsized":"2024-09-21T02:37:39.866648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:08.712477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604344+0000","last_change":"2024-09-21T02:36:23.730710+0000","last_active":"2024-09-21T02:37:39.604344+0000","last_peered":"2024-09-21T02:37:39.604344+0000","last_clean":"2024-09-21T02:37:39.604344+0000","last_became_active":"2024-09-21T02:36:23.730247+0000","last_became_peered":"2024-09-21T02:36:23.730247+0000","last_unstale":"2024-09-21T02:37:39.604344+0000","last_undegraded":"2024-09-21T02:37:39.604344+0000","last_fullsized":"2024-09-21T02:37:39.604344+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:43.863444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605253+0000","last_change":"2024-09-21T02:35:09.949841+0000","last_active":"2024-09-21T02:37:39.605253+0000","last_peered":"2024-09-21T02:37:39.605253+0000","last_clean":"2024-09-21T02:37:39.605253+0000","last_became_active":"2024-09-21T02:35:09.944224+0000","last_became_peered":"2024-09-21T02:35:09.944224+0000","last_unstale":"2024-09-21T02:37:39.605253+0000","last_undegraded":"2024-09-21T02:37:39.605253+0000","last_fullsized":"2024-09-21T02:37:39.605253+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_clean_scrub_stamp":"2024-09-21T02:31:38.930125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:56.467576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.874595+0000","last_change":"2024-09-21T02:37:39.874486+0000","last_active":"2024-09-21T02:37:39.874595+0000","last_peered":"2024-09-21T02:37:39.874595+0000","last_clean":"2024-09-21T02:37:39.874595+0000","last_became_active":"2024-09-21T02:37:39.869523+0000","last_became_peered":"2024-09-21T02:37:39.869523+0000","last_unstale":"2024-09-21T02:37:39.874595+0000","last_undegraded":"2024-09-21T02:37:39.874595+0000","last_fullsized":"2024-09-21T02:37:39.874595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:21:33.524428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606052+0000","last_change":"2024-09-21T02:36:25.741176+0000","last_active":"2024-09-21T02:37:39.606052+0000","last_peered":"2024-09-21T02:37:39.606052+0000","last_clean":"2024-09-21T02:37:39.606052+0000","last_became_active":"2024-09-21T02:36:25.740355+0000","last_became_peered":"2024-09-21T02:36:25.740355+0000","last_unstale":"2024-09-21T02:37:39.606052+0000","last_undegraded":"2024-09-21T02:37:39.606052+0000","last_fullsized":"2024-09-21T02:37:39.606052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:47:48.204768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:06.131232+0000","last_change":"2024-09-21T02:36:24.261500+0000","last_active":"2024-09-21T02:38:06.131232+0000","last_peered":"2024-09-21T02:38:06.131232+0000","last_clean":"2024-09-21T02:38:06.131232+0000","last_became_active":"2024-09-21T02:36:24.261089+0000","last_became_peered":"2024-09-21T02:36:24.261089+0000","last_unstale":"2024-09-21T02:38:06.131232+0000","last_undegraded":"2024-09-21T02:38:06.131232+0000","last_fullsized":"2024-09-21T02:38:06.131232+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:33:33.158177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866524+0000","last_change":"2024-09-21T02:36:32.796980+0000","last_active":"2024-09-21T02:37:39.866524+0000","last_peered":"2024-09-21T02:37:39.866524+0000","last_clean":"2024-09-21T02:37:39.866524+0000","last_became_active":"2024-09-21T02:36:32.796517+0000","last_became_peered":"2024-09-21T02:36:32.796517+0000","last_unstale":"2024-09-21T02:37:39.866524+0000","last_undegraded":"2024-09-21T02:37:39.866524+0000","last_fullsized":"2024-09-21T02:37:39.866524+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:07.230177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605201+0000","last_change":"2024-09-21T02:36:25.750931+0000","last_active":"2024-09-21T02:37:39.605201+0000","last_peered":"2024-09-21T02:37:39.605201+0000","last_clean":"2024-09-21T02:37:39.605201+0000","last_became_active":"2024-09-21T02:36:25.750604+0000","last_became_peered":"2024-09-21T02:36:25.750604+0000","last_unstale":"2024-09-21T02:37:39.605201+0000","last_undegraded":"2024-09-21T02:37:39.605201+0000","last_fullsized":"2024-09-21T02:37:39.605201+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:21.325095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609694+0000","last_change":"2024-09-21T02:36:23.730700+0000","last_active":"2024-09-21T02:37:39.609694+0000","last_peered":"2024-09-21T02:37:39.609694+0000","last_clean":"2024-09-21T02:37:39.609694+0000","last_became_active":"2024-09-21T02:36:23.730367+0000","last_became_peered":"2024-09-21T02:36:23.730367+0000","last_unstale":"2024-09-21T02:37:39.609694+0000","last_undegraded":"2024-09-21T02:37:39.609694+0000","last_fullsized":"2024-09-21T02:37:39.609694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:27:57.373077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607709+0000","last_change":"2024-09-21T02:36:32.805885+0000","last_active":"2024-09-21T02:37:39.607709+0000","last_peered":"2024-09-21T02:37:39.607709+0000","last_clean":"2024-09-21T02:37:39.607709+0000","last_became_active":"2024-09-21T02:36:32.805479+0000","last_became_peered":"2024-09-21T02:36:32.805479+0000","last_unstale":"2024-09-21T02:37:39.607709+0000","last_undegraded":"2024-09-21T02:37:39.607709+0000","last_fullsized":"2024-09-21T02:37:39.607709+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:11:29.910642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605971+0000","last_change":"2024-09-21T02:36:25.742542+0000","last_active":"2024-09-21T02:37:39.605971+0000","last_peered":"2024-09-21T02:37:39.605971+0000","last_clean":"2024-09-21T02:37:39.605971+0000","last_became_active":"2024-09-21T02:36:25.742000+0000","last_became_peered":"2024-09-21T02:36:25.742000+0000","last_unstale":"2024-09-21T02:37:39.605971+0000","last_undegraded":"2024-09-21T02:37:39.605971+0000","last_fullsized":"2024-09-21T02:37:39.605971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:52.300212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605081+0000","last_change":"2024-09-21T02:36:23.723896+0000","last_active":"2024-09-21T02:37:39.605081+0000","last_peered":"2024-09-21T02:37:39.605081+0000","last_clean":"2024-09-21T02:37:39.605081+0000","last_became_active":"2024-09-21T02:36:23.723518+0000","last_became_peered":"2024-09-21T02:36:23.723518+0000","last_unstale":"2024-09-21T02:37:39.605081+0000","last_undegraded":"2024-09-21T02:37:39.605081+0000","last_fullsized":"2024-09-21T02:37:39.605081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:37.250247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608598+0000","last_change":"2024-09-21T02:36:32.799924+0000","last_active":"2024-09-21T02:37:39.608598+0000","last_peered":"2024-09-21T02:37:39.608598+0000","last_clean":"2024-09-21T02:37:39.608598+0000","last_became_active":"2024-09-21T02:36:32.799437+0000","last_became_peered":"2024-09-21T02:36:32.799437+0000","last_unstale":"2024-09-21T02:37:39.608598+0000","last_undegraded":"2024-09-21T02:37:39.608598+0000","last_fullsized":"2024-09-21T02:37:39.608598+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:19.258293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604170+0000","last_change":"2024-09-21T02:36:25.747321+0000","last_active":"2024-09-21T02:37:39.604170+0000","last_peered":"2024-09-21T02:37:39.604170+0000","last_clean":"2024-09-21T02:37:39.604170+0000","last_became_active":"2024-09-21T02:36:25.746955+0000","last_became_peered":"2024-09-21T02:36:25.746955+0000","last_unstale":"2024-09-21T02:37:39.604170+0000","last_undegraded":"2024-09-21T02:37:39.604170+0000","last_fullsized":"2024-09-21T02:37:39.604170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:25.148300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606409+0000","last_change":"2024-09-21T02:36:23.725350+0000","last_active":"2024-09-21T02:37:39.606409+0000","last_peered":"2024-09-21T02:37:39.606409+0000","last_clean":"2024-09-21T02:37:39.606409+0000","last_became_active":"2024-09-21T02:36:23.725090+0000","last_became_peered":"2024-09-21T02:36:23.725090+0000","last_unstale":"2024-09-21T02:37:39.606409+0000","last_undegraded":"2024-09-21T02:37:39.606409+0000","last_fullsized":"2024-09-21T02:37:39.606409+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:52.130345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610054+0000","last_change":"2024-09-21T02:36:32.801355+0000","last_active":"2024-09-21T02:37:39.610054+0000","last_peered":"2024-09-21T02:37:39.610054+0000","last_clean":"2024-09-21T02:37:39.610054+0000","last_became_active":"2024-09-21T02:36:32.800803+0000","last_became_peered":"2024-09-21T02:36:32.800803+0000","last_unstale":"2024-09-21T02:37:39.610054+0000","last_undegraded":"2024-09-21T02:37:39.610054+0000","last_fullsized":"2024-09-21T02:37:39.610054+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:54.755733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609524+0000","last_change":"2024-09-21T02:36:25.743433+0000","last_active":"2024-09-21T02:37:39.609524+0000","last_peered":"2024-09-21T02:37:39.609524+0000","last_clean":"2024-09-21T02:37:39.609524+0000","last_became_active":"2024-09-21T02:36:25.742643+0000","last_became_peered":"2024-09-21T02:36:25.742643+0000","last_unstale":"2024-09-21T02:37:39.609524+0000","last_undegraded":"2024-09-21T02:37:39.609524+0000","last_fullsized":"2024-09-21T02:37:39.609524+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:20:09.849806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.625433+0000","last_change":"2024-09-21T02:37:39.625360+0000","last_active":"2024-09-21T02:37:39.625433+0000","last_peered":"2024-09-21T02:37:39.625433+0000","last_clean":"2024-09-21T02:37:39.625433+0000","last_became_active":"2024-09-21T02:37:39.611781+0000","last_became_peered":"2024-09-21T02:37:39.611781+0000","last_unstale":"2024-09-21T02:37:39.625433+0000","last_undegraded":"2024-09-21T02:37:39.625433+0000","last_fullsized":"2024-09-21T02:37:39.625433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:11.846383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605044+0000","last_change":"2024-09-21T02:36:32.799728+0000","last_active":"2024-09-21T02:37:39.605044+0000","last_peered":"2024-09-21T02:37:39.605044+0000","last_clean":"2024-09-21T02:37:39.605044+0000","last_became_active":"2024-09-21T02:36:32.798419+0000","last_became_peered":"2024-09-21T02:36:32.798419+0000","last_unstale":"2024-09-21T02:37:39.605044+0000","last_undegraded":"2024-09-21T02:37:39.605044+0000","last_fullsized":"2024-09-21T02:37:39.605044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:53.681473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604652+0000","last_change":"2024-09-21T02:36:25.747712+0000","last_active":"2024-09-21T02:37:39.604652+0000","last_peered":"2024-09-21T02:37:39.604652+0000","last_clean":"2024-09-21T02:37:39.604652+0000","last_became_active":"2024-09-21T02:36:25.747197+0000","last_became_peered":"2024-09-21T02:36:25.747197+0000","last_unstale":"2024-09-21T02:37:39.604652+0000","last_undegraded":"2024-09-21T02:37:39.604652+0000","last_fullsized":"2024-09-21T02:37:39.604652+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:03.900360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":98,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:04.128143+0000","last_change":"2024-09-21T02:36:23.732156+0000","last_active":"2024-09-21T02:38:04.128143+0000","last_peered":"2024-09-21T02:38:04.128143+0000","last_clean":"2024-09-21T02:38:04.128143+0000","last_became_active":"2024-09-21T02:36:23.730238+0000","last_became_peered":"2024-09-21T02:36:23.730238+0000","last_unstale":"2024-09-21T02:38:04.128143+0000","last_undegraded":"2024-09-21T02:38:04.128143+0000","last_fullsized":"2024-09-21T02:38:04.128143+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:18.053890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":59,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602096+0000","last_change":"2024-09-21T02:36:32.806674+0000","last_active":"2024-09-21T02:37:39.602096+0000","last_peered":"2024-09-21T02:37:39.602096+0000","last_clean":"2024-09-21T02:37:39.602096+0000","last_became_active":"2024-09-21T02:36:32.805373+0000","last_became_peered":"2024-09-21T02:36:32.805373+0000","last_unstale":"2024-09-21T02:37:39.602096+0000","last_undegraded":"2024-09-21T02:37:39.602096+0000","last_fullsized":"2024-09-21T02:37:39.602096+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:29.249243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605083+0000","last_change":"2024-09-21T02:36:25.748703+0000","last_active":"2024-09-21T02:37:39.605083+0000","last_peered":"2024-09-21T02:37:39.605083+0000","last_clean":"2024-09-21T02:37:39.605083+0000","last_became_active":"2024-09-21T02:36:25.748340+0000","last_became_peered":"2024-09-21T02:36:25.748340+0000","last_unstale":"2024-09-21T02:37:39.605083+0000","last_undegraded":"2024-09-21T02:37:39.605083+0000","last_fullsized":"2024-09-21T02:37:39.605083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:07.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610285+0000","last_change":"2024-09-21T02:36:23.732332+0000","last_active":"2024-09-21T02:37:39.610285+0000","last_peered":"2024-09-21T02:37:39.610285+0000","last_clean":"2024-09-21T02:37:39.610285+0000","last_became_active":"2024-09-21T02:36:23.731974+0000","last_became_peered":"2024-09-21T02:36:23.731974+0000","last_unstale":"2024-09-21T02:37:39.610285+0000","last_undegraded":"2024-09-21T02:37:39.610285+0000","last_fullsized":"2024-09-21T02:37:39.610285+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:52.284192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610819+0000","last_change":"2024-09-21T02:37:39.610698+0000","last_active":"2024-09-21T02:37:39.610819+0000","last_peered":"2024-09-21T02:37:39.610819+0000","last_clean":"2024-09-21T02:37:39.610819+0000","last_became_active":"2024-09-21T02:37:39.610185+0000","last_became_peered":"2024-09-21T02:37:39.610185+0000","last_unstale":"2024-09-21T02:37:39.610819+0000","last_undegraded":"2024-09-21T02:37:39.610819+0000","last_fullsized":"2024-09-21T02:37:39.610819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:54:51.050463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604687+0000","last_change":"2024-09-21T02:36:25.751720+0000","last_active":"2024-09-21T02:37:39.604687+0000","last_peered":"2024-09-21T02:37:39.604687+0000","last_clean":"2024-09-21T02:37:39.604687+0000","last_became_active":"2024-09-21T02:36:25.751351+0000","last_became_peered":"2024-09-21T02:36:25.751351+0000","last_unstale":"2024-09-21T02:37:39.604687+0000","last_undegraded":"2024-09-21T02:37:39.604687+0000","last_fullsized":"2024-09-21T02:37:39.604687+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:01.201453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":46,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603023+0000","last_change":"2024-09-21T02:36:23.729981+0000","last_active":"2024-09-21T02:37:39.603023+0000","last_peered":"2024-09-21T02:37:39.603023+0000","last_clean":"2024-09-21T02:37:39.603023+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:37:39.603023+0000","last_undegraded":"2024-09-21T02:37:39.603023+0000","last_fullsized":"2024-09-21T02:37:39.603023+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:03.115712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609295+0000","last_change":"2024-09-21T02:36:32.805286+0000","last_active":"2024-09-21T02:37:39.609295+0000","last_peered":"2024-09-21T02:37:39.609295+0000","last_clean":"2024-09-21T02:37:39.609295+0000","last_became_active":"2024-09-21T02:36:32.804770+0000","last_became_peered":"2024-09-21T02:36:32.804770+0000","last_unstale":"2024-09-21T02:37:39.609295+0000","last_undegraded":"2024-09-21T02:37:39.609295+0000","last_fullsized":"2024-09-21T02:37:39.609295+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:18:34.155892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606225+0000","last_change":"2024-09-21T02:36:25.747812+0000","last_active":"2024-09-21T02:37:39.606225+0000","last_peered":"2024-09-21T02:37:39.606225+0000","last_clean":"2024-09-21T02:37:39.606225+0000","last_became_active":"2024-09-21T02:36:25.747187+0000","last_became_peered":"2024-09-21T02:36:25.747187+0000","last_unstale":"2024-09-21T02:37:39.606225+0000","last_undegraded":"2024-09-21T02:37:39.606225+0000","last_fullsized":"2024-09-21T02:37:39.606225+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:51.395869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"81'35","reported_seq":195,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:07.131904+0000","last_change":"2024-09-21T02:36:23.735676+0000","last_active":"2024-09-21T02:38:07.131904+0000","last_peered":"2024-09-21T02:38:07.131904+0000","last_clean":"2024-09-21T02:38:07.131904+0000","last_became_active":"2024-09-21T02:36:23.735281+0000","last_became_peered":"2024-09-21T02:36:23.735281+0000","last_unstale":"2024-09-21T02:38:07.131904+0000","last_undegraded":"2024-09-21T02:38:07.131904+0000","last_fullsized":"2024-09-21T02:38:07.131904+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:28:01.547245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67620,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":125,"num_read_kb":128,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601441+0000","last_change":"2024-09-21T02:36:32.806585+0000","last_active":"2024-09-21T02:37:39.601441+0000","last_peered":"2024-09-21T02:37:39.601441+0000","last_clean":"2024-09-21T02:37:39.601441+0000","last_became_active":"2024-09-21T02:36:32.805913+0000","last_became_peered":"2024-09-21T02:36:32.805913+0000","last_unstale":"2024-09-21T02:37:39.601441+0000","last_undegraded":"2024-09-21T02:37:39.601441+0000","last_fullsized":"2024-09-21T02:37:39.601441+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:59:31.888645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867132+0000","last_change":"2024-09-21T02:36:25.742208+0000","last_active":"2024-09-21T02:37:39.867132+0000","last_peered":"2024-09-21T02:37:39.867132+0000","last_clean":"2024-09-21T02:37:39.867132+0000","last_became_active":"2024-09-21T02:36:25.741849+0000","last_became_peered":"2024-09-21T02:36:25.741849+0000","last_unstale":"2024-09-21T02:37:39.867132+0000","last_undegraded":"2024-09-21T02:37:39.867132+0000","last_fullsized":"2024-09-21T02:37:39.867132+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:21:43.720481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609540+0000","last_change":"2024-09-21T02:36:23.800751+0000","last_active":"2024-09-21T02:37:39.609540+0000","last_peered":"2024-09-21T02:37:39.609540+0000","last_clean":"2024-09-21T02:37:39.609540+0000","last_became_active":"2024-09-21T02:36:23.800341+0000","last_became_peered":"2024-09-21T02:36:23.800341+0000","last_unstale":"2024-09-21T02:37:39.609540+0000","last_undegraded":"2024-09-21T02:37:39.609540+0000","last_fullsized":"2024-09-21T02:37:39.609540+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:08:02.556549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604004+0000","last_change":"2024-09-21T02:36:32.802879+0000","last_active":"2024-09-21T02:37:39.604004+0000","last_peered":"2024-09-21T02:37:39.604004+0000","last_clean":"2024-09-21T02:37:39.604004+0000","last_became_active":"2024-09-21T02:36:32.802301+0000","last_became_peered":"2024-09-21T02:36:32.802301+0000","last_unstale":"2024-09-21T02:37:39.604004+0000","last_undegraded":"2024-09-21T02:37:39.604004+0000","last_fullsized":"2024-09-21T02:37:39.604004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:08.135814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608374+0000","last_change":"2024-09-21T02:36:25.744647+0000","last_active":"2024-09-21T02:37:39.608374+0000","last_peered":"2024-09-21T02:37:39.608374+0000","last_clean":"2024-09-21T02:37:39.608374+0000","last_became_active":"2024-09-21T02:36:25.744049+0000","last_became_peered":"2024-09-21T02:36:25.744049+0000","last_unstale":"2024-09-21T02:37:39.608374+0000","last_undegraded":"2024-09-21T02:37:39.608374+0000","last_fullsized":"2024-09-21T02:37:39.608374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:52:02.210380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605972+0000","last_change":"2024-09-21T02:36:23.733590+0000","last_active":"2024-09-21T02:37:39.605972+0000","last_peered":"2024-09-21T02:37:39.605972+0000","last_clean":"2024-09-21T02:37:39.605972+0000","last_became_active":"2024-09-21T02:36:23.732922+0000","last_became_peered":"2024-09-21T02:36:23.732922+0000","last_unstale":"2024-09-21T02:37:39.605972+0000","last_undegraded":"2024-09-21T02:37:39.605972+0000","last_fullsized":"2024-09-21T02:37:39.605972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601278+0000","last_change":"2024-09-21T02:36:32.806540+0000","last_active":"2024-09-21T02:37:39.601278+0000","last_peered":"2024-09-21T02:37:39.601278+0000","last_clean":"2024-09-21T02:37:39.601278+0000","last_became_active":"2024-09-21T02:36:32.805612+0000","last_became_peered":"2024-09-21T02:36:32.805612+0000","last_unstale":"2024-09-21T02:37:39.601278+0000","last_undegraded":"2024-09-21T02:37:39.601278+0000","last_fullsized":"2024-09-21T02:37:39.601278+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:23:24.686383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867826+0000","last_change":"2024-09-21T02:36:25.746229+0000","last_active":"2024-09-21T02:37:39.867826+0000","last_peered":"2024-09-21T02:37:39.867826+0000","last_clean":"2024-09-21T02:37:39.867826+0000","last_became_active":"2024-09-21T02:36:25.745892+0000","last_became_peered":"2024-09-21T02:36:25.745892+0000","last_unstale":"2024-09-21T02:37:39.867826+0000","last_undegraded":"2024-09-21T02:37:39.867826+0000","last_fullsized":"2024-09-21T02:37:39.867826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:55.154499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":155,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:07.131918+0000","last_change":"2024-09-21T02:36:23.730987+0000","last_active":"2024-09-21T02:38:07.131918+0000","last_peered":"2024-09-21T02:38:07.131918+0000","last_clean":"2024-09-21T02:38:07.131918+0000","last_became_active":"2024-09-21T02:36:23.730566+0000","last_became_peered":"2024-09-21T02:36:23.730566+0000","last_unstale":"2024-09-21T02:38:07.131918+0000","last_undegraded":"2024-09-21T02:38:07.131918+0000","last_fullsized":"2024-09-21T02:38:07.131918+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:18:12.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866092+0000","last_change":"2024-09-21T02:36:32.804608+0000","last_active":"2024-09-21T02:37:39.866092+0000","last_peered":"2024-09-21T02:37:39.866092+0000","last_clean":"2024-09-21T02:37:39.866092+0000","last_became_active":"2024-09-21T02:36:32.803567+0000","last_became_peered":"2024-09-21T02:36:32.803567+0000","last_unstale":"2024-09-21T02:37:39.866092+0000","last_undegraded":"2024-09-21T02:37:39.866092+0000","last_fullsized":"2024-09-21T02:37:39.866092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:41.692644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607920+0000","last_change":"2024-09-21T02:36:32.804687+0000","last_active":"2024-09-21T02:37:39.607920+0000","last_peered":"2024-09-21T02:37:39.607920+0000","last_clean":"2024-09-21T02:37:39.607920+0000","last_became_active":"2024-09-21T02:36:32.804232+0000","last_became_peered":"2024-09-21T02:36:32.804232+0000","last_unstale":"2024-09-21T02:37:39.607920+0000","last_undegraded":"2024-09-21T02:37:39.607920+0000","last_fullsized":"2024-09-21T02:37:39.607920+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:43.899924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601400+0000","last_change":"2024-09-21T02:36:25.749114+0000","last_active":"2024-09-21T02:37:39.601400+0000","last_peered":"2024-09-21T02:37:39.601400+0000","last_clean":"2024-09-21T02:37:39.601400+0000","last_became_active":"2024-09-21T02:36:25.748758+0000","last_became_peered":"2024-09-21T02:36:25.748758+0000","last_unstale":"2024-09-21T02:37:39.601400+0000","last_undegraded":"2024-09-21T02:37:39.601400+0000","last_fullsized":"2024-09-21T02:37:39.601400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:57.958533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608906+0000","last_change":"2024-09-21T02:36:23.731473+0000","last_active":"2024-09-21T02:37:39.608906+0000","last_peered":"2024-09-21T02:37:39.608906+0000","last_clean":"2024-09-21T02:37:39.608906+0000","last_became_active":"2024-09-21T02:36:23.731134+0000","last_became_peered":"2024-09-21T02:36:23.731134+0000","last_unstale":"2024-09-21T02:37:39.608906+0000","last_undegraded":"2024-09-21T02:37:39.608906+0000","last_fullsized":"2024-09-21T02:37:39.608906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:05.368941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604072+0000","last_change":"2024-09-21T02:36:32.809630+0000","last_active":"2024-09-21T02:37:39.604072+0000","last_peered":"2024-09-21T02:37:39.604072+0000","last_clean":"2024-09-21T02:37:39.604072+0000","last_became_active":"2024-09-21T02:36:32.809030+0000","last_became_peered":"2024-09-21T02:36:32.809030+0000","last_unstale":"2024-09-21T02:37:39.604072+0000","last_undegraded":"2024-09-21T02:37:39.604072+0000","last_fullsized":"2024-09-21T02:37:39.604072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:57:42.436590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866309+0000","last_change":"2024-09-21T02:36:25.742558+0000","last_active":"2024-09-21T02:37:39.866309+0000","last_peered":"2024-09-21T02:37:39.866309+0000","last_clean":"2024-09-21T02:37:39.866309+0000","last_became_active":"2024-09-21T02:36:25.742045+0000","last_became_peered":"2024-09-21T02:36:25.742045+0000","last_unstale":"2024-09-21T02:37:39.866309+0000","last_undegraded":"2024-09-21T02:37:39.866309+0000","last_fullsized":"2024-09-21T02:37:39.866309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:57:06.705561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604389+0000","last_change":"2024-09-21T02:36:23.724345+0000","last_active":"2024-09-21T02:37:39.604389+0000","last_peered":"2024-09-21T02:37:39.604389+0000","last_clean":"2024-09-21T02:37:39.604389+0000","last_became_active":"2024-09-21T02:36:23.723635+0000","last_became_peered":"2024-09-21T02:36:23.723635+0000","last_unstale":"2024-09-21T02:37:39.604389+0000","last_undegraded":"2024-09-21T02:37:39.604389+0000","last_fullsized":"2024-09-21T02:37:39.604389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:18.812176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867234+0000","last_change":"2024-09-21T02:36:23.732897+0000","last_active":"2024-09-21T02:37:39.867234+0000","last_peered":"2024-09-21T02:37:39.867234+0000","last_clean":"2024-09-21T02:37:39.867234+0000","last_became_active":"2024-09-21T02:36:23.732509+0000","last_became_peered":"2024-09-21T02:36:23.732509+0000","last_unstale":"2024-09-21T02:37:39.867234+0000","last_undegraded":"2024-09-21T02:37:39.867234+0000","last_fullsized":"2024-09-21T02:37:39.867234+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:12:27.117962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609775+0000","last_change":"2024-09-21T02:36:25.748929+0000","last_active":"2024-09-21T02:37:39.609775+0000","last_peered":"2024-09-21T02:37:39.609775+0000","last_clean":"2024-09-21T02:37:39.609775+0000","last_became_active":"2024-09-21T02:36:25.748588+0000","last_became_peered":"2024-09-21T02:36:25.748588+0000","last_unstale":"2024-09-21T02:37:39.609775+0000","last_undegraded":"2024-09-21T02:37:39.609775+0000","last_fullsized":"2024-09-21T02:37:39.609775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.086472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602636+0000","last_change":"2024-09-21T02:36:32.807655+0000","last_active":"2024-09-21T02:37:39.602636+0000","last_peered":"2024-09-21T02:37:39.602636+0000","last_clean":"2024-09-21T02:37:39.602636+0000","last_became_active":"2024-09-21T02:36:32.806995+0000","last_became_peered":"2024-09-21T02:36:32.806995+0000","last_unstale":"2024-09-21T02:37:39.602636+0000","last_undegraded":"2024-09-21T02:37:39.602636+0000","last_fullsized":"2024-09-21T02:37:39.602636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:28:20.420591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":101,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:05.130041+0000","last_change":"2024-09-21T02:36:23.729427+0000","last_active":"2024-09-21T02:38:05.130041+0000","last_peered":"2024-09-21T02:38:05.130041+0000","last_clean":"2024-09-21T02:38:05.130041+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:38:05.130041+0000","last_undegraded":"2024-09-21T02:38:05.130041+0000","last_fullsized":"2024-09-21T02:38:05.130041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:10:41.251665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605668+0000","last_change":"2024-09-21T02:36:25.743212+0000","last_active":"2024-09-21T02:37:39.605668+0000","last_peered":"2024-09-21T02:37:39.605668+0000","last_clean":"2024-09-21T02:37:39.605668+0000","last_became_active":"2024-09-21T02:36:25.742777+0000","last_became_peered":"2024-09-21T02:36:25.742777+0000","last_unstale":"2024-09-21T02:37:39.605668+0000","last_undegraded":"2024-09-21T02:37:39.605668+0000","last_fullsized":"2024-09-21T02:37:39.605668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:12:19.917214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602914+0000","last_change":"2024-09-21T02:36:32.801880+0000","last_active":"2024-09-21T02:37:39.602914+0000","last_peered":"2024-09-21T02:37:39.602914+0000","last_clean":"2024-09-21T02:37:39.602914+0000","last_became_active":"2024-09-21T02:36:32.801389+0000","last_became_peered":"2024-09-21T02:36:32.801389+0000","last_unstale":"2024-09-21T02:37:39.602914+0000","last_undegraded":"2024-09-21T02:37:39.602914+0000","last_fullsized":"2024-09-21T02:37:39.602914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:11:44.877062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605000+0000","last_change":"2024-09-21T02:36:25.754014+0000","last_active":"2024-09-21T02:37:39.605000+0000","last_peered":"2024-09-21T02:37:39.605000+0000","last_clean":"2024-09-21T02:37:39.605000+0000","last_became_active":"2024-09-21T02:36:25.753548+0000","last_became_peered":"2024-09-21T02:36:25.753548+0000","last_unstale":"2024-09-21T02:37:39.605000+0000","last_undegraded":"2024-09-21T02:37:39.605000+0000","last_fullsized":"2024-09-21T02:37:39.605000+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:03.850625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610997+0000","last_change":"2024-09-21T02:37:39.610894+0000","last_active":"2024-09-21T02:37:39.610997+0000","last_peered":"2024-09-21T02:37:39.610997+0000","last_clean":"2024-09-21T02:37:39.610997+0000","last_became_active":"2024-09-21T02:37:39.610404+0000","last_became_peered":"2024-09-21T02:37:39.610404+0000","last_unstale":"2024-09-21T02:37:39.610997+0000","last_undegraded":"2024-09-21T02:37:39.610997+0000","last_fullsized":"2024-09-21T02:37:39.610997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:54.347708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602163+0000","last_change":"2024-09-21T02:36:25.752839+0000","last_active":"2024-09-21T02:37:39.602163+0000","last_peered":"2024-09-21T02:37:39.602163+0000","last_clean":"2024-09-21T02:37:39.602163+0000","last_became_active":"2024-09-21T02:36:25.750882+0000","last_became_peered":"2024-09-21T02:36:25.750882+0000","last_unstale":"2024-09-21T02:37:39.602163+0000","last_undegraded":"2024-09-21T02:37:39.602163+0000","last_fullsized":"2024-09-21T02:37:39.602163+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:53.028611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607755+0000","last_change":"2024-09-21T02:36:25.744624+0000","last_active":"2024-09-21T02:37:39.607755+0000","last_peered":"2024-09-21T02:37:39.607755+0000","last_clean":"2024-09-21T02:37:39.607755+0000","last_became_active":"2024-09-21T02:36:25.744173+0000","last_became_peered":"2024-09-21T02:36:25.744173+0000","last_unstale":"2024-09-21T02:37:39.607755+0000","last_undegraded":"2024-09-21T02:37:39.607755+0000","last_fullsized":"2024-09-21T02:37:39.607755+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:55.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609153+0000","last_change":"2024-09-21T02:37:39.609050+0000","last_active":"2024-09-21T02:37:39.609153+0000","last_peered":"2024-09-21T02:37:39.609153+0000","last_clean":"2024-09-21T02:37:39.609153+0000","last_became_active":"2024-09-21T02:37:39.608595+0000","last_became_peered":"2024-09-21T02:37:39.608595+0000","last_unstale":"2024-09-21T02:37:39.609153+0000","last_undegraded":"2024-09-21T02:37:39.609153+0000","last_fullsized":"2024-09-21T02:37:39.609153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:26.831121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605670+0000","last_change":"2024-09-21T02:36:25.743476+0000","last_active":"2024-09-21T02:37:39.605670+0000","last_peered":"2024-09-21T02:37:39.605670+0000","last_clean":"2024-09-21T02:37:39.605670+0000","last_became_active":"2024-09-21T02:36:25.743117+0000","last_became_peered":"2024-09-21T02:36:25.743117+0000","last_unstale":"2024-09-21T02:37:39.605670+0000","last_undegraded":"2024-09-21T02:37:39.605670+0000","last_fullsized":"2024-09-21T02:37:39.605670+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:08.084109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603172+0000","last_change":"2024-09-21T02:36:25.753014+0000","last_active":"2024-09-21T02:37:39.603172+0000","last_peered":"2024-09-21T02:37:39.603172+0000","last_clean":"2024-09-21T02:37:39.603172+0000","last_became_active":"2024-09-21T02:36:25.751232+0000","last_became_peered":"2024-09-21T02:36:25.751232+0000","last_unstale":"2024-09-21T02:37:39.603172+0000","last_undegraded":"2024-09-21T02:37:39.603172+0000","last_fullsized":"2024-09-21T02:37:39.603172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:03.052737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607167+0000","last_change":"2024-09-21T02:37:39.606525+0000","last_active":"2024-09-21T02:37:39.607167+0000","last_peered":"2024-09-21T02:37:39.607167+0000","last_clean":"2024-09-21T02:37:39.607167+0000","last_became_active":"2024-09-21T02:37:39.606078+0000","last_became_peered":"2024-09-21T02:37:39.606078+0000","last_unstale":"2024-09-21T02:37:39.607167+0000","last_undegraded":"2024-09-21T02:37:39.607167+0000","last_fullsized":"2024-09-21T02:37:39.607167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:22.423457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608763+0000","last_change":"2024-09-21T02:36:25.749605+0000","last_active":"2024-09-21T02:37:39.608763+0000","last_peered":"2024-09-21T02:37:39.608763+0000","last_clean":"2024-09-21T02:37:39.608763+0000","last_became_active":"2024-09-21T02:36:25.749165+0000","last_became_peered":"2024-09-21T02:36:25.749165+0000","last_unstale":"2024-09-21T02:37:39.608763+0000","last_undegraded":"2024-09-21T02:37:39.608763+0000","last_fullsized":"2024-09-21T02:37:39.608763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:15:51.412761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866165+0000","last_change":"2024-09-21T02:36:25.747494+0000","last_active":"2024-09-21T02:37:39.866165+0000","last_peered":"2024-09-21T02:37:39.866165+0000","last_clean":"2024-09-21T02:37:39.866165+0000","last_became_active":"2024-09-21T02:36:25.746965+0000","last_became_peered":"2024-09-21T02:36:25.746965+0000","last_unstale":"2024-09-21T02:37:39.866165+0000","last_undegraded":"2024-09-21T02:37:39.866165+0000","last_fullsized":"2024-09-21T02:37:39.866165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:51.309307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867530+0000","last_change":"2024-09-21T02:36:25.749517+0000","last_active":"2024-09-21T02:37:39.867530+0000","last_peered":"2024-09-21T02:37:39.867530+0000","last_clean":"2024-09-21T02:37:39.867530+0000","last_became_active":"2024-09-21T02:36:25.749117+0000","last_became_peered":"2024-09-21T02:36:25.749117+0000","last_unstale":"2024-09-21T02:37:39.867530+0000","last_undegraded":"2024-09-21T02:37:39.867530+0000","last_fullsized":"2024-09-21T02:37:39.867530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:38.758019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604855+0000","last_change":"2024-09-21T02:36:25.736067+0000","last_active":"2024-09-21T02:37:39.604855+0000","last_peered":"2024-09-21T02:37:39.604855+0000","last_clean":"2024-09-21T02:37:39.604855+0000","last_became_active":"2024-09-21T02:36:25.735710+0000","last_became_peered":"2024-09-21T02:36:25.735710+0000","last_unstale":"2024-09-21T02:37:39.604855+0000","last_undegraded":"2024-09-21T02:37:39.604855+0000","last_fullsized":"2024-09-21T02:37:39.604855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.559448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604519+0000","last_change":"2024-09-21T02:36:25.746307+0000","last_active":"2024-09-21T02:37:39.604519+0000","last_peered":"2024-09-21T02:37:39.604519+0000","last_clean":"2024-09-21T02:37:39.604519+0000","last_became_active":"2024-09-21T02:36:25.745872+0000","last_became_peered":"2024-09-21T02:36:25.745872+0000","last_unstale":"2024-09-21T02:37:39.604519+0000","last_undegraded":"2024-09-21T02:37:39.604519+0000","last_fullsized":"2024-09-21T02:37:39.604519+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:58:53.074559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603439+0000","last_change":"2024-09-21T02:36:25.753393+0000","last_active":"2024-09-21T02:37:39.603439+0000","last_peered":"2024-09-21T02:37:39.603439+0000","last_clean":"2024-09-21T02:37:39.603439+0000","last_became_active":"2024-09-21T02:36:25.751871+0000","last_became_peered":"2024-09-21T02:36:25.751871+0000","last_unstale":"2024-09-21T02:37:39.603439+0000","last_undegraded":"2024-09-21T02:37:39.603439+0000","last_fullsized":"2024-09-21T02:37:39.603439+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:28.199451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601888+0000","last_change":"2024-09-21T02:36:25.744845+0000","last_active":"2024-09-21T02:37:39.601888+0000","last_peered":"2024-09-21T02:37:39.601888+0000","last_clean":"2024-09-21T02:37:39.601888+0000","last_became_active":"2024-09-21T02:36:25.744492+0000","last_became_peered":"2024-09-21T02:36:25.744492+0000","last_unstale":"2024-09-21T02:37:39.601888+0000","last_undegraded":"2024-09-21T02:37:39.601888+0000","last_fullsized":"2024-09-21T02:37:39.601888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:08.524871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604947+0000","last_change":"2024-09-21T02:36:25.751187+0000","last_active":"2024-09-21T02:37:39.604947+0000","last_peered":"2024-09-21T02:37:39.604947+0000","last_clean":"2024-09-21T02:37:39.604947+0000","last_became_active":"2024-09-21T02:36:25.750813+0000","last_became_peered":"2024-09-21T02:36:25.750813+0000","last_unstale":"2024-09-21T02:37:39.604947+0000","last_undegraded":"2024-09-21T02:37:39.604947+0000","last_fullsized":"2024-09-21T02:37:39.604947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.590210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866405+0000","last_change":"2024-09-21T02:36:25.744046+0000","last_active":"2024-09-21T02:37:39.866405+0000","last_peered":"2024-09-21T02:37:39.866405+0000","last_clean":"2024-09-21T02:37:39.866405+0000","last_became_active":"2024-09-21T02:36:25.743695+0000","last_became_peered":"2024-09-21T02:36:25.743695+0000","last_unstale":"2024-09-21T02:37:39.866405+0000","last_undegraded":"2024-09-21T02:37:39.866405+0000","last_fullsized":"2024-09-21T02:37:39.866405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:12.092940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607554+0000","last_change":"2024-09-21T02:36:25.749113+0000","last_active":"2024-09-21T02:37:39.607554+0000","last_peered":"2024-09-21T02:37:39.607554+0000","last_clean":"2024-09-21T02:37:39.607554+0000","last_became_active":"2024-09-21T02:36:25.748759+0000","last_became_peered":"2024-09-21T02:36:25.748759+0000","last_unstale":"2024-09-21T02:37:39.607554+0000","last_undegraded":"2024-09-21T02:37:39.607554+0000","last_fullsized":"2024-09-21T02:37:39.607554+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:22.785273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607560+0000","last_change":"2024-09-21T02:36:25.745454+0000","last_active":"2024-09-21T02:37:39.607560+0000","last_peered":"2024-09-21T02:37:39.607560+0000","last_clean":"2024-09-21T02:37:39.607560+0000","last_became_active":"2024-09-21T02:36:25.745108+0000","last_became_peered":"2024-09-21T02:36:25.745108+0000","last_unstale":"2024-09-21T02:37:39.607560+0000","last_undegraded":"2024-09-21T02:37:39.607560+0000","last_fullsized":"2024-09-21T02:37:39.607560+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:36.710456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604836+0000","last_change":"2024-09-21T02:36:25.737213+0000","last_active":"2024-09-21T02:37:39.604836+0000","last_peered":"2024-09-21T02:37:39.604836+0000","last_clean":"2024-09-21T02:37:39.604836+0000","last_became_active":"2024-09-21T02:36:25.736641+0000","last_became_peered":"2024-09-21T02:36:25.736641+0000","last_unstale":"2024-09-21T02:37:39.604836+0000","last_undegraded":"2024-09-21T02:37:39.604836+0000","last_fullsized":"2024-09-21T02:37:39.604836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:47:50.258805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604582+0000","last_change":"2024-09-21T02:36:25.748230+0000","last_active":"2024-09-21T02:37:39.604582+0000","last_peered":"2024-09-21T02:37:39.604582+0000","last_clean":"2024-09-21T02:37:39.604582+0000","last_became_active":"2024-09-21T02:36:25.747859+0000","last_became_peered":"2024-09-21T02:36:25.747859+0000","last_unstale":"2024-09-21T02:37:39.604582+0000","last_undegraded":"2024-09-21T02:37:39.604582+0000","last_fullsized":"2024-09-21T02:37:39.604582+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:15:21.244825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605976+0000","last_change":"2024-09-21T02:36:25.744379+0000","last_active":"2024-09-21T02:37:39.605976+0000","last_peered":"2024-09-21T02:37:39.605976+0000","last_clean":"2024-09-21T02:37:39.605976+0000","last_became_active":"2024-09-21T02:36:25.744003+0000","last_became_peered":"2024-09-21T02:36:25.744003+0000","last_unstale":"2024-09-21T02:37:39.605976+0000","last_undegraded":"2024-09-21T02:37:39.605976+0000","last_fullsized":"2024-09-21T02:37:39.605976+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:59:59.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.868567+0000","last_change":"2024-09-21T02:36:25.755616+0000","last_active":"2024-09-21T02:37:39.868567+0000","last_peered":"2024-09-21T02:37:39.868567+0000","last_clean":"2024-09-21T02:37:39.868567+0000","last_became_active":"2024-09-21T02:36:25.755304+0000","last_became_peered":"2024-09-21T02:36:25.755304+0000","last_unstale":"2024-09-21T02:37:39.868567+0000","last_undegraded":"2024-09-21T02:37:39.868567+0000","last_fullsized":"2024-09-21T02:37:39.868567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:24.813780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867940+0000","last_change":"2024-09-21T02:36:25.739747+0000","last_active":"2024-09-21T02:37:39.867940+0000","last_peered":"2024-09-21T02:37:39.867940+0000","last_clean":"2024-09-21T02:37:39.867940+0000","last_became_active":"2024-09-21T02:36:25.739404+0000","last_became_peered":"2024-09-21T02:36:25.739404+0000","last_unstale":"2024-09-21T02:37:39.867940+0000","last_undegraded":"2024-09-21T02:37:39.867940+0000","last_fullsized":"2024-09-21T02:37:39.867940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:17.975419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866105+0000","last_change":"2024-09-21T02:36:25.747723+0000","last_active":"2024-09-21T02:37:39.866105+0000","last_peered":"2024-09-21T02:37:39.866105+0000","last_clean":"2024-09-21T02:37:39.866105+0000","last_became_active":"2024-09-21T02:36:25.747368+0000","last_became_peered":"2024-09-21T02:36:25.747368+0000","last_unstale":"2024-09-21T02:37:39.866105+0000","last_undegraded":"2024-09-21T02:37:39.866105+0000","last_fullsized":"2024-09-21T02:37:39.866105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.319756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605349+0000","last_change":"2024-09-21T02:36:25.743772+0000","last_active":"2024-09-21T02:37:39.605349+0000","last_peered":"2024-09-21T02:37:39.605349+0000","last_clean":"2024-09-21T02:37:39.605349+0000","last_became_active":"2024-09-21T02:36:25.743403+0000","last_became_peered":"2024-09-21T02:36:25.743403+0000","last_unstale":"2024-09-21T02:37:39.605349+0000","last_undegraded":"2024-09-21T02:37:39.605349+0000","last_fullsized":"2024-09-21T02:37:39.605349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:11:29.930248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867226+0000","last_change":"2024-09-21T02:36:25.746911+0000","last_active":"2024-09-21T02:37:39.867226+0000","last_peered":"2024-09-21T02:37:39.867226+0000","last_clean":"2024-09-21T02:37:39.867226+0000","last_became_active":"2024-09-21T02:36:25.746426+0000","last_became_peered":"2024-09-21T02:36:25.746426+0000","last_unstale":"2024-09-21T02:37:39.867226+0000","last_undegraded":"2024-09-21T02:37:39.867226+0000","last_fullsized":"2024-09-21T02:37:39.867226+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:12.835851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606065+0000","last_change":"2024-09-21T02:36:25.751278+0000","last_active":"2024-09-21T02:37:39.606065+0000","last_peered":"2024-09-21T02:37:39.606065+0000","last_clean":"2024-09-21T02:37:39.606065+0000","last_became_active":"2024-09-21T02:36:25.750414+0000","last_became_peered":"2024-09-21T02:36:25.750414+0000","last_unstale":"2024-09-21T02:37:39.606065+0000","last_undegraded":"2024-09-21T02:37:39.606065+0000","last_fullsized":"2024-09-21T02:37:39.606065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:13:56.080122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605503+0000","last_change":"2024-09-21T02:36:25.752852+0000","last_active":"2024-09-21T02:37:39.605503+0000","last_peered":"2024-09-21T02:37:39.605503+0000","last_clean":"2024-09-21T02:37:39.605503+0000","last_became_active":"2024-09-21T02:36:25.752514+0000","last_became_peered":"2024-09-21T02:36:25.752514+0000","last_unstale":"2024-09-21T02:37:39.605503+0000","last_undegraded":"2024-09-21T02:37:39.605503+0000","last_fullsized":"2024-09-21T02:37:39.605503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:55.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607756+0000","last_change":"2024-09-21T02:36:25.750081+0000","last_active":"2024-09-21T02:37:39.607756+0000","last_peered":"2024-09-21T02:37:39.607756+0000","last_clean":"2024-09-21T02:37:39.607756+0000","last_became_active":"2024-09-21T02:36:25.749602+0000","last_became_peered":"2024-09-21T02:36:25.749602+0000","last_unstale":"2024-09-21T02:37:39.607756+0000","last_undegraded":"2024-09-21T02:37:39.607756+0000","last_fullsized":"2024-09-21T02:37:39.607756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:06.785998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608041+0000","last_change":"2024-09-21T02:36:25.746897+0000","last_active":"2024-09-21T02:37:39.608041+0000","last_peered":"2024-09-21T02:37:39.608041+0000","last_clean":"2024-09-21T02:37:39.608041+0000","last_became_active":"2024-09-21T02:36:25.746477+0000","last_became_peered":"2024-09-21T02:36:25.746477+0000","last_unstale":"2024-09-21T02:37:39.608041+0000","last_undegraded":"2024-09-21T02:37:39.608041+0000","last_fullsized":"2024-09-21T02:37:39.608041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:34:00.739410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604837+0000","last_change":"2024-09-21T02:36:25.745233+0000","last_active":"2024-09-21T02:37:39.604837+0000","last_peered":"2024-09-21T02:37:39.604837+0000","last_clean":"2024-09-21T02:37:39.604837+0000","last_became_active":"2024-09-21T02:36:25.744637+0000","last_became_peered":"2024-09-21T02:36:25.744637+0000","last_unstale":"2024-09-21T02:37:39.604837+0000","last_undegraded":"2024-09-21T02:37:39.604837+0000","last_fullsized":"2024-09-21T02:37:39.604837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:34:59.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74824,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":620,"num_read_kb":392,"num_write":252,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":224472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249064,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":69,"seq":296352743458,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":467025,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906983,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":1125054,"data_compressed":59828,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070509,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":942080,"data_stored":466452,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266738,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":929792,"data_stored":466718,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462968,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":991232,"data_stored":1063327,"data_compressed":59828,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659197,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":917504,"data_stored":461956,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855426,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":1058909,"data_compressed":59828,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018950,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":925696,"data_stored":466030,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215179,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":929792,"data_stored":461664,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1011712,"data_stored":531608,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":991232,"data_stored":530606,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":942080,"data_stored":463803,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:38:12.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:38:12.801 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:12.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:12 smithi029 ceph-mon[31006]: pgmap v360: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:13.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:12 smithi066 ceph-mon[33436]: pgmap v360: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:13.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:12 smithi190 ceph-mon[33871]: pgmap v360: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.332+0000 7f5cec0fd640 1 -- 172.21.15.29:0/22362431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 msgr2=0x7f5ce4110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.332+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/22362431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 0x7f5ce4110620 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009ca0 tx=0x7f5cdc02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.332+0000 7f5cec0fd640 1 -- 172.21.15.29:0/22362431 shutdown_connections 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.332+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/22362431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 0x7f5ce4110620 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.332+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/22362431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ce41094a0 0x7f5ce41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.332+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/22362431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ce4103b60 0x7f5ce4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.332+0000 7f5cec0fd640 1 -- 172.21.15.29:0/22362431 >> 172.21.15.29:0/22362431 conn(0x7f5ce407b850 msgr2=0x7f5ce407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.333+0000 7f5cec0fd640 1 -- 172.21.15.29:0/22362431 shutdown_connections 2024-09-21T02:38:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.333+0000 7f5cec0fd640 1 -- 172.21.15.29:0/22362431 wait complete. 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.333+0000 7f5cec0fd640 1 Processor -- start 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.333+0000 7f5cec0fd640 1 -- start start 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.333+0000 7f5cec0fd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ce4103b60 0x7f5ce419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cec0fd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ce41094a0 0x7f5ce419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cec0fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 0x7f5ce419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cec0fd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41139b0 con 0x7f5ce4109df0 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cec0fd640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4113830 con 0x7f5ce4103b60 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cec0fd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4113b30 con 0x7f5ce41094a0 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 0x7f5ce419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 0x7f5ce419fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44988/0 (socket says 172.21.15.29:44988) 2024-09-21T02:38:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 -- 172.21.15.29:0/2796763933 learned_addr learned my addr 172.21.15.29:0/2796763933 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5ce9698640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ce41094a0 0x7f5ce419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5ce9e99640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ce4103b60 0x7f5ce419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 -- 172.21.15.29:0/2796763933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ce41094a0 msgr2=0x7f5ce419a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ce41094a0 0x7f5ce419a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 -- 172.21.15.29:0/2796763933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ce4103b60 msgr2=0x7f5ce419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ce4103b60 0x7f5ce419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 -- 172.21.15.29:0/2796763933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce41a0170 con 0x7f5ce4109df0 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cea69a640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 0x7f5ce419fb00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005cc0 tx=0x7f5cdc032c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.334+0000 7f5cdaffd640 1 -- 172.21.15.29:0/2796763933 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc004040 con 0x7f5ce4109df0 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.335+0000 7f5cdaffd640 1 -- 172.21.15.29:0/2796763933 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5cdc004670 con 0x7f5ce4109df0 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.335+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce41a0400 con 0x7f5ce4109df0 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.335+0000 7f5cdaffd640 1 -- 172.21.15.29:0/2796763933 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc030cc0 con 0x7f5ce4109df0 2024-09-21T02:38:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.335+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce41a0940 con 0x7f5ce4109df0 2024-09-21T02:38:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.336+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce4103f70 con 0x7f5ce4109df0 2024-09-21T02:38:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.337+0000 7f5cdaffd640 1 -- 172.21.15.29:0/2796763933 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f5cdc0041e0 con 0x7f5ce4109df0 2024-09-21T02:38:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.338+0000 7f5cdaffd640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5cd00b65d0 0x7f5cd00b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.338+0000 7f5ce9e99640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5cd00b65d0 0x7f5cd00b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.338+0000 7f5cdaffd640 1 -- 172.21.15.29:0/2796763933 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f5cdc108d00 con 0x7f5ce4109df0 2024-09-21T02:38:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.339+0000 7f5ce9e99640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5cd00b65d0 0x7f5cd00b8ae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000abe0 tx=0x7f5ce000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.341+0000 7f5cdaffd640 1 -- 172.21.15.29:0/2796763933 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5cdc0cf190 con 0x7f5ce4109df0 2024-09-21T02:38:13.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.528+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5ce4063800 con 0x7f5cd00b65d0 2024-09-21T02:38:13.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.534+0000 7f5cdaffd640 1 -- 172.21.15.29:0/2796763933 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340651 (secure 0 0 0) 0x7f5ce4063800 con 0x7f5cd00b65d0 2024-09-21T02:38:13.538 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:13.539 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T02:38:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5cd00b65d0 msgr2=0x7f5cd00b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5cd00b65d0 0x7f5cd00b8ae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000abe0 tx=0x7f5ce000a2d0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 msgr2=0x7f5ce419fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 0x7f5ce419fb00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005cc0 tx=0x7f5cdc032c40 comp rx=0 tx=0).stop 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 shutdown_connections 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5cd00b65d0 0x7f5cd00b8ae0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ce4109df0 0x7f5ce419fb00 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ce41094a0 0x7f5ce419a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 --2- 172.21.15.29:0/2796763933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ce4103b60 0x7f5ce419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 >> 172.21.15.29:0/2796763933 conn(0x7f5ce407b850 msgr2=0x7f5ce4107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 shutdown_connections 2024-09-21T02:38:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:13.538+0000 7f5cec0fd640 1 -- 172.21.15.29:0/2796763933 wait complete. 2024-09-21T02:38:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:13 smithi029 ceph-mon[31006]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:38:14.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:13 smithi066 ceph-mon[33436]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:38:14.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:13 smithi190 ceph-mon[33871]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:38:14.318 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-21T02:38:11.958394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665426,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":706,"num_read_kb":456,"num_write":377,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":2952069,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364796,"kb_used_data":11312,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124577028,"statfs":{"total":1151940427776,"available":1151566876672,"internally_reserved":0,"allocated":11583488,"data_stored":7563152,"data_compressed":391470,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003946"},"pg_stats":[{"pgid":"2.1f","version":"81'13","reported_seq":107,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:06.129508+0000","last_change":"2024-09-21T02:36:23.729337+0000","last_active":"2024-09-21T02:38:06.129508+0000","last_peered":"2024-09-21T02:38:06.129508+0000","last_clean":"2024-09-21T02:38:06.129508+0000","last_became_active":"2024-09-21T02:36:23.728576+0000","last_became_peered":"2024-09-21T02:36:23.728576+0000","last_unstale":"2024-09-21T02:38:06.129508+0000","last_undegraded":"2024-09-21T02:38:06.129508+0000","last_fullsized":"2024-09-21T02:38:06.129508+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:31.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":59,"num_read_kb":59,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866296+0000","last_change":"2024-09-21T02:36:25.748925+0000","last_active":"2024-09-21T02:37:39.866296+0000","last_peered":"2024-09-21T02:37:39.866296+0000","last_clean":"2024-09-21T02:37:39.866296+0000","last_became_active":"2024-09-21T02:36:25.748333+0000","last_became_peered":"2024-09-21T02:36:25.748333+0000","last_unstale":"2024-09-21T02:37:39.866296+0000","last_undegraded":"2024-09-21T02:37:39.866296+0000","last_fullsized":"2024-09-21T02:37:39.866296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:46.889497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602004+0000","last_change":"2024-09-21T02:36:32.807771+0000","last_active":"2024-09-21T02:37:39.602004+0000","last_peered":"2024-09-21T02:37:39.602004+0000","last_clean":"2024-09-21T02:37:39.602004+0000","last_became_active":"2024-09-21T02:36:32.807291+0000","last_became_peered":"2024-09-21T02:36:32.807291+0000","last_unstale":"2024-09-21T02:37:39.602004+0000","last_undegraded":"2024-09-21T02:37:39.602004+0000","last_fullsized":"2024-09-21T02:37:39.602004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:41.468318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866613+0000","last_change":"2024-09-21T02:36:23.800779+0000","last_active":"2024-09-21T02:37:39.866613+0000","last_peered":"2024-09-21T02:37:39.866613+0000","last_clean":"2024-09-21T02:37:39.866613+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:37:39.866613+0000","last_undegraded":"2024-09-21T02:37:39.866613+0000","last_fullsized":"2024-09-21T02:37:39.866613+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:24.843233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608328+0000","last_change":"2024-09-21T02:36:25.745277+0000","last_active":"2024-09-21T02:37:39.608328+0000","last_peered":"2024-09-21T02:37:39.608328+0000","last_clean":"2024-09-21T02:37:39.608328+0000","last_became_active":"2024-09-21T02:36:25.744501+0000","last_became_peered":"2024-09-21T02:36:25.744501+0000","last_unstale":"2024-09-21T02:37:39.608328+0000","last_undegraded":"2024-09-21T02:37:39.608328+0000","last_fullsized":"2024-09-21T02:37:39.608328+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:14.776250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603390+0000","last_change":"2024-09-21T02:36:32.801978+0000","last_active":"2024-09-21T02:37:39.603390+0000","last_peered":"2024-09-21T02:37:39.603390+0000","last_clean":"2024-09-21T02:37:39.603390+0000","last_became_active":"2024-09-21T02:36:32.801436+0000","last_became_peered":"2024-09-21T02:36:32.801436+0000","last_unstale":"2024-09-21T02:37:39.603390+0000","last_undegraded":"2024-09-21T02:37:39.603390+0000","last_fullsized":"2024-09-21T02:37:39.603390+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:37.380417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606474+0000","last_change":"2024-09-21T02:36:23.729336+0000","last_active":"2024-09-21T02:37:39.606474+0000","last_peered":"2024-09-21T02:37:39.606474+0000","last_clean":"2024-09-21T02:37:39.606474+0000","last_became_active":"2024-09-21T02:36:23.728724+0000","last_became_peered":"2024-09-21T02:36:23.728724+0000","last_unstale":"2024-09-21T02:37:39.606474+0000","last_undegraded":"2024-09-21T02:37:39.606474+0000","last_fullsized":"2024-09-21T02:37:39.606474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:31:52.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867454+0000","last_change":"2024-09-21T02:36:25.737577+0000","last_active":"2024-09-21T02:37:39.867454+0000","last_peered":"2024-09-21T02:37:39.867454+0000","last_clean":"2024-09-21T02:37:39.867454+0000","last_became_active":"2024-09-21T02:36:25.737221+0000","last_became_peered":"2024-09-21T02:36:25.737221+0000","last_unstale":"2024-09-21T02:37:39.867454+0000","last_undegraded":"2024-09-21T02:37:39.867454+0000","last_fullsized":"2024-09-21T02:37:39.867454+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:42:54.786464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604191+0000","last_change":"2024-09-21T02:36:32.802507+0000","last_active":"2024-09-21T02:37:39.604191+0000","last_peered":"2024-09-21T02:37:39.604191+0000","last_clean":"2024-09-21T02:37:39.604191+0000","last_became_active":"2024-09-21T02:36:32.801879+0000","last_became_peered":"2024-09-21T02:36:32.801879+0000","last_unstale":"2024-09-21T02:37:39.604191+0000","last_undegraded":"2024-09-21T02:37:39.604191+0000","last_fullsized":"2024-09-21T02:37:39.604191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:09.873825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605829+0000","last_change":"2024-09-21T02:36:23.730889+0000","last_active":"2024-09-21T02:37:39.605829+0000","last_peered":"2024-09-21T02:37:39.605829+0000","last_clean":"2024-09-21T02:37:39.605829+0000","last_became_active":"2024-09-21T02:36:23.730551+0000","last_became_peered":"2024-09-21T02:36:23.730551+0000","last_unstale":"2024-09-21T02:37:39.605829+0000","last_undegraded":"2024-09-21T02:37:39.605829+0000","last_fullsized":"2024-09-21T02:37:39.605829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:17.178848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.869371+0000","last_change":"2024-09-21T02:37:39.869272+0000","last_active":"2024-09-21T02:37:39.869371+0000","last_peered":"2024-09-21T02:37:39.869371+0000","last_clean":"2024-09-21T02:37:39.869371+0000","last_became_active":"2024-09-21T02:37:39.868824+0000","last_became_peered":"2024-09-21T02:37:39.868824+0000","last_unstale":"2024-09-21T02:37:39.869371+0000","last_undegraded":"2024-09-21T02:37:39.869371+0000","last_fullsized":"2024-09-21T02:37:39.869371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:05:31.503359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603327+0000","last_change":"2024-09-21T02:36:32.800489+0000","last_active":"2024-09-21T02:37:39.603327+0000","last_peered":"2024-09-21T02:37:39.603327+0000","last_clean":"2024-09-21T02:37:39.603327+0000","last_became_active":"2024-09-21T02:36:32.799876+0000","last_became_peered":"2024-09-21T02:36:32.799876+0000","last_unstale":"2024-09-21T02:37:39.603327+0000","last_undegraded":"2024-09-21T02:37:39.603327+0000","last_fullsized":"2024-09-21T02:37:39.603327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:02:56.711257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604876+0000","last_change":"2024-09-21T02:36:25.744561+0000","last_active":"2024-09-21T02:37:39.604876+0000","last_peered":"2024-09-21T02:37:39.604876+0000","last_clean":"2024-09-21T02:37:39.604876+0000","last_became_active":"2024-09-21T02:36:25.743757+0000","last_became_peered":"2024-09-21T02:36:25.743757+0000","last_unstale":"2024-09-21T02:37:39.604876+0000","last_undegraded":"2024-09-21T02:37:39.604876+0000","last_fullsized":"2024-09-21T02:37:39.604876+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:30.235517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.616747+0000","last_change":"2024-09-21T02:37:39.616628+0000","last_active":"2024-09-21T02:37:39.616747+0000","last_peered":"2024-09-21T02:37:39.616747+0000","last_clean":"2024-09-21T02:37:39.616747+0000","last_became_active":"2024-09-21T02:37:39.609108+0000","last_became_peered":"2024-09-21T02:37:39.609108+0000","last_unstale":"2024-09-21T02:37:39.616747+0000","last_undegraded":"2024-09-21T02:37:39.616747+0000","last_fullsized":"2024-09-21T02:37:39.616747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:11:28.944244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603713+0000","last_change":"2024-09-21T02:36:33.003670+0000","last_active":"2024-09-21T02:37:39.603713+0000","last_peered":"2024-09-21T02:37:39.603713+0000","last_clean":"2024-09-21T02:37:39.603713+0000","last_became_active":"2024-09-21T02:36:33.003022+0000","last_became_peered":"2024-09-21T02:36:33.003022+0000","last_unstale":"2024-09-21T02:37:39.603713+0000","last_undegraded":"2024-09-21T02:37:39.603713+0000","last_fullsized":"2024-09-21T02:37:39.603713+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:12.753375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866246+0000","last_change":"2024-09-21T02:36:25.749404+0000","last_active":"2024-09-21T02:37:39.866246+0000","last_peered":"2024-09-21T02:37:39.866246+0000","last_clean":"2024-09-21T02:37:39.866246+0000","last_became_active":"2024-09-21T02:36:25.747854+0000","last_became_peered":"2024-09-21T02:36:25.747854+0000","last_unstale":"2024-09-21T02:37:39.866246+0000","last_undegraded":"2024-09-21T02:37:39.866246+0000","last_fullsized":"2024-09-21T02:37:39.866246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:41.224740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602302+0000","last_change":"2024-09-21T02:36:23.726689+0000","last_active":"2024-09-21T02:37:39.602302+0000","last_peered":"2024-09-21T02:37:39.602302+0000","last_clean":"2024-09-21T02:37:39.602302+0000","last_became_active":"2024-09-21T02:36:23.726165+0000","last_became_peered":"2024-09-21T02:36:23.726165+0000","last_unstale":"2024-09-21T02:37:39.602302+0000","last_undegraded":"2024-09-21T02:37:39.602302+0000","last_fullsized":"2024-09-21T02:37:39.602302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:58.166457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604112+0000","last_change":"2024-09-21T02:36:32.809594+0000","last_active":"2024-09-21T02:37:39.604112+0000","last_peered":"2024-09-21T02:37:39.604112+0000","last_clean":"2024-09-21T02:37:39.604112+0000","last_became_active":"2024-09-21T02:36:32.808893+0000","last_became_peered":"2024-09-21T02:36:32.808893+0000","last_unstale":"2024-09-21T02:37:39.604112+0000","last_undegraded":"2024-09-21T02:37:39.604112+0000","last_fullsized":"2024-09-21T02:37:39.604112+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:55.187241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605126+0000","last_change":"2024-09-21T02:36:25.755955+0000","last_active":"2024-09-21T02:37:39.605126+0000","last_peered":"2024-09-21T02:37:39.605126+0000","last_clean":"2024-09-21T02:37:39.605126+0000","last_became_active":"2024-09-21T02:36:25.755577+0000","last_became_peered":"2024-09-21T02:36:25.755577+0000","last_unstale":"2024-09-21T02:37:39.605126+0000","last_undegraded":"2024-09-21T02:37:39.605126+0000","last_fullsized":"2024-09-21T02:37:39.605126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:48.447758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866328+0000","last_change":"2024-09-21T02:36:23.730043+0000","last_active":"2024-09-21T02:37:39.866328+0000","last_peered":"2024-09-21T02:37:39.866328+0000","last_clean":"2024-09-21T02:37:39.866328+0000","last_became_active":"2024-09-21T02:36:23.729576+0000","last_became_peered":"2024-09-21T02:36:23.729576+0000","last_unstale":"2024-09-21T02:37:39.866328+0000","last_undegraded":"2024-09-21T02:37:39.866328+0000","last_fullsized":"2024-09-21T02:37:39.866328+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:48:09.530913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601976+0000","last_change":"2024-09-21T02:36:32.806923+0000","last_active":"2024-09-21T02:37:39.601976+0000","last_peered":"2024-09-21T02:37:39.601976+0000","last_clean":"2024-09-21T02:37:39.601976+0000","last_became_active":"2024-09-21T02:36:32.806176+0000","last_became_peered":"2024-09-21T02:36:32.806176+0000","last_unstale":"2024-09-21T02:37:39.601976+0000","last_undegraded":"2024-09-21T02:37:39.601976+0000","last_fullsized":"2024-09-21T02:37:39.601976+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.165638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605188+0000","last_change":"2024-09-21T02:36:25.747501+0000","last_active":"2024-09-21T02:37:39.605188+0000","last_peered":"2024-09-21T02:37:39.605188+0000","last_clean":"2024-09-21T02:37:39.605188+0000","last_became_active":"2024-09-21T02:36:25.747111+0000","last_became_peered":"2024-09-21T02:36:25.747111+0000","last_unstale":"2024-09-21T02:37:39.605188+0000","last_undegraded":"2024-09-21T02:37:39.605188+0000","last_fullsized":"2024-09-21T02:37:39.605188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:03:04.061537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866583+0000","last_change":"2024-09-21T02:36:23.800802+0000","last_active":"2024-09-21T02:37:39.866583+0000","last_peered":"2024-09-21T02:37:39.866583+0000","last_clean":"2024-09-21T02:37:39.866583+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:37:39.866583+0000","last_undegraded":"2024-09-21T02:37:39.866583+0000","last_fullsized":"2024-09-21T02:37:39.866583+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:44:34.165406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.870161+0000","last_change":"2024-09-21T02:37:39.870041+0000","last_active":"2024-09-21T02:37:39.870161+0000","last_peered":"2024-09-21T02:37:39.870161+0000","last_clean":"2024-09-21T02:37:39.870161+0000","last_became_active":"2024-09-21T02:37:39.869518+0000","last_became_peered":"2024-09-21T02:37:39.869518+0000","last_unstale":"2024-09-21T02:37:39.870161+0000","last_undegraded":"2024-09-21T02:37:39.870161+0000","last_fullsized":"2024-09-21T02:37:39.870161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:20.276471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608088+0000","last_change":"2024-09-21T02:36:25.746139+0000","last_active":"2024-09-21T02:37:39.608088+0000","last_peered":"2024-09-21T02:37:39.608088+0000","last_clean":"2024-09-21T02:37:39.608088+0000","last_became_active":"2024-09-21T02:36:25.744418+0000","last_became_peered":"2024-09-21T02:36:25.744418+0000","last_unstale":"2024-09-21T02:37:39.608088+0000","last_undegraded":"2024-09-21T02:37:39.608088+0000","last_fullsized":"2024-09-21T02:37:39.608088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:21.706782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603890+0000","last_change":"2024-09-21T02:36:24.261945+0000","last_active":"2024-09-21T02:37:39.603890+0000","last_peered":"2024-09-21T02:37:39.603890+0000","last_clean":"2024-09-21T02:37:39.603890+0000","last_became_active":"2024-09-21T02:36:24.261486+0000","last_became_peered":"2024-09-21T02:36:24.261486+0000","last_unstale":"2024-09-21T02:37:39.603890+0000","last_undegraded":"2024-09-21T02:37:39.603890+0000","last_fullsized":"2024-09-21T02:37:39.603890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:45:18.331035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602209+0000","last_change":"2024-09-21T02:36:32.799464+0000","last_active":"2024-09-21T02:37:39.602209+0000","last_peered":"2024-09-21T02:37:39.602209+0000","last_clean":"2024-09-21T02:37:39.602209+0000","last_became_active":"2024-09-21T02:36:32.799000+0000","last_became_peered":"2024-09-21T02:36:32.799000+0000","last_unstale":"2024-09-21T02:37:39.602209+0000","last_undegraded":"2024-09-21T02:37:39.602209+0000","last_fullsized":"2024-09-21T02:37:39.602209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:56:36.271172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866824+0000","last_change":"2024-09-21T02:36:25.741151+0000","last_active":"2024-09-21T02:37:39.866824+0000","last_peered":"2024-09-21T02:37:39.866824+0000","last_clean":"2024-09-21T02:37:39.866824+0000","last_became_active":"2024-09-21T02:36:25.740803+0000","last_became_peered":"2024-09-21T02:36:25.740803+0000","last_unstale":"2024-09-21T02:37:39.866824+0000","last_undegraded":"2024-09-21T02:37:39.866824+0000","last_fullsized":"2024-09-21T02:37:39.866824+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:44.585136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'16","reported_seq":51,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601919+0000","last_change":"2024-09-21T02:36:23.725155+0000","last_active":"2024-09-21T02:37:39.601919+0000","last_peered":"2024-09-21T02:37:39.601919+0000","last_clean":"2024-09-21T02:37:39.601919+0000","last_became_active":"2024-09-21T02:36:23.724791+0000","last_became_peered":"2024-09-21T02:36:23.724791+0000","last_unstale":"2024-09-21T02:37:39.601919+0000","last_undegraded":"2024-09-21T02:37:39.601919+0000","last_fullsized":"2024-09-21T02:37:39.601919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:06.520465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.865963+0000","last_change":"2024-09-21T02:36:33.003401+0000","last_active":"2024-09-21T02:37:39.865963+0000","last_peered":"2024-09-21T02:37:39.865963+0000","last_clean":"2024-09-21T02:37:39.865963+0000","last_became_active":"2024-09-21T02:36:33.002930+0000","last_became_peered":"2024-09-21T02:36:33.002930+0000","last_unstale":"2024-09-21T02:37:39.865963+0000","last_undegraded":"2024-09-21T02:37:39.865963+0000","last_fullsized":"2024-09-21T02:37:39.865963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:40:23.629763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605001+0000","last_change":"2024-09-21T02:36:25.735501+0000","last_active":"2024-09-21T02:37:39.605001+0000","last_peered":"2024-09-21T02:37:39.605001+0000","last_clean":"2024-09-21T02:37:39.605001+0000","last_became_active":"2024-09-21T02:36:25.734453+0000","last_became_peered":"2024-09-21T02:36:25.734453+0000","last_unstale":"2024-09-21T02:37:39.605001+0000","last_undegraded":"2024-09-21T02:37:39.605001+0000","last_fullsized":"2024-09-21T02:37:39.605001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:50.238529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606569+0000","last_change":"2024-09-21T02:36:23.800338+0000","last_active":"2024-09-21T02:37:39.606569+0000","last_peered":"2024-09-21T02:37:39.606569+0000","last_clean":"2024-09-21T02:37:39.606569+0000","last_became_active":"2024-09-21T02:36:23.799796+0000","last_became_peered":"2024-09-21T02:36:23.799796+0000","last_unstale":"2024-09-21T02:37:39.606569+0000","last_undegraded":"2024-09-21T02:37:39.606569+0000","last_fullsized":"2024-09-21T02:37:39.606569+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:00:18.459515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.611641+0000","last_change":"2024-09-21T02:37:39.611513+0000","last_active":"2024-09-21T02:37:39.611641+0000","last_peered":"2024-09-21T02:37:39.611641+0000","last_clean":"2024-09-21T02:37:39.611641+0000","last_became_active":"2024-09-21T02:37:39.610843+0000","last_became_peered":"2024-09-21T02:37:39.610843+0000","last_unstale":"2024-09-21T02:37:39.611641+0000","last_undegraded":"2024-09-21T02:37:39.611641+0000","last_fullsized":"2024-09-21T02:37:39.611641+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:51.911830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.868107+0000","last_change":"2024-09-21T02:36:25.745497+0000","last_active":"2024-09-21T02:37:39.868107+0000","last_peered":"2024-09-21T02:37:39.868107+0000","last_clean":"2024-09-21T02:37:39.868107+0000","last_became_active":"2024-09-21T02:36:25.745126+0000","last_became_peered":"2024-09-21T02:36:25.745126+0000","last_unstale":"2024-09-21T02:37:39.868107+0000","last_undegraded":"2024-09-21T02:37:39.868107+0000","last_fullsized":"2024-09-21T02:37:39.868107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:42:17.835784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601889+0000","last_change":"2024-09-21T02:36:23.802440+0000","last_active":"2024-09-21T02:37:39.601889+0000","last_peered":"2024-09-21T02:37:39.601889+0000","last_clean":"2024-09-21T02:37:39.601889+0000","last_became_active":"2024-09-21T02:36:23.802065+0000","last_became_peered":"2024-09-21T02:36:23.802065+0000","last_unstale":"2024-09-21T02:37:39.601889+0000","last_undegraded":"2024-09-21T02:37:39.601889+0000","last_fullsized":"2024-09-21T02:37:39.601889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:22:20.647476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605448+0000","last_change":"2024-09-21T02:36:33.003465+0000","last_active":"2024-09-21T02:37:39.605448+0000","last_peered":"2024-09-21T02:37:39.605448+0000","last_clean":"2024-09-21T02:37:39.605448+0000","last_became_active":"2024-09-21T02:36:33.002874+0000","last_became_peered":"2024-09-21T02:36:33.002874+0000","last_unstale":"2024-09-21T02:37:39.605448+0000","last_undegraded":"2024-09-21T02:37:39.605448+0000","last_fullsized":"2024-09-21T02:37:39.605448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:02.388770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867451+0000","last_change":"2024-09-21T02:36:25.746092+0000","last_active":"2024-09-21T02:37:39.867451+0000","last_peered":"2024-09-21T02:37:39.867451+0000","last_clean":"2024-09-21T02:37:39.867451+0000","last_became_active":"2024-09-21T02:36:25.745730+0000","last_became_peered":"2024-09-21T02:36:25.745730+0000","last_unstale":"2024-09-21T02:37:39.867451+0000","last_undegraded":"2024-09-21T02:37:39.867451+0000","last_fullsized":"2024-09-21T02:37:39.867451+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:24:13.212450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":101,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:08.133112+0000","last_change":"2024-09-21T02:36:23.732090+0000","last_active":"2024-09-21T02:38:08.133112+0000","last_peered":"2024-09-21T02:38:08.133112+0000","last_clean":"2024-09-21T02:38:08.133112+0000","last_became_active":"2024-09-21T02:36:23.731653+0000","last_became_peered":"2024-09-21T02:36:23.731653+0000","last_unstale":"2024-09-21T02:38:08.133112+0000","last_undegraded":"2024-09-21T02:38:08.133112+0000","last_fullsized":"2024-09-21T02:38:08.133112+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:02:42.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604825+0000","last_change":"2024-09-21T02:36:32.794893+0000","last_active":"2024-09-21T02:37:39.604825+0000","last_peered":"2024-09-21T02:37:39.604825+0000","last_clean":"2024-09-21T02:37:39.604825+0000","last_became_active":"2024-09-21T02:36:32.794406+0000","last_became_peered":"2024-09-21T02:36:32.794406+0000","last_unstale":"2024-09-21T02:37:39.604825+0000","last_undegraded":"2024-09-21T02:37:39.604825+0000","last_fullsized":"2024-09-21T02:37:39.604825+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:40:39.621502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608789+0000","last_change":"2024-09-21T02:36:25.748220+0000","last_active":"2024-09-21T02:37:39.608789+0000","last_peered":"2024-09-21T02:37:39.608789+0000","last_clean":"2024-09-21T02:37:39.608789+0000","last_became_active":"2024-09-21T02:36:25.747553+0000","last_became_peered":"2024-09-21T02:36:25.747553+0000","last_unstale":"2024-09-21T02:37:39.608789+0000","last_undegraded":"2024-09-21T02:37:39.608789+0000","last_fullsized":"2024-09-21T02:37:39.608789+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:28.757981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":100,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:07.131874+0000","last_change":"2024-09-21T02:36:23.801087+0000","last_active":"2024-09-21T02:38:07.131874+0000","last_peered":"2024-09-21T02:38:07.131874+0000","last_clean":"2024-09-21T02:38:07.131874+0000","last_became_active":"2024-09-21T02:36:23.800077+0000","last_became_peered":"2024-09-21T02:36:23.800077+0000","last_unstale":"2024-09-21T02:38:07.131874+0000","last_undegraded":"2024-09-21T02:38:07.131874+0000","last_fullsized":"2024-09-21T02:38:07.131874+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:50.074650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":60,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605077+0000","last_change":"2024-09-21T02:36:33.003354+0000","last_active":"2024-09-21T02:37:39.605077+0000","last_peered":"2024-09-21T02:37:39.605077+0000","last_clean":"2024-09-21T02:37:39.605077+0000","last_became_active":"2024-09-21T02:36:33.002912+0000","last_became_peered":"2024-09-21T02:36:33.002912+0000","last_unstale":"2024-09-21T02:37:39.605077+0000","last_undegraded":"2024-09-21T02:37:39.605077+0000","last_fullsized":"2024-09-21T02:37:39.605077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:21.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604307+0000","last_change":"2024-09-21T02:36:25.741701+0000","last_active":"2024-09-21T02:37:39.604307+0000","last_peered":"2024-09-21T02:37:39.604307+0000","last_clean":"2024-09-21T02:37:39.604307+0000","last_became_active":"2024-09-21T02:36:25.741222+0000","last_became_peered":"2024-09-21T02:36:25.741222+0000","last_unstale":"2024-09-21T02:37:39.604307+0000","last_undegraded":"2024-09-21T02:37:39.604307+0000","last_fullsized":"2024-09-21T02:37:39.604307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:35:39.896852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604926+0000","last_change":"2024-09-21T02:36:23.728793+0000","last_active":"2024-09-21T02:37:39.604926+0000","last_peered":"2024-09-21T02:37:39.604926+0000","last_clean":"2024-09-21T02:37:39.604926+0000","last_became_active":"2024-09-21T02:36:23.728362+0000","last_became_peered":"2024-09-21T02:36:23.728362+0000","last_unstale":"2024-09-21T02:37:39.604926+0000","last_undegraded":"2024-09-21T02:37:39.604926+0000","last_fullsized":"2024-09-21T02:37:39.604926+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:28:15.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610665+0000","last_change":"2024-09-21T02:36:33.005225+0000","last_active":"2024-09-21T02:37:39.610665+0000","last_peered":"2024-09-21T02:37:39.610665+0000","last_clean":"2024-09-21T02:37:39.610665+0000","last_became_active":"2024-09-21T02:36:33.004756+0000","last_became_peered":"2024-09-21T02:36:33.004756+0000","last_unstale":"2024-09-21T02:37:39.610665+0000","last_undegraded":"2024-09-21T02:37:39.610665+0000","last_fullsized":"2024-09-21T02:37:39.610665+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:02:16.686284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866648+0000","last_change":"2024-09-21T02:36:25.745639+0000","last_active":"2024-09-21T02:37:39.866648+0000","last_peered":"2024-09-21T02:37:39.866648+0000","last_clean":"2024-09-21T02:37:39.866648+0000","last_became_active":"2024-09-21T02:36:25.745202+0000","last_became_peered":"2024-09-21T02:36:25.745202+0000","last_unstale":"2024-09-21T02:37:39.866648+0000","last_undegraded":"2024-09-21T02:37:39.866648+0000","last_fullsized":"2024-09-21T02:37:39.866648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:08.712477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604344+0000","last_change":"2024-09-21T02:36:23.730710+0000","last_active":"2024-09-21T02:37:39.604344+0000","last_peered":"2024-09-21T02:37:39.604344+0000","last_clean":"2024-09-21T02:37:39.604344+0000","last_became_active":"2024-09-21T02:36:23.730247+0000","last_became_peered":"2024-09-21T02:36:23.730247+0000","last_unstale":"2024-09-21T02:37:39.604344+0000","last_undegraded":"2024-09-21T02:37:39.604344+0000","last_fullsized":"2024-09-21T02:37:39.604344+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:43.863444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605253+0000","last_change":"2024-09-21T02:35:09.949841+0000","last_active":"2024-09-21T02:37:39.605253+0000","last_peered":"2024-09-21T02:37:39.605253+0000","last_clean":"2024-09-21T02:37:39.605253+0000","last_became_active":"2024-09-21T02:35:09.944224+0000","last_became_peered":"2024-09-21T02:35:09.944224+0000","last_unstale":"2024-09-21T02:37:39.605253+0000","last_undegraded":"2024-09-21T02:37:39.605253+0000","last_fullsized":"2024-09-21T02:37:39.605253+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_clean_scrub_stamp":"2024-09-21T02:31:38.930125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:56.467576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.874595+0000","last_change":"2024-09-21T02:37:39.874486+0000","last_active":"2024-09-21T02:37:39.874595+0000","last_peered":"2024-09-21T02:37:39.874595+0000","last_clean":"2024-09-21T02:37:39.874595+0000","last_became_active":"2024-09-21T02:37:39.869523+0000","last_became_peered":"2024-09-21T02:37:39.869523+0000","last_unstale":"2024-09-21T02:37:39.874595+0000","last_undegraded":"2024-09-21T02:37:39.874595+0000","last_fullsized":"2024-09-21T02:37:39.874595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:21:33.524428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606052+0000","last_change":"2024-09-21T02:36:25.741176+0000","last_active":"2024-09-21T02:37:39.606052+0000","last_peered":"2024-09-21T02:37:39.606052+0000","last_clean":"2024-09-21T02:37:39.606052+0000","last_became_active":"2024-09-21T02:36:25.740355+0000","last_became_peered":"2024-09-21T02:36:25.740355+0000","last_unstale":"2024-09-21T02:37:39.606052+0000","last_undegraded":"2024-09-21T02:37:39.606052+0000","last_fullsized":"2024-09-21T02:37:39.606052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:47:48.204768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'6","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:06.131232+0000","last_change":"2024-09-21T02:36:24.261500+0000","last_active":"2024-09-21T02:38:06.131232+0000","last_peered":"2024-09-21T02:38:06.131232+0000","last_clean":"2024-09-21T02:38:06.131232+0000","last_became_active":"2024-09-21T02:36:24.261089+0000","last_became_peered":"2024-09-21T02:36:24.261089+0000","last_unstale":"2024-09-21T02:38:06.131232+0000","last_undegraded":"2024-09-21T02:38:06.131232+0000","last_fullsized":"2024-09-21T02:38:06.131232+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:33:33.158177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866524+0000","last_change":"2024-09-21T02:36:32.796980+0000","last_active":"2024-09-21T02:37:39.866524+0000","last_peered":"2024-09-21T02:37:39.866524+0000","last_clean":"2024-09-21T02:37:39.866524+0000","last_became_active":"2024-09-21T02:36:32.796517+0000","last_became_peered":"2024-09-21T02:36:32.796517+0000","last_unstale":"2024-09-21T02:37:39.866524+0000","last_undegraded":"2024-09-21T02:37:39.866524+0000","last_fullsized":"2024-09-21T02:37:39.866524+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:07.230177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605201+0000","last_change":"2024-09-21T02:36:25.750931+0000","last_active":"2024-09-21T02:37:39.605201+0000","last_peered":"2024-09-21T02:37:39.605201+0000","last_clean":"2024-09-21T02:37:39.605201+0000","last_became_active":"2024-09-21T02:36:25.750604+0000","last_became_peered":"2024-09-21T02:36:25.750604+0000","last_unstale":"2024-09-21T02:37:39.605201+0000","last_undegraded":"2024-09-21T02:37:39.605201+0000","last_fullsized":"2024-09-21T02:37:39.605201+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:21.325095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609694+0000","last_change":"2024-09-21T02:36:23.730700+0000","last_active":"2024-09-21T02:37:39.609694+0000","last_peered":"2024-09-21T02:37:39.609694+0000","last_clean":"2024-09-21T02:37:39.609694+0000","last_became_active":"2024-09-21T02:36:23.730367+0000","last_became_peered":"2024-09-21T02:36:23.730367+0000","last_unstale":"2024-09-21T02:37:39.609694+0000","last_undegraded":"2024-09-21T02:37:39.609694+0000","last_fullsized":"2024-09-21T02:37:39.609694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:27:57.373077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607709+0000","last_change":"2024-09-21T02:36:32.805885+0000","last_active":"2024-09-21T02:37:39.607709+0000","last_peered":"2024-09-21T02:37:39.607709+0000","last_clean":"2024-09-21T02:37:39.607709+0000","last_became_active":"2024-09-21T02:36:32.805479+0000","last_became_peered":"2024-09-21T02:36:32.805479+0000","last_unstale":"2024-09-21T02:37:39.607709+0000","last_undegraded":"2024-09-21T02:37:39.607709+0000","last_fullsized":"2024-09-21T02:37:39.607709+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:11:29.910642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605971+0000","last_change":"2024-09-21T02:36:25.742542+0000","last_active":"2024-09-21T02:37:39.605971+0000","last_peered":"2024-09-21T02:37:39.605971+0000","last_clean":"2024-09-21T02:37:39.605971+0000","last_became_active":"2024-09-21T02:36:25.742000+0000","last_became_peered":"2024-09-21T02:36:25.742000+0000","last_unstale":"2024-09-21T02:37:39.605971+0000","last_undegraded":"2024-09-21T02:37:39.605971+0000","last_fullsized":"2024-09-21T02:37:39.605971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:52.300212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605081+0000","last_change":"2024-09-21T02:36:23.723896+0000","last_active":"2024-09-21T02:37:39.605081+0000","last_peered":"2024-09-21T02:37:39.605081+0000","last_clean":"2024-09-21T02:37:39.605081+0000","last_became_active":"2024-09-21T02:36:23.723518+0000","last_became_peered":"2024-09-21T02:36:23.723518+0000","last_unstale":"2024-09-21T02:37:39.605081+0000","last_undegraded":"2024-09-21T02:37:39.605081+0000","last_fullsized":"2024-09-21T02:37:39.605081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:37.250247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608598+0000","last_change":"2024-09-21T02:36:32.799924+0000","last_active":"2024-09-21T02:37:39.608598+0000","last_peered":"2024-09-21T02:37:39.608598+0000","last_clean":"2024-09-21T02:37:39.608598+0000","last_became_active":"2024-09-21T02:36:32.799437+0000","last_became_peered":"2024-09-21T02:36:32.799437+0000","last_unstale":"2024-09-21T02:37:39.608598+0000","last_undegraded":"2024-09-21T02:37:39.608598+0000","last_fullsized":"2024-09-21T02:37:39.608598+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:19.258293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604170+0000","last_change":"2024-09-21T02:36:25.747321+0000","last_active":"2024-09-21T02:37:39.604170+0000","last_peered":"2024-09-21T02:37:39.604170+0000","last_clean":"2024-09-21T02:37:39.604170+0000","last_became_active":"2024-09-21T02:36:25.746955+0000","last_became_peered":"2024-09-21T02:36:25.746955+0000","last_unstale":"2024-09-21T02:37:39.604170+0000","last_undegraded":"2024-09-21T02:37:39.604170+0000","last_fullsized":"2024-09-21T02:37:39.604170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:25.148300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606409+0000","last_change":"2024-09-21T02:36:23.725350+0000","last_active":"2024-09-21T02:37:39.606409+0000","last_peered":"2024-09-21T02:37:39.606409+0000","last_clean":"2024-09-21T02:37:39.606409+0000","last_became_active":"2024-09-21T02:36:23.725090+0000","last_became_peered":"2024-09-21T02:36:23.725090+0000","last_unstale":"2024-09-21T02:37:39.606409+0000","last_undegraded":"2024-09-21T02:37:39.606409+0000","last_fullsized":"2024-09-21T02:37:39.606409+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:52.130345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610054+0000","last_change":"2024-09-21T02:36:32.801355+0000","last_active":"2024-09-21T02:37:39.610054+0000","last_peered":"2024-09-21T02:37:39.610054+0000","last_clean":"2024-09-21T02:37:39.610054+0000","last_became_active":"2024-09-21T02:36:32.800803+0000","last_became_peered":"2024-09-21T02:36:32.800803+0000","last_unstale":"2024-09-21T02:37:39.610054+0000","last_undegraded":"2024-09-21T02:37:39.610054+0000","last_fullsized":"2024-09-21T02:37:39.610054+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:54.755733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609524+0000","last_change":"2024-09-21T02:36:25.743433+0000","last_active":"2024-09-21T02:37:39.609524+0000","last_peered":"2024-09-21T02:37:39.609524+0000","last_clean":"2024-09-21T02:37:39.609524+0000","last_became_active":"2024-09-21T02:36:25.742643+0000","last_became_peered":"2024-09-21T02:36:25.742643+0000","last_unstale":"2024-09-21T02:37:39.609524+0000","last_undegraded":"2024-09-21T02:37:39.609524+0000","last_fullsized":"2024-09-21T02:37:39.609524+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:20:09.849806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.625433+0000","last_change":"2024-09-21T02:37:39.625360+0000","last_active":"2024-09-21T02:37:39.625433+0000","last_peered":"2024-09-21T02:37:39.625433+0000","last_clean":"2024-09-21T02:37:39.625433+0000","last_became_active":"2024-09-21T02:37:39.611781+0000","last_became_peered":"2024-09-21T02:37:39.611781+0000","last_unstale":"2024-09-21T02:37:39.625433+0000","last_undegraded":"2024-09-21T02:37:39.625433+0000","last_fullsized":"2024-09-21T02:37:39.625433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:11.846383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605044+0000","last_change":"2024-09-21T02:36:32.799728+0000","last_active":"2024-09-21T02:37:39.605044+0000","last_peered":"2024-09-21T02:37:39.605044+0000","last_clean":"2024-09-21T02:37:39.605044+0000","last_became_active":"2024-09-21T02:36:32.798419+0000","last_became_peered":"2024-09-21T02:36:32.798419+0000","last_unstale":"2024-09-21T02:37:39.605044+0000","last_undegraded":"2024-09-21T02:37:39.605044+0000","last_fullsized":"2024-09-21T02:37:39.605044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:53.681473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604652+0000","last_change":"2024-09-21T02:36:25.747712+0000","last_active":"2024-09-21T02:37:39.604652+0000","last_peered":"2024-09-21T02:37:39.604652+0000","last_clean":"2024-09-21T02:37:39.604652+0000","last_became_active":"2024-09-21T02:36:25.747197+0000","last_became_peered":"2024-09-21T02:36:25.747197+0000","last_unstale":"2024-09-21T02:37:39.604652+0000","last_undegraded":"2024-09-21T02:37:39.604652+0000","last_fullsized":"2024-09-21T02:37:39.604652+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:03.900360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":103,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:09.135191+0000","last_change":"2024-09-21T02:36:23.732156+0000","last_active":"2024-09-21T02:38:09.135191+0000","last_peered":"2024-09-21T02:38:09.135191+0000","last_clean":"2024-09-21T02:38:09.135191+0000","last_became_active":"2024-09-21T02:36:23.730238+0000","last_became_peered":"2024-09-21T02:36:23.730238+0000","last_unstale":"2024-09-21T02:38:09.135191+0000","last_undegraded":"2024-09-21T02:38:09.135191+0000","last_fullsized":"2024-09-21T02:38:09.135191+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:18.053890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":64,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602096+0000","last_change":"2024-09-21T02:36:32.806674+0000","last_active":"2024-09-21T02:37:39.602096+0000","last_peered":"2024-09-21T02:37:39.602096+0000","last_clean":"2024-09-21T02:37:39.602096+0000","last_became_active":"2024-09-21T02:36:32.805373+0000","last_became_peered":"2024-09-21T02:36:32.805373+0000","last_unstale":"2024-09-21T02:37:39.602096+0000","last_undegraded":"2024-09-21T02:37:39.602096+0000","last_fullsized":"2024-09-21T02:37:39.602096+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:29.249243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605083+0000","last_change":"2024-09-21T02:36:25.748703+0000","last_active":"2024-09-21T02:37:39.605083+0000","last_peered":"2024-09-21T02:37:39.605083+0000","last_clean":"2024-09-21T02:37:39.605083+0000","last_became_active":"2024-09-21T02:36:25.748340+0000","last_became_peered":"2024-09-21T02:36:25.748340+0000","last_unstale":"2024-09-21T02:37:39.605083+0000","last_undegraded":"2024-09-21T02:37:39.605083+0000","last_fullsized":"2024-09-21T02:37:39.605083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:07.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610285+0000","last_change":"2024-09-21T02:36:23.732332+0000","last_active":"2024-09-21T02:37:39.610285+0000","last_peered":"2024-09-21T02:37:39.610285+0000","last_clean":"2024-09-21T02:37:39.610285+0000","last_became_active":"2024-09-21T02:36:23.731974+0000","last_became_peered":"2024-09-21T02:36:23.731974+0000","last_unstale":"2024-09-21T02:37:39.610285+0000","last_undegraded":"2024-09-21T02:37:39.610285+0000","last_fullsized":"2024-09-21T02:37:39.610285+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:52.284192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610819+0000","last_change":"2024-09-21T02:37:39.610698+0000","last_active":"2024-09-21T02:37:39.610819+0000","last_peered":"2024-09-21T02:37:39.610819+0000","last_clean":"2024-09-21T02:37:39.610819+0000","last_became_active":"2024-09-21T02:37:39.610185+0000","last_became_peered":"2024-09-21T02:37:39.610185+0000","last_unstale":"2024-09-21T02:37:39.610819+0000","last_undegraded":"2024-09-21T02:37:39.610819+0000","last_fullsized":"2024-09-21T02:37:39.610819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:54:51.050463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604687+0000","last_change":"2024-09-21T02:36:25.751720+0000","last_active":"2024-09-21T02:37:39.604687+0000","last_peered":"2024-09-21T02:37:39.604687+0000","last_clean":"2024-09-21T02:37:39.604687+0000","last_became_active":"2024-09-21T02:36:25.751351+0000","last_became_peered":"2024-09-21T02:36:25.751351+0000","last_unstale":"2024-09-21T02:37:39.604687+0000","last_undegraded":"2024-09-21T02:37:39.604687+0000","last_fullsized":"2024-09-21T02:37:39.604687+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:01.201453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":46,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603023+0000","last_change":"2024-09-21T02:36:23.729981+0000","last_active":"2024-09-21T02:37:39.603023+0000","last_peered":"2024-09-21T02:37:39.603023+0000","last_clean":"2024-09-21T02:37:39.603023+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:37:39.603023+0000","last_undegraded":"2024-09-21T02:37:39.603023+0000","last_fullsized":"2024-09-21T02:37:39.603023+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:03.115712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609295+0000","last_change":"2024-09-21T02:36:32.805286+0000","last_active":"2024-09-21T02:37:39.609295+0000","last_peered":"2024-09-21T02:37:39.609295+0000","last_clean":"2024-09-21T02:37:39.609295+0000","last_became_active":"2024-09-21T02:36:32.804770+0000","last_became_peered":"2024-09-21T02:36:32.804770+0000","last_unstale":"2024-09-21T02:37:39.609295+0000","last_undegraded":"2024-09-21T02:37:39.609295+0000","last_fullsized":"2024-09-21T02:37:39.609295+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:18:34.155892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606225+0000","last_change":"2024-09-21T02:36:25.747812+0000","last_active":"2024-09-21T02:37:39.606225+0000","last_peered":"2024-09-21T02:37:39.606225+0000","last_clean":"2024-09-21T02:37:39.606225+0000","last_became_active":"2024-09-21T02:36:25.747187+0000","last_became_peered":"2024-09-21T02:36:25.747187+0000","last_unstale":"2024-09-21T02:37:39.606225+0000","last_undegraded":"2024-09-21T02:37:39.606225+0000","last_fullsized":"2024-09-21T02:37:39.606225+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:51.395869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"81'35","reported_seq":195,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:07.131904+0000","last_change":"2024-09-21T02:36:23.735676+0000","last_active":"2024-09-21T02:38:07.131904+0000","last_peered":"2024-09-21T02:38:07.131904+0000","last_clean":"2024-09-21T02:38:07.131904+0000","last_became_active":"2024-09-21T02:36:23.735281+0000","last_became_peered":"2024-09-21T02:36:23.735281+0000","last_unstale":"2024-09-21T02:38:07.131904+0000","last_undegraded":"2024-09-21T02:38:07.131904+0000","last_fullsized":"2024-09-21T02:38:07.131904+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:28:01.547245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67620,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":125,"num_read_kb":128,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601441+0000","last_change":"2024-09-21T02:36:32.806585+0000","last_active":"2024-09-21T02:37:39.601441+0000","last_peered":"2024-09-21T02:37:39.601441+0000","last_clean":"2024-09-21T02:37:39.601441+0000","last_became_active":"2024-09-21T02:36:32.805913+0000","last_became_peered":"2024-09-21T02:36:32.805913+0000","last_unstale":"2024-09-21T02:37:39.601441+0000","last_undegraded":"2024-09-21T02:37:39.601441+0000","last_fullsized":"2024-09-21T02:37:39.601441+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:59:31.888645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867132+0000","last_change":"2024-09-21T02:36:25.742208+0000","last_active":"2024-09-21T02:37:39.867132+0000","last_peered":"2024-09-21T02:37:39.867132+0000","last_clean":"2024-09-21T02:37:39.867132+0000","last_became_active":"2024-09-21T02:36:25.741849+0000","last_became_peered":"2024-09-21T02:36:25.741849+0000","last_unstale":"2024-09-21T02:37:39.867132+0000","last_undegraded":"2024-09-21T02:37:39.867132+0000","last_fullsized":"2024-09-21T02:37:39.867132+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:21:43.720481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609540+0000","last_change":"2024-09-21T02:36:23.800751+0000","last_active":"2024-09-21T02:37:39.609540+0000","last_peered":"2024-09-21T02:37:39.609540+0000","last_clean":"2024-09-21T02:37:39.609540+0000","last_became_active":"2024-09-21T02:36:23.800341+0000","last_became_peered":"2024-09-21T02:36:23.800341+0000","last_unstale":"2024-09-21T02:37:39.609540+0000","last_undegraded":"2024-09-21T02:37:39.609540+0000","last_fullsized":"2024-09-21T02:37:39.609540+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:08:02.556549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604004+0000","last_change":"2024-09-21T02:36:32.802879+0000","last_active":"2024-09-21T02:37:39.604004+0000","last_peered":"2024-09-21T02:37:39.604004+0000","last_clean":"2024-09-21T02:37:39.604004+0000","last_became_active":"2024-09-21T02:36:32.802301+0000","last_became_peered":"2024-09-21T02:36:32.802301+0000","last_unstale":"2024-09-21T02:37:39.604004+0000","last_undegraded":"2024-09-21T02:37:39.604004+0000","last_fullsized":"2024-09-21T02:37:39.604004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:08.135814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608374+0000","last_change":"2024-09-21T02:36:25.744647+0000","last_active":"2024-09-21T02:37:39.608374+0000","last_peered":"2024-09-21T02:37:39.608374+0000","last_clean":"2024-09-21T02:37:39.608374+0000","last_became_active":"2024-09-21T02:36:25.744049+0000","last_became_peered":"2024-09-21T02:36:25.744049+0000","last_unstale":"2024-09-21T02:37:39.608374+0000","last_undegraded":"2024-09-21T02:37:39.608374+0000","last_fullsized":"2024-09-21T02:37:39.608374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:52:02.210380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605972+0000","last_change":"2024-09-21T02:36:23.733590+0000","last_active":"2024-09-21T02:37:39.605972+0000","last_peered":"2024-09-21T02:37:39.605972+0000","last_clean":"2024-09-21T02:37:39.605972+0000","last_became_active":"2024-09-21T02:36:23.732922+0000","last_became_peered":"2024-09-21T02:36:23.732922+0000","last_unstale":"2024-09-21T02:37:39.605972+0000","last_undegraded":"2024-09-21T02:37:39.605972+0000","last_fullsized":"2024-09-21T02:37:39.605972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601278+0000","last_change":"2024-09-21T02:36:32.806540+0000","last_active":"2024-09-21T02:37:39.601278+0000","last_peered":"2024-09-21T02:37:39.601278+0000","last_clean":"2024-09-21T02:37:39.601278+0000","last_became_active":"2024-09-21T02:36:32.805612+0000","last_became_peered":"2024-09-21T02:36:32.805612+0000","last_unstale":"2024-09-21T02:37:39.601278+0000","last_undegraded":"2024-09-21T02:37:39.601278+0000","last_fullsized":"2024-09-21T02:37:39.601278+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:23:24.686383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867826+0000","last_change":"2024-09-21T02:36:25.746229+0000","last_active":"2024-09-21T02:37:39.867826+0000","last_peered":"2024-09-21T02:37:39.867826+0000","last_clean":"2024-09-21T02:37:39.867826+0000","last_became_active":"2024-09-21T02:36:25.745892+0000","last_became_peered":"2024-09-21T02:36:25.745892+0000","last_unstale":"2024-09-21T02:37:39.867826+0000","last_undegraded":"2024-09-21T02:37:39.867826+0000","last_fullsized":"2024-09-21T02:37:39.867826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:55.154499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":155,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:07.131918+0000","last_change":"2024-09-21T02:36:23.730987+0000","last_active":"2024-09-21T02:38:07.131918+0000","last_peered":"2024-09-21T02:38:07.131918+0000","last_clean":"2024-09-21T02:38:07.131918+0000","last_became_active":"2024-09-21T02:36:23.730566+0000","last_became_peered":"2024-09-21T02:36:23.730566+0000","last_unstale":"2024-09-21T02:38:07.131918+0000","last_undegraded":"2024-09-21T02:38:07.131918+0000","last_fullsized":"2024-09-21T02:38:07.131918+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:18:12.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866092+0000","last_change":"2024-09-21T02:36:32.804608+0000","last_active":"2024-09-21T02:37:39.866092+0000","last_peered":"2024-09-21T02:37:39.866092+0000","last_clean":"2024-09-21T02:37:39.866092+0000","last_became_active":"2024-09-21T02:36:32.803567+0000","last_became_peered":"2024-09-21T02:36:32.803567+0000","last_unstale":"2024-09-21T02:37:39.866092+0000","last_undegraded":"2024-09-21T02:37:39.866092+0000","last_fullsized":"2024-09-21T02:37:39.866092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:41.692644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607920+0000","last_change":"2024-09-21T02:36:32.804687+0000","last_active":"2024-09-21T02:37:39.607920+0000","last_peered":"2024-09-21T02:37:39.607920+0000","last_clean":"2024-09-21T02:37:39.607920+0000","last_became_active":"2024-09-21T02:36:32.804232+0000","last_became_peered":"2024-09-21T02:36:32.804232+0000","last_unstale":"2024-09-21T02:37:39.607920+0000","last_undegraded":"2024-09-21T02:37:39.607920+0000","last_fullsized":"2024-09-21T02:37:39.607920+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:43.899924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601400+0000","last_change":"2024-09-21T02:36:25.749114+0000","last_active":"2024-09-21T02:37:39.601400+0000","last_peered":"2024-09-21T02:37:39.601400+0000","last_clean":"2024-09-21T02:37:39.601400+0000","last_became_active":"2024-09-21T02:36:25.748758+0000","last_became_peered":"2024-09-21T02:36:25.748758+0000","last_unstale":"2024-09-21T02:37:39.601400+0000","last_undegraded":"2024-09-21T02:37:39.601400+0000","last_fullsized":"2024-09-21T02:37:39.601400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:57.958533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608906+0000","last_change":"2024-09-21T02:36:23.731473+0000","last_active":"2024-09-21T02:37:39.608906+0000","last_peered":"2024-09-21T02:37:39.608906+0000","last_clean":"2024-09-21T02:37:39.608906+0000","last_became_active":"2024-09-21T02:36:23.731134+0000","last_became_peered":"2024-09-21T02:36:23.731134+0000","last_unstale":"2024-09-21T02:37:39.608906+0000","last_undegraded":"2024-09-21T02:37:39.608906+0000","last_fullsized":"2024-09-21T02:37:39.608906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:05.368941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604072+0000","last_change":"2024-09-21T02:36:32.809630+0000","last_active":"2024-09-21T02:37:39.604072+0000","last_peered":"2024-09-21T02:37:39.604072+0000","last_clean":"2024-09-21T02:37:39.604072+0000","last_became_active":"2024-09-21T02:36:32.809030+0000","last_became_peered":"2024-09-21T02:36:32.809030+0000","last_unstale":"2024-09-21T02:37:39.604072+0000","last_undegraded":"2024-09-21T02:37:39.604072+0000","last_fullsized":"2024-09-21T02:37:39.604072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:57:42.436590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866309+0000","last_change":"2024-09-21T02:36:25.742558+0000","last_active":"2024-09-21T02:37:39.866309+0000","last_peered":"2024-09-21T02:37:39.866309+0000","last_clean":"2024-09-21T02:37:39.866309+0000","last_became_active":"2024-09-21T02:36:25.742045+0000","last_became_peered":"2024-09-21T02:36:25.742045+0000","last_unstale":"2024-09-21T02:37:39.866309+0000","last_undegraded":"2024-09-21T02:37:39.866309+0000","last_fullsized":"2024-09-21T02:37:39.866309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:57:06.705561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604389+0000","last_change":"2024-09-21T02:36:23.724345+0000","last_active":"2024-09-21T02:37:39.604389+0000","last_peered":"2024-09-21T02:37:39.604389+0000","last_clean":"2024-09-21T02:37:39.604389+0000","last_became_active":"2024-09-21T02:36:23.723635+0000","last_became_peered":"2024-09-21T02:36:23.723635+0000","last_unstale":"2024-09-21T02:37:39.604389+0000","last_undegraded":"2024-09-21T02:37:39.604389+0000","last_fullsized":"2024-09-21T02:37:39.604389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:18.812176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867234+0000","last_change":"2024-09-21T02:36:23.732897+0000","last_active":"2024-09-21T02:37:39.867234+0000","last_peered":"2024-09-21T02:37:39.867234+0000","last_clean":"2024-09-21T02:37:39.867234+0000","last_became_active":"2024-09-21T02:36:23.732509+0000","last_became_peered":"2024-09-21T02:36:23.732509+0000","last_unstale":"2024-09-21T02:37:39.867234+0000","last_undegraded":"2024-09-21T02:37:39.867234+0000","last_fullsized":"2024-09-21T02:37:39.867234+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:12:27.117962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609775+0000","last_change":"2024-09-21T02:36:25.748929+0000","last_active":"2024-09-21T02:37:39.609775+0000","last_peered":"2024-09-21T02:37:39.609775+0000","last_clean":"2024-09-21T02:37:39.609775+0000","last_became_active":"2024-09-21T02:36:25.748588+0000","last_became_peered":"2024-09-21T02:36:25.748588+0000","last_unstale":"2024-09-21T02:37:39.609775+0000","last_undegraded":"2024-09-21T02:37:39.609775+0000","last_fullsized":"2024-09-21T02:37:39.609775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.086472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602636+0000","last_change":"2024-09-21T02:36:32.807655+0000","last_active":"2024-09-21T02:37:39.602636+0000","last_peered":"2024-09-21T02:37:39.602636+0000","last_clean":"2024-09-21T02:37:39.602636+0000","last_became_active":"2024-09-21T02:36:32.806995+0000","last_became_peered":"2024-09-21T02:36:32.806995+0000","last_unstale":"2024-09-21T02:37:39.602636+0000","last_undegraded":"2024-09-21T02:37:39.602636+0000","last_fullsized":"2024-09-21T02:37:39.602636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:28:20.420591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":106,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:38:10.137175+0000","last_change":"2024-09-21T02:36:23.729427+0000","last_active":"2024-09-21T02:38:10.137175+0000","last_peered":"2024-09-21T02:38:10.137175+0000","last_clean":"2024-09-21T02:38:10.137175+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:38:10.137175+0000","last_undegraded":"2024-09-21T02:38:10.137175+0000","last_fullsized":"2024-09-21T02:38:10.137175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:10:41.251665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":65,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605668+0000","last_change":"2024-09-21T02:36:25.743212+0000","last_active":"2024-09-21T02:37:39.605668+0000","last_peered":"2024-09-21T02:37:39.605668+0000","last_clean":"2024-09-21T02:37:39.605668+0000","last_became_active":"2024-09-21T02:36:25.742777+0000","last_became_peered":"2024-09-21T02:36:25.742777+0000","last_unstale":"2024-09-21T02:37:39.605668+0000","last_undegraded":"2024-09-21T02:37:39.605668+0000","last_fullsized":"2024-09-21T02:37:39.605668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:12:19.917214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602914+0000","last_change":"2024-09-21T02:36:32.801880+0000","last_active":"2024-09-21T02:37:39.602914+0000","last_peered":"2024-09-21T02:37:39.602914+0000","last_clean":"2024-09-21T02:37:39.602914+0000","last_became_active":"2024-09-21T02:36:32.801389+0000","last_became_peered":"2024-09-21T02:36:32.801389+0000","last_unstale":"2024-09-21T02:37:39.602914+0000","last_undegraded":"2024-09-21T02:37:39.602914+0000","last_fullsized":"2024-09-21T02:37:39.602914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:11:44.877062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605000+0000","last_change":"2024-09-21T02:36:25.754014+0000","last_active":"2024-09-21T02:37:39.605000+0000","last_peered":"2024-09-21T02:37:39.605000+0000","last_clean":"2024-09-21T02:37:39.605000+0000","last_became_active":"2024-09-21T02:36:25.753548+0000","last_became_peered":"2024-09-21T02:36:25.753548+0000","last_unstale":"2024-09-21T02:37:39.605000+0000","last_undegraded":"2024-09-21T02:37:39.605000+0000","last_fullsized":"2024-09-21T02:37:39.605000+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:03.850625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.610997+0000","last_change":"2024-09-21T02:37:39.610894+0000","last_active":"2024-09-21T02:37:39.610997+0000","last_peered":"2024-09-21T02:37:39.610997+0000","last_clean":"2024-09-21T02:37:39.610997+0000","last_became_active":"2024-09-21T02:37:39.610404+0000","last_became_peered":"2024-09-21T02:37:39.610404+0000","last_unstale":"2024-09-21T02:37:39.610997+0000","last_undegraded":"2024-09-21T02:37:39.610997+0000","last_fullsized":"2024-09-21T02:37:39.610997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:54.347708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.602163+0000","last_change":"2024-09-21T02:36:25.752839+0000","last_active":"2024-09-21T02:37:39.602163+0000","last_peered":"2024-09-21T02:37:39.602163+0000","last_clean":"2024-09-21T02:37:39.602163+0000","last_became_active":"2024-09-21T02:36:25.750882+0000","last_became_peered":"2024-09-21T02:36:25.750882+0000","last_unstale":"2024-09-21T02:37:39.602163+0000","last_undegraded":"2024-09-21T02:37:39.602163+0000","last_fullsized":"2024-09-21T02:37:39.602163+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:53.028611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607755+0000","last_change":"2024-09-21T02:36:25.744624+0000","last_active":"2024-09-21T02:37:39.607755+0000","last_peered":"2024-09-21T02:37:39.607755+0000","last_clean":"2024-09-21T02:37:39.607755+0000","last_became_active":"2024-09-21T02:36:25.744173+0000","last_became_peered":"2024-09-21T02:36:25.744173+0000","last_unstale":"2024-09-21T02:37:39.607755+0000","last_undegraded":"2024-09-21T02:37:39.607755+0000","last_fullsized":"2024-09-21T02:37:39.607755+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:55.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.609153+0000","last_change":"2024-09-21T02:37:39.609050+0000","last_active":"2024-09-21T02:37:39.609153+0000","last_peered":"2024-09-21T02:37:39.609153+0000","last_clean":"2024-09-21T02:37:39.609153+0000","last_became_active":"2024-09-21T02:37:39.608595+0000","last_became_peered":"2024-09-21T02:37:39.608595+0000","last_unstale":"2024-09-21T02:37:39.609153+0000","last_undegraded":"2024-09-21T02:37:39.609153+0000","last_fullsized":"2024-09-21T02:37:39.609153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:26.831121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605670+0000","last_change":"2024-09-21T02:36:25.743476+0000","last_active":"2024-09-21T02:37:39.605670+0000","last_peered":"2024-09-21T02:37:39.605670+0000","last_clean":"2024-09-21T02:37:39.605670+0000","last_became_active":"2024-09-21T02:36:25.743117+0000","last_became_peered":"2024-09-21T02:36:25.743117+0000","last_unstale":"2024-09-21T02:37:39.605670+0000","last_undegraded":"2024-09-21T02:37:39.605670+0000","last_fullsized":"2024-09-21T02:37:39.605670+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:08.084109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603172+0000","last_change":"2024-09-21T02:36:25.753014+0000","last_active":"2024-09-21T02:37:39.603172+0000","last_peered":"2024-09-21T02:37:39.603172+0000","last_clean":"2024-09-21T02:37:39.603172+0000","last_became_active":"2024-09-21T02:36:25.751232+0000","last_became_peered":"2024-09-21T02:36:25.751232+0000","last_unstale":"2024-09-21T02:37:39.603172+0000","last_undegraded":"2024-09-21T02:37:39.603172+0000","last_fullsized":"2024-09-21T02:37:39.603172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:03.052737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607167+0000","last_change":"2024-09-21T02:37:39.606525+0000","last_active":"2024-09-21T02:37:39.607167+0000","last_peered":"2024-09-21T02:37:39.607167+0000","last_clean":"2024-09-21T02:37:39.607167+0000","last_became_active":"2024-09-21T02:37:39.606078+0000","last_became_peered":"2024-09-21T02:37:39.606078+0000","last_unstale":"2024-09-21T02:37:39.607167+0000","last_undegraded":"2024-09-21T02:37:39.607167+0000","last_fullsized":"2024-09-21T02:37:39.607167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:22.423457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608763+0000","last_change":"2024-09-21T02:36:25.749605+0000","last_active":"2024-09-21T02:37:39.608763+0000","last_peered":"2024-09-21T02:37:39.608763+0000","last_clean":"2024-09-21T02:37:39.608763+0000","last_became_active":"2024-09-21T02:36:25.749165+0000","last_became_peered":"2024-09-21T02:36:25.749165+0000","last_unstale":"2024-09-21T02:37:39.608763+0000","last_undegraded":"2024-09-21T02:37:39.608763+0000","last_fullsized":"2024-09-21T02:37:39.608763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:15:51.412761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866165+0000","last_change":"2024-09-21T02:36:25.747494+0000","last_active":"2024-09-21T02:37:39.866165+0000","last_peered":"2024-09-21T02:37:39.866165+0000","last_clean":"2024-09-21T02:37:39.866165+0000","last_became_active":"2024-09-21T02:36:25.746965+0000","last_became_peered":"2024-09-21T02:36:25.746965+0000","last_unstale":"2024-09-21T02:37:39.866165+0000","last_undegraded":"2024-09-21T02:37:39.866165+0000","last_fullsized":"2024-09-21T02:37:39.866165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:51.309307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867530+0000","last_change":"2024-09-21T02:36:25.749517+0000","last_active":"2024-09-21T02:37:39.867530+0000","last_peered":"2024-09-21T02:37:39.867530+0000","last_clean":"2024-09-21T02:37:39.867530+0000","last_became_active":"2024-09-21T02:36:25.749117+0000","last_became_peered":"2024-09-21T02:36:25.749117+0000","last_unstale":"2024-09-21T02:37:39.867530+0000","last_undegraded":"2024-09-21T02:37:39.867530+0000","last_fullsized":"2024-09-21T02:37:39.867530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:38.758019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604855+0000","last_change":"2024-09-21T02:36:25.736067+0000","last_active":"2024-09-21T02:37:39.604855+0000","last_peered":"2024-09-21T02:37:39.604855+0000","last_clean":"2024-09-21T02:37:39.604855+0000","last_became_active":"2024-09-21T02:36:25.735710+0000","last_became_peered":"2024-09-21T02:36:25.735710+0000","last_unstale":"2024-09-21T02:37:39.604855+0000","last_undegraded":"2024-09-21T02:37:39.604855+0000","last_fullsized":"2024-09-21T02:37:39.604855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.559448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604519+0000","last_change":"2024-09-21T02:36:25.746307+0000","last_active":"2024-09-21T02:37:39.604519+0000","last_peered":"2024-09-21T02:37:39.604519+0000","last_clean":"2024-09-21T02:37:39.604519+0000","last_became_active":"2024-09-21T02:36:25.745872+0000","last_became_peered":"2024-09-21T02:36:25.745872+0000","last_unstale":"2024-09-21T02:37:39.604519+0000","last_undegraded":"2024-09-21T02:37:39.604519+0000","last_fullsized":"2024-09-21T02:37:39.604519+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:58:53.074559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.603439+0000","last_change":"2024-09-21T02:36:25.753393+0000","last_active":"2024-09-21T02:37:39.603439+0000","last_peered":"2024-09-21T02:37:39.603439+0000","last_clean":"2024-09-21T02:37:39.603439+0000","last_became_active":"2024-09-21T02:36:25.751871+0000","last_became_peered":"2024-09-21T02:36:25.751871+0000","last_unstale":"2024-09-21T02:37:39.603439+0000","last_undegraded":"2024-09-21T02:37:39.603439+0000","last_fullsized":"2024-09-21T02:37:39.603439+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:28.199451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.601888+0000","last_change":"2024-09-21T02:36:25.744845+0000","last_active":"2024-09-21T02:37:39.601888+0000","last_peered":"2024-09-21T02:37:39.601888+0000","last_clean":"2024-09-21T02:37:39.601888+0000","last_became_active":"2024-09-21T02:36:25.744492+0000","last_became_peered":"2024-09-21T02:36:25.744492+0000","last_unstale":"2024-09-21T02:37:39.601888+0000","last_undegraded":"2024-09-21T02:37:39.601888+0000","last_fullsized":"2024-09-21T02:37:39.601888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:08.524871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604947+0000","last_change":"2024-09-21T02:36:25.751187+0000","last_active":"2024-09-21T02:37:39.604947+0000","last_peered":"2024-09-21T02:37:39.604947+0000","last_clean":"2024-09-21T02:37:39.604947+0000","last_became_active":"2024-09-21T02:36:25.750813+0000","last_became_peered":"2024-09-21T02:36:25.750813+0000","last_unstale":"2024-09-21T02:37:39.604947+0000","last_undegraded":"2024-09-21T02:37:39.604947+0000","last_fullsized":"2024-09-21T02:37:39.604947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.590210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866405+0000","last_change":"2024-09-21T02:36:25.744046+0000","last_active":"2024-09-21T02:37:39.866405+0000","last_peered":"2024-09-21T02:37:39.866405+0000","last_clean":"2024-09-21T02:37:39.866405+0000","last_became_active":"2024-09-21T02:36:25.743695+0000","last_became_peered":"2024-09-21T02:36:25.743695+0000","last_unstale":"2024-09-21T02:37:39.866405+0000","last_undegraded":"2024-09-21T02:37:39.866405+0000","last_fullsized":"2024-09-21T02:37:39.866405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:12.092940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607554+0000","last_change":"2024-09-21T02:36:25.749113+0000","last_active":"2024-09-21T02:37:39.607554+0000","last_peered":"2024-09-21T02:37:39.607554+0000","last_clean":"2024-09-21T02:37:39.607554+0000","last_became_active":"2024-09-21T02:36:25.748759+0000","last_became_peered":"2024-09-21T02:36:25.748759+0000","last_unstale":"2024-09-21T02:37:39.607554+0000","last_undegraded":"2024-09-21T02:37:39.607554+0000","last_fullsized":"2024-09-21T02:37:39.607554+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:22.785273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607560+0000","last_change":"2024-09-21T02:36:25.745454+0000","last_active":"2024-09-21T02:37:39.607560+0000","last_peered":"2024-09-21T02:37:39.607560+0000","last_clean":"2024-09-21T02:37:39.607560+0000","last_became_active":"2024-09-21T02:36:25.745108+0000","last_became_peered":"2024-09-21T02:36:25.745108+0000","last_unstale":"2024-09-21T02:37:39.607560+0000","last_undegraded":"2024-09-21T02:37:39.607560+0000","last_fullsized":"2024-09-21T02:37:39.607560+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:36.710456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604836+0000","last_change":"2024-09-21T02:36:25.737213+0000","last_active":"2024-09-21T02:37:39.604836+0000","last_peered":"2024-09-21T02:37:39.604836+0000","last_clean":"2024-09-21T02:37:39.604836+0000","last_became_active":"2024-09-21T02:36:25.736641+0000","last_became_peered":"2024-09-21T02:36:25.736641+0000","last_unstale":"2024-09-21T02:37:39.604836+0000","last_undegraded":"2024-09-21T02:37:39.604836+0000","last_fullsized":"2024-09-21T02:37:39.604836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:47:50.258805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604582+0000","last_change":"2024-09-21T02:36:25.748230+0000","last_active":"2024-09-21T02:37:39.604582+0000","last_peered":"2024-09-21T02:37:39.604582+0000","last_clean":"2024-09-21T02:37:39.604582+0000","last_became_active":"2024-09-21T02:36:25.747859+0000","last_became_peered":"2024-09-21T02:36:25.747859+0000","last_unstale":"2024-09-21T02:37:39.604582+0000","last_undegraded":"2024-09-21T02:37:39.604582+0000","last_fullsized":"2024-09-21T02:37:39.604582+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:15:21.244825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605976+0000","last_change":"2024-09-21T02:36:25.744379+0000","last_active":"2024-09-21T02:37:39.605976+0000","last_peered":"2024-09-21T02:37:39.605976+0000","last_clean":"2024-09-21T02:37:39.605976+0000","last_became_active":"2024-09-21T02:36:25.744003+0000","last_became_peered":"2024-09-21T02:36:25.744003+0000","last_unstale":"2024-09-21T02:37:39.605976+0000","last_undegraded":"2024-09-21T02:37:39.605976+0000","last_fullsized":"2024-09-21T02:37:39.605976+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:59:59.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.868567+0000","last_change":"2024-09-21T02:36:25.755616+0000","last_active":"2024-09-21T02:37:39.868567+0000","last_peered":"2024-09-21T02:37:39.868567+0000","last_clean":"2024-09-21T02:37:39.868567+0000","last_became_active":"2024-09-21T02:36:25.755304+0000","last_became_peered":"2024-09-21T02:36:25.755304+0000","last_unstale":"2024-09-21T02:37:39.868567+0000","last_undegraded":"2024-09-21T02:37:39.868567+0000","last_fullsized":"2024-09-21T02:37:39.868567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:24.813780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867940+0000","last_change":"2024-09-21T02:36:25.739747+0000","last_active":"2024-09-21T02:37:39.867940+0000","last_peered":"2024-09-21T02:37:39.867940+0000","last_clean":"2024-09-21T02:37:39.867940+0000","last_became_active":"2024-09-21T02:36:25.739404+0000","last_became_peered":"2024-09-21T02:36:25.739404+0000","last_unstale":"2024-09-21T02:37:39.867940+0000","last_undegraded":"2024-09-21T02:37:39.867940+0000","last_fullsized":"2024-09-21T02:37:39.867940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:17.975419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.866105+0000","last_change":"2024-09-21T02:36:25.747723+0000","last_active":"2024-09-21T02:37:39.866105+0000","last_peered":"2024-09-21T02:37:39.866105+0000","last_clean":"2024-09-21T02:37:39.866105+0000","last_became_active":"2024-09-21T02:36:25.747368+0000","last_became_peered":"2024-09-21T02:36:25.747368+0000","last_unstale":"2024-09-21T02:37:39.866105+0000","last_undegraded":"2024-09-21T02:37:39.866105+0000","last_fullsized":"2024-09-21T02:37:39.866105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.319756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605349+0000","last_change":"2024-09-21T02:36:25.743772+0000","last_active":"2024-09-21T02:37:39.605349+0000","last_peered":"2024-09-21T02:37:39.605349+0000","last_clean":"2024-09-21T02:37:39.605349+0000","last_became_active":"2024-09-21T02:36:25.743403+0000","last_became_peered":"2024-09-21T02:36:25.743403+0000","last_unstale":"2024-09-21T02:37:39.605349+0000","last_undegraded":"2024-09-21T02:37:39.605349+0000","last_fullsized":"2024-09-21T02:37:39.605349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:11:29.930248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.867226+0000","last_change":"2024-09-21T02:36:25.746911+0000","last_active":"2024-09-21T02:37:39.867226+0000","last_peered":"2024-09-21T02:37:39.867226+0000","last_clean":"2024-09-21T02:37:39.867226+0000","last_became_active":"2024-09-21T02:36:25.746426+0000","last_became_peered":"2024-09-21T02:36:25.746426+0000","last_unstale":"2024-09-21T02:37:39.867226+0000","last_undegraded":"2024-09-21T02:37:39.867226+0000","last_fullsized":"2024-09-21T02:37:39.867226+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:12.835851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.606065+0000","last_change":"2024-09-21T02:36:25.751278+0000","last_active":"2024-09-21T02:37:39.606065+0000","last_peered":"2024-09-21T02:37:39.606065+0000","last_clean":"2024-09-21T02:37:39.606065+0000","last_became_active":"2024-09-21T02:36:25.750414+0000","last_became_peered":"2024-09-21T02:36:25.750414+0000","last_unstale":"2024-09-21T02:37:39.606065+0000","last_undegraded":"2024-09-21T02:37:39.606065+0000","last_fullsized":"2024-09-21T02:37:39.606065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:13:56.080122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.605503+0000","last_change":"2024-09-21T02:36:25.752852+0000","last_active":"2024-09-21T02:37:39.605503+0000","last_peered":"2024-09-21T02:37:39.605503+0000","last_clean":"2024-09-21T02:37:39.605503+0000","last_became_active":"2024-09-21T02:36:25.752514+0000","last_became_peered":"2024-09-21T02:36:25.752514+0000","last_unstale":"2024-09-21T02:37:39.605503+0000","last_undegraded":"2024-09-21T02:37:39.605503+0000","last_fullsized":"2024-09-21T02:37:39.605503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:55.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.607756+0000","last_change":"2024-09-21T02:36:25.750081+0000","last_active":"2024-09-21T02:37:39.607756+0000","last_peered":"2024-09-21T02:37:39.607756+0000","last_clean":"2024-09-21T02:37:39.607756+0000","last_became_active":"2024-09-21T02:36:25.749602+0000","last_became_peered":"2024-09-21T02:36:25.749602+0000","last_unstale":"2024-09-21T02:37:39.607756+0000","last_undegraded":"2024-09-21T02:37:39.607756+0000","last_fullsized":"2024-09-21T02:37:39.607756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:06.785998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.608041+0000","last_change":"2024-09-21T02:36:25.746897+0000","last_active":"2024-09-21T02:37:39.608041+0000","last_peered":"2024-09-21T02:37:39.608041+0000","last_clean":"2024-09-21T02:37:39.608041+0000","last_became_active":"2024-09-21T02:36:25.746477+0000","last_became_peered":"2024-09-21T02:36:25.746477+0000","last_unstale":"2024-09-21T02:37:39.608041+0000","last_undegraded":"2024-09-21T02:37:39.608041+0000","last_fullsized":"2024-09-21T02:37:39.608041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:34:00.739410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-21T02:37:39.604837+0000","last_change":"2024-09-21T02:36:25.745233+0000","last_active":"2024-09-21T02:37:39.604837+0000","last_peered":"2024-09-21T02:37:39.604837+0000","last_clean":"2024-09-21T02:37:39.604837+0000","last_became_active":"2024-09-21T02:36:25.744637+0000","last_became_peered":"2024-09-21T02:36:25.744637+0000","last_unstale":"2024-09-21T02:37:39.604837+0000","last_undegraded":"2024-09-21T02:37:39.604837+0000","last_fullsized":"2024-09-21T02:37:39.604837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:34:59.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74824,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":630,"num_read_kb":392,"num_write":252,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":224472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249064,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":69,"seq":296352743458,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":467025,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906983,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":1125054,"data_compressed":59828,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070509,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":942080,"data_stored":466452,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266739,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":929792,"data_stored":466718,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462968,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":991232,"data_stored":1063327,"data_compressed":59828,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659197,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":917504,"data_stored":461956,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855427,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":1058909,"data_compressed":59828,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018951,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":925696,"data_stored":466030,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215179,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":929792,"data_stored":461664,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1011712,"data_stored":531608,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":991232,"data_stored":530606,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":942080,"data_stored":463803,"data_compressed":23554,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:38:14.323 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T02:38:14.323 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T02:38:14.323 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T02:38:14.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-21T02:38:14.582 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:15.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:14 smithi066 ceph-mon[33436]: pgmap v361: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:15.080 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:14 smithi029 ceph-mon[31006]: pgmap v361: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:15.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:14 smithi190 ceph-mon[33871]: pgmap v361: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 -- 172.21.15.29:0/1779542612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 msgr2=0x7fb62c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 --2- 172.21.15.29:0/1779542612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 0x7fb62c110300 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb624009ca0 tx=0x7fb62402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 -- 172.21.15.29:0/1779542612 shutdown_connections 2024-09-21T02:38:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 --2- 172.21.15.29:0/1779542612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 0x7fb62c110300 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 --2- 172.21.15.29:0/1779542612 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c105300 0x7fb62c10d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 --2- 172.21.15.29:0/1779542612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1049b0 0x7fb62c104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 -- 172.21.15.29:0/1779542612 >> 172.21.15.29:0/1779542612 conn(0x7fb62c079730 msgr2=0x7fb62c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 -- 172.21.15.29:0/1779542612 shutdown_connections 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.171+0000 7fb631e2c640 1 -- 172.21.15.29:0/1779542612 wait complete. 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb631e2c640 1 Processor -- start 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb631e2c640 1 -- start start 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb631e2c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1049b0 0x7fb62c19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb631e2c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c105300 0x7fb62c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb631e2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 0x7fb62c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb631e2c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c113970 con 0x7fb62c10de10 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb631e2c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c1137f0 con 0x7fb62c105300 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb631e2c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c113af0 con 0x7fb62c1049b0 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb62bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 0x7fb62c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb62bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 0x7fb62c19fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41210/0 (socket says 172.21.15.29:41210) 2024-09-21T02:38:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb62bfff640 1 -- 172.21.15.29:0/357382236 learned_addr learned my addr 172.21.15.29:0/357382236 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.172+0000 7fb62affd640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c105300 0x7fb62c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb62b7fe640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1049b0 0x7fb62c19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb62bfff640 1 -- 172.21.15.29:0/357382236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1049b0 msgr2=0x7fb62c19a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb62bfff640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1049b0 0x7fb62c19a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb62bfff640 1 -- 172.21.15.29:0/357382236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c105300 msgr2=0x7fb62c19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb62bfff640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c105300 0x7fb62c19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb62bfff640 1 -- 172.21.15.29:0/357382236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c1a0390 con 0x7fb62c10de10 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb62b7fe640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1049b0 0x7fb62c19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb62bfff640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 0x7fb62c19fc60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb624005cc0 tx=0x7fb624032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb628ff9640 1 -- 172.21.15.29:0/357382236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb624032e00 con 0x7fb62c10de10 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb628ff9640 1 -- 172.21.15.29:0/357382236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6240044d0 con 0x7fb62c10de10 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62c1a0620 con 0x7fb62c10de10 2024-09-21T02:38:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb628ff9640 1 -- 172.21.15.29:0/357382236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb624030b70 con 0x7fb62c10de10 2024-09-21T02:38:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.173+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb62c1a6de0 con 0x7fb62c10de10 2024-09-21T02:38:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.175+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb62c104dc0 con 0x7fb62c10de10 2024-09-21T02:38:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.176+0000 7fb628ff9640 1 -- 172.21.15.29:0/357382236 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fb624004680 con 0x7fb62c10de10 2024-09-21T02:38:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.177+0000 7fb628ff9640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6140b65a0 0x7fb6140b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.177+0000 7fb62b7fe640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6140b65a0 0x7fb6140b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.177+0000 7fb628ff9640 1 -- 172.21.15.29:0/357382236 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fb624108ac0 con 0x7fb62c10de10 2024-09-21T02:38:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.177+0000 7fb62b7fe640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6140b65a0 0x7fb6140b8ab0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0058c0 tx=0x7fb61c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.181+0000 7fb628ff9640 1 -- 172.21.15.29:0/357382236 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb6240cee30 con 0x7fb62c10de10 2024-09-21T02:38:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.412+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb62c1a7010 con 0x7fb62c10de10 2024-09-21T02:38:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.412+0000 7fb628ff9640 1 -- 172.21.15.29:0/357382236 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb6240d3210 con 0x7fb62c10de10 2024-09-21T02:38:15.413 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:15.413 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6140b65a0 msgr2=0x7fb6140b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6140b65a0 0x7fb6140b8ab0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0058c0 tx=0x7fb61c005850 comp rx=0 tx=0).stop 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 msgr2=0x7fb62c19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 0x7fb62c19fc60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb624005cc0 tx=0x7fb624032a10 comp rx=0 tx=0).stop 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 shutdown_connections 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6140b65a0 0x7fb6140b8ab0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c10de10 0x7fb62c19fc60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb62c105300 0x7fb62c19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 --2- 172.21.15.29:0/357382236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb62c1049b0 0x7fb62c19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:15.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.414+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 >> 172.21.15.29:0/357382236 conn(0x7fb62c079730 msgr2=0x7fb62c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:15.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.415+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 shutdown_connections 2024-09-21T02:38:15.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:15.415+0000 7fb631e2c640 1 -- 172.21.15.29:0/357382236 wait complete. 2024-09-21T02:38:16.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:15 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/357382236' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T02:38:16.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:15 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/357382236' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T02:38:16.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:15 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/357382236' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T02:38:16.291 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T02:38:16.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:38:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:16.998 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:16 smithi029 ceph-mon[31006]: pgmap v362: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:17.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:16 smithi066 ceph-mon[33436]: pgmap v362: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:17.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 -- 172.21.15.29:0/401110150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb020109df0 msgr2=0x7fb020110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 --2- 172.21.15.29:0/401110150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb020109df0 0x7fb020110620 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb018009ca0 tx=0x7fb01802a8e0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 -- 172.21.15.29:0/401110150 shutdown_connections 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 --2- 172.21.15.29:0/401110150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb020109df0 0x7fb020110620 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 --2- 172.21.15.29:0/401110150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0201094a0 0x7fb0201098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 --2- 172.21.15.29:0/401110150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb020103b60 0x7fb020103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 -- 172.21.15.29:0/401110150 >> 172.21.15.29:0/401110150 conn(0x7fb02007b850 msgr2=0x7fb02007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 -- 172.21.15.29:0/401110150 shutdown_connections 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.090+0000 7fb024e1d640 1 -- 172.21.15.29:0/401110150 wait complete. 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb024e1d640 1 Processor -- start 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb024e1d640 1 -- start start 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb024e1d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb020103b60 0x7fb02019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb024e1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0201094a0 0x7fb02019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb024e1d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb020109df0 0x7fb0201a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb024e1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb020113a20 con 0x7fb0201094a0 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb024e1d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb0201138a0 con 0x7fb020103b60 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb024e1d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb020113ba0 con 0x7fb020109df0 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb01dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0201094a0 0x7fb02019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb01dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0201094a0 0x7fb02019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41228/0 (socket says 172.21.15.29:41228) 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb01dd9b640 1 -- 172.21.15.29:0/961957440 learned_addr learned my addr 172.21.15.29:0/961957440 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb01e59c640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb020103b60 0x7fb02019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb01ed9d640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb020109df0 0x7fb0201a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb01dd9b640 1 -- 172.21.15.29:0/961957440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb020109df0 msgr2=0x7fb0201a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.091+0000 7fb01dd9b640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb020109df0 0x7fb0201a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb01dd9b640 1 -- 172.21.15.29:0/961957440 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb020103b60 msgr2=0x7fb02019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb01dd9b640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb020103b60 0x7fb02019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb01dd9b640 1 -- 172.21.15.29:0/961957440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0201a2660 con 0x7fb0201094a0 2024-09-21T02:38:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb01ed9d640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb020109df0 0x7fb0201a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb01dd9b640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0201094a0 0x7fb02019cc90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb014006fd0 tx=0x7fb01400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb00f7fe640 1 -- 172.21.15.29:0/961957440 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014011830 con 0x7fb0201094a0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0201a2950 con 0x7fb0201094a0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0201a9130 con 0x7fb0201094a0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb00f7fe640 1 -- 172.21.15.29:0/961957440 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb014016070 con 0x7fb0201094a0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.092+0000 7fb00f7fe640 1 -- 172.21.15.29:0/961957440 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014010cf0 con 0x7fb0201094a0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.093+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0000052c0 con 0x7fb0201094a0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.095+0000 7fb00f7fe640 1 -- 172.21.15.29:0/961957440 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fb0140119d0 con 0x7fb0201094a0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.095+0000 7fb00f7fe640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0080b6620 0x7fb0080b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.096+0000 7fb01e59c640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0080b6620 0x7fb0080b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.096+0000 7fb00f7fe640 1 -- 172.21.15.29:0/961957440 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fb0140e7310 con 0x7fb0201094a0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.096+0000 7fb01e59c640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0080b6620 0x7fb0080b8b30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb010009890 tx=0x7fb010008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.099+0000 7fb00f7fe640 1 -- 172.21.15.29:0/961957440 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb0140ad680 con 0x7fb0201094a0 2024-09-21T02:38:17.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:16 smithi190 ceph-mon[33871]: pgmap v362: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.322+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0000055b0 con 0x7fb0201094a0 2024-09-21T02:38:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.323+0000 7fb00f7fe640 1 -- 172.21.15.29:0/961957440 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11971 (secure 0 0 0) 0x7fb0140b1a60 con 0x7fb0201094a0 2024-09-21T02:38:17.325 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:17.325 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-09-21T02:37:10:039454+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:37:10.039429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:38:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-09-21T02:38:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0080b6620 msgr2=0x7fb0080b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0080b6620 0x7fb0080b8b30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb010009890 tx=0x7fb010008040 comp rx=0 tx=0).stop 2024-09-21T02:38:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0201094a0 msgr2=0x7fb02019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0201094a0 0x7fb02019cc90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb014006fd0 tx=0x7fb01400b8e0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 shutdown_connections 2024-09-21T02:38:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0080b6620 0x7fb0080b8b30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb020109df0 0x7fb0201a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0201094a0 0x7fb02019cc90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 --2- 172.21.15.29:0/961957440 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb020103b60 0x7fb02019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 >> 172.21.15.29:0/961957440 conn(0x7fb02007b850 msgr2=0x7fb020107350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 shutdown_connections 2024-09-21T02:38:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:17.326+0000 7fb024e1d640 1 -- 172.21.15.29:0/961957440 wait complete. 2024-09-21T02:38:18.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:17 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/961957440' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:18.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:17 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/961957440' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:18.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:17 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/961957440' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:18.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:38:18.475 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:18 smithi029 ceph-mon[31006]: pgmap v363: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:19.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 -- 172.21.15.29:0/1443520105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5160102a60 msgr2=0x7f5160102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:19.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 --2- 172.21.15.29:0/1443520105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5160102a60 0x7f5160102e70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f515c009900 tx=0x7f515c02a410 comp rx=0 tx=0).stop 2024-09-21T02:38:19.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 -- 172.21.15.29:0/1443520105 shutdown_connections 2024-09-21T02:38:19.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 --2- 172.21.15.29:0/1443520105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5160108cf0 0x7f516010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 --2- 172.21.15.29:0/1443520105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51601083a0 0x7f51601087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 --2- 172.21.15.29:0/1443520105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5160102a60 0x7f5160102e70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 -- 172.21.15.29:0/1443520105 >> 172.21.15.29:0/1443520105 conn(0x7f5160079730 msgr2=0x7f5160079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 -- 172.21.15.29:0/1443520105 shutdown_connections 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 -- 172.21.15.29:0/1443520105 wait complete. 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.011+0000 7f5167798640 1 Processor -- start 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5167798640 1 -- start start 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5167798640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5160102a60 0x7f516019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5167798640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51601083a0 0x7f516019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5167798640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5160108cf0 0x7f516019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5167798640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112810 con 0x7f51601083a0 2024-09-21T02:38:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5167798640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112690 con 0x7f5160108cf0 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5167798640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112990 con 0x7f5160102a60 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5164d33640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51601083a0 0x7f516019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5164d33640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51601083a0 0x7f516019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41240/0 (socket says 172.21.15.29:41240) 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5164d33640 1 -- 172.21.15.29:0/3124714736 learned_addr learned my addr 172.21.15.29:0/3124714736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5165534640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5160102a60 0x7f516019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5165d35640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5160108cf0 0x7f516019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5164d33640 1 -- 172.21.15.29:0/3124714736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5160102a60 msgr2=0x7f516019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5164d33640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5160102a60 0x7f516019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5164d33640 1 -- 172.21.15.29:0/3124714736 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5160108cf0 msgr2=0x7f516019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5164d33640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5160108cf0 0x7f516019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5164d33640 1 -- 172.21.15.29:0/3124714736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51601a0220 con 0x7f51601083a0 2024-09-21T02:38:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.012+0000 7f5165d35640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5160108cf0 0x7f516019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.013+0000 7f5164d33640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51601083a0 0x7f516019a910 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f515000c970 tx=0x7f515000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.013+0000 7f51567fc640 1 -- 172.21.15.29:0/3124714736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5150014070 con 0x7f51601083a0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.013+0000 7f51567fc640 1 -- 172.21.15.29:0/3124714736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5150004440 con 0x7f51601083a0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.013+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51601a0510 con 0x7f51601083a0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.013+0000 7f51567fc640 1 -- 172.21.15.29:0/3124714736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5150003d40 con 0x7f51601083a0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.013+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51601a6d30 con 0x7f51601083a0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.014+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5160102e70 con 0x7f51601083a0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.015+0000 7f51567fc640 1 -- 172.21.15.29:0/3124714736 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f5150017070 con 0x7f51601083a0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.016+0000 7f51567fc640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f514c0b6650 0x7f514c0b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.016+0000 7f51567fc640 1 -- 172.21.15.29:0/3124714736 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f51500e7aa0 con 0x7f51601083a0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.016+0000 7f5165534640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f514c0b6650 0x7f514c0b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.017+0000 7f5165534640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f514c0b6650 0x7f514c0b8b60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f515c0057c0 tx=0x7f515c0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.019+0000 7f51567fc640 1 -- 172.21.15.29:0/3124714736 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f515001e030 con 0x7f51601083a0 2024-09-21T02:38:19.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:18 smithi190 ceph-mon[33871]: pgmap v363: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:19.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.248+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f516019b4a0 con 0x7f51601083a0 2024-09-21T02:38:19.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.248+0000 7f51567fc640 1 -- 172.21.15.29:0/3124714736 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11971 (secure 0 0 0) 0x7f51500adef0 con 0x7f51601083a0 2024-09-21T02:38:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:19.250 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-09-21T02:37:10:039454+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:37:10.039429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:38:19.250 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-09-21T02:38:19.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f514c0b6650 msgr2=0x7f514c0b8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:19.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f514c0b6650 0x7f514c0b8b60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f515c0057c0 tx=0x7f515c0058d0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51601083a0 msgr2=0x7f516019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:19.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51601083a0 0x7f516019a910 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f515000c970 tx=0x7f515000ce30 comp rx=0 tx=0).stop 2024-09-21T02:38:19.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 shutdown_connections 2024-09-21T02:38:19.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f514c0b6650 0x7f514c0b8b60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5160108cf0 0x7f516019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51601083a0 0x7f516019a910 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 --2- 172.21.15.29:0/3124714736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5160102a60 0x7f516019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:19.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 >> 172.21.15.29:0/3124714736 conn(0x7f5160079730 msgr2=0x7f5160101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:19.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 shutdown_connections 2024-09-21T02:38:19.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:19.251+0000 7f5167798640 1 -- 172.21.15.29:0/3124714736 wait complete. 2024-09-21T02:38:19.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:18 smithi066 ceph-mon[33436]: pgmap v363: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:20.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:19 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3124714736' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:20.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:19 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3124714736' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:20.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:38:20.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:19 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3124714736' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:20.393 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:21.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.985+0000 7f38b46b8640 1 -- 172.21.15.29:0/19815312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 msgr2=0x7f38ac10d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.985+0000 7f38b46b8640 1 --2- 172.21.15.29:0/19815312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 0x7f38ac10d900 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f389c009900 tx=0x7f389c02a410 comp rx=0 tx=0).stop 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 -- 172.21.15.29:0/19815312 shutdown_connections 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 --2- 172.21.15.29:0/19815312 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38ac10dec0 0x7f38ac1103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 --2- 172.21.15.29:0/19815312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 0x7f38ac10d900 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 --2- 172.21.15.29:0/19815312 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac104a60 0x7f38ac104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 -- 172.21.15.29:0/19815312 >> 172.21.15.29:0/19815312 conn(0x7f38ac0796d0 msgr2=0x7f38ac079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 -- 172.21.15.29:0/19815312 shutdown_connections 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 -- 172.21.15.29:0/19815312 wait complete. 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 Processor -- start 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.986+0000 7f38b46b8640 1 -- start start 2024-09-21T02:38:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b46b8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac104a60 0x7f38ac19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b46b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 0x7f38ac19aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b46b8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38ac10dec0 0x7f38ac19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b46b8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac1139d0 con 0x7f38ac1053b0 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b46b8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac113850 con 0x7f38ac10dec0 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b46b8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac113b50 con 0x7f38ac104a60 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 0x7f38ac19aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 0x7f38ac19aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41256/0 (socket says 172.21.15.29:41256) 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 -- 172.21.15.29:0/1352228791 learned_addr learned my addr 172.21.15.29:0/1352228791 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b2454640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac104a60 0x7f38ac19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b2c55640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38ac10dec0 0x7f38ac19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 -- 172.21.15.29:0/1352228791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac104a60 msgr2=0x7f38ac19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac104a60 0x7f38ac19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 -- 172.21.15.29:0/1352228791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38ac10dec0 msgr2=0x7f38ac19fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38ac10dec0 0x7f38ac19fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 -- 172.21.15.29:0/1352228791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac1a0410 con 0x7f38ac1053b0 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b2454640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac104a60 0x7f38ac19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b2c55640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38ac10dec0 0x7f38ac19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38b1c53640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 0x7f38ac19aa40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f389c005940 tx=0x7f389c0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.987+0000 7f38a37fe640 1 -- 172.21.15.29:0/1352228791 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c03f030 con 0x7f38ac1053b0 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.988+0000 7f38a37fe640 1 -- 172.21.15.29:0/1352228791 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f389c003a80 con 0x7f38ac1053b0 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.988+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ac1a06a0 con 0x7f38ac1053b0 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.988+0000 7f38a37fe640 1 -- 172.21.15.29:0/1352228791 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c035070 con 0x7f38ac1053b0 2024-09-21T02:38:21.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.988+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac1a6e60 con 0x7f38ac1053b0 2024-09-21T02:38:21.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.989+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38ac06c260 con 0x7f38ac1053b0 2024-09-21T02:38:21.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.990+0000 7f38a37fe640 1 -- 172.21.15.29:0/1352228791 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f389c042030 con 0x7f38ac1053b0 2024-09-21T02:38:21.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.991+0000 7f38a37fe640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f38980b65a0 0x7f38980b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:21.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.991+0000 7f38a37fe640 1 -- 172.21.15.29:0/1352228791 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f389c108210 con 0x7f38ac1053b0 2024-09-21T02:38:21.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.991+0000 7f38b2454640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f38980b65a0 0x7f38980b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:21.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.992+0000 7f38b2454640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f38980b65a0 0x7f38980b8ab0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f38a80058c0 tx=0x7f38a8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:21.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:20.994+0000 7f38a37fe640 1 -- 172.21.15.29:0/1352228791 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f389c0ce5e0 con 0x7f38ac1053b0 2024-09-21T02:38:21.025 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:20 smithi190 ceph-mon[33871]: pgmap v364: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:21.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:20 smithi029 ceph-mon[31006]: pgmap v364: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:21.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.186+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38ac108fc0 con 0x7f38ac1053b0 2024-09-21T02:38:21.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.187+0000 7f38a37fe640 1 -- 172.21.15.29:0/1352228791 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+24726 (secure 0 0 0) 0x7f389c0d29c0 con 0x7f38ac1053b0 2024-09-21T02:38:21.189 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:21.190 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:37:39.593209+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:38:21.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f38980b65a0 msgr2=0x7f38980b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:21.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f38980b65a0 0x7f38980b8ab0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f38a80058c0 tx=0x7f38a8005850 comp rx=0 tx=0).stop 2024-09-21T02:38:21.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 msgr2=0x7f38ac19aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:21.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 0x7f38ac19aa40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f389c005940 tx=0x7f389c0040e0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 shutdown_connections 2024-09-21T02:38:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f38980b65a0 0x7f38980b8ab0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38ac10dec0 0x7f38ac19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38ac1053b0 0x7f38ac19aa40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 --2- 172.21.15.29:0/1352228791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac104a60 0x7f38ac19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 >> 172.21.15.29:0/1352228791 conn(0x7f38ac0796d0 msgr2=0x7f38ac100a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 shutdown_connections 2024-09-21T02:38:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:21.190+0000 7f38b46b8640 1 -- 172.21.15.29:0/1352228791 wait complete. 2024-09-21T02:38:21.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:20 smithi066 ceph-mon[33436]: pgmap v364: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:21 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1352228791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:38:22.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:38:22.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:21 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1352228791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:38:22.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:21 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1352228791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:38:22.305 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.824+0000 7fb1bab5f640 1 -- 172.21.15.29:0/3697493376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b4102a80 msgr2=0x7fb1b4102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.824+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/3697493376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b4102a80 0x7fb1b4102e90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009900 tx=0x7fb1a802a410 comp rx=0 tx=0).stop 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 -- 172.21.15.29:0/3697493376 shutdown_connections 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/3697493376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4108d10 0x7fb1b410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/3697493376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb1b41083c0 0x7fb1b41087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/3697493376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b4102a80 0x7fb1b4102e90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 -- 172.21.15.29:0/3697493376 >> 172.21.15.29:0/3697493376 conn(0x7fb1b4079740 msgr2=0x7fb1b4079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 -- 172.21.15.29:0/3697493376 shutdown_connections 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 -- 172.21.15.29:0/3697493376 wait complete. 2024-09-21T02:38:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 Processor -- start 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.825+0000 7fb1bab5f640 1 -- start start 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1bab5f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4102a80 0x7fb1b419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1bab5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b41083c0 0x7fb1b419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1bab5f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb1b4108d10 0x7fb1b419fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1bab5f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b4113950 con 0x7fb1b41083c0 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1bab5f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41137d0 con 0x7fb1b4108d10 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1bab5f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b4113ad0 con 0x7fb1b4102a80 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b41083c0 0x7fb1b419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b41083c0 0x7fb1b419a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41284/0 (socket says 172.21.15.29:41284) 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 -- 172.21.15.29:0/622354759 learned_addr learned my addr 172.21.15.29:0/622354759 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b88fb640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4102a80 0x7fb1b419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 -- 172.21.15.29:0/622354759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4102a80 msgr2=0x7fb1b419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b90fc640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb1b4108d10 0x7fb1b419fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:22.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4102a80 0x7fb1b419a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 -- 172.21.15.29:0/622354759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb1b4108d10 msgr2=0x7fb1b419fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb1b4108d10 0x7fb1b419fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 -- 172.21.15.29:0/622354759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b41a02f0 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.826+0000 7fb1b3fff640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b41083c0 0x7fb1b419a920 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000d770 tx=0x7fb1a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.827+0000 7fb1b1ffb640 1 -- 172.21.15.29:0/622354759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0017070 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.827+0000 7fb1b1ffb640 1 -- 172.21.15.29:0/622354759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1a0009c30 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.827+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b41a05e0 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.827+0000 7fb1b1ffb640 1 -- 172.21.15.29:0/622354759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a000f620 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.827+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b41a6d40 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.828+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b4103d00 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.831+0000 7fb1b1ffb640 1 -- 172.21.15.29:0/622354759 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fb1a001a070 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.831+0000 7fb1b1ffb640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb1a40b65a0 0x7fb1a40b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.832+0000 7fb1b88fb640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb1a40b65a0 0x7fb1a40b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.832+0000 7fb1b1ffb640 1 -- 172.21.15.29:0/622354759 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fb1a00e7360 con 0x7fb1b41083c0 2024-09-21T02:38:22.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.832+0000 7fb1b88fb640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb1a40b65a0 0x7fb1a40b8ab0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8005f70 tx=0x7fb1a8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:22.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:22.835+0000 7fb1b1ffb640 1 -- 172.21.15.29:0/622354759 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb1a0009de0 con 0x7fb1b41083c0 2024-09-21T02:38:23.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.061+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1b4102e90 con 0x7fb1b41083c0 2024-09-21T02:38:23.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:22 smithi029 ceph-mon[31006]: pgmap v365: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:23.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.062+0000 7fb1b1ffb640 1 -- 172.21.15.29:0/622354759 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11971 (secure 0 0 0) 0x7fb1a00ad730 con 0x7fb1b41083c0 2024-09-21T02:38:23.064 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:23.064 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-09-21T02:37:10:039454+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:37:10.039429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:38:23.064 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-09-21T02:38:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb1a40b65a0 msgr2=0x7fb1a40b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb1a40b65a0 0x7fb1a40b8ab0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8005f70 tx=0x7fb1a8000bc0 comp rx=0 tx=0).stop 2024-09-21T02:38:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b41083c0 msgr2=0x7fb1b419a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b41083c0 0x7fb1b419a920 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000d770 tx=0x7fb1a000dc30 comp rx=0 tx=0).stop 2024-09-21T02:38:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 shutdown_connections 2024-09-21T02:38:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb1a40b65a0 0x7fb1a40b8ab0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb1b4108d10 0x7fb1b419fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1b41083c0 0x7fb1b419a920 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 --2- 172.21.15.29:0/622354759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4102a80 0x7fb1b419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 >> 172.21.15.29:0/622354759 conn(0x7fb1b4079740 msgr2=0x7fb1b41014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 shutdown_connections 2024-09-21T02:38:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:23.066+0000 7fb1bab5f640 1 -- 172.21.15.29:0/622354759 wait complete. 2024-09-21T02:38:23.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:22 smithi190 ceph-mon[33871]: pgmap v365: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:23.160 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:22 smithi066 ceph-mon[33436]: pgmap v365: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:23.966 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T02:36:26.562695+0000', 'modified': '2024-09-21T02:37:10.039429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24637, 'mds_1': 24632, 'mds_2': 24625, 'mds_3': 24613, 'mds_4': 24599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14781': {'gid': 14781, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6839/886485890', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 886485890}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 886485890}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14787': {'gid': 14787, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6841/1488187270', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6840', 'nonce': 1488187270}, {'type': 'v1', 'addr': '172.21.15.29:6841', 'nonce': 1488187270}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.66:6833/2291646688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6832', 'nonce': 2291646688}, {'type': 'v1', 'addr': '172.21.15.66:6833', 'nonce': 2291646688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24574': {'gid': 24574, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6833/3478087331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6832', 'nonce': 3478087331}, {'type': 'v1', 'addr': '172.21.15.190:6833', 'nonce': 3478087331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.66:6835/4067532455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6834', 'nonce': 4067532455}, {'type': 'v1', 'addr': '172.21.15.66:6835', 'nonce': 4067532455}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24599': {'gid': 24599, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.66:6837/3184195773', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6836', 'nonce': 3184195773}, {'type': 'v1', 'addr': '172.21.15.66:6837', 'nonce': 3184195773}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24613': {'gid': 24613, 'name': 'f', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.190:6835/131555194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 131555194}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 131555194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24625': {'gid': 24625, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.190:6837/3758896499', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 3758896499}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 3758896499}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24632': {'gid': 24632, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.66:6839/3618856137', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6838', 'nonce': 3618856137}, {'type': 'v1', 'addr': '172.21.15.66:6839', 'nonce': 3618856137}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24637': {'gid': 24637, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.190:6839/3083654868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 3083654868}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 3083654868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24637, 'qdb_cluster': [24637, 24632, 24613, 24625, 24599]} 2024-09-21T02:38:23.966 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-21T02:38:23.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.h status 2024-09-21T02:38:24.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:23 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/622354759' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:23 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/622354759' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:24.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:23 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/622354759' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 -- 172.21.15.29:0/199768122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 msgr2=0x7f89cc110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 --2- 172.21.15.29:0/199768122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 0x7f89cc110670 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009900 tx=0x7f89c402a410 comp rx=0 tx=0).stop 2024-09-21T02:38:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 -- 172.21.15.29:0/199768122 shutdown_connections 2024-09-21T02:38:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 --2- 172.21.15.29:0/199768122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 0x7f89cc110670 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 --2- 172.21.15.29:0/199768122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89cc1094f0 0x7f89cc109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 --2- 172.21.15.29:0/199768122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89cc103bb0 0x7f89cc103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 -- 172.21.15.29:0/199768122 >> 172.21.15.29:0/199768122 conn(0x7f89cc0ff600 msgr2=0x7f89cc101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 -- 172.21.15.29:0/199768122 shutdown_connections 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.755+0000 7f89d1307640 1 -- 172.21.15.29:0/199768122 wait complete. 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.756+0000 7f89d1307640 1 Processor -- start 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.756+0000 7f89d1307640 1 -- start start 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.756+0000 7f89d1307640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89cc103bb0 0x7f89cc19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.756+0000 7f89d1307640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89cc1094f0 0x7f89cc19a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.756+0000 7f89d1307640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 0x7f89cc19fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.756+0000 7f89d1307640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f89cc1139b0 con 0x7f89cc109e40 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.756+0000 7f89d1307640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f89cc113830 con 0x7f89cc103bb0 2024-09-21T02:38:24.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.756+0000 7f89d1307640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f89cc113b30 con 0x7f89cc1094f0 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 0x7f89cc19fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 0x7f89cc19fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46144/0 (socket says 172.21.15.29:46144) 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 -- 172.21.15.29:0/1357324530 learned_addr learned my addr 172.21.15.29:0/1357324530 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89cc1094f0 0x7f89cc19a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89caffd640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89cc103bb0 0x7f89cc19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 -- 172.21.15.29:0/1357324530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89cc1094f0 msgr2=0x7f89cc19a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89cc1094f0 0x7f89cc19a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 -- 172.21.15.29:0/1357324530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89cc103bb0 msgr2=0x7f89cc19a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89cc103bb0 0x7f89cc19a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 -- 172.21.15.29:0/1357324530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89cc1a02d0 con 0x7f89cc109e40 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89caffd640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89cc103bb0 0x7f89cc19a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89cb7fe640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 0x7f89cc19fba0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f89c4005ab0 tx=0x7f89c40040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89b3fff640 1 -- 172.21.15.29:0/1357324530 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c403f030 con 0x7f89cc109e40 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89d1307640 1 -- 172.21.15.29:0/1357324530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89cc1a0560 con 0x7f89cc109e40 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89b3fff640 1 -- 172.21.15.29:0/1357324530 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89c4003a80 con 0x7f89cc109e40 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89b3fff640 1 -- 172.21.15.29:0/1357324530 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4035070 con 0x7f89cc109e40 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.757+0000 7f89d1307640 1 -- 172.21.15.29:0/1357324530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89cc1a6d20 con 0x7f89cc109e40 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.760+0000 7f89d787b740 1 Processor -- start 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89b3fff640 1 -- 172.21.15.29:0/1357324530 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f89c4042030 con 0x7f89cc109e40 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89d787b740 1 -- start start 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89d787b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d8af60 0x558e98130f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89d787b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d620a0 0x558e98131440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89d787b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d6a280 0x558e98131980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89d787b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558e980a8c10 con 0x558e97d6a280 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89d787b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x558e98098040 con 0x558e97d8af60 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89d787b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558e9809be50 con 0x558e97d620a0 2024-09-21T02:38:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89ca7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d6a280 0x558e98131980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89ca7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d6a280 0x558e98131980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46152/0 (socket says 172.21.15.29:46152) 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89ca7fc640 1 -- 172.21.15.29:0/3932548983 learned_addr learned my addr 172.21.15.29:0/3932548983 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.761+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d8af60 0x558e98130f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89b3fff640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89b40b65a0 0x7f89b40b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89caffd640 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d620a0 0x558e98131440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89ca7fc640 1 -- 172.21.15.29:0/3932548983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d620a0 msgr2=0x558e98131440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d620a0 0x558e98131440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89ca7fc640 1 -- 172.21.15.29:0/3932548983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d8af60 msgr2=0x558e98130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d8af60 0x558e98130f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89cb7fe640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89b40b65a0 0x7f89b40b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89ca7fc640 1 -- 172.21.15.29:0/3932548983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e97df6540 con 0x558e97d6a280 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89b3fff640 1 -- 172.21.15.29:0/1357324530 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f89c4107d40 con 0x7f89cc109e40 2024-09-21T02:38:24.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d6a280 0x558e98131980 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f89c00098a0 tx=0x7f89c0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f375d2852accb54f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89b17fa640 1 -- 172.21.15.29:0/3932548983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0026020 con 0x558e97d6a280 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89cb7fe640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89b40b65a0 0x7f89b40b8ab0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f89c40cd710 tx=0x7f89c40c22c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89b17fa640 1 -- 172.21.15.29:0/3932548983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89c0003e00 con 0x558e97d6a280 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.762+0000 7f89b17fa640 1 -- 172.21.15.29:0/3932548983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0010d20 con 0x558e97d6a280 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 -- 172.21.15.29:0/3932548983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d6a280 msgr2=0x558e98131980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d6a280 0x558e98131980 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f89c00098a0 tx=0x7f89c0005bd0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 -- 172.21.15.29:0/3932548983 shutdown_connections 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d6a280 0x558e98131980 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d620a0 0x558e98131440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 --2- 172.21.15.29:0/3932548983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d8af60 0x558e98130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 -- 172.21.15.29:0/3932548983 >> 172.21.15.29:0/3932548983 conn(0x558e97d7f350 msgr2=0x558e9812edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 -- 172.21.15.29:0/3932548983 shutdown_connections 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.763+0000 7f89d787b740 1 -- 172.21.15.29:0/3932548983 wait complete. 2024-09-21T02:38:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 1 -- start start 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 1 Processor -- start 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e9813cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e9813d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d8af60 0x558e98142450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558e97ef95f0 con 0x558e97d620a0 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x558e98033010 con 0x558e97d6a280 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558e98137410 con 0x558e97d8af60 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89d787b740 10 client.0 _ll_register_callbacks cb 0x558e98092340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89caffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e9813d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89caffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e9813d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:45920/0 (socket says 172.21.15.29:45920) 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.764+0000 7f89caffd640 1 -- 172.21.15.29:0/2845510234 learned_addr learned my addr 172.21.15.29:0/2845510234 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d8af60 0x558e98142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e9813cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89caffd640 1 -- 172.21.15.29:0/2845510234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d8af60 msgr2=0x558e98142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89caffd640 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d8af60 0x558e98142450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89caffd640 1 -- 172.21.15.29:0/2845510234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 msgr2=0x558e9813cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89caffd640 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e9813cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89caffd640 1 -- 172.21.15.29:0/2845510234 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e98060a00 con 0x558e97d6a280 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89caffd640 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e9813d400 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f89bc011a60 tx=0x7f89bc011f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f7dfdc0734739e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.765+0000 7f89aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89aeff5640 1 -- 172.21.15.29:0/2845510234 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc00fdb0 con 0x558e97d6a280 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89d787b740 1 -- 172.21.15.29:0/2845510234 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89c0003e00 con 0x558e97d6a280 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89aeff5640 1 -- 172.21.15.29:0/2845510234 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f89bc00e3b0 con 0x558e97d6a280 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89aeff5640 1 -- 172.21.15.29:0/2845510234 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f89bc00e5b0 con 0x558e97d6a280 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89d787b740 10 client.24950 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89d787b740 1 -- 172.21.15.29:0/2845510234 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558e9814b610 con 0x558e97d6a280 2024-09-21T02:38:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89aeff5640 1 -- 172.21.15.29:0/2845510234 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89c0003e00 con 0x558e97d6a280 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89aeff5640 1 -- 172.21.15.29:0/2845510234 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f89bc01b790 con 0x558e97d6a280 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89d787b740 10 client.24950 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.766+0000 7f89d787b740 10 client.24950 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.767+0000 7f89d787b740 20 client.24950 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.767+0000 7f89d787b740 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x558e97d498e0 0x558e9814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.767+0000 7f89d787b740 4 client.24950 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.767+0000 7f89d787b740 1 -- 172.21.15.29:0/2845510234 --> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e980aab60 con 0x558e97d498e0 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.767+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x558e97d498e0 0x558e9814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.767+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x558e97d498e0 0x558e9814f0b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f5791125e8c934ea server_cookie=b1db19e40797eb88 in_seq=0 out_seq=0 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.767+0000 7f89aeff5640 10 client.24950 ms_handle_connect on v2:172.21.15.66:6836/3184195773 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89aeff5640 1 -- 172.21.15.29:0/2845510234 <== mds.4 v2:172.21.15.66:6836/3184195773 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558e980aab60 con 0x558e97d498e0 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89aeff5640 10 client.24950 handle_command_reply: tid=0 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 client.24950 shutdown 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 -- 172.21.15.29:0/2845510234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 msgr2=0x558e9813d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e9813d400 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f89bc011a60 tx=0x7f89bc011f20 comp rx=0 tx=0).stop 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 -- 172.21.15.29:0/2845510234 shutdown_connections 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x558e97d498e0 0x558e9814f0b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d8af60 0x558e98142450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e9813d400 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89aeff5640 0 client.24950 ms_handle_reset on v2:172.21.15.66:6836/3184195773 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 --2- 172.21.15.29:0/2845510234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e9813cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 -- 172.21.15.29:0/2845510234 >> 172.21.15.29:0/2845510234 conn(0x558e97d7f350 msgr2=0x558e981328f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 -- 172.21.15.29:0/2845510234 shutdown_connections 2024-09-21T02:38:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.768+0000 7f89d787b740 1 -- 172.21.15.29:0/2845510234 wait complete. 2024-09-21T02:38:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.769+0000 7f89d787b740 20 client.24950 trim_cache size 0 max 16384 2024-09-21T02:38:24.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89d787b740 1 Processor -- start 2024-09-21T02:38:24.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89d787b740 1 -- start start 2024-09-21T02:38:24.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89d787b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d498e0 0x558e9814ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89d787b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e98136030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89d787b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e98137e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89d787b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558e9814d3f0 con 0x558e97d620a0 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89d787b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x558e9814dac0 con 0x558e97d6a280 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89d787b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558e9814e190 con 0x558e97d498e0 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.788+0000 7f89caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e98136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e98136030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46174/0 (socket says 172.21.15.29:46174) 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89caffd640 1 -- 172.21.15.29:0/3515865513 learned_addr learned my addr 172.21.15.29:0/3515865513 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d498e0 0x558e9814ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e98137e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89caffd640 1 -- 172.21.15.29:0/3515865513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d498e0 msgr2=0x558e9814ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89caffd640 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d498e0 0x558e9814ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89caffd640 1 -- 172.21.15.29:0/3515865513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 msgr2=0x558e98137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89caffd640 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e98137e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89caffd640 1 -- 172.21.15.29:0/3515865513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e98131d40 con 0x558e97d620a0 2024-09-21T02:38:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e98137e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89caffd640 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e98136030 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f89bc01bc80 tx=0x7f89bc005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68a67c87720517a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89b0ff9640 1 -- 172.21.15.29:0/3515865513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc02e070 con 0x558e97d620a0 2024-09-21T02:38:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89b0ff9640 1 -- 172.21.15.29:0/3515865513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89bc004020 con 0x558e97d620a0 2024-09-21T02:38:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.789+0000 7f89b0ff9640 1 -- 172.21.15.29:0/3515865513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc0054b0 con 0x558e97d620a0 2024-09-21T02:38:24.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 -- 172.21.15.29:0/3515865513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 msgr2=0x558e98136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e98136030 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f89bc01bc80 tx=0x7f89bc005a90 comp rx=0 tx=0).stop 2024-09-21T02:38:24.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 -- 172.21.15.29:0/3515865513 shutdown_connections 2024-09-21T02:38:24.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d6a280 0x558e98137e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e98136030 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 --2- 172.21.15.29:0/3515865513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d498e0 0x558e9814ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 -- 172.21.15.29:0/3515865513 >> 172.21.15.29:0/3515865513 conn(0x558e97d7f350 msgr2=0x558e98136a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 -- 172.21.15.29:0/3515865513 shutdown_connections 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.790+0000 7f89d787b740 1 -- 172.21.15.29:0/3515865513 wait complete. 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 1 -- start start 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 1 Processor -- start 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d498e0 0x558e9812dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e981487e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d6a280 0x558e98148de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558e9814ef30 con 0x558e97d620a0 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x558e9814e860 con 0x558e97d498e0 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558e9814f600 con 0x558e97d6a280 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89d787b740 10 client.0 _ll_register_callbacks cb 0x7f89bc00c830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:24.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e981487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e981487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46178/0 (socket says 172.21.15.29:46178) 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89caffd640 1 -- 172.21.15.29:0/453417229 learned_addr learned my addr 172.21.15.29:0/453417229 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d498e0 0x558e9812dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.791+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d6a280 0x558e98148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89caffd640 1 -- 172.21.15.29:0/453417229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d6a280 msgr2=0x558e98148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89caffd640 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d6a280 0x558e98148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89caffd640 1 -- 172.21.15.29:0/453417229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d498e0 msgr2=0x558e9812dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89caffd640 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d498e0 0x558e9812dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89caffd640 1 -- 172.21.15.29:0/453417229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e98149570 con 0x558e97d620a0 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d6a280 0x558e98148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89caffd640 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e981487e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f89bc000c00 tx=0x7f89bc00fde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c76e9ca140bce4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:24.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89aeff5640 1 -- 172.21.15.29:0/453417229 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc02e050 con 0x558e97d620a0 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89aeff5640 1 -- 172.21.15.29:0/453417229 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f89bc0053c0 con 0x558e97d620a0 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89d787b740 1 -- 172.21.15.29:0/453417229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89bc004020 con 0x558e97d620a0 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89aeff5640 1 -- 172.21.15.29:0/453417229 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f89bc043050 con 0x558e97d620a0 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89d787b740 10 client.15252 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89d787b740 1 -- 172.21.15.29:0/453417229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558e981ba580 con 0x558e97d620a0 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.792+0000 7f89aeff5640 1 -- 172.21.15.29:0/453417229 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89bc004020 con 0x558e97d620a0 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89aeff5640 1 -- 172.21.15.29:0/453417229 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f89bc00e8e0 con 0x558e97d620a0 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89d787b740 10 client.15252 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89d787b740 10 client.15252 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89d787b740 20 client.15252 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89d787b740 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x558e97d8af60 0x558e981bd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89d787b740 4 client.15252 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89d787b740 1 -- 172.21.15.29:0/453417229 --> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] -- command(tid 0: {"prefix": "status"}) -- 0x558e981434e0 con 0x558e97d8af60 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x558e97d8af60 0x558e981bd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89ca7fc640 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x558e97d8af60 0x558e981bd9b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4d74b3d71bc45732 server_cookie=435c607fbe166f0c in_seq=0 out_seq=0 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.793+0000 7f89aeff5640 10 client.15252 ms_handle_connect on v2:172.21.15.66:6836/3184195773 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89aeff5640 1 -- 172.21.15.29:0/453417229 <== mds.4 v2:172.21.15.66:6836/3184195773 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558e981434e0 con 0x558e97d8af60 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89aeff5640 10 client.15252 handle_command_reply: tid=0 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 client.15252 shutdown 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 -- 172.21.15.29:0/453417229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 msgr2=0x558e981487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e981487e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f89bc000c00 tx=0x7f89bc00fde0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 -- 172.21.15.29:0/453417229 shutdown_connections 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x558e97d8af60 0x558e981bd9b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e97d6a280 0x558e98148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e97d620a0 0x558e981487e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 --2- 172.21.15.29:0/453417229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558e97d498e0 0x558e9812dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 -- 172.21.15.29:0/453417229 >> 172.21.15.29:0/453417229 conn(0x558e97d7f350 msgr2=0x558e98137e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89aeff5640 0 client.15252 ms_handle_reset on v2:172.21.15.66:6836/3184195773 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 -- 172.21.15.29:0/453417229 shutdown_connections 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 1 -- 172.21.15.29:0/453417229 wait complete. 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.794+0000 7f89d787b740 20 client.15252 trim_cache size 0 max 16384 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24599, 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 75.762238382999996, 2024-09-21T02:38:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-09-21T02:38:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 141.01353841299999 2024-09-21T02:38:24.801 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 -- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89b40b65a0 msgr2=0x7f89b40b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89b40b65a0 0x7f89b40b8ab0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f89c40cd710 tx=0x7f89c40c22c0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 -- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 msgr2=0x7f89cc19fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 0x7f89cc19fba0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f89c4005ab0 tx=0x7f89c40040e0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 -- 172.21.15.29:0/1357324530 shutdown_connections 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89b40b65a0 0x7f89b40b8ab0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89cc109e40 0x7f89cc19fba0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89cc1094f0 0x7f89cc19a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 --2- 172.21.15.29:0/1357324530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89cc103bb0 0x7f89cc19a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 -- 172.21.15.29:0/1357324530 >> 172.21.15.29:0/1357324530 conn(0x7f89cc0ff600 msgr2=0x7f89cc101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 -- 172.21.15.29:0/1357324530 shutdown_connections 2024-09-21T02:38:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:24.795+0000 7f89d1307640 1 -- 172.21.15.29:0/1357324530 wait complete. 2024-09-21T02:38:25.127 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:24 smithi190 ceph-mon[33871]: pgmap v366: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:25.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:24 smithi029 ceph-mon[31006]: pgmap v366: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:25.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:24 smithi066 ceph-mon[33436]: pgmap v366: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:25.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.f status 2024-09-21T02:38:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:38:26.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:38:26.209 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:38:26.488 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 -- 172.21.15.29:0/3272974978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 msgr2=0x7f41ac1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 --2- 172.21.15.29:0/3272974978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 0x7f41ac1098d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009900 tx=0x7f41a002a410 comp rx=0 tx=0).stop 2024-09-21T02:38:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 -- 172.21.15.29:0/3272974978 shutdown_connections 2024-09-21T02:38:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 --2- 172.21.15.29:0/3272974978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41ac109e10 0x7f41ac110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 --2- 172.21.15.29:0/3272974978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 0x7f41ac1098d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 --2- 172.21.15.29:0/3272974978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac103b80 0x7f41ac103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 -- 172.21.15.29:0/3272974978 >> 172.21.15.29:0/3272974978 conn(0x7f41ac0ff5d0 msgr2=0x7f41ac101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 -- 172.21.15.29:0/3272974978 shutdown_connections 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 -- 172.21.15.29:0/3272974978 wait complete. 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 Processor -- start 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.036+0000 7f41b24df640 1 -- start start 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41b24df640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac103b80 0x7f41ac19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41b24df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 0x7f41ac19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41b24df640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41ac109e10 0x7f41ac19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41b24df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac113980 con 0x7f41ac1094c0 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41b24df640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac113800 con 0x7f41ac109e10 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41b24df640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac113b00 con 0x7f41ac103b80 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 0x7f41ac19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 0x7f41ac19a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46202/0 (socket says 172.21.15.29:46202) 2024-09-21T02:38:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41ab7fe640 1 -- 172.21.15.29:0/2645524991 learned_addr learned my addr 172.21.15.29:0/2645524991 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41ac109e10 0x7f41ac19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41abfff640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac103b80 0x7f41ac19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41ab7fe640 1 -- 172.21.15.29:0/2645524991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac103b80 msgr2=0x7f41ac19a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41ab7fe640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac103b80 0x7f41ac19a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41ab7fe640 1 -- 172.21.15.29:0/2645524991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41ac109e10 msgr2=0x7f41ac19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41ab7fe640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41ac109e10 0x7f41ac19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.037+0000 7f41ab7fe640 1 -- 172.21.15.29:0/2645524991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac1a0320 con 0x7f41ac1094c0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.038+0000 7f41ab7fe640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 0x7f41ac19a950 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f41a0005940 tx=0x7f41a00040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.038+0000 7f41a97fa640 1 -- 172.21.15.29:0/2645524991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a003f030 con 0x7f41ac1094c0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.038+0000 7f41a97fa640 1 -- 172.21.15.29:0/2645524991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41a0003a80 con 0x7f41ac1094c0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.038+0000 7f41b24df640 1 -- 172.21.15.29:0/2645524991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ac1a05b0 con 0x7f41ac1094c0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.038+0000 7f41a97fa640 1 -- 172.21.15.29:0/2645524991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0035070 con 0x7f41ac1094c0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.038+0000 7f41b24df640 1 -- 172.21.15.29:0/2645524991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ac1a6d70 con 0x7f41ac1094c0 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.040+0000 7f41b8a53740 1 Processor -- start 2024-09-21T02:38:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.040+0000 7f41b8a53740 1 -- start start 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41a97fa640 1 -- 172.21.15.29:0/2645524991 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f41a0042030 con 0x7f41ac1094c0 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb81bf60 0x555ccbbc1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbc2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb7fb280 0x555ccbbc2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b8a53740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555ccbb39c10 con 0x555ccb7fb280 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b8a53740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555ccbb29040 con 0x555ccb81bf60 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b8a53740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555ccbb2ce50 con 0x555ccb7f30a0 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb7fb280 0x555ccbbc2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb7fb280 0x555ccbbc2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46208/0 (socket says 172.21.15.29:46208) 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41ab7fe640 1 -- 172.21.15.29:0/292136367 learned_addr learned my addr 172.21.15.29:0/292136367 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b0a7c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb81bf60 0x555ccbbc1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41abfff640 1 --2- 172.21.15.29:0/292136367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbc2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b0a7c640 1 -- 172.21.15.29:0/292136367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 msgr2=0x555ccbbc2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/292136367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbc2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b0a7c640 1 -- 172.21.15.29:0/292136367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb7fb280 msgr2=0x555ccbbc2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/292136367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb7fb280 0x555ccbbc2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.041+0000 7f41b0a7c640 1 -- 172.21.15.29:0/292136367 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ccb887540 con 0x555ccb81bf60 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.042+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/292136367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb81bf60 0x555ccbbc1f00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009eb0 tx=0x7f41a4010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e81d63c622b1ff server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.042+0000 7f41a97fa640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41940b6620 0x7f41940b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.042+0000 7f41abfff640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41940b6620 0x7f41940b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.042+0000 7f41927fc640 1 -- 172.21.15.29:0/292136367 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4016070 con 0x555ccb81bf60 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.042+0000 7f41a97fa640 1 -- 172.21.15.29:0/2645524991 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f41a0107d70 con 0x7f41ac1094c0 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.042+0000 7f41927fc640 1 -- 172.21.15.29:0/292136367 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41a4004d20 con 0x555ccb81bf60 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.042+0000 7f41927fc640 1 -- 172.21.15.29:0/292136367 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a40126b0 con 0x555ccb81bf60 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 -- 172.21.15.29:0/292136367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb81bf60 msgr2=0x555ccbbc1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 --2- 172.21.15.29:0/292136367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb81bf60 0x555ccbbc1f00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009eb0 tx=0x7f41a4010740 comp rx=0 tx=0).stop 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41abfff640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41940b6620 0x7f41940b8b30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x555ccbbc0b80 tx=0x7f419c00d580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 -- 172.21.15.29:0/292136367 shutdown_connections 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 --2- 172.21.15.29:0/292136367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb7fb280 0x555ccbbc2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 --2- 172.21.15.29:0/292136367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbc2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 --2- 172.21.15.29:0/292136367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb81bf60 0x555ccbbc1f00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 -- 172.21.15.29:0/292136367 >> 172.21.15.29:0/292136367 conn(0x555ccb810350 msgr2=0x555ccbbbfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 -- 172.21.15.29:0/292136367 shutdown_connections 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 -- 172.21.15.29:0/292136367 wait complete. 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 -- start start 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.043+0000 7f41b8a53740 1 Processor -- start 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbcd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbcdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbd9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b8a53740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555ccb98a5f0 con 0x555ccb81bf60 2024-09-21T02:38:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b8a53740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555ccbac4010 con 0x555ccb7fb280 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b8a53740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555ccbbd06e0 con 0x555ccb7f30a0 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b8a53740 10 client.0 _ll_register_callbacks cb 0x555ccbb23340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b0a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbd9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41ab7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbcdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b0a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbd9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46216/0 (socket says 172.21.15.29:46216) 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b0a7c640 1 -- 172.21.15.29:0/2434058829 learned_addr learned my addr 172.21.15.29:0/2434058829 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41abfff640 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbcd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b0a7c640 1 -- 172.21.15.29:0/2434058829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 msgr2=0x555ccbbcd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbcd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b0a7c640 1 -- 172.21.15.29:0/2434058829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 msgr2=0x555ccbbcdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbcdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.044+0000 7f41b0a7c640 1 -- 172.21.15.29:0/2434058829 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ccbaf1a00 con 0x555ccb81bf60 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbd9a00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f41a4005a20 tx=0x7f41a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03943383df5d823 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f418b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f418b7fe640 1 -- 172.21.15.29:0/2434058829 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4016070 con 0x555ccb81bf60 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f418b7fe640 1 -- 172.21.15.29:0/2434058829 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f41a4003d80 con 0x555ccb81bf60 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b8a53740 1 -- 172.21.15.29:0/2434058829 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41a4004d20 con 0x555ccb81bf60 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f418b7fe640 1 -- 172.21.15.29:0/2434058829 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41a402f050 con 0x555ccb81bf60 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b8a53740 10 client.15264 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b8a53740 1 -- 172.21.15.29:0/2434058829 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ccbbdc650 con 0x555ccb81bf60 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f418b7fe640 1 -- 172.21.15.29:0/2434058829 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41a4004d20 con 0x555ccb81bf60 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f418b7fe640 1 -- 172.21.15.29:0/2434058829 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f41a400f040 con 0x555ccb81bf60 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b8a53740 10 client.15264 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b8a53740 10 client.15264 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b8a53740 20 client.15264 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x555ccb7da8e0 0x555ccbbdfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.045+0000 7f41b8a53740 4 client.15264 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.046+0000 7f41b8a53740 1 -- 172.21.15.29:0/2434058829 --> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ccbb3bb60 con 0x555ccb7da8e0 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.046+0000 7f41abfff640 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x555ccb7da8e0 0x555ccbbdfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.046+0000 7f41abfff640 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x555ccb7da8e0 0x555ccbbdfce0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ea34417cf0a1a781 server_cookie=73137bfc3c164e2 in_seq=0 out_seq=0 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.046+0000 7f418b7fe640 10 client.15264 ms_handle_connect on v2:172.21.15.190:6834/131555194 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f418b7fe640 1 -- 172.21.15.29:0/2434058829 <== mds.3 v2:172.21.15.190:6834/131555194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555ccbb3bb60 con 0x555ccb7da8e0 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f418b7fe640 10 client.15264 handle_command_reply: tid=0 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 client.15264 shutdown 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 -- 172.21.15.29:0/2434058829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 msgr2=0x555ccbbd9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbd9a00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f41a4005a20 tx=0x7f41a40057f0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 -- 172.21.15.29:0/2434058829 shutdown_connections 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x555ccb7da8e0 0x555ccbbdfce0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbd9a00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbcdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2434058829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbcd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f418b7fe640 0 client.15264 ms_handle_reset on v2:172.21.15.190:6834/131555194 2024-09-21T02:38:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 -- 172.21.15.29:0/2434058829 >> 172.21.15.29:0/2434058829 conn(0x555ccb810350 msgr2=0x555ccbbc5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.047+0000 7f41b8a53740 1 -- 172.21.15.29:0/2434058829 shutdown_connections 2024-09-21T02:38:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.048+0000 7f41b8a53740 1 -- 172.21.15.29:0/2434058829 wait complete. 2024-09-21T02:38:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.048+0000 7f41b8a53740 20 client.15264 trim_cache size 0 max 16384 2024-09-21T02:38:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.067+0000 7f41b8a53740 1 Processor -- start 2024-09-21T02:38:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.067+0000 7f41b8a53740 1 -- start start 2024-09-21T02:38:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbdfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbd9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbcc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b8a53740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555ccbbde020 con 0x555ccb81bf60 2024-09-21T02:38:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b8a53740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555ccbbde6f0 con 0x555ccb7fb280 2024-09-21T02:38:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b8a53740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555ccbbdedc0 con 0x555ccb7f30a0 2024-09-21T02:38:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41ab7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbd9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41ab7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbd9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46008/0 (socket says 172.21.15.29:46008) 2024-09-21T02:38:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b0a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbcc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41ab7fe640 1 -- 172.21.15.29:0/1216094825 learned_addr learned my addr 172.21.15.29:0/1216094825 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41abfff640 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbdfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b0a7c640 1 -- 172.21.15.29:0/1216094825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 msgr2=0x555ccbbdfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbdfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b0a7c640 1 -- 172.21.15.29:0/1216094825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 msgr2=0x555ccbbd9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbd9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41b0a7c640 1 -- 172.21.15.29:0/1216094825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ccbbceaa0 con 0x555ccb81bf60 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41ab7fe640 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbd9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.068+0000 7f41abfff640 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbdfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.069+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbcc060 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f41a4005ff0 tx=0x7f41a4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fb957600365b7a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.069+0000 7f4191ffb640 1 -- 172.21.15.29:0/1216094825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4016040 con 0x555ccb81bf60 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.069+0000 7f4191ffb640 1 -- 172.21.15.29:0/1216094825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41a4004010 con 0x555ccb81bf60 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.069+0000 7f4191ffb640 1 -- 172.21.15.29:0/1216094825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4005150 con 0x555ccb81bf60 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 -- 172.21.15.29:0/1216094825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 msgr2=0x555ccbbcc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbcc060 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f41a4005ff0 tx=0x7f41a4010740 comp rx=0 tx=0).stop 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 -- 172.21.15.29:0/1216094825 shutdown_connections 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbcc060 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbd9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 --2- 172.21.15.29:0/1216094825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbdfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 -- 172.21.15.29:0/1216094825 >> 172.21.15.29:0/1216094825 conn(0x555ccb810350 msgr2=0x555ccbbc89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 -- 172.21.15.29:0/1216094825 shutdown_connections 2024-09-21T02:38:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 -- 172.21.15.29:0/1216094825 wait complete. 2024-09-21T02:38:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.070+0000 7f41b8a53740 1 -- start start 2024-09-21T02:38:27.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b8a53740 1 Processor -- start 2024-09-21T02:38:27.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbe26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbe2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b8a53740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbe7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b8a53740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555ccbbdfb60 con 0x555ccb81bf60 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b8a53740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555ccbbdf490 con 0x555ccb7fb280 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b8a53740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555ccbbe0230 con 0x555ccb7f30a0 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b8a53740 10 client.0 _ll_register_callbacks cb 0x7f41a010b6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b0a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbe7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b0a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbe7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46230/0 (socket says 172.21.15.29:46230) 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41b0a7c640 1 -- 172.21.15.29:0/2761257968 learned_addr learned my addr 172.21.15.29:0/2761257968 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41ab7fe640 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbe2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.071+0000 7f41abfff640 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbe26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41ab7fe640 1 -- 172.21.15.29:0/2761257968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 msgr2=0x555ccbbe26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41ab7fe640 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbe26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41ab7fe640 1 -- 172.21.15.29:0/2761257968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 msgr2=0x555ccbbe7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41ab7fe640 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbe7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41ab7fe640 1 -- 172.21.15.29:0/2761257968 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ccbbe7b20 con 0x555ccb7fb280 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41b0a7c640 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbe7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41ab7fe640 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbe2c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41a0032a50 tx=0x7f41a010b350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a62bf04a04ac0b08 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f418b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f418b7fe640 1 -- 172.21.15.29:0/2761257968 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0049040 con 0x555ccb7fb280 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f418b7fe640 1 -- 172.21.15.29:0/2761257968 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f41a00d7170 con 0x555ccb7fb280 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41b8a53740 1 -- 172.21.15.29:0/2761257968 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41a4004010 con 0x555ccb7fb280 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f418b7fe640 1 -- 172.21.15.29:0/2761257968 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41a00c9880 con 0x555ccb7fb280 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41b8a53740 10 client.24971 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.072+0000 7f41b8a53740 1 -- 172.21.15.29:0/2761257968 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ccbc4b1b0 con 0x555ccb7fb280 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f418b7fe640 1 -- 172.21.15.29:0/2761257968 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41a4004010 con 0x555ccb7fb280 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f418b7fe640 1 -- 172.21.15.29:0/2761257968 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f41a00d2020 con 0x555ccb7fb280 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f41b8a53740 10 client.24971 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f41b8a53740 10 client.24971 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f41b8a53740 20 client.24971 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x555ccb7afc90 0x555ccbc4e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f41b8a53740 4 client.24971 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f41b8a53740 1 -- 172.21.15.29:0/2761257968 --> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] -- command(tid 0: {"prefix": "status"}) -- 0x555ccbb37610 con 0x555ccb7afc90 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.073+0000 7f41abfff640 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x555ccb7afc90 0x555ccbc4e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f418b7fe640 10 client.24971 ms_handle_connect on v2:172.21.15.190:6834/131555194 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41abfff640 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x555ccb7afc90 0x555ccbc4e990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3ec06dfa1be7bd94 server_cookie=9dc7da3ae074787 in_seq=0 out_seq=0 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f418b7fe640 1 -- 172.21.15.29:0/2761257968 <== mds.3 v2:172.21.15.190:6834/131555194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x555ccbb37610 con 0x555ccb7afc90 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f418b7fe640 10 client.24971 handle_command_reply: tid=0 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 client.24971 shutdown 2024-09-21T02:38:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 -- 172.21.15.29:0/2761257968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 msgr2=0x555ccbbe2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbe2c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41a0032a50 tx=0x7f41a010b350 comp rx=0 tx=0).stop 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 -- 172.21.15.29:0/2761257968 shutdown_connections 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x555ccb7afc90 0x555ccbc4e990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f418b7fe640 0 client.24971 ms_handle_reset on v2:172.21.15.190:6834/131555194 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ccb81bf60 0x555ccbbe7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555ccb7fb280 0x555ccbbe2c40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 --2- 172.21.15.29:0/2761257968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ccb7f30a0 0x555ccbbe26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 -- 172.21.15.29:0/2761257968 >> 172.21.15.29:0/2761257968 conn(0x555ccb810350 msgr2=0x555ccbbcd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 -- 172.21.15.29:0/2761257968 shutdown_connections 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.074+0000 7f41b8a53740 1 -- 172.21.15.29:0/2761257968 wait complete. 2024-09-21T02:38:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b8a53740 20 client.24971 trim_cache size 0 max 16384 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24613, 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 79.051256801999997, 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 148.431454395 2024-09-21T02:38:27.082 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 -- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41940b6620 msgr2=0x7f41940b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41940b6620 0x7f41940b8b30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x555ccbbc0b80 tx=0x7f419c00d580 comp rx=0 tx=0).stop 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 -- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 msgr2=0x7f41ac19a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 0x7f41ac19a950 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f41a0005940 tx=0x7f41a00040e0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 -- 172.21.15.29:0/2645524991 shutdown_connections 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41940b6620 0x7f41940b8b30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41ac109e10 0x7f41ac19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41ac1094c0 0x7f41ac19a950 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 --2- 172.21.15.29:0/2645524991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac103b80 0x7f41ac19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 -- 172.21.15.29:0/2645524991 >> 172.21.15.29:0/2645524991 conn(0x7f41ac0ff5d0 msgr2=0x7f41ac1019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 -- 172.21.15.29:0/2645524991 shutdown_connections 2024-09-21T02:38:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:27.075+0000 7f41b24df640 1 -- 172.21.15.29:0/2645524991 wait complete. 2024-09-21T02:38:27.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:27 smithi029 ceph-mon[31006]: pgmap v367: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:27.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:27.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:27.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:27.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:27 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:27.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.c status 2024-09-21T02:38:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:27 smithi066 ceph-mon[33436]: pgmap v367: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:27 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:27 smithi190 ceph-mon[33871]: pgmap v367: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:27 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.717+0000 7fc1857df640 1 -- 172.21.15.29:0/141618290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180103b60 msgr2=0x7fc180103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.717+0000 7fc1857df640 1 --2- 172.21.15.29:0/141618290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180103b60 0x7fc180103f70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc170009900 tx=0x7fc17002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.717+0000 7fc1857df640 1 -- 172.21.15.29:0/141618290 shutdown_connections 2024-09-21T02:38:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.717+0000 7fc1857df640 1 --2- 172.21.15.29:0/141618290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc180109df0 0x7fc180110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.717+0000 7fc1857df640 1 --2- 172.21.15.29:0/141618290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1801094a0 0x7fc1801098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.717+0000 7fc1857df640 1 --2- 172.21.15.29:0/141618290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180103b60 0x7fc180103f70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.718+0000 7fc1857df640 1 -- 172.21.15.29:0/141618290 >> 172.21.15.29:0/141618290 conn(0x7fc18007b850 msgr2=0x7fc18007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.718+0000 7fc1857df640 1 -- 172.21.15.29:0/141618290 shutdown_connections 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.718+0000 7fc1857df640 1 -- 172.21.15.29:0/141618290 wait complete. 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.718+0000 7fc1857df640 1 Processor -- start 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.718+0000 7fc1857df640 1 -- start start 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc1857df640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc180103b60 0x7fc180075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc1857df640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1801094a0 0x7fc180075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc1857df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180109df0 0x7fc180076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc1857df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc1801139b0 con 0x7fc180109df0 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc1857df640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc180113830 con 0x7fc180103b60 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc1857df640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc180113b30 con 0x7fc1801094a0 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180109df0 0x7fc180076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180109df0 0x7fc180076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46246/0 (socket says 172.21.15.29:46246) 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17f7fe640 1 -- 172.21.15.29:0/1860006549 learned_addr learned my addr 172.21.15.29:0/1860006549 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1801094a0 0x7fc180075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17effd640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc180103b60 0x7fc180075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17f7fe640 1 -- 172.21.15.29:0/1860006549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1801094a0 msgr2=0x7fc180075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1801094a0 0x7fc180075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17f7fe640 1 -- 172.21.15.29:0/1860006549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc180103b60 msgr2=0x7fc180075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc180103b60 0x7fc180075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17f7fe640 1 -- 172.21.15.29:0/1860006549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1801a8610 con 0x7fc180109df0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.719+0000 7fc17effd640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc180103b60 0x7fc180075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.720+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180109df0 0x7fc180076190 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc17800cce0 tx=0x7fc17800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.720+0000 7fc167fff640 1 -- 172.21.15.29:0/1860006549 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17800f450 con 0x7fc180109df0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.720+0000 7fc1857df640 1 -- 172.21.15.29:0/1860006549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1801a8900 con 0x7fc180109df0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.720+0000 7fc167fff640 1 -- 172.21.15.29:0/1860006549 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc17800fa80 con 0x7fc180109df0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.720+0000 7fc167fff640 1 -- 172.21.15.29:0/1860006549 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178018ad0 con 0x7fc180109df0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.720+0000 7fc1857df640 1 -- 172.21.15.29:0/1860006549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1801a8df0 con 0x7fc180109df0 2024-09-21T02:38:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc18bd53740 1 Processor -- start 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc167fff640 1 -- 172.21.15.29:0/1860006549 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fc1780213f0 con 0x7fc180109df0 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc18bd53740 1 -- start start 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 0x56258f115f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed470a0 0x56258f116440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f116980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc18bd53740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56258f08dc10 con 0x56258ed4f280 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc18bd53740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56258f07d040 con 0x56258ed470a0 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.723+0000 7fc18bd53740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56258f080e50 con 0x56258ed6ff60 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f116980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f116980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46252/0 (socket says 172.21.15.29:46252) 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 -- 172.21.15.29:0/4177221320 learned_addr learned my addr 172.21.15.29:0/4177221320 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17effd640 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed470a0 0x56258f116440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 0x56258f115f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 -- 172.21.15.29:0/4177221320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 msgr2=0x56258f115f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 0x56258f115f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 -- 172.21.15.29:0/4177221320 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed470a0 msgr2=0x56258f116440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed470a0 0x56258f116440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 -- 172.21.15.29:0/4177221320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56258eddb540 con 0x56258ed4f280 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc167fff640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1680b65a0 0x7fc1680b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f116980 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc1740124c0 tx=0x7fc174005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f865e8f473ae451b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc17effd640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1680b65a0 0x7fc1680b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc167fff640 1 -- 172.21.15.29:0/1860006549 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fc1780e80d0 con 0x7fc180109df0 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc1657fa640 1 -- 172.21.15.29:0/4177221320 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174004220 con 0x56258ed4f280 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc1657fa640 1 -- 172.21.15.29:0/4177221320 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1740043c0 con 0x56258ed4f280 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.724+0000 7fc1657fa640 1 -- 172.21.15.29:0/4177221320 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1740035f0 con 0x56258ed4f280 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 -- 172.21.15.29:0/4177221320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 msgr2=0x56258f116980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc17effd640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1680b65a0 0x7fc1680b8ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc180077160 tx=0x7fc170005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f116980 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc1740124c0 tx=0x7fc174005c90 comp rx=0 tx=0).stop 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 -- 172.21.15.29:0/4177221320 shutdown_connections 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f116980 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed470a0 0x56258f116440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4177221320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 0x56258f115f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 -- 172.21.15.29:0/4177221320 >> 172.21.15.29:0/4177221320 conn(0x56258ed64350 msgr2=0x56258f113df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 -- 172.21.15.29:0/4177221320 shutdown_connections 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.725+0000 7fc18bd53740 1 -- 172.21.15.29:0/4177221320 wait complete. 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 1 -- start start 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 1 Processor -- start 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 0x56258f121aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f121fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed6ff60 0x56258f12d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56258eede5f0 con 0x56258ed4f280 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56258f018010 con 0x56258ed6ff60 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56258f1246e0 con 0x56258ed470a0 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc18bd53740 10 client.0 _ll_register_callbacks cb 0x56258f077340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc17e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f121fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc17e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f121fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46256/0 (socket says 172.21.15.29:46256) 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.726+0000 7fc17e7fc640 1 -- 172.21.15.29:0/2243008364 learned_addr learned my addr 172.21.15.29:0/2243008364 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17effd640 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 0x56258f121aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed6ff60 0x56258f12d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17e7fc640 1 -- 172.21.15.29:0/2243008364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 msgr2=0x56258f121aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 0x56258f121aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17e7fc640 1 -- 172.21.15.29:0/2243008364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed6ff60 msgr2=0x56258f12d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed6ff60 0x56258f12d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17e7fc640 1 -- 172.21.15.29:0/2243008364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56258f045a00 con 0x56258ed4f280 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed6ff60 0x56258f12d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f121fe0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc174004a90 tx=0x7fc174004ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c427f9eb2879f9a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc162ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc162ff5640 1 -- 172.21.15.29:0/2243008364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174005430 con 0x56258ed4f280 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc162ff5640 1 -- 172.21.15.29:0/2243008364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fc1740055d0 con 0x56258ed4f280 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc18bd53740 1 -- 172.21.15.29:0/2243008364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1740043c0 con 0x56258ed4f280 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc162ff5640 1 -- 172.21.15.29:0/2243008364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc174034050 con 0x56258ed4f280 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc18bd53740 10 client.15282 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.727+0000 7fc18bd53740 1 -- 172.21.15.29:0/2243008364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56258f1306e0 con 0x56258ed4f280 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc162ff5640 1 -- 172.21.15.29:0/2243008364 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1740043c0 con 0x56258ed4f280 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc162ff5640 1 -- 172.21.15.29:0/2243008364 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fc17402f030 con 0x56258ed4f280 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc18bd53740 10 client.15282 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc18bd53740 10 client.15282 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:38:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc18bd53740 20 client.15282 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc18bd53740 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56258ed2e8e0 0x56258f133eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc18bd53740 4 client.15282 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc18bd53740 1 -- 172.21.15.29:0/2243008364 --> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56258f08fb60 con 0x56258ed2e8e0 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.728+0000 7fc17effd640 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56258ed2e8e0 0x56258f133eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.729+0000 7fc17effd640 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56258ed2e8e0 0x56258f133eb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c8d2fbbd83d21bc3 server_cookie=56e8fcb058a8bc44 in_seq=0 out_seq=0 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.729+0000 7fc162ff5640 10 client.15282 ms_handle_connect on v2:172.21.15.190:6836/3758896499 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc162ff5640 1 -- 172.21.15.29:0/2243008364 <== mds.2 v2:172.21.15.190:6836/3758896499 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56258f08fb60 con 0x56258ed2e8e0 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc162ff5640 10 client.15282 handle_command_reply: tid=0 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 client.15282 shutdown 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 -- 172.21.15.29:0/2243008364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 msgr2=0x56258f121fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f121fe0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc174004a90 tx=0x7fc174004ea0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 -- 172.21.15.29:0/2243008364 shutdown_connections 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56258ed2e8e0 0x56258f133eb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed6ff60 0x56258f12d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed4f280 0x56258f121fe0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 --2- 172.21.15.29:0/2243008364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 0x56258f121aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc162ff5640 0 client.15282 ms_handle_reset on v2:172.21.15.190:6836/3758896499 2024-09-21T02:38:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 -- 172.21.15.29:0/2243008364 >> 172.21.15.29:0/2243008364 conn(0x56258ed64350 msgr2=0x56258f1190e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 -- 172.21.15.29:0/2243008364 shutdown_connections 2024-09-21T02:38:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 1 -- 172.21.15.29:0/2243008364 wait complete. 2024-09-21T02:38:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.730+0000 7fc18bd53740 20 client.15282 trim_cache size 0 max 16384 2024-09-21T02:38:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.748+0000 7fc18bd53740 1 Processor -- start 2024-09-21T02:38:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc18bd53740 1 -- start start 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed470a0 0x56258f132a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f12d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 0x56258f12de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc18bd53740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56258f1321f0 con 0x56258ed470a0 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc18bd53740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56258f1328c0 con 0x56258ed4f280 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc18bd53740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56258f132f90 con 0x56258ed6ff60 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed470a0 0x56258f132a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed470a0 0x56258f132a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46272/0 (socket says 172.21.15.29:46272) 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 -- 172.21.15.29:0/118463195 learned_addr learned my addr 172.21.15.29:0/118463195 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f12d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 -- 172.21.15.29:0/118463195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 msgr2=0x56258f12de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 0x56258f12de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 0x56258f12de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 -- 172.21.15.29:0/118463195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 msgr2=0x56258f12d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f12d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 -- 172.21.15.29:0/118463195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56258f10e080 con 0x56258ed470a0 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.749+0000 7fc17effd640 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed470a0 0x56258f132a40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc17002a9e0 tx=0x7fc17003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c03928dad1f1df server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc164ff9640 1 -- 172.21.15.29:0/118463195 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170035040 con 0x56258ed470a0 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc164ff9640 1 -- 172.21.15.29:0/118463195 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc170056070 con 0x56258ed470a0 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc164ff9640 1 -- 172.21.15.29:0/118463195 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17005b6d0 con 0x56258ed470a0 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc18bd53740 1 -- 172.21.15.29:0/118463195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed470a0 msgr2=0x56258f132a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc18bd53740 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed470a0 0x56258f132a40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc17002a9e0 tx=0x7fc17003e000 comp rx=0 tx=0).stop 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc18bd53740 1 -- 172.21.15.29:0/118463195 shutdown_connections 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc18bd53740 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed6ff60 0x56258f12de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc18bd53740 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f12d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc18bd53740 1 --2- 172.21.15.29:0/118463195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed470a0 0x56258f132a40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.750+0000 7fc18bd53740 1 -- 172.21.15.29:0/118463195 >> 172.21.15.29:0/118463195 conn(0x56258ed64350 msgr2=0x56258f11c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 -- 172.21.15.29:0/118463195 shutdown_connections 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 -- 172.21.15.29:0/118463195 wait complete. 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 -- start start 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 Processor -- start 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 0x56258f13c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f135ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed6ff60 0x56258f1364d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56258f133d30 con 0x56258ed6ff60 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56258f133660 con 0x56258ed4f280 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56258f134400 con 0x56258ed470a0 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc18bd53740 10 client.0 _ll_register_callbacks cb 0x7fc170009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.751+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed6ff60 0x56258f1364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed6ff60 0x56258f1364d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46276/0 (socket says 172.21.15.29:46276) 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17f7fe640 1 -- 172.21.15.29:0/4017131723 learned_addr learned my addr 172.21.15.29:0/4017131723 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17effd640 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 0x56258f13c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f135ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17f7fe640 1 -- 172.21.15.29:0/4017131723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 msgr2=0x56258f13c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 0x56258f13c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17f7fe640 1 -- 172.21.15.29:0/4017131723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 msgr2=0x56258f135ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f135ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17f7fe640 1 -- 172.21.15.29:0/4017131723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56258f136c30 con 0x56258ed6ff60 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17e7fc640 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f135ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc17f7fe640 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed6ff60 0x56258f1364d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc17800afd0 tx=0x7fc1780ad750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e89b26eaa5350d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc162ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc162ff5640 1 -- 172.21.15.29:0/4017131723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1780b2820 con 0x56258ed6ff60 2024-09-21T02:38:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc162ff5640 1 -- 172.21.15.29:0/4017131723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fc1780b5ba0 con 0x56258ed6ff60 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc18bd53740 1 -- 172.21.15.29:0/4017131723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc170056070 con 0x56258ed6ff60 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc162ff5640 1 -- 172.21.15.29:0/4017131723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc1780b7760 con 0x56258ed6ff60 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc18bd53740 10 client.15288 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.752+0000 7fc18bd53740 1 -- 172.21.15.29:0/4017131723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56258f19f380 con 0x56258ed6ff60 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc162ff5640 1 -- 172.21.15.29:0/4017131723 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc170056070 con 0x56258ed6ff60 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc162ff5640 1 -- 172.21.15.29:0/4017131723 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fc178093bf0 con 0x56258ed6ff60 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc18bd53740 10 client.15288 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc18bd53740 10 client.15288 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc18bd53740 20 client.15288 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56258ed03c90 0x56258f1a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc18bd53740 4 client.15288 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc18bd53740 1 -- 172.21.15.29:0/4017131723 --> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] -- command(tid 0: {"prefix": "status"}) -- 0x56258f08b610 con 0x56258ed03c90 2024-09-21T02:38:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.753+0000 7fc17effd640 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56258ed03c90 0x56258f1a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc162ff5640 10 client.15288 ms_handle_connect on v2:172.21.15.190:6836/3758896499 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc17effd640 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56258ed03c90 0x56258f1a2c20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=247b653eb07ae849 server_cookie=a8b154a381951db in_seq=0 out_seq=0 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc162ff5640 1 -- 172.21.15.29:0/4017131723 <== mds.2 v2:172.21.15.190:6836/3758896499 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56258f08b610 con 0x56258ed03c90 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc162ff5640 10 client.15288 handle_command_reply: tid=0 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 client.15288 shutdown 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 -- 172.21.15.29:0/4017131723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed6ff60 msgr2=0x56258f1364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed6ff60 0x56258f1364d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc17800afd0 tx=0x7fc1780ad750 comp rx=0 tx=0).stop 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 -- 172.21.15.29:0/4017131723 shutdown_connections 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56258ed03c90 0x56258f1a2c20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56258ed6ff60 0x56258f1364d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56258ed4f280 0x56258f135ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc162ff5640 0 client.15288 ms_handle_reset on v2:172.21.15.190:6836/3758896499 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 --2- 172.21.15.29:0/4017131723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56258ed470a0 0x56258f13c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 -- 172.21.15.29:0/4017131723 >> 172.21.15.29:0/4017131723 conn(0x56258ed64350 msgr2=0x56258f11fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 -- 172.21.15.29:0/4017131723 shutdown_connections 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 1 -- 172.21.15.29:0/4017131723 wait complete. 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.754+0000 7fc18bd53740 20 client.15288 trim_cache size 0 max 16384 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24625, 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 81.740049792999997, 2024-09-21T02:38:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-09-21T02:38:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 147.60721447200001 2024-09-21T02:38:28.769 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 -- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1680b65a0 msgr2=0x7fc1680b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1680b65a0 0x7fc1680b8ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc180077160 tx=0x7fc170005a20 comp rx=0 tx=0).stop 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 -- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180109df0 msgr2=0x7fc180076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180109df0 0x7fc180076190 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc17800cce0 tx=0x7fc17800a620 comp rx=0 tx=0).stop 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 -- 172.21.15.29:0/1860006549 shutdown_connections 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1680b65a0 0x7fc1680b8ab0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc180109df0 0x7fc180076190 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1801094a0 0x7fc180075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 --2- 172.21.15.29:0/1860006549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc180103b60 0x7fc180075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 -- 172.21.15.29:0/1860006549 >> 172.21.15.29:0/1860006549 conn(0x7fc18007b850 msgr2=0x7fc180107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 -- 172.21.15.29:0/1860006549 shutdown_connections 2024-09-21T02:38:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:28.755+0000 7fc1857df640 1 -- 172.21.15.29:0/1860006549 wait complete. 2024-09-21T02:38:29.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:29 smithi029 ceph-mon[31006]: pgmap v368: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:29.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:38:29.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:38:29.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:29 smithi066 ceph-mon[33436]: pgmap v368: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:29.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:38:29.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:38:29.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T02:38:29.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:29 smithi190 ceph-mon[33871]: pgmap v368: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:38:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:38:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:38:29.824 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 -- 172.21.15.29:0/249001198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 msgr2=0x7fe364108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 --2- 172.21.15.29:0/249001198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 0x7fe364108a20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe358009900 tx=0x7fe35802a410 comp rx=0 tx=0).stop 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 -- 172.21.15.29:0/249001198 shutdown_connections 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 --2- 172.21.15.29:0/249001198 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe364108f60 0x7fe36410f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 --2- 172.21.15.29:0/249001198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 0x7fe364108a20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 --2- 172.21.15.29:0/249001198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3641031d0 0x7fe3641035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 -- 172.21.15.29:0/249001198 >> 172.21.15.29:0/249001198 conn(0x7fe364079e40 msgr2=0x7fe36407a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 -- 172.21.15.29:0/249001198 shutdown_connections 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.352+0000 7fe369c4e640 1 -- 172.21.15.29:0/249001198 wait complete. 2024-09-21T02:38:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe369c4e640 1 Processor -- start 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe369c4e640 1 -- start start 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe369c4e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3641031d0 0x7fe364110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe369c4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 0x7fe3641107b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe369c4e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe364108f60 0x7fe364110cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe369c4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe364113bf0 con 0x7fe364108610 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe369c4e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe364113a70 con 0x7fe3641031d0 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe369c4e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe364113d70 con 0x7fe364108f60 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.353+0000 7fe362ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 0x7fe3641107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe362ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 0x7fe3641107b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46294/0 (socket says 172.21.15.29:46294) 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe362ffd640 1 -- 172.21.15.29:0/1473461431 learned_addr learned my addr 172.21.15.29:0/1473461431 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe3637fe640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3641031d0 0x7fe364110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe363fff640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe364108f60 0x7fe364110cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe362ffd640 1 -- 172.21.15.29:0/1473461431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe364108f60 msgr2=0x7fe364110cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe362ffd640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe364108f60 0x7fe364110cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe362ffd640 1 -- 172.21.15.29:0/1473461431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3641031d0 msgr2=0x7fe364110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe362ffd640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3641031d0 0x7fe364110270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe362ffd640 1 -- 172.21.15.29:0/1473461431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3641a88e0 con 0x7fe364108610 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe3637fe640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3641031d0 0x7fe364110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe362ffd640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 0x7fe3641107b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe358004770 tx=0x7fe358004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe360ff9640 1 -- 172.21.15.29:0/1473461431 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35803f030 con 0x7fe364108610 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe360ff9640 1 -- 172.21.15.29:0/1473461431 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe358004440 con 0x7fe364108610 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe369c4e640 1 -- 172.21.15.29:0/1473461431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3641a8b70 con 0x7fe364108610 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe360ff9640 1 -- 172.21.15.29:0/1473461431 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358035070 con 0x7fe364108610 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.354+0000 7fe369c4e640 1 -- 172.21.15.29:0/1473461431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3641a9050 con 0x7fe364108610 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.357+0000 7fe3701c2740 1 Processor -- start 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.357+0000 7fe3701c2740 1 -- start start 2024-09-21T02:38:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d3496f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30c80a0 0x5584d3497440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d3497980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe3701c2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5584d340ec10 con 0x5584d30d0280 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe3701c2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5584d33fe040 con 0x5584d30f0f60 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe3701c2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5584d3401e50 con 0x5584d30c80a0 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe360ff9640 1 -- 172.21.15.29:0/1473461431 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fe358042030 con 0x7fe364108610 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe363fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d3496f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe3637fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30c80a0 0x5584d3497440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe363fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d3496f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46104/0 (socket says 172.21.15.29:46104) 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe363fff640 1 -- 172.21.15.29:0/2767460812 learned_addr learned my addr 172.21.15.29:0/2767460812 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe362ffd640 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d3497980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe363fff640 1 -- 172.21.15.29:0/2767460812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30c80a0 msgr2=0x5584d3497440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe363fff640 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30c80a0 0x5584d3497440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe363fff640 1 -- 172.21.15.29:0/2767460812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 msgr2=0x5584d3497980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.358+0000 7fe363fff640 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d3497980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe363fff640 1 -- 172.21.15.29:0/2767460812 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584d315c540 con 0x5584d30f0f60 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe362ffd640 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d3497980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe363fff640 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d3496f00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009e70 tx=0x7fe35c00e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1be7c59e2b1f9998 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe360ff9640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe34c0b65a0 0x7fe34c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe349ffb640 1 -- 172.21.15.29:0/2767460812 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c016070 con 0x5584d30f0f60 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe349ffb640 1 -- 172.21.15.29:0/2767460812 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe35c00ee90 con 0x5584d30f0f60 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe349ffb640 1 -- 172.21.15.29:0/2767460812 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c011670 con 0x5584d30f0f60 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe3637fe640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe34c0b65a0 0x7fe34c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.359+0000 7fe360ff9640 1 -- 172.21.15.29:0/1473461431 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fe358107c90 con 0x7fe364108610 2024-09-21T02:38:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 -- 172.21.15.29:0/2767460812 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 msgr2=0x5584d3496f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3637fe640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe34c0b65a0 0x7fe34c0b8ab0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5584d3495b80 tx=0x7fe354005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:30.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d3496f00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009e70 tx=0x7fe35c00e6c0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 -- 172.21.15.29:0/2767460812 shutdown_connections 2024-09-21T02:38:30.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d3497980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30c80a0 0x5584d3497440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 --2- 172.21.15.29:0/2767460812 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d3496f00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 -- 172.21.15.29:0/2767460812 >> 172.21.15.29:0/2767460812 conn(0x5584d30e5350 msgr2=0x5584d3494df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 -- 172.21.15.29:0/2767460812 shutdown_connections 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 -- 172.21.15.29:0/2767460812 wait complete. 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.360+0000 7fe3701c2740 1 -- start start 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3701c2740 1 Processor -- start 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 0x5584d34a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d34a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30f0f60 0x5584d34aea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3701c2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5584d325f5f0 con 0x5584d30d0280 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3701c2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5584d3399010 con 0x5584d30c80a0 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3701c2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5584d34a56e0 con 0x5584d30f0f60 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3701c2740 10 client.0 _ll_register_callbacks cb 0x5584d33f8340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe362ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d34a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe362ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d34a2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46320/0 (socket says 172.21.15.29:46320) 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe362ffd640 1 -- 172.21.15.29:0/481360667 learned_addr learned my addr 172.21.15.29:0/481360667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe3637fe640 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 0x5584d34a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe363fff640 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30f0f60 0x5584d34aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe362ffd640 1 -- 172.21.15.29:0/481360667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30f0f60 msgr2=0x5584d34aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe362ffd640 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30f0f60 0x5584d34aea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe362ffd640 1 -- 172.21.15.29:0/481360667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 msgr2=0x5584d34a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe362ffd640 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 0x5584d34a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe362ffd640 1 -- 172.21.15.29:0/481360667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584d33c6a00 con 0x5584d30d0280 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.361+0000 7fe363fff640 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30f0f60 0x5584d34aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe362ffd640 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d34a2ec0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe358005940 tx=0x7fe35802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92560df25632a36 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe342ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe342ffd640 1 -- 172.21.15.29:0/481360667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358049070 con 0x5584d30d0280 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe342ffd640 1 -- 172.21.15.29:0/481360667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fe3580d70f0 con 0x5584d30d0280 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe3701c2740 1 -- 172.21.15.29:0/481360667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe35c00ee90 con 0x5584d30d0280 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe342ffd640 1 -- 172.21.15.29:0/481360667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3580cd730 con 0x5584d30d0280 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe3701c2740 10 client.15300 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe3701c2740 1 -- 172.21.15.29:0/481360667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5584d34b15e0 con 0x5584d30d0280 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe342ffd640 1 -- 172.21.15.29:0/481360667 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe35c00ee90 con 0x5584d30d0280 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe342ffd640 1 -- 172.21.15.29:0/481360667 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fe3580d2090 con 0x5584d30d0280 2024-09-21T02:38:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.362+0000 7fe3701c2740 10 client.15300 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.363+0000 7fe3701c2740 10 client.15300 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.363+0000 7fe3701c2740 20 client.15300 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.363+0000 7fe3701c2740 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x5584d30af8e0 0x5584d34b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.363+0000 7fe3701c2740 4 client.15300 mds_command: new command op to 24632 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.363+0000 7fe3701c2740 1 -- 172.21.15.29:0/481360667 --> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584d3410b60 con 0x5584d30af8e0 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.363+0000 7fe3637fe640 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x5584d30af8e0 0x5584d34b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.363+0000 7fe3637fe640 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x5584d30af8e0 0x5584d34b5010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d7d689712c6d6bb server_cookie=a1b4e875d1420586 in_seq=0 out_seq=0 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.363+0000 7fe342ffd640 10 client.15300 ms_handle_connect on v2:172.21.15.66:6838/3618856137 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.364+0000 7fe342ffd640 1 -- 172.21.15.29:0/481360667 <== mds.1 v2:172.21.15.66:6838/3618856137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5584d3410b60 con 0x5584d30af8e0 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.364+0000 7fe342ffd640 10 client.15300 handle_command_reply: tid=0 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.364+0000 7fe3701c2740 1 client.15300 shutdown 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.364+0000 7fe3701c2740 1 -- 172.21.15.29:0/481360667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 msgr2=0x5584d34a2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.364+0000 7fe3701c2740 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d34a2ec0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe358005940 tx=0x7fe35802a410 comp rx=0 tx=0).stop 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 1 -- 172.21.15.29:0/481360667 shutdown_connections 2024-09-21T02:38:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x5584d30af8e0 0x5584d34b5010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30f0f60 0x5584d34aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30d0280 0x5584d34a2ec0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe342ffd640 0 client.15300 ms_handle_reset on v2:172.21.15.66:6838/3618856137 2024-09-21T02:38:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 1 --2- 172.21.15.29:0/481360667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 0x5584d34a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 1 -- 172.21.15.29:0/481360667 >> 172.21.15.29:0/481360667 conn(0x5584d30e5350 msgr2=0x5584d349a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 1 -- 172.21.15.29:0/481360667 shutdown_connections 2024-09-21T02:38:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 1 -- 172.21.15.29:0/481360667 wait complete. 2024-09-21T02:38:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.365+0000 7fe3701c2740 20 client.15300 trim_cache size 0 max 16384 2024-09-21T02:38:30.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3701c2740 1 Processor -- start 2024-09-21T02:38:30.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3701c2740 1 -- start start 2024-09-21T02:38:30.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 0x5584d34a1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34a1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30f0f60 0x5584d3494730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3701c2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5584d34b3350 con 0x5584d30f0f60 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3701c2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5584d34b3a20 con 0x5584d30c80a0 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3701c2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5584d34b40f0 con 0x5584d30d0280 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe363fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30f0f60 0x5584d3494730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe363fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30f0f60 0x5584d3494730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46336/0 (socket says 172.21.15.29:46336) 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe363fff640 1 -- 172.21.15.29:0/2271354608 learned_addr learned my addr 172.21.15.29:0/2271354608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe3637fe640 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 0x5584d34a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.383+0000 7fe362ffd640 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe363fff640 1 -- 172.21.15.29:0/2271354608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 msgr2=0x5584d34a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe363fff640 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe363fff640 1 -- 172.21.15.29:0/2271354608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 msgr2=0x5584d34a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe363fff640 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 0x5584d34a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe363fff640 1 -- 172.21.15.29:0/2271354608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584d3499420 con 0x5584d30f0f60 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe362ffd640 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe363fff640 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30f0f60 0x5584d3494730 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009e70 tx=0x7fe35c005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da155257735003fb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe3497fa640 1 -- 172.21.15.29:0/2271354608 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c02a070 con 0x5584d30f0f60 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe3497fa640 1 -- 172.21.15.29:0/2271354608 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe35c011900 con 0x5584d30f0f60 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.384+0000 7fe3497fa640 1 -- 172.21.15.29:0/2271354608 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c016040 con 0x5584d30f0f60 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 -- 172.21.15.29:0/2271354608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30f0f60 msgr2=0x5584d3494730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30f0f60 0x5584d3494730 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009e70 tx=0x7fe35c005c60 comp rx=0 tx=0).stop 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 -- 172.21.15.29:0/2271354608 shutdown_connections 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30f0f60 0x5584d3494730 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 --2- 172.21.15.29:0/2271354608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30c80a0 0x5584d34a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 -- 172.21.15.29:0/2271354608 >> 172.21.15.29:0/2271354608 conn(0x5584d30e5350 msgr2=0x5584d349d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 -- 172.21.15.29:0/2271354608 shutdown_connections 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 -- 172.21.15.29:0/2271354608 wait complete. 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.385+0000 7fe3701c2740 1 -- start start 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3701c2740 1 Processor -- start 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30c80a0 0x5584d34baa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34baf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3701c2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d34bb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3701c2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5584d34b4e90 con 0x5584d30c80a0 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3701c2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5584d34b47c0 con 0x5584d30f0f60 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3701c2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5584d34b5560 con 0x5584d30d0280 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3701c2740 10 client.0 _ll_register_callbacks cb 0x7fe358005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30c80a0 0x5584d34baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30c80a0 0x5584d34baa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46342/0 (socket says 172.21.15.29:46342) 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3637fe640 1 -- 172.21.15.29:0/279596855 learned_addr learned my addr 172.21.15.29:0/279596855 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe363fff640 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d34bb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe362ffd640 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34baf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3637fe640 1 -- 172.21.15.29:0/279596855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 msgr2=0x5584d34baf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3637fe640 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34baf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3637fe640 1 -- 172.21.15.29:0/279596855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 msgr2=0x5584d34bb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3637fe640 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d34bb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.386+0000 7fe3637fe640 1 -- 172.21.15.29:0/279596855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584d34b9b50 con 0x5584d30c80a0 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe3637fe640 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30c80a0 0x5584d34baa20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe354009b80 tx=0x7fe354015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6ea2cad54c6247 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe342ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe342ffd640 1 -- 172.21.15.29:0/279596855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe354004220 con 0x5584d30c80a0 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe342ffd640 1 -- 172.21.15.29:0/279596855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fe3540043c0 con 0x5584d30c80a0 2024-09-21T02:38:30.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe3701c2740 1 -- 172.21.15.29:0/279596855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe35c011900 con 0x5584d30c80a0 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe342ffd640 1 -- 172.21.15.29:0/279596855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe35403a050 con 0x5584d30c80a0 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe3701c2740 10 client.15306 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe3701c2740 1 -- 172.21.15.29:0/279596855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5584d3520930 con 0x5584d30c80a0 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe342ffd640 1 -- 172.21.15.29:0/279596855 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe35c011900 con 0x5584d30c80a0 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.387+0000 7fe342ffd640 1 -- 172.21.15.29:0/279596855 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fe35400f040 con 0x5584d30c80a0 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe3701c2740 10 client.15306 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe3701c2740 10 client.15306 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe3701c2740 20 client.15306 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe3701c2740 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x5584d3084c90 0x5584d3524250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe3701c2740 4 client.15306 mds_command: new command op to 24632 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe3701c2740 1 -- 172.21.15.29:0/279596855 --> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] -- command(tid 0: {"prefix": "status"}) -- 0x5584d340c610 con 0x5584d3084c90 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe362ffd640 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x5584d3084c90 0x5584d3524250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe342ffd640 10 client.15306 ms_handle_connect on v2:172.21.15.66:6838/3618856137 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.388+0000 7fe362ffd640 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x5584d3084c90 0x5584d3524250 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c4e30f451a8fb84 server_cookie=cb7552c359475833 in_seq=0 out_seq=0 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe342ffd640 1 -- 172.21.15.29:0/279596855 <== mds.1 v2:172.21.15.66:6838/3618856137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5584d340c610 con 0x5584d3084c90 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe342ffd640 10 client.15306 handle_command_reply: tid=0 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 client.15306 shutdown 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 -- 172.21.15.29:0/279596855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30c80a0 msgr2=0x5584d34baa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30c80a0 0x5584d34baa20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe354009b80 tx=0x7fe354015040 comp rx=0 tx=0).stop 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 -- 172.21.15.29:0/279596855 shutdown_connections 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x5584d3084c90 0x5584d3524250 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5584d30f0f60 0x5584d34bb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5584d30d0280 0x5584d34baf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 --2- 172.21.15.29:0/279596855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5584d30c80a0 0x5584d34baa20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 -- 172.21.15.29:0/279596855 >> 172.21.15.29:0/279596855 conn(0x5584d30e5350 msgr2=0x5584d34946d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe342ffd640 0 client.15306 ms_handle_reset on v2:172.21.15.66:6838/3618856137 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 -- 172.21.15.29:0/279596855 shutdown_connections 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 1 -- 172.21.15.29:0/279596855 wait complete. 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.389+0000 7fe3701c2740 20 client.15306 trim_cache size 0 max 16384 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24632, 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:30.400 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:30.401 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 84.381628766999995, 2024-09-21T02:38:30.401 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:30.401 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-09-21T02:38:30.401 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:30.401 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 134.753146208 2024-09-21T02:38:30.401 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 -- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe34c0b65a0 msgr2=0x7fe34c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe34c0b65a0 0x7fe34c0b8ab0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5584d3495b80 tx=0x7fe354005f90 comp rx=0 tx=0).stop 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 -- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 msgr2=0x7fe3641107b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 0x7fe3641107b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe358004770 tx=0x7fe358004040 comp rx=0 tx=0).stop 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 -- 172.21.15.29:0/1473461431 shutdown_connections 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe34c0b65a0 0x7fe34c0b8ab0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe364108f60 0x7fe364110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe364108610 0x7fe3641107b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 --2- 172.21.15.29:0/1473461431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3641031d0 0x7fe364110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 -- 172.21.15.29:0/1473461431 >> 172.21.15.29:0/1473461431 conn(0x7fe364079e40 msgr2=0x7fe3641046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 -- 172.21.15.29:0/1473461431 shutdown_connections 2024-09-21T02:38:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:30.390+0000 7fe369c4e640 1 -- 172.21.15.29:0/1473461431 wait complete. 2024-09-21T02:38:31.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T02:38:31.357 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:31.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:31 smithi029 ceph-mon[31006]: pgmap v369: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:31.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:31 smithi066 ceph-mon[33436]: pgmap v369: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:31.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:31 smithi190 ceph-mon[33871]: pgmap v369: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:38:32.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 -- 172.21.15.29:0/344999594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74102a70 msgr2=0x7f1f74102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 --2- 172.21.15.29:0/344999594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74102a70 0x7f1f74102e80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009900 tx=0x7f1f6402a410 comp rx=0 tx=0).stop 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 -- 172.21.15.29:0/344999594 shutdown_connections 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 --2- 172.21.15.29:0/344999594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f74108d00 0x7f1f7410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 --2- 172.21.15.29:0/344999594 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1f741083b0 0x7f1f741087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 --2- 172.21.15.29:0/344999594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74102a70 0x7f1f74102e80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 -- 172.21.15.29:0/344999594 >> 172.21.15.29:0/344999594 conn(0x7f1f74079730 msgr2=0x7f1f74079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 -- 172.21.15.29:0/344999594 shutdown_connections 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.032+0000 7f1f79844640 1 -- 172.21.15.29:0/344999594 wait complete. 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f79844640 1 Processor -- start 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f79844640 1 -- start start 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f79844640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1f74102a70 0x7f1f74198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f79844640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741083b0 0x7f1f741994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f79844640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f74108d00 0x7f1f741999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f79844640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1f74112820 con 0x7f1f741083b0 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f79844640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1f741126a0 con 0x7f1f74102a70 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f79844640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1f741129a0 con 0x7f1f74108d00 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741083b0 0x7f1f741994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741083b0 0x7f1f741994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46368/0 (socket says 172.21.15.29:46368) 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f727fc640 1 -- 172.21.15.29:0/3968321550 learned_addr learned my addr 172.21.15.29:0/3968321550 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1f74102a70 0x7f1f74198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.033+0000 7f1f737fe640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f74108d00 0x7f1f741999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f727fc640 1 -- 172.21.15.29:0/3968321550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f74108d00 msgr2=0x7f1f741999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f727fc640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f74108d00 0x7f1f741999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f727fc640 1 -- 172.21.15.29:0/3968321550 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1f74102a70 msgr2=0x7f1f74198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f727fc640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1f74102a70 0x7f1f74198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f727fc640 1 -- 172.21.15.29:0/3968321550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f741a8610 con 0x7f1f741083b0 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f737fe640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f74108d00 0x7f1f741999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f727fc640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741083b0 0x7f1f741994a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800c970 tx=0x7f1f6800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f78842640 1 -- 172.21.15.29:0/3968321550 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f68007bb0 con 0x7f1f741083b0 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f78842640 1 -- 172.21.15.29:0/3968321550 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f68007d50 con 0x7f1f741083b0 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f78842640 1 -- 172.21.15.29:0/3968321550 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f68004e10 con 0x7f1f741083b0 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f79844640 1 -- 172.21.15.29:0/3968321550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f741a8900 con 0x7f1f741083b0 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.034+0000 7f1f79844640 1 -- 172.21.15.29:0/3968321550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f741a8df0 con 0x7f1f741083b0 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.037+0000 7f1f78842640 1 -- 172.21.15.29:0/3968321550 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f1f68005000 con 0x7f1f741083b0 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.037+0000 7f1f7fdb8740 1 Processor -- start 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.037+0000 7f1f7fdb8740 1 -- start start 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.037+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61213f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.037+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61214440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.037+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61214980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bd6118bc10 con 0x55bd60e4d280 2024-09-21T02:38:32.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55bd6117b040 con 0x55bd60e450a0 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bd6117ee50 con 0x55bd60e6df60 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61214980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f737fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61213f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f737fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61213f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:33106/0 (socket says 172.21.15.29:33106) 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f737fe640 1 -- 172.21.15.29:0/1803337290 learned_addr learned my addr 172.21.15.29:0/1803337290 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61214440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f78842640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1f5c0b6620 0x7f1f5c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f737fe640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1f5c0b6620 0x7f1f5c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f727fc640 1 -- 172.21.15.29:0/1803337290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 msgr2=0x55bd61213f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f727fc640 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61213f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f727fc640 1 -- 172.21.15.29:0/1803337290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 msgr2=0x55bd61214440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f727fc640 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61214440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f727fc640 1 -- 172.21.15.29:0/1803337290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd60ed9540 con 0x55bd60e4d280 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.038+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61214440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.039+0000 7f1f78842640 1 -- 172.21.15.29:0/3968321550 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f1f680e7830 con 0x7f1f741083b0 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.039+0000 7f1f727fc640 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61214980 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1f680e4e80 tx=0x7f1f68065f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64405c7f69f168c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.039+0000 7f1f737fe640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1f5c0b6620 0x7f1f5c0b8b30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c014a10 tx=0x7f1f6c004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.039+0000 7f1f59ffb640 1 -- 172.21.15.29:0/1803337290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6801d070 con 0x55bd60e4d280 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.039+0000 7f1f59ffb640 1 -- 172.21.15.29:0/1803337290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f680055b0 con 0x55bd60e4d280 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.039+0000 7f1f59ffb640 1 -- 172.21.15.29:0/1803337290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f680adc00 con 0x55bd60e4d280 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1803337290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 msgr2=0x55bd61214980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61214980 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1f680e4e80 tx=0x7f1f68065f30 comp rx=0 tx=0).stop 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1803337290 shutdown_connections 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61214980 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61214440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1803337290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61213f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1803337290 >> 172.21.15.29:0/1803337290 conn(0x55bd60e62350 msgr2=0x55bd61211df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1803337290 shutdown_connections 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1803337290 wait complete. 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.040+0000 7f1f7fdb8740 1 -- start start 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f7fdb8740 1 Processor -- start 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61228220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61228760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61234080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bd60fdc5f0 con 0x55bd60e4d280 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55bd61116010 con 0x55bd60e450a0 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bd6122ad50 con 0x55bd60e6df60 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f7fdb8740 10 client.0 _ll_register_callbacks cb 0x55bd61175340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61228760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61228760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46380/0 (socket says 172.21.15.29:46380) 2024-09-21T02:38:32.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f727fc640 1 -- 172.21.15.29:0/3276109635 learned_addr learned my addr 172.21.15.29:0/3276109635 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61228220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f737fe640 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61234080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f727fc640 1 -- 172.21.15.29:0/3276109635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 msgr2=0x55bd61234080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f727fc640 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61234080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f727fc640 1 -- 172.21.15.29:0/3276109635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 msgr2=0x55bd61228220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f727fc640 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61228220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f727fc640 1 -- 172.21.15.29:0/3276109635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd61143a00 con 0x55bd60e4d280 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.041+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61228220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f727fc640 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61228760 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1f680a9ad0 tx=0x7f1f6800a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8002771943aa8713 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f52ffd640 1 -- 172.21.15.29:0/3276109635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6801d040 con 0x55bd60e4d280 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f52ffd640 1 -- 172.21.15.29:0/3276109635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f1f680b6c90 con 0x55bd60e4d280 2024-09-21T02:38:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/3276109635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd6118db60 con 0x55bd60e4d280 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f52ffd640 1 -- 172.21.15.29:0/3276109635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1f680acc70 con 0x55bd60e4d280 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f7fdb8740 10 client.15318 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/3276109635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bd61236cf0 con 0x55bd60e4d280 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f52ffd640 1 -- 172.21.15.29:0/3276109635 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55bd6118db60 con 0x55bd60e4d280 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.042+0000 7f1f52ffd640 1 -- 172.21.15.29:0/3276109635 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f1f680a8ee0 con 0x55bd60e4d280 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f7fdb8740 10 client.15318 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f7fdb8740 10 client.15318 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f7fdb8740 20 client.15318 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bd60e2c8e0 0x55bd6123a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f7fdb8740 4 client.15318 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/3276109635 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd61189610 con 0x55bd60e2c8e0 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bd60e2c8e0 0x55bd6123a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f52ffd640 10 client.15318 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.043+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bd60e2c8e0 0x55bd6123a260 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11d027932a6bfd6a server_cookie=a22b6eca1a4217ed in_seq=0 out_seq=0 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f52ffd640 1 -- 172.21.15.29:0/3276109635 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bd61189610 con 0x55bd60e2c8e0 2024-09-21T02:38:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f52ffd640 10 client.15318 handle_command_reply: tid=0 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 client.15318 shutdown 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/3276109635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 msgr2=0x55bd61228760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61228760 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1f680a9ad0 tx=0x7f1f6800a510 comp rx=0 tx=0).stop 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/3276109635 shutdown_connections 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61234080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bd60e2c8e0 0x55bd6123a260 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61228760 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/3276109635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61228220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/3276109635 >> 172.21.15.29:0/3276109635 conn(0x55bd60e62350 msgr2=0x55bd612103b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.044+0000 7f1f52ffd640 0 client.15318 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.045+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/3276109635 shutdown_connections 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.045+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/3276109635 wait complete. 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.045+0000 7f1f7fdb8740 20 client.15318 trim_cache size 0 max 16384 2024-09-21T02:38:32.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f7fdb8740 1 Processor -- start 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f7fdb8740 1 -- start start 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61215a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61233ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61234530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bd6118bc10 con 0x55bd60e4d280 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55bd6122ad50 con 0x55bd60e450a0 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bd61239340 con 0x55bd60e6df60 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61233ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61233ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46388/0 (socket says 172.21.15.29:46388) 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.063+0000 7f1f727fc640 1 -- 172.21.15.29:0/548864895 learned_addr learned my addr 172.21.15.29:0/548864895 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f737fe640 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61234530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61215a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f727fc640 1 -- 172.21.15.29:0/548864895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 msgr2=0x55bd61234530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:32.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f727fc640 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61234530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f727fc640 1 -- 172.21.15.29:0/548864895 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 msgr2=0x55bd61215a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f727fc640 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61215a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f727fc640 1 -- 172.21.15.29:0/548864895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd612292e0 con 0x55bd60e4d280 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61215a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f727fc640 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61233ff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1f680ad200 tx=0x7f1f6800a4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f076531a0edf310 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f597fa640 1 -- 172.21.15.29:0/548864895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6801d070 con 0x55bd60e4d280 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f597fa640 1 -- 172.21.15.29:0/548864895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f68009290 con 0x55bd60e4d280 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.064+0000 7f1f597fa640 1 -- 172.21.15.29:0/548864895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f680ed3d0 con 0x55bd60e4d280 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/548864895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 msgr2=0x55bd61233ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61233ff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1f680ad200 tx=0x7f1f6800a4a0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/548864895 shutdown_connections 2024-09-21T02:38:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd61234530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61233ff0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/548864895 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61215a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/548864895 >> 172.21.15.29:0/548864895 conn(0x55bd60e62350 msgr2=0x55bd612268a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/548864895 shutdown_connections 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/548864895 wait complete. 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.065+0000 7f1f7fdb8740 1 -- start start 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f7fdb8740 1 Processor -- start 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61242910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61242ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f7fdb8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd6123bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bd6123a0e0 con 0x55bd60e4d280 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55bd61239a10 con 0x55bd60e450a0 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f7fdb8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bd6123a7b0 con 0x55bd60e6df60 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f7fdb8740 10 client.0 _ll_register_callbacks cb 0x7f1f64005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61242ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61242ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46396/0 (socket says 172.21.15.29:46396) 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f727fc640 1 -- 172.21.15.29:0/1460835628 learned_addr learned my addr 172.21.15.29:0/1460835628 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61242910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f727fc640 1 -- 172.21.15.29:0/1460835628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 msgr2=0x55bd6123bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.066+0000 7f1f727fc640 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd6123bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f727fc640 1 -- 172.21.15.29:0/1460835628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 msgr2=0x55bd61242910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f727fc640 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61242910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f727fc640 1 -- 172.21.15.29:0/1460835628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd6123c6b0 con 0x55bd60e4d280 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61242910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f727fc640 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61242ef0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1f680a9a80 tx=0x7f1f680a9b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4857a55bc32c958 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f52ffd640 1 -- 172.21.15.29:0/1460835628 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f680c7060 con 0x55bd60e4d280 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f52ffd640 1 -- 172.21.15.29:0/1460835628 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f1f680b6a90 con 0x55bd60e4d280 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1460835628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd61186a40 con 0x55bd60e4d280 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f52ffd640 1 -- 172.21.15.29:0/1460835628 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1f6801d070 con 0x55bd60e4d280 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f7fdb8740 10 client.15324 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1460835628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bd6123c6b0 con 0x55bd60e4d280 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f52ffd640 1 -- 172.21.15.29:0/1460835628 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55bd61186a40 con 0x55bd60e4d280 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f52ffd640 1 -- 172.21.15.29:0/1460835628 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f1f680c8c70 con 0x55bd60e4d280 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.067+0000 7f1f7fdb8740 10 client.15324 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f7fdb8740 10 client.15324 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f7fdb8740 20 client.15324 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bd60e01c90 0x55bd612a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f7fdb8740 4 client.15324 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1460835628 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "status"}) -- 0x55bd610fc950 con 0x55bd60e01c90 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bd60e01c90 0x55bd612a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24637, 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:32.148 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:32.149 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:32.149 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 125.334327157, 2024-09-21T02:38:32.149 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:32.149 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-09-21T02:38:32.149 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:32.149 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 145.774595113 2024-09-21T02:38:32.149 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:32.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bd60e01c90 0x55bd612a84f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2b789a1747ddfa2 server_cookie=550e9ccc90775675 in_seq=0 out_seq=0 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f52ffd640 10 client.15324 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f52ffd640 1 -- 172.21.15.29:0/1460835628 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55bd610fc950 con 0x55bd60e01c90 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.068+0000 7f1f52ffd640 10 client.15324 handle_command_reply: tid=0 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 client.15324 shutdown 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1460835628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 msgr2=0x55bd61242ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61242ef0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1f680a9a80 tx=0x7f1f680a9b50 comp rx=0 tx=0).stop 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1460835628 shutdown_connections 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bd60e6df60 0x55bd6123bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bd60e01c90 0x55bd612a84f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f52ffd640 0 client.15324 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd60e4d280 0x55bd61242ef0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 --2- 172.21.15.29:0/1460835628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bd60e450a0 0x55bd61242910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1460835628 >> 172.21.15.29:0/1460835628 conn(0x55bd60e62350 msgr2=0x55bd612103b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1460835628 shutdown_connections 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 1 -- 172.21.15.29:0/1460835628 wait complete. 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f7fdb8740 20 client.15324 trim_cache size 0 max 16384 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f79844640 1 -- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1f5c0b6620 msgr2=0x7f1f5c0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f79844640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1f5c0b6620 0x7f1f5c0b8b30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c014a10 tx=0x7f1f6c004ab0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f79844640 1 -- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741083b0 msgr2=0x7f1f741994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.069+0000 7f1f79844640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741083b0 0x7f1f741994a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800c970 tx=0x7f1f6800ce30 comp rx=0 tx=0).stop 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.070+0000 7f1f79844640 1 -- 172.21.15.29:0/3968321550 shutdown_connections 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.070+0000 7f1f79844640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1f5c0b6620 0x7f1f5c0b8b30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.070+0000 7f1f79844640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f74108d00 0x7f1f741999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.070+0000 7f1f79844640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741083b0 0x7f1f741994a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.070+0000 7f1f79844640 1 --2- 172.21.15.29:0/3968321550 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1f74102a70 0x7f1f74198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:32.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.070+0000 7f1f79844640 1 -- 172.21.15.29:0/3968321550 >> 172.21.15.29:0/3968321550 conn(0x7f1f74079730 msgr2=0x7f1f741014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:32.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.070+0000 7f1f79844640 1 -- 172.21.15.29:0/3968321550 shutdown_connections 2024-09-21T02:38:32.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:32.070+0000 7f1f79844640 1 -- 172.21.15.29:0/3968321550 wait complete. 2024-09-21T02:38:32.941 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T02:38:32.942 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T02:38:32.957 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T02:38:32.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- bash -c 'ceph orch status' 2024-09-21T02:38:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:33.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:33 smithi066 ceph-mon[33436]: pgmap v370: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:33.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:33 smithi029 ceph-mon[31006]: pgmap v370: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:33.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:33 smithi190 ceph-mon[33871]: pgmap v370: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 -- 172.21.15.29:0/4105881440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c1036d0 msgr2=0x7f851c103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 --2- 172.21.15.29:0/4105881440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c1036d0 0x7f851c103ae0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8510009900 tx=0x7f851002a480 comp rx=0 tx=0).stop 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 -- 172.21.15.29:0/4105881440 shutdown_connections 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 --2- 172.21.15.29:0/4105881440 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f851c104070 0x7f851c10aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 --2- 172.21.15.29:0/4105881440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c1036d0 0x7f851c103ae0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 --2- 172.21.15.29:0/4105881440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851c102640 0x7f851c102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 -- 172.21.15.29:0/4105881440 >> 172.21.15.29:0/4105881440 conn(0x7f851c0fdfb0 msgr2=0x7f851c100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 -- 172.21.15.29:0/4105881440 shutdown_connections 2024-09-21T02:38:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.791+0000 7f85220af640 1 -- 172.21.15.29:0/4105881440 wait complete. 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f85220af640 1 Processor -- start 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f85220af640 1 -- start start 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f85220af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c102640 0x7f851c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f85220af640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851c1036d0 0x7f851c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f85220af640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f851c104070 0x7f851c19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f85220af640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f851c10e120 con 0x7f851c102640 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f85220af640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f851c10dfa0 con 0x7f851c104070 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f85220af640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f851c10e2a0 con 0x7f851c1036d0 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.792+0000 7f851b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c102640 0x7f851c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c102640 0x7f851c199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46420/0 (socket says 172.21.15.29:46420) 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851b7fe640 1 -- 172.21.15.29:0/116282899 learned_addr learned my addr 172.21.15.29:0/116282899 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851bfff640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f851c104070 0x7f851c19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851affd640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851c1036d0 0x7f851c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851b7fe640 1 -- 172.21.15.29:0/116282899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851c1036d0 msgr2=0x7f851c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851b7fe640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851c1036d0 0x7f851c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851b7fe640 1 -- 172.21.15.29:0/116282899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f851c104070 msgr2=0x7f851c19b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851b7fe640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f851c104070 0x7f851c19b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851b7fe640 1 -- 172.21.15.29:0/116282899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851c19b870 con 0x7f851c102640 2024-09-21T02:38:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f851b7fe640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c102640 0x7f851c199ed0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f850c00c920 tx=0x7f850c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f8518ff9640 1 -- 172.21.15.29:0/116282899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c007c70 con 0x7f851c102640 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f8518ff9640 1 -- 172.21.15.29:0/116282899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f850c007e10 con 0x7f851c102640 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f851c19bb60 con 0x7f851c102640 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f8518ff9640 1 -- 172.21.15.29:0/116282899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c018710 con 0x7f851c102640 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.793+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f851c1a22f0 con 0x7f851c102640 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.795+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f851c1a24c0 con 0x7f851c102640 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.796+0000 7f8518ff9640 1 -- 172.21.15.29:0/116282899 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f850c00f450 con 0x7f851c102640 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.797+0000 7f8518ff9640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f85040b65a0 0x7f85040b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.797+0000 7f8518ff9640 1 -- 172.21.15.29:0/116282899 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f850c0e7940 con 0x7f851c102640 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.797+0000 7f851affd640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f85040b65a0 0x7f85040b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.798+0000 7f851affd640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f85040b65a0 0x7f85040b8ab0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f85100057c0 tx=0x7f8510000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:33.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:33.801+0000 7f8518ff9640 1 -- 172.21.15.29:0/116282899 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f850c014030 con 0x7f851c102640 2024-09-21T02:38:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.011+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f851c100030 con 0x7f85040b65a0 2024-09-21T02:38:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.012+0000 7f8518ff9640 1 -- 172.21.15.29:0/116282899 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f851c100030 con 0x7f85040b65a0 2024-09-21T02:38:34.013 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: cephadm 2024-09-21T02:38:34.014 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2024-09-21T02:38:34.014 INFO:teuthology.orchestra.run.smithi029.stdout:Paused: No 2024-09-21T02:38:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f85040b65a0 msgr2=0x7f85040b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f85040b65a0 0x7f85040b8ab0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f85100057c0 tx=0x7f8510000bc0 comp rx=0 tx=0).stop 2024-09-21T02:38:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c102640 msgr2=0x7f851c199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c102640 0x7f851c199ed0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f850c00c920 tx=0x7f850c00cde0 comp rx=0 tx=0).stop 2024-09-21T02:38:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 shutdown_connections 2024-09-21T02:38:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f85040b65a0 0x7f85040b8ab0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f851c104070 0x7f851c19b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851c1036d0 0x7f851c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 --2- 172.21.15.29:0/116282899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f851c102640 0x7f851c199ed0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 >> 172.21.15.29:0/116282899 conn(0x7f851c0fdfb0 msgr2=0x7f851c0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 shutdown_connections 2024-09-21T02:38:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:34.015+0000 7f85220af640 1 -- 172.21.15.29:0/116282899 wait complete. 2024-09-21T02:38:34.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ps' 2024-09-21T02:38:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:35 smithi029 ceph-mon[31006]: pgmap v371: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:35 smithi029 ceph-mon[31006]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:35.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:35 smithi066 ceph-mon[33436]: pgmap v371: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:35 smithi066 ceph-mon[33436]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 -- 172.21.15.29:0/2749474884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 msgr2=0x7f5344102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 --2- 172.21.15.29:0/2749474884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 0x7f5344102af0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5334009900 tx=0x7f533402a480 comp rx=0 tx=0).stop 2024-09-21T02:38:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 -- 172.21.15.29:0/2749474884 shutdown_connections 2024-09-21T02:38:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 --2- 172.21.15.29:0/2749474884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53441040b0 0x7f534410aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 --2- 172.21.15.29:0/2749474884 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5344103710 0x7f5344103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 --2- 172.21.15.29:0/2749474884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 0x7f5344102af0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 -- 172.21.15.29:0/2749474884 >> 172.21.15.29:0/2749474884 conn(0x7f53440fdff0 msgr2=0x7f53441004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 -- 172.21.15.29:0/2749474884 shutdown_connections 2024-09-21T02:38:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 -- 172.21.15.29:0/2749474884 wait complete. 2024-09-21T02:38:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.601+0000 7f534918e640 1 Processor -- start 2024-09-21T02:38:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534918e640 1 -- start start 2024-09-21T02:38:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534918e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 0x7f5344198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534918e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344103710 0x7f53441994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534918e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53441040b0 0x7f534419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534918e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f534410e170 con 0x7f5344102680 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534918e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f534410dff0 con 0x7f53441040b0 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534918e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f534410e2f0 con 0x7f5344103710 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f5342d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 0x7f5344198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f5342d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 0x7f5344198e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46586/0 (socket says 172.21.15.29:46586) 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f5342d9d640 1 -- 172.21.15.29:0/2797600729 learned_addr learned my addr 172.21.15.29:0/2797600729 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534259c640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344103710 0x7f53441994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f534359e640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53441040b0 0x7f534419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f5342d9d640 1 -- 172.21.15.29:0/2797600729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344103710 msgr2=0x7f53441994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f5342d9d640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344103710 0x7f53441994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f5342d9d640 1 -- 172.21.15.29:0/2797600729 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53441040b0 msgr2=0x7f534419a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f5342d9d640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53441040b0 0x7f534419a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.602+0000 7f5342d9d640 1 -- 172.21.15.29:0/2797600729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534419a880 con 0x7f5344102680 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.603+0000 7f5342d9d640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 0x7f5344198e50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5334009230 tx=0x7f53340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.603+0000 7f532bfff640 1 -- 172.21.15.29:0/2797600729 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533403f030 con 0x7f5344102680 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.603+0000 7f532bfff640 1 -- 172.21.15.29:0/2797600729 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5334003c90 con 0x7f5344102680 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.603+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534419ab10 con 0x7f5344102680 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.603+0000 7f532bfff640 1 -- 172.21.15.29:0/2797600729 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5334035070 con 0x7f5344102680 2024-09-21T02:38:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.603+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53441a10a0 con 0x7f5344102680 2024-09-21T02:38:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.604+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5344109860 con 0x7f5344102680 2024-09-21T02:38:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.605+0000 7f532bfff640 1 -- 172.21.15.29:0/2797600729 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f5334042030 con 0x7f5344102680 2024-09-21T02:38:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.606+0000 7f532bfff640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f532c0b65a0 0x7f532c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.606+0000 7f534259c640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f532c0b65a0 0x7f532c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.606+0000 7f532bfff640 1 -- 172.21.15.29:0/2797600729 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f5334108080 con 0x7f5344102680 2024-09-21T02:38:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.607+0000 7f534259c640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f532c0b65a0 0x7f532c0b8ab0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f53380058c0 tx=0x7f533800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.611+0000 7f532bfff640 1 -- 172.21.15.29:0/2797600729 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f53340ce3f0 con 0x7f5344102680 2024-09-21T02:38:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:35 smithi190 ceph-mon[33871]: pgmap v371: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T02:38:35.626 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:35 smithi190 ceph-mon[33871]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:35.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.809+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f5344063800 con 0x7f532c0b65a0 2024-09-21T02:38:35.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.823+0000 7f532bfff640 1 -- 172.21.15.29:0/2797600729 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f5344063800 con 0x7f532c0b65a0 2024-09-21T02:38:35.824 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:38:35.824 INFO:teuthology.orchestra.run.smithi029.stdout:mds.a smithi029 running (2m) 7s ago 2m 86.0M - 19.3.0-5120-g18432218 ddce307e1f32 53953a68aa93 2024-09-21T02:38:35.824 INFO:teuthology.orchestra.run.smithi029.stdout:mds.b smithi066 running (2m) 8s ago 2m 86.8M - 19.3.0-5120-g18432218 ddce307e1f32 5e59c4f1839e 2024-09-21T02:38:35.824 INFO:teuthology.orchestra.run.smithi029.stdout:mds.c smithi190 running (2m) 9s ago 2m 86.2M - 19.3.0-5120-g18432218 ddce307e1f32 783b37f837dc 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.d smithi029 running (2m) 7s ago 2m 82.5M - 19.3.0-5120-g18432218 ddce307e1f32 a77e9981ac91 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.e smithi066 running (2m) 8s ago 2m 88.2M - 19.3.0-5120-g18432218 ddce307e1f32 00e9dcb7646e 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.f smithi190 running (2m) 9s ago 2m 89.3M - 19.3.0-5120-g18432218 ddce307e1f32 a231aab13ca9 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.g smithi029 running (2m) 7s ago 2m 87.8M - 19.3.0-5120-g18432218 ddce307e1f32 23c9bd050707 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.h smithi066 running (2m) 8s ago 2m 86.9M - 19.3.0-5120-g18432218 ddce307e1f32 51d18469cafe 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.i smithi190 running (2m) 9s ago 2m 94.2M - 19.3.0-5120-g18432218 ddce307e1f32 bc6ea4992baf 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.j smithi029 running (2m) 7s ago 2m 83.3M - 19.3.0-5120-g18432218 ddce307e1f32 a110a80bdb15 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.k smithi066 running (2m) 8s ago 2m 85.8M - 19.3.0-5120-g18432218 ddce307e1f32 328ec19fddf2 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mds.l smithi190 running (2m) 9s ago 2m 85.9M - 19.3.0-5120-g18432218 ddce307e1f32 7019d1c01fae 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi029 *:9283,8765 running (10m) 7s ago 10m 593M - 19.3.0-5120-g18432218 ddce307e1f32 d8db2ab5cded 2024-09-21T02:38:35.825 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi066 *:8443,8765 running (8m) 8s ago 8m 524M - 19.3.0-5120-g18432218 ddce307e1f32 cf829af23cc1 2024-09-21T02:38:35.826 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.z smithi190 *:8443,8765 running (8m) 9s ago 8m 524M - 19.3.0-5120-g18432218 ddce307e1f32 a0293db2ee1a 2024-09-21T02:38:35.826 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (10m) 7s ago 10m 123M 2048M 19.3.0-5120-g18432218 ddce307e1f32 d52d85c4a57f 2024-09-21T02:38:35.826 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi066 running (9m) 8s ago 9m 109M 2048M 19.3.0-5120-g18432218 ddce307e1f32 e664fe5b69d1 2024-09-21T02:38:35.826 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi190 running (9m) 9s ago 9m 108M 2048M 19.3.0-5120-g18432218 ddce307e1f32 898283fdb943 2024-09-21T02:38:35.826 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (7m) 7s ago 7m 123M 4273M 19.3.0-5120-g18432218 ddce307e1f32 17d5d9cf7bda 2024-09-21T02:38:35.826 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi066 running (7m) 8s ago 7m 124M 4272M 19.3.0-5120-g18432218 ddce307e1f32 b629e6e8dde2 2024-09-21T02:38:35.826 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi190 running (7m) 9s ago 7m 126M 4272M 19.3.0-5120-g18432218 ddce307e1f32 f6d8c94129b7 2024-09-21T02:38:35.826 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (6m) 7s ago 6m 111M 4273M 19.3.0-5120-g18432218 ddce307e1f32 4f61b525a23b 2024-09-21T02:38:35.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi066 running (6m) 8s ago 6m 112M 4272M 19.3.0-5120-g18432218 ddce307e1f32 709db2b8fb7a 2024-09-21T02:38:35.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi190 running (5m) 9s ago 5m 111M 4272M 19.3.0-5120-g18432218 ddce307e1f32 49aa38da51f2 2024-09-21T02:38:35.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi029 running (5m) 7s ago 5m 110M 4273M 19.3.0-5120-g18432218 ddce307e1f32 9ed1f1be55f7 2024-09-21T02:38:35.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi066 running (4m) 8s ago 4m 125M 4272M 19.3.0-5120-g18432218 ddce307e1f32 a200deb3f211 2024-09-21T02:38:35.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 smithi190 running (4m) 9s ago 4m 113M 4272M 19.3.0-5120-g18432218 ddce307e1f32 4d428e62ab43 2024-09-21T02:38:35.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 smithi029 running (4m) 7s ago 4m 111M 4273M 19.3.0-5120-g18432218 ddce307e1f32 72e6208f77f4 2024-09-21T02:38:35.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 smithi066 running (3m) 8s ago 3m 110M 4272M 19.3.0-5120-g18432218 ddce307e1f32 6f1b78f955d1 2024-09-21T02:38:35.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 smithi190 running (3m) 9s ago 3m 110M 4272M 19.3.0-5120-g18432218 ddce307e1f32 7f999657ec19 2024-09-21T02:38:35.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f532c0b65a0 msgr2=0x7f532c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:35.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f532c0b65a0 0x7f532c0b8ab0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f53380058c0 tx=0x7f533800a4c0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 msgr2=0x7f5344198e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:35.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 0x7f5344198e50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5334009230 tx=0x7f53340042f0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 shutdown_connections 2024-09-21T02:38:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f532c0b65a0 0x7f532c0b8ab0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53441040b0 0x7f534419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344103710 0x7f53441994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 --2- 172.21.15.29:0/2797600729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5344102680 0x7f5344198e50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 >> 172.21.15.29:0/2797600729 conn(0x7f53440fdff0 msgr2=0x7f5344106a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 shutdown_connections 2024-09-21T02:38:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:35.825+0000 7f534918e640 1 -- 172.21.15.29:0/2797600729 wait complete. 2024-09-21T02:38:36.568 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ls' 2024-09-21T02:38:36.833 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:37.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:37 smithi029 ceph-mon[31006]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:37.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:37 smithi029 ceph-mon[31006]: pgmap v372: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.331+0000 7fd675122640 1 -- 172.21.15.29:0/888341087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd670103750 msgr2=0x7fd670103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.331+0000 7fd675122640 1 --2- 172.21.15.29:0/888341087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd670103750 0x7fd670103bc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd660009900 tx=0x7fd66002a480 comp rx=0 tx=0).stop 2024-09-21T02:38:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.332+0000 7fd675122640 1 -- 172.21.15.29:0/888341087 shutdown_connections 2024-09-21T02:38:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.332+0000 7fd675122640 1 --2- 172.21.15.29:0/888341087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd670105180 0x7fd67010bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.332+0000 7fd675122640 1 --2- 172.21.15.29:0/888341087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6701047e0 0x7fd670104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.332+0000 7fd675122640 1 --2- 172.21.15.29:0/888341087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd670103750 0x7fd670103bc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.332+0000 7fd675122640 1 -- 172.21.15.29:0/888341087 >> 172.21.15.29:0/888341087 conn(0x7fd6700ff0c0 msgr2=0x7fd670101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.332+0000 7fd675122640 1 -- 172.21.15.29:0/888341087 shutdown_connections 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.332+0000 7fd675122640 1 -- 172.21.15.29:0/888341087 wait complete. 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.332+0000 7fd675122640 1 Processor -- start 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd675122640 1 -- start start 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd675122640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd670103750 0x7fd670199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd675122640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6701047e0 0x7fd67019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd675122640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd670105180 0x7fd67019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd675122640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd67010f220 con 0x7fd6701047e0 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd675122640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd67010f0a0 con 0x7fd670105180 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd675122640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd67010f3a0 con 0x7fd670103750 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6701047e0 0x7fd67019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6701047e0 0x7fd67019a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46596/0 (socket says 172.21.15.29:46596) 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66e59c640 1 -- 172.21.15.29:0/3089910830 learned_addr learned my addr 172.21.15.29:0/3089910830 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66ed9d640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd670103750 0x7fd670199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66f59e640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd670105180 0x7fd67019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66e59c640 1 -- 172.21.15.29:0/3089910830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd670103750 msgr2=0x7fd670199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66e59c640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd670103750 0x7fd670199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66e59c640 1 -- 172.21.15.29:0/3089910830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd670105180 msgr2=0x7fd67019b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.333+0000 7fd66e59c640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd670105180 0x7fd67019b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.334+0000 7fd66e59c640 1 -- 172.21.15.29:0/3089910830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67019b900 con 0x7fd6701047e0 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.334+0000 7fd66e59c640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6701047e0 0x7fd67019a540 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd65800d950 tx=0x7fd65800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.334+0000 7fd667fff640 1 -- 172.21.15.29:0/3089910830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658017030 con 0x7fd6701047e0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.334+0000 7fd667fff640 1 -- 172.21.15.29:0/3089910830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd658004020 con 0x7fd6701047e0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.334+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67019bbf0 con 0x7fd6701047e0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.334+0000 7fd667fff640 1 -- 172.21.15.29:0/3089910830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6580055d0 con 0x7fd6701047e0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.334+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6701a22f0 con 0x7fd6701047e0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.335+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd67010a930 con 0x7fd6701047e0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.337+0000 7fd667fff640 1 -- 172.21.15.29:0/3089910830 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7fd658009a30 con 0x7fd6701047e0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.338+0000 7fd667fff640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd65c0b65a0 0x7fd65c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.338+0000 7fd66ed9d640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd65c0b65a0 0x7fd65c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.338+0000 7fd667fff640 1 -- 172.21.15.29:0/3089910830 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fd6580e79b0 con 0x7fd6701047e0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.339+0000 7fd66ed9d640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd65c0b65a0 0x7fd65c0b8ab0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd660000c00 tx=0x7fd660005720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.341+0000 7fd667fff640 1 -- 172.21.15.29:0/3089910830 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd6580add80 con 0x7fd6701047e0 2024-09-21T02:38:37.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:37 smithi066 ceph-mon[33436]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:37.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:37 smithi066 ceph-mon[33436]: pgmap v372: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:37.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.542+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fd670063800 con 0x7fd65c0b65a0 2024-09-21T02:38:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.548+0000 7fd667fff640 1 -- 172.21.15.29:0/3089910830 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fd670063800 con 0x7fd65c0b65a0 2024-09-21T02:38:37.549 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-21T02:38:37.549 INFO:teuthology.orchestra.run.smithi029.stdout:mds.all 12/12 10s ago 2m smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi066=b;smithi066=e;smithi066=h;smithi066=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2024-09-21T02:38:37.549 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 3/3 10s ago 9m smithi029=x;smithi066=y;smithi190=z;count:3 2024-09-21T02:38:37.550 INFO:teuthology.orchestra.run.smithi029.stdout:mon 3/3 10s ago 9m smithi029:172.21.15.29=a;smithi066:172.21.15.66=b;smithi190:172.21.15.190=c;count:3 2024-09-21T02:38:37.550 INFO:teuthology.orchestra.run.smithi029.stdout:osd 12 10s ago - 2024-09-21T02:38:37.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd65c0b65a0 msgr2=0x7fd65c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd65c0b65a0 0x7fd65c0b8ab0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd660000c00 tx=0x7fd660005720 comp rx=0 tx=0).stop 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6701047e0 msgr2=0x7fd67019a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6701047e0 0x7fd67019a540 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd65800d950 tx=0x7fd65800de10 comp rx=0 tx=0).stop 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 shutdown_connections 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd65c0b65a0 0x7fd65c0b8ab0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd670105180 0x7fd67019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6701047e0 0x7fd67019a540 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 --2- 172.21.15.29:0/3089910830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd670103750 0x7fd670199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.550+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 >> 172.21.15.29:0/3089910830 conn(0x7fd6700ff0c0 msgr2=0x7fd670107b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.551+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 shutdown_connections 2024-09-21T02:38:37.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:37.551+0000 7fd675122640 1 -- 172.21.15.29:0/3089910830 wait complete. 2024-09-21T02:38:37.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:37 smithi190 ceph-mon[33871]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:37.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:37 smithi190 ceph-mon[33871]: pgmap v372: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:38.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- bash -c 'ceph orch host ls' 2024-09-21T02:38:38.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:38 smithi066 ceph-mon[33436]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:38.570 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:38.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:38 smithi190 ceph-mon[33871]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:38.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:38 smithi029 ceph-mon[31006]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 -- 172.21.15.29:0/929673653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b4104070 msgr2=0x7f28b410aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 --2- 172.21.15.29:0/929673653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b4104070 0x7f28b410aa10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009ca0 tx=0x7f28ac02a950 comp rx=0 tx=0).stop 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 -- 172.21.15.29:0/929673653 shutdown_connections 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 --2- 172.21.15.29:0/929673653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b4104070 0x7f28b410aa10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 --2- 172.21.15.29:0/929673653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28b41036d0 0x7f28b4103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 --2- 172.21.15.29:0/929673653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28b4102640 0x7f28b4102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 -- 172.21.15.29:0/929673653 >> 172.21.15.29:0/929673653 conn(0x7f28b40fdfb0 msgr2=0x7f28b4100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 -- 172.21.15.29:0/929673653 shutdown_connections 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.100+0000 7f28b9047640 1 -- 172.21.15.29:0/929673653 wait complete. 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b9047640 1 Processor -- start 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b9047640 1 -- start start 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b9047640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28b4102640 0x7f28b4198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b9047640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b41036d0 0x7f28b41993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b9047640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28b4104070 0x7f28b4199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:39.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b9047640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28b410e120 con 0x7f28b41036d0 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b9047640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f28b410dfa0 con 0x7f28b4102640 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b9047640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f28b410e2a0 con 0x7f28b4104070 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b41036d0 0x7f28b41993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b41036d0 0x7f28b41993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46614/0 (socket says 172.21.15.29:46614) 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b259c640 1 -- 172.21.15.29:0/715695982 learned_addr learned my addr 172.21.15.29:0/715695982 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b2d9d640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28b4102640 0x7f28b4198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.101+0000 7f28b359e640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28b4104070 0x7f28b4199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f28b259c640 1 -- 172.21.15.29:0/715695982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28b4104070 msgr2=0x7f28b4199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f28b259c640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28b4104070 0x7f28b4199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f28b259c640 1 -- 172.21.15.29:0/715695982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28b4102640 msgr2=0x7f28b4198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f28b259c640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28b4102640 0x7f28b4198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f28b259c640 1 -- 172.21.15.29:0/715695982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b419a6f0 con 0x7f28b41036d0 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f28b259c640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b41036d0 0x7f28b41993c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c890 tx=0x7f28a800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f289bfff640 1 -- 172.21.15.29:0/715695982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a8004220 con 0x7f28b41036d0 2024-09-21T02:38:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b419a9e0 con 0x7f28b41036d0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f289bfff640 1 -- 172.21.15.29:0/715695982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28a8007e40 con 0x7f28b41036d0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f289bfff640 1 -- 172.21.15.29:0/715695982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a80047a0 con 0x7f28b41036d0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.102+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b41a1170 con 0x7f28b41036d0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.103+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b4102ab0 con 0x7f28b41036d0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.105+0000 7f289bfff640 1 -- 172.21.15.29:0/715695982 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 154563+0+0 (secure 0 0 0) 0x7f28a80079b0 con 0x7f28b41036d0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.105+0000 7f289bfff640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f289c0b65a0 0x7f289c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.105+0000 7f289bfff640 1 -- 172.21.15.29:0/715695982 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f28a80e7910 con 0x7f28b41036d0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.105+0000 7f28b2d9d640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f289c0b65a0 0x7f289c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.106+0000 7f28b2d9d640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f289c0b65a0 0x7f289c0b8ab0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f28a400abe0 tx=0x7f28a400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:39.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.109+0000 7f289bfff640 1 -- 172.21.15.29:0/715695982 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f28a8014030 con 0x7f28b41036d0 2024-09-21T02:38:39.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.307+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f28b4063800 con 0x7f289c0b65a0 2024-09-21T02:38:39.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:39 smithi029 ceph-mon[31006]: pgmap v373: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:39.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]} : dispatch 2024-09-21T02:38:39.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.309+0000 7f289bfff640 1 -- 172.21.15.29:0/715695982 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f28b4063800 con 0x7f289c0b65a0 2024-09-21T02:38:39.310 INFO:teuthology.orchestra.run.smithi029.stdout:HOST ADDR LABELS STATUS 2024-09-21T02:38:39.310 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 172.21.15.29 2024-09-21T02:38:39.310 INFO:teuthology.orchestra.run.smithi029.stdout:smithi066 172.21.15.66 2024-09-21T02:38:39.310 INFO:teuthology.orchestra.run.smithi029.stdout:smithi190 172.21.15.190 2024-09-21T02:38:39.310 INFO:teuthology.orchestra.run.smithi029.stdout:3 hosts in cluster 2024-09-21T02:38:39.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f289c0b65a0 msgr2=0x7f289c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:39.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f289c0b65a0 0x7f289c0b8ab0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f28a400abe0 tx=0x7f28a400a2d0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b41036d0 msgr2=0x7f28b41993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b41036d0 0x7f28b41993c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c890 tx=0x7f28a800cc90 comp rx=0 tx=0).stop 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 shutdown_connections 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f289c0b65a0 0x7f289c0b8ab0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28b4104070 0x7f28b4199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b41036d0 0x7f28b41993c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 --2- 172.21.15.29:0/715695982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28b4102640 0x7f28b4198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 >> 172.21.15.29:0/715695982 conn(0x7f28b40fdfb0 msgr2=0x7f28b41069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 shutdown_connections 2024-09-21T02:38:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:39.311+0000 7f28b9047640 1 -- 172.21.15.29:0/715695982 wait complete. 2024-09-21T02:38:39.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:39 smithi066 ceph-mon[33436]: pgmap v373: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:39.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]} : dispatch 2024-09-21T02:38:39.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:39 smithi190 ceph-mon[33871]: pgmap v373: 129 pgs: 129 active+clean; 650 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:39.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]} : dispatch 2024-09-21T02:38:40.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- bash -c 'ceph orch device ls' 2024-09-21T02:38:40.313 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:40.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-09-21T02:38:40.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:40 smithi029 ceph-mon[31006]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T02:38:40.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:40 smithi029 ceph-mon[31006]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:40.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-09-21T02:38:40.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:40 smithi066 ceph-mon[33436]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T02:38:40.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:40 smithi066 ceph-mon[33436]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:40.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-09-21T02:38:40.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:40 smithi190 ceph-mon[33871]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T02:38:40.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:40 smithi190 ceph-mon[33871]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:40.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 -- 172.21.15.29:0/2059590845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6510104070 msgr2=0x7f651010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:40.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 --2- 172.21.15.29:0/2059590845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6510104070 0x7f651010aa10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6508009ca0 tx=0x7f650802a950 comp rx=0 tx=0).stop 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 -- 172.21.15.29:0/2059590845 shutdown_connections 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 --2- 172.21.15.29:0/2059590845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6510104070 0x7f651010aa10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 --2- 172.21.15.29:0/2059590845 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65101036d0 0x7f6510103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 --2- 172.21.15.29:0/2059590845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6510102640 0x7f6510102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 -- 172.21.15.29:0/2059590845 >> 172.21.15.29:0/2059590845 conn(0x7f65100fdfb0 msgr2=0x7f6510100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 -- 172.21.15.29:0/2059590845 shutdown_connections 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.839+0000 7f6517032640 1 -- 172.21.15.29:0/2059590845 wait complete. 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f6517032640 1 Processor -- start 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f6517032640 1 -- start start 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f6517032640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6510102640 0x7f6510198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f6517032640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65101036d0 0x7f65101993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:40.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f6517032640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6510104070 0x7f6510199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f6517032640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f651010e120 con 0x7f65101036d0 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f6517032640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f651010dfa0 con 0x7f6510102640 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f6517032640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f651010e2a0 con 0x7f6510104070 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f650ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65101036d0 0x7f65101993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f650ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65101036d0 0x7f65101993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46618/0 (socket says 172.21.15.29:46618) 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.840+0000 7f650ffff640 1 -- 172.21.15.29:0/3633726828 learned_addr learned my addr 172.21.15.29:0/3633726828 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f65155cf640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6510104070 0x7f6510199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f6514dce640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6510102640 0x7f6510198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650ffff640 1 -- 172.21.15.29:0/3633726828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6510104070 msgr2=0x7f6510199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650ffff640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6510104070 0x7f6510199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650ffff640 1 -- 172.21.15.29:0/3633726828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6510102640 msgr2=0x7f6510198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650ffff640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6510102640 0x7f6510198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650ffff640 1 -- 172.21.15.29:0/3633726828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651019a780 con 0x7f65101036d0 2024-09-21T02:38:40.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650ffff640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65101036d0 0x7f65101993c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f650400d950 tx=0x7f650400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650dffb640 1 -- 172.21.15.29:0/3633726828 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504017070 con 0x7f65101036d0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650dffb640 1 -- 172.21.15.29:0/3633726828 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6504009c30 con 0x7f65101036d0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651019aa70 con 0x7f65101036d0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.841+0000 7f650dffb640 1 -- 172.21.15.29:0/3633726828 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650400f620 con 0x7f65101036d0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.842+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65101a1170 con 0x7f65101036d0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.842+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6510102ab0 con 0x7f65101036d0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.844+0000 7f650dffb640 1 -- 172.21.15.29:0/3633726828 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f6504010460 con 0x7f65101036d0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.845+0000 7f650dffb640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64f80b6510 0x7f64f80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.845+0000 7f6514dce640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64f80b6510 0x7f64f80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.845+0000 7f650dffb640 1 -- 172.21.15.29:0/3633726828 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f65040e7270 con 0x7f65101036d0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.845+0000 7f6514dce640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64f80b6510 0x7f64f80b8a20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65000058c0 tx=0x7f650000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:40.849+0000 7f650dffb640 1 -- 172.21.15.29:0/3633726828 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6504009de0 con 0x7f65101036d0 2024-09-21T02:38:41.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.046+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f6510063800 con 0x7f64f80b6510 2024-09-21T02:38:41.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.048+0000 7f650dffb640 1 -- 172.21.15.29:0/3633726828 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f6510063800 con 0x7f64f80b6510 2024-09-21T02:38:41.049 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-21T02:38:41.049 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-09-21T02:38:41.049 INFO:teuthology.orchestra.run.smithi029.stdout:smithi066 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53430091400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-21T02:38:41.049 INFO:teuthology.orchestra.run.smithi029.stdout:smithi190 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT63210012400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-21T02:38:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64f80b6510 msgr2=0x7f64f80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64f80b6510 0x7f64f80b8a20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65000058c0 tx=0x7f650000a2d0 comp rx=0 tx=0).stop 2024-09-21T02:38:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65101036d0 msgr2=0x7f65101993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65101036d0 0x7f65101993c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f650400d950 tx=0x7f650400de10 comp rx=0 tx=0).stop 2024-09-21T02:38:41.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 shutdown_connections 2024-09-21T02:38:41.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f64f80b6510 0x7f64f80b8a20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:41.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6510104070 0x7f6510199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:41.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65101036d0 0x7f65101993c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:41.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 --2- 172.21.15.29:0/3633726828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6510102640 0x7f6510198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:41.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 >> 172.21.15.29:0/3633726828 conn(0x7f65100fdfb0 msgr2=0x7f65101069d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:41.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 shutdown_connections 2024-09-21T02:38:41.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:41.050+0000 7f6517032640 1 -- 172.21.15.29:0/3633726828 wait complete. 2024-09-21T02:38:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:41 smithi029 ceph-mon[31006]: pgmap v375: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:38:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:41 smithi029 ceph-mon[31006]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T02:38:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:41 smithi029 ceph-mon[31006]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-09-21T02:38:41.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:41 smithi066 ceph-mon[33436]: pgmap v375: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:38:41.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:41 smithi066 ceph-mon[33436]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T02:38:41.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:41 smithi066 ceph-mon[33436]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-09-21T02:38:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:41 smithi190 ceph-mon[33871]: pgmap v375: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:38:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:41 smithi190 ceph-mon[33871]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T02:38:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:41 smithi190 ceph-mon[33871]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-09-21T02:38:41.804 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T02:38:41.813 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T02:38:41.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- bash -c 'ceph fs dump' 2024-09-21T02:38:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:42.498 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:42 smithi029 ceph-mon[31006]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:42.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:42 smithi066 ceph-mon[33436]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:42.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.589+0000 7f201d628640 1 -- 172.21.15.29:0/1680636329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2018109480 msgr2=0x7f201810b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:42.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.589+0000 7f201d628640 1 --2- 172.21.15.29:0/1680636329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2018109480 0x7f201810b990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2010009ca0 tx=0x7f201002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.590+0000 7f201d628640 1 -- 172.21.15.29:0/1680636329 shutdown_connections 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.590+0000 7f201d628640 1 --2- 172.21.15.29:0/1680636329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2018109480 0x7f201810b990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.590+0000 7f201d628640 1 --2- 172.21.15.29:0/1680636329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20181068e0 0x7f2018108d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.590+0000 7f201d628640 1 --2- 172.21.15.29:0/1680636329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f201806c660 0x7f20181063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.590+0000 7f201d628640 1 -- 172.21.15.29:0/1680636329 >> 172.21.15.29:0/1680636329 conn(0x7f20180fdfe0 msgr2=0x7f2018100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.590+0000 7f201d628640 1 -- 172.21.15.29:0/1680636329 shutdown_connections 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.590+0000 7f201d628640 1 -- 172.21.15.29:0/1680636329 wait complete. 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.590+0000 7f201d628640 1 Processor -- start 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f201d628640 1 -- start start 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f201d628640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201806c660 0x7f201819c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f201d628640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20181068e0 0x7f201819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:42.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f201d628640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2018109480 0x7f201819d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f201d628640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f201810f2a0 con 0x7f20181068e0 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f201d628640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f201810f120 con 0x7f2018109480 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f201d628640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f201810f420 con 0x7f201806c660 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20167fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20181068e0 0x7f201819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20167fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20181068e0 0x7f201819c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46634/0 (socket says 172.21.15.29:46634) 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20167fc640 1 -- 172.21.15.29:0/2910940329 learned_addr learned my addr 172.21.15.29:0/2910940329 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f2016ffd640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201806c660 0x7f201819c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20177fe640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2018109480 0x7f201819d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20167fc640 1 -- 172.21.15.29:0/2910940329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201806c660 msgr2=0x7f201819c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20167fc640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201806c660 0x7f201819c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20167fc640 1 -- 172.21.15.29:0/2910940329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2018109480 msgr2=0x7f201819d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20167fc640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2018109480 0x7f201819d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20167fc640 1 -- 172.21.15.29:0/2910940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201819dae0 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.591+0000 7f20177fe640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2018109480 0x7f201819d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.592+0000 7f20167fc640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20181068e0 0x7f201819c720 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f200c00c970 tx=0x7f200c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.592+0000 7f1ffffff640 1 -- 172.21.15.29:0/2910940329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c007bb0 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.592+0000 7f1ffffff640 1 -- 172.21.15.29:0/2910940329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f200c007d50 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.592+0000 7f1ffffff640 1 -- 172.21.15.29:0/2910940329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c004e40 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.592+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201819ddd0 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.592+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20181a44d0 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.593+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20181a46a0 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.594+0000 7f1ffffff640 1 -- 172.21.15.29:0/2910940329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f200c005020 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.596+0000 7f1ffffff640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20000b6590 0x7f20000b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.596+0000 7f2016ffd640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20000b6590 0x7f20000b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.596+0000 7f1ffffff640 1 -- 172.21.15.29:0/2910940329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f200c0e7300 con 0x7f20181068e0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.596+0000 7f2016ffd640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20000b6590 0x7f20000b8aa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f200800bbe0 tx=0x7f200800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:42.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.600+0000 7f1ffffff640 1 -- 172.21.15.29:0/2910940329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f200c014030 con 0x7f20181068e0 2024-09-21T02:38:42.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:42 smithi190 ceph-mon[33871]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:42.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.829+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f20181a4870 con 0x7f20181068e0 2024-09-21T02:38:42.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.829+0000 7f1ffffff640 1 -- 172.21.15.29:0/2910940329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3128 (secure 0 0 0) 0x7f200c0ad810 con 0x7f20181068e0 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:e28 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:btime 2024-09-21T02:37:10:039454+0000 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 28 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-09-21T02:36:26.562695+0000 2024-09-21T02:38:42.831 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-09-21T02:37:10.039429+0000 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 300 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:max_xattr_size 65536 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 0 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 5 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:in 0,1,2,3,4 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=24637,1=24632,2=24625,3=24613,4=24599} 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-09-21T02:38:42.832 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3,4] 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data disabled 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 0 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:qdb_cluster leader: 24637 members: 24637,24632,24613,24625,24599 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.i{0:24637} state up:active seq 7 addr [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.e{0:24587} state up:standby-replay seq 1 addr [v2:172.21.15.66:6834/4067532455,v1:172.21.15.66:6835/4067532455] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b{1:24632} state up:active seq 14 addr [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.l{1:24574} state up:standby-replay seq 1 addr [v2:172.21.15.190:6832/3478087331,v1:172.21.15.190:6833/3478087331] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c{2:24625} state up:active seq 18 addr [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.k{2:24560} state up:standby-replay seq 1 addr [v2:172.21.15.66:6832/2291646688,v1:172.21.15.66:6833/2291646688] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.f{3:24613} state up:active seq 19 addr [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.833 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a{3:14787} state up:standby-replay seq 1 addr [v2:172.21.15.29:6840/1488187270,v1:172.21.15.29:6841/1488187270] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.834 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.h{4:24599} state up:active seq 18 addr [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.834 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.g{4:14781} state up:standby-replay seq 1 addr [v2:172.21.15.29:6838/886485890,v1:172.21.15.29:6839/886485890] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:42.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:42.834 INFO:teuthology.orchestra.run.smithi029.stdout:Standby daemons: 2024-09-21T02:38:42.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:42.834 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.j{-1:14721} state up:standby seq 1 addr [v2:172.21.15.29:6834/502969019,v1:172.21.15.29:6835/502969019] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.834 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d{-1:14733} state up:standby seq 1 addr [v2:172.21.15.29:6836/2096055950,v1:172.21.15.29:6837/2096055950] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20000b6590 msgr2=0x7f20000b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20000b6590 0x7f20000b8aa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f200800bbe0 tx=0x7f200800b390 comp rx=0 tx=0).stop 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20181068e0 msgr2=0x7f201819c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20181068e0 0x7f201819c720 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f200c00c970 tx=0x7f200c00ce30 comp rx=0 tx=0).stop 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 shutdown_connections 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20000b6590 0x7f20000b8aa0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2018109480 0x7f201819d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20181068e0 0x7f201819c720 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 --2- 172.21.15.29:0/2910940329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201806c660 0x7f201819c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:42.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 >> 172.21.15.29:0/2910940329 conn(0x7f20180fdfe0 msgr2=0x7f2018100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:42.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 shutdown_connections 2024-09-21T02:38:42.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:42.832+0000 7f201d628640 1 -- 172.21.15.29:0/2910940329 wait complete. 2024-09-21T02:38:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:43 smithi066 ceph-mon[33436]: pgmap v377: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 127 B/s wr, 8 op/s 2024-09-21T02:38:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:43 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2910940329' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T02:38:43.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:43 smithi190 ceph-mon[33871]: pgmap v377: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 127 B/s wr, 8 op/s 2024-09-21T02:38:43.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:43 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2910940329' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T02:38:43.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:43 smithi029 ceph-mon[31006]: pgmap v377: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 127 B/s wr, 8 op/s 2024-09-21T02:38:43.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:43 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2910940329' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T02:38:43.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- bash -c 'ceph osd dump' 2024-09-21T02:38:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 -- 172.21.15.29:0/3153823542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd2406c640 msgr2=0x7fdd24106430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/3153823542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd2406c640 0x7fdd24106430 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c009900 tx=0x7fdd0c02a410 comp rx=0 tx=0).stop 2024-09-21T02:38:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 -- 172.21.15.29:0/3153823542 shutdown_connections 2024-09-21T02:38:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/3153823542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd24109480 0x7fdd2410b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/3153823542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd24106970 0x7fdd24108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/3153823542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd2406c640 0x7fdd24106430 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 -- 172.21.15.29:0/3153823542 >> 172.21.15.29:0/3153823542 conn(0x7fdd240fdfd0 msgr2=0x7fdd24100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 -- 172.21.15.29:0/3153823542 shutdown_connections 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 -- 172.21.15.29:0/3153823542 wait complete. 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 Processor -- start 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.504+0000 7fdd2af2f640 1 -- start start 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd2af2f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd2406c640 0x7fdd2419c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd2af2f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd24106970 0x7fdd2419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd2af2f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd24109480 0x7fdd2419d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd2af2f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410f230 con 0x7fdd24106970 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd2af2f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410f0b0 con 0x7fdd2406c640 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd2af2f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410f3b0 con 0x7fdd24109480 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd24106970 0x7fdd2419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd24106970 0x7fdd2419c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42532/0 (socket says 172.21.15.29:42532) 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 -- 172.21.15.29:0/1142304636 learned_addr learned my addr 172.21.15.29:0/1142304636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:44.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd294cc640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd24109480 0x7fdd2419d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd28ccb640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd2406c640 0x7fdd2419c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 -- 172.21.15.29:0/1142304636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd24109480 msgr2=0x7fdd2419d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd24109480 0x7fdd2419d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 -- 172.21.15.29:0/1142304636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd2406c640 msgr2=0x7fdd2419c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd2406c640 0x7fdd2419c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 -- 172.21.15.29:0/1142304636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2419db10 con 0x7fdd24106970 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd28ccb640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd2406c640 0x7fdd2419c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd23fff640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd24106970 0x7fdd2419c750 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400db80 tx=0x7fdd1400b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd21ffb640 1 -- 172.21.15.29:0/1142304636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14017070 con 0x7fdd24106970 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd21ffb640 1 -- 172.21.15.29:0/1142304636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd14004020 con 0x7fdd24106970 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd21ffb640 1 -- 172.21.15.29:0/1142304636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1400be60 con 0x7fdd24106970 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.505+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2419de00 con 0x7fdd24106970 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.506+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd241a4500 con 0x7fdd24106970 2024-09-21T02:38:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.507+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd241a46d0 con 0x7fdd24106970 2024-09-21T02:38:44.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.508+0000 7fdd21ffb640 1 -- 172.21.15.29:0/1142304636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7fdd14003b70 con 0x7fdd24106970 2024-09-21T02:38:44.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.509+0000 7fdd21ffb640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd100b6510 0x7fdd100b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:44.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.509+0000 7fdd28ccb640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd100b6510 0x7fdd100b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:44.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.509+0000 7fdd21ffb640 1 -- 172.21.15.29:0/1142304636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdd140e7c00 con 0x7fdd24106970 2024-09-21T02:38:44.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.510+0000 7fdd28ccb640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd100b6510 0x7fdd100b8a20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c005f70 tx=0x7fdd0c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:44.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.512+0000 7fdd21ffb640 1 -- 172.21.15.29:0/1142304636 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdd140ae110 con 0x7fdd24106970 2024-09-21T02:38:44.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.703+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fdd241a48a0 con 0x7fdd24106970 2024-09-21T02:38:44.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.704+0000 7fdd21ffb640 1 -- 172.21.15.29:0/1142304636 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v85) ==== 56+0+5619 (secure 0 0 0) 0x7fdd140b24f0 con 0x7fdd24106970 2024-09-21T02:38:44.705 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 85 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:fsid e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-09-21T02:27:47.925089+0000 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-09-21T02:38:40.213155+0000 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:crush_version 27 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:full_ratio 0.95 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:backfillfull_ratio 0.9 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:nearfull_ratio 0.85 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:require_min_compat_client luminous 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:min_compat_client luminous 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:require_osd_release squid 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:stretch_mode_enabled false 2024-09-21T02:38:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:max_osd 12 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 up in weight 1 up_from 8 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6802/3066733118,v1:172.21.15.29:6803/3066733118] [v2:172.21.15.29:6804/3066733118,v1:172.21.15.29:6805/3066733118] exists,up daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.66:6800/3928353915,v1:172.21.15.66:6801/3928353915] [v2:172.21.15.66:6802/3928353915,v1:172.21.15.66:6803/3928353915] exists,up 53d7d39c-63c1-4b1c-905d-cbbad4c7e516 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 up in weight 1 up_from 18 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/334233995,v1:172.21.15.190:6801/334233995] [v2:172.21.15.190:6802/334233995,v1:172.21.15.190:6803/334233995] exists,up 70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 up in weight 1 up_from 24 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6810/1263922693,v1:172.21.15.29:6811/1263922693] [v2:172.21.15.29:6812/1263922693,v1:172.21.15.29:6813/1263922693] exists,up 2c4edcdb-857c-4f2a-a290-ba400aea7930 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 up in weight 1 up_from 30 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.66:6808/3255192229,v1:172.21.15.66:6809/3255192229] [v2:172.21.15.66:6810/3255192229,v1:172.21.15.66:6811/3255192229] exists,up f3c143be-ba4f-4bd0-b14d-5c09661d58a4 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 up in weight 1 up_from 35 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/4042925750,v1:172.21.15.190:6809/4042925750] [v2:172.21.15.190:6810/4042925750,v1:172.21.15.190:6811/4042925750] exists,up 53d4db37-3682-40a4-ba57-c957525f8a09 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 up in weight 1 up_from 41 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6818/2439227427,v1:172.21.15.29:6819/2439227427] [v2:172.21.15.29:6820/2439227427,v1:172.21.15.29:6821/2439227427] exists,up 22b00286-ed8f-4f2d-954b-c055126544f4 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 up in weight 1 up_from 47 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.66:6816/3852321903,v1:172.21.15.66:6817/3852321903] [v2:172.21.15.66:6818/3852321903,v1:172.21.15.66:6819/3852321903] exists,up 8b2ea8fa-8bdb-4065-9c6b-be4157c46f31 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 up in weight 1 up_from 53 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/2392824828,v1:172.21.15.190:6817/2392824828] [v2:172.21.15.190:6818/2392824828,v1:172.21.15.190:6819/2392824828] exists,up a0995b89-dfe7-4dc5-b927-6d57c8e71d27 2024-09-21T02:38:44.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 up in weight 1 up_from 59 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6826/1846064827,v1:172.21.15.29:6827/1846064827] [v2:172.21.15.29:6828/1846064827,v1:172.21.15.29:6829/1846064827] exists,up 33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 up in weight 1 up_from 64 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.66:6824/2015526133,v1:172.21.15.66:6825/2015526133] [v2:172.21.15.66:6826/2015526133,v1:172.21.15.66:6827/2015526133] exists,up e6234f49-76e4-49a0-9468-5dba3e2f378a 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 up in weight 1 up_from 69 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/2545559018,v1:172.21.15.190:6825/2545559018] [v2:172.21.15.190:6826/2545559018,v1:172.21.15.190:6827/2545559018] exists,up 75c747c5-0647-4baa-b2fa-a9651588604f 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.8 [2,11] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.b [2,4] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.10 [8,11] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.1d [4,8] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.21 [5,9] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.24 [4,9] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.27 [3,9] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 4.3 [5,8] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 4.5 [9,10] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 4.e [9,11] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 4.14 [9,8] 2024-09-21T02:38:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/3488265372 expires 2024-09-22T02:28:37.826591+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/3273694575 expires 2024-09-22T02:28:37.826591+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1830887273 expires 2024-09-22T02:28:10.590358+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/313500774 expires 2024-09-22T02:28:10.590358+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/2882183575 expires 2024-09-22T02:28:37.826591+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/2882183575 expires 2024-09-22T02:28:37.826591+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/3241279793 expires 2024-09-22T02:28:10.590358+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/595746591 expires 2024-09-22T02:28:10.590358+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1927307007 expires 2024-09-22T02:28:37.826591+0000 2024-09-21T02:38:44.709 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/595746591 expires 2024-09-22T02:28:10.590358+0000 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.706+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd100b6510 msgr2=0x7fdd100b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.706+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd100b6510 0x7fdd100b8a20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c005f70 tx=0x7fdd0c000bc0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.706+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd24106970 msgr2=0x7fdd2419c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.706+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd24106970 0x7fdd2419c750 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400db80 tx=0x7fdd1400b630 comp rx=0 tx=0).stop 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.707+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 shutdown_connections 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.707+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd100b6510 0x7fdd100b8a20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.707+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd24109480 0x7fdd2419d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.707+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd24106970 0x7fdd2419c750 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.707+0000 7fdd2af2f640 1 --2- 172.21.15.29:0/1142304636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd2406c640 0x7fdd2419c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.707+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 >> 172.21.15.29:0/1142304636 conn(0x7fdd240fdfd0 msgr2=0x7fdd241003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.707+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 shutdown_connections 2024-09-21T02:38:44.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:44.707+0000 7fdd2af2f640 1 -- 172.21.15.29:0/1142304636 wait complete. 2024-09-21T02:38:45.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:45 smithi029 ceph-mon[31006]: pgmap v378: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:45.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:45 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1142304636' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T02:38:45.437 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-21T02:38:45.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:38:45.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:45 smithi066 ceph-mon[33436]: pgmap v378: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:45.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:45 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1142304636' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T02:38:45.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:45 smithi190 ceph-mon[33871]: pgmap v378: 129 pgs: 129 active+clean; 650 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:45.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:45 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1142304636' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T02:38:45.713 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 -- 172.21.15.29:0/75394243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084103b60 msgr2=0x7f2084103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 --2- 172.21.15.29:0/75394243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084103b60 0x7f2084103f70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2074009900 tx=0x7f207402a410 comp rx=0 tx=0).stop 2024-09-21T02:38:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 -- 172.21.15.29:0/75394243 shutdown_connections 2024-09-21T02:38:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 --2- 172.21.15.29:0/75394243 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2084109df0 0x7f2084110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 --2- 172.21.15.29:0/75394243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20841094a0 0x7f20841098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 --2- 172.21.15.29:0/75394243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084103b60 0x7f2084103f70 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 -- 172.21.15.29:0/75394243 >> 172.21.15.29:0/75394243 conn(0x7f208407b850 msgr2=0x7f208407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 -- 172.21.15.29:0/75394243 shutdown_connections 2024-09-21T02:38:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.236+0000 7f208a09f640 1 -- 172.21.15.29:0/75394243 wait complete. 2024-09-21T02:38:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f208a09f640 1 Processor -- start 2024-09-21T02:38:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f208a09f640 1 -- start start 2024-09-21T02:38:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f208a09f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2084103b60 0x7f208419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f208a09f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f20841094a0 0x7f208419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f208a09f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084109df0 0x7f208419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f208a09f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2084113960 con 0x7f2084109df0 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f208a09f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f20841137e0 con 0x7f20841094a0 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f208a09f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2084113ae0 con 0x7f2084103b60 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f2083fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084109df0 0x7f208419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f2082ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f20841094a0 0x7f208419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f2083fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084109df0 0x7f208419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42546/0 (socket says 172.21.15.29:42546) 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f20837fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2084103b60 0x7f208419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.237+0000 7f2083fff640 1 -- 172.21.15.29:0/3209107877 learned_addr learned my addr 172.21.15.29:0/3209107877 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2083fff640 1 -- 172.21.15.29:0/3209107877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2084103b60 msgr2=0x7f208419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2083fff640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2084103b60 0x7f208419a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2083fff640 1 -- 172.21.15.29:0/3209107877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f20841094a0 msgr2=0x7f208419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2083fff640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f20841094a0 0x7f208419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2083fff640 1 -- 172.21.15.29:0/3209107877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20841a03a0 con 0x7f2084109df0 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2082ffd640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f20841094a0 0x7f208419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2083fff640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084109df0 0x7f208419fc70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f207c00eaa0 tx=0x7f207c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2080ff9640 1 -- 172.21.15.29:0/3209107877 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c00cdb0 con 0x7f2084109df0 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2080ff9640 1 -- 172.21.15.29:0/3209107877 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f207c005020 con 0x7f2084109df0 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f2080ff9640 1 -- 172.21.15.29:0/3209107877 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c0116d0 con 0x7f2084109df0 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.238+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20841a0690 con 0x7f2084109df0 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.239+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20841a6df0 con 0x7f2084109df0 2024-09-21T02:38:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.240+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2084103f70 con 0x7f2084109df0 2024-09-21T02:38:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.242+0000 7f2080ff9640 1 -- 172.21.15.29:0/3209107877 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f207c0051d0 con 0x7f2084109df0 2024-09-21T02:38:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.243+0000 7f2080ff9640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20700b6590 0x7f20700b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.243+0000 7f20837fe640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20700b6590 0x7f20700b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.243+0000 7f2080ff9640 1 -- 172.21.15.29:0/3209107877 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f207c0e7b00 con 0x7f2084109df0 2024-09-21T02:38:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.244+0000 7f20837fe640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20700b6590 0x7f20700b8aa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f20740057c0 tx=0x7f20740058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.246+0000 7f2080ff9640 1 -- 172.21.15.29:0/3209107877 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f207c016030 con 0x7f2084109df0 2024-09-21T02:38:46.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.472+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f208419b560 con 0x7f2084109df0 2024-09-21T02:38:46.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.473+0000 7f2080ff9640 1 -- 172.21.15.29:0/3209107877 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11971 (secure 0 0 0) 0x7f208419b560 con 0x7f2084109df0 2024-09-21T02:38:46.474 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:46.474 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-09-21T02:37:10:039454+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:37:10.039429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:38:46.474 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-09-21T02:38:46.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20700b6590 msgr2=0x7f20700b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:46.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20700b6590 0x7f20700b8aa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f20740057c0 tx=0x7f20740058d0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084109df0 msgr2=0x7f208419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084109df0 0x7f208419fc70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f207c00eaa0 tx=0x7f207c00ef60 comp rx=0 tx=0).stop 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 shutdown_connections 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f20700b6590 0x7f20700b8aa0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2084109df0 0x7f208419fc70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f20841094a0 0x7f208419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 --2- 172.21.15.29:0/3209107877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2084103b60 0x7f208419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 >> 172.21.15.29:0/3209107877 conn(0x7f208407b850 msgr2=0x7f2084107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 shutdown_connections 2024-09-21T02:38:46.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:46.475+0000 7f208a09f640 1 -- 172.21.15.29:0/3209107877 wait complete. 2024-09-21T02:38:47.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:38:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:47.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:47 smithi066 ceph-mon[33436]: pgmap v379: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:47.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:47 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3209107877' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:47.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:47 smithi029 ceph-mon[31006]: pgmap v379: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:47.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:47 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3209107877' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:47.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:47 smithi190 ceph-mon[33871]: pgmap v379: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:47.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:47 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3209107877' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 -- 172.21.15.29:0/668537929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf68104860 msgr2=0x7faf68104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/668537929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf68104860 0x7faf68104c70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faf58009ca0 tx=0x7faf5802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:38:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 -- 172.21.15.29:0/668537929 shutdown_connections 2024-09-21T02:38:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/668537929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf6810de10 0x7faf68110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/668537929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf68105290 0x7faf6810d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/668537929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf68104860 0x7faf68104c70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 -- 172.21.15.29:0/668537929 >> 172.21.15.29:0/668537929 conn(0x7faf68079740 msgr2=0x7faf68079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 -- 172.21.15.29:0/668537929 shutdown_connections 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.979+0000 7faf6d8dd640 1 -- 172.21.15.29:0/668537929 wait complete. 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf6d8dd640 1 Processor -- start 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf6d8dd640 1 -- start start 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf6d8dd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf68104860 0x7faf68102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf6d8dd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf68105290 0x7faf68102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf6d8dd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf6810de10 0x7faf681031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf6d8dd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faf68113990 con 0x7faf6810de10 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf6d8dd640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faf68113810 con 0x7faf68105290 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf6d8dd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faf68113b10 con 0x7faf68104860 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf6810de10 0x7faf681031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf6810de10 0x7faf681031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42566/0 (socket says 172.21.15.29:42566) 2024-09-21T02:38:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf677fe640 1 -- 172.21.15.29:0/995613454 learned_addr learned my addr 172.21.15.29:0/995613454 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf667fc640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf68105290 0x7faf68102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.980+0000 7faf66ffd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf68104860 0x7faf68102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf677fe640 1 -- 172.21.15.29:0/995613454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf68104860 msgr2=0x7faf68102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf677fe640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf68104860 0x7faf68102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf677fe640 1 -- 172.21.15.29:0/995613454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf68105290 msgr2=0x7faf68102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf677fe640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf68105290 0x7faf68102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf677fe640 1 -- 172.21.15.29:0/995613454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf681a86c0 con 0x7faf6810de10 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf677fe640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf6810de10 0x7faf681031f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7faf6000d750 tx=0x7faf6000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf6c8db640 1 -- 172.21.15.29:0/995613454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf60017070 con 0x7faf6810de10 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf6c8db640 1 -- 172.21.15.29:0/995613454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf600054b0 con 0x7faf6810de10 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf681a89b0 con 0x7faf6810de10 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf6c8db640 1 -- 172.21.15.29:0/995613454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf60004b90 con 0x7faf6810de10 2024-09-21T02:38:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.981+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf681a8ea0 con 0x7faf6810de10 2024-09-21T02:38:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.982+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf6806c1e0 con 0x7faf6810de10 2024-09-21T02:38:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.983+0000 7faf6c8db640 1 -- 172.21.15.29:0/995613454 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7faf60004d30 con 0x7faf6810de10 2024-09-21T02:38:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.983+0000 7faf6c8db640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf540b6510 0x7faf540b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.983+0000 7faf66ffd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf540b6510 0x7faf540b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.984+0000 7faf6c8db640 1 -- 172.21.15.29:0/995613454 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faf600e7090 con 0x7faf6810de10 2024-09-21T02:38:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.984+0000 7faf66ffd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf540b6510 0x7faf540b8a20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faf58006010 tx=0x7faf58000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:47.988+0000 7faf6c8db640 1 -- 172.21.15.29:0/995613454 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faf600ad5a0 con 0x7faf6810de10 2024-09-21T02:38:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.218+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf6802e640 con 0x7faf6810de10 2024-09-21T02:38:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.219+0000 7faf6c8db640 1 -- 172.21.15.29:0/995613454 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11971 (secure 0 0 0) 0x7faf600b1980 con 0x7faf6810de10 2024-09-21T02:38:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:48.221 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-09-21T02:37:10:039454+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:37:10.039429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:38:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-09-21T02:38:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf540b6510 msgr2=0x7faf540b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf540b6510 0x7faf540b8a20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faf58006010 tx=0x7faf58000f40 comp rx=0 tx=0).stop 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf6810de10 msgr2=0x7faf681031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf6810de10 0x7faf681031f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7faf6000d750 tx=0x7faf6000dc10 comp rx=0 tx=0).stop 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 shutdown_connections 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf540b6510 0x7faf540b8a20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf6810de10 0x7faf681031f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf68105290 0x7faf68102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 --2- 172.21.15.29:0/995613454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf68104860 0x7faf68102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.221+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 >> 172.21.15.29:0/995613454 conn(0x7faf68079740 msgr2=0x7faf68100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.222+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 shutdown_connections 2024-09-21T02:38:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:48.222+0000 7faf6d8dd640 1 -- 172.21.15.29:0/995613454 wait complete. 2024-09-21T02:38:48.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:48 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/995613454' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:48.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:48 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/995613454' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:48.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:48 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/995613454' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:38:48.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:38:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:49.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:49 smithi066 ceph-mon[33436]: pgmap v380: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:49.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:49 smithi029 ceph-mon[31006]: pgmap v380: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:49.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:49 smithi190 ceph-mon[33871]: pgmap v380: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:49.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.801+0000 7f9920966640 1 -- 172.21.15.29:0/2159640665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c1083c0 msgr2=0x7f991c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.801+0000 7f9920966640 1 --2- 172.21.15.29:0/2159640665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c1083c0 0x7f991c1087d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9910009900 tx=0x7f991002a410 comp rx=0 tx=0).stop 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.802+0000 7f9920966640 1 -- 172.21.15.29:0/2159640665 shutdown_connections 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.802+0000 7f9920966640 1 --2- 172.21.15.29:0/2159640665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c108d10 0x7f991c10f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.802+0000 7f9920966640 1 --2- 172.21.15.29:0/2159640665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c1083c0 0x7f991c1087d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.802+0000 7f9920966640 1 --2- 172.21.15.29:0/2159640665 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f991c102a80 0x7f991c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.802+0000 7f9920966640 1 -- 172.21.15.29:0/2159640665 >> 172.21.15.29:0/2159640665 conn(0x7f991c079740 msgr2=0x7f991c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.802+0000 7f9920966640 1 -- 172.21.15.29:0/2159640665 shutdown_connections 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.802+0000 7f9920966640 1 -- 172.21.15.29:0/2159640665 wait complete. 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9920966640 1 Processor -- start 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9920966640 1 -- start start 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9920966640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c102a80 0x7f991c19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9920966640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f991c1083c0 0x7f991c19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9920966640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c108d10 0x7f991c19fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9920966640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f991c112830 con 0x7f991c108d10 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9920966640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f991c1126b0 con 0x7f991c1083c0 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9920966640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f991c1129b0 con 0x7f991c102a80 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f991ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c108d10 0x7f991c19fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f991ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c108d10 0x7f991c19fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42588/0 (socket says 172.21.15.29:42588) 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f991ad9d640 1 -- 172.21.15.29:0/2129466640 learned_addr learned my addr 172.21.15.29:0/2129466640 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f9919d9b640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f991c1083c0 0x7f991c19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.803+0000 7f991a59c640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c102a80 0x7f991c19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f991ad9d640 1 -- 172.21.15.29:0/2129466640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c102a80 msgr2=0x7f991c19a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f991ad9d640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c102a80 0x7f991c19a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f991ad9d640 1 -- 172.21.15.29:0/2129466640 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f991c1083c0 msgr2=0x7f991c19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f991ad9d640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f991c1083c0 0x7f991c19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f991ad9d640 1 -- 172.21.15.29:0/2129466640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991c1a0460 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f991a59c640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c102a80 0x7f991c19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f991ad9d640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c108d10 0x7f991c19fd30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f991400cad0 tx=0x7f991400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f990b7fe640 1 -- 172.21.15.29:0/2129466640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991400ae20 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f990b7fe640 1 -- 172.21.15.29:0/2129466640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9914005020 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991c1a0750 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f990b7fe640 1 -- 172.21.15.29:0/2129466640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991400f6b0 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.804+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f991c1a6eb0 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.805+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f991c103d00 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.807+0000 7f990b7fe640 1 -- 172.21.15.29:0/2129466640 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f99140051d0 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.808+0000 7f990b7fe640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f99040b6510 0x7f99040b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.808+0000 7f991a59c640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f99040b6510 0x7f99040b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.808+0000 7f990b7fe640 1 -- 172.21.15.29:0/2129466640 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f99140e7990 con 0x7f991c108d10 2024-09-21T02:38:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.808+0000 7f991a59c640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f99040b6510 0x7f99040b8a20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f990c005d00 tx=0x7f990c009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:49.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.811+0000 7f990b7fe640 1 -- 172.21.15.29:0/2129466640 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9914014030 con 0x7f991c108d10 2024-09-21T02:38:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:49.999+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f991c102e90 con 0x7f991c108d10 2024-09-21T02:38:50.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.000+0000 7f990b7fe640 1 -- 172.21.15.29:0/2129466640 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24773 (secure 0 0 0) 0x7f991c102e90 con 0x7f991c108d10 2024-09-21T02:38:50.003 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:38:50.003 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:38:40.213155+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.002+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f99040b6510 msgr2=0x7f99040b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.002+0000 7f9920966640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f99040b6510 0x7f99040b8a20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f990c005d00 tx=0x7f990c009550 comp rx=0 tx=0).stop 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c108d10 msgr2=0x7f991c19fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c108d10 0x7f991c19fd30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f991400cad0 tx=0x7f991400cf90 comp rx=0 tx=0).stop 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 shutdown_connections 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f99040b6510 0x7f99040b8a20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c108d10 0x7f991c19fd30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f991c1083c0 0x7f991c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 --2- 172.21.15.29:0/2129466640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c102a80 0x7f991c19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 >> 172.21.15.29:0/2129466640 conn(0x7f991c079740 msgr2=0x7f991c06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 shutdown_connections 2024-09-21T02:38:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:50.003+0000 7f9920966640 1 -- 172.21.15.29:0/2129466640 wait complete. 2024-09-21T02:38:50.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:50 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2129466640' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:38:50.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:50 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2129466640' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:38:50.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:50 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2129466640' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:38:50.711 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T02:36:26.562695+0000', 'modified': '2024-09-21T02:37:10.039429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24637, 'mds_1': 24632, 'mds_2': 24625, 'mds_3': 24613, 'mds_4': 24599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14781': {'gid': 14781, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6839/886485890', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 886485890}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 886485890}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14787': {'gid': 14787, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6841/1488187270', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6840', 'nonce': 1488187270}, {'type': 'v1', 'addr': '172.21.15.29:6841', 'nonce': 1488187270}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.66:6833/2291646688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6832', 'nonce': 2291646688}, {'type': 'v1', 'addr': '172.21.15.66:6833', 'nonce': 2291646688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24574': {'gid': 24574, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6833/3478087331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6832', 'nonce': 3478087331}, {'type': 'v1', 'addr': '172.21.15.190:6833', 'nonce': 3478087331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.66:6835/4067532455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6834', 'nonce': 4067532455}, {'type': 'v1', 'addr': '172.21.15.66:6835', 'nonce': 4067532455}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24599': {'gid': 24599, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.66:6837/3184195773', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6836', 'nonce': 3184195773}, {'type': 'v1', 'addr': '172.21.15.66:6837', 'nonce': 3184195773}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24613': {'gid': 24613, 'name': 'f', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.190:6835/131555194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 131555194}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 131555194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24625': {'gid': 24625, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.190:6837/3758896499', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 3758896499}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 3758896499}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24632': {'gid': 24632, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.66:6839/3618856137', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6838', 'nonce': 3618856137}, {'type': 'v1', 'addr': '172.21.15.66:6839', 'nonce': 3618856137}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24637': {'gid': 24637, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.190:6839/3083654868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 3083654868}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 3083654868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24637, 'qdb_cluster': [24637, 24632, 24613, 24625, 24599]} 2024-09-21T02:38:50.712 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-21T02:38:50.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.h status 2024-09-21T02:38:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:51.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:51 smithi029 ceph-mon[31006]: pgmap v381: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 -- 172.21.15.29:0/3939094399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e01083c0 msgr2=0x7f03e01087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 --2- 172.21.15.29:0/3939094399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e01083c0 0x7f03e01087d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009900 tx=0x7f03c802a410 comp rx=0 tx=0).stop 2024-09-21T02:38:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 -- 172.21.15.29:0/3939094399 shutdown_connections 2024-09-21T02:38:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 --2- 172.21.15.29:0/3939094399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03e0108d10 0x7f03e010f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 --2- 172.21.15.29:0/3939094399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e01083c0 0x7f03e01087d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 --2- 172.21.15.29:0/3939094399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03e0102a80 0x7f03e0102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 -- 172.21.15.29:0/3939094399 >> 172.21.15.29:0/3939094399 conn(0x7f03e0079740 msgr2=0x7f03e0079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 -- 172.21.15.29:0/3939094399 shutdown_connections 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.481+0000 7f03e69cb640 1 -- 172.21.15.29:0/3939094399 wait complete. 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e69cb640 1 Processor -- start 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e69cb640 1 -- start start 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e69cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e0102a80 0x7f03e01992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e69cb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03e01083c0 0x7f03e0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e69cb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03e0108d10 0x7f03e019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e69cb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03e0112880 con 0x7f03e0102a80 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e69cb640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f03e0112700 con 0x7f03e01083c0 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e69cb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f03e0112a00 con 0x7f03e0108d10 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e0102a80 0x7f03e01992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e0102a80 0x7f03e01992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42612/0 (socket says 172.21.15.29:42612) 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03dffff640 1 -- 172.21.15.29:0/909160009 learned_addr learned my addr 172.21.15.29:0/909160009 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03d7fff640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03e01083c0 0x7f03e0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e4f68640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03e0108d10 0x7f03e019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03dffff640 1 -- 172.21.15.29:0/909160009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03e0108d10 msgr2=0x7f03e019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03dffff640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03e0108d10 0x7f03e019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03dffff640 1 -- 172.21.15.29:0/909160009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03e01083c0 msgr2=0x7f03e0199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03dffff640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03e01083c0 0x7f03e0199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03dffff640 1 -- 172.21.15.29:0/909160009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e019f180 con 0x7f03e0102a80 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03e4f68640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03e0108d10 0x7f03e019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.482+0000 7f03d7fff640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03e01083c0 0x7f03e0199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.483+0000 7f03dffff640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e0102a80 0x7f03e01992d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f03d000d770 tx=0x7f03d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.483+0000 7f03ddffb640 1 -- 172.21.15.29:0/909160009 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0017070 con 0x7f03e0102a80 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.483+0000 7f03e69cb640 1 -- 172.21.15.29:0/909160009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e019f470 con 0x7f03e0102a80 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.483+0000 7f03ddffb640 1 -- 172.21.15.29:0/909160009 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03d00054b0 con 0x7f03e0102a80 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.483+0000 7f03ddffb640 1 -- 172.21.15.29:0/909160009 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0004c60 con 0x7f03e0102a80 2024-09-21T02:38:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.483+0000 7f03e69cb640 1 -- 172.21.15.29:0/909160009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e01a5c90 con 0x7f03e0102a80 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.486+0000 7f03ddffb640 1 -- 172.21.15.29:0/909160009 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f03d0004e00 con 0x7f03e0102a80 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.486+0000 7f03ecf3f740 1 Processor -- start 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.486+0000 7f03ecf3f740 1 -- start start 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287d8f60 0x560228b7ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b00a0 0x560228b7f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b8280 0x560228b7f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560228af6c10 con 0x5602287b00a0 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560228ae6040 con 0x5602287d8f60 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560228ae9e50 con 0x5602287b8280 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b00a0 0x560228b7f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b00a0 0x560228b7f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42616/0 (socket says 172.21.15.29:42616) 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03dffff640 1 -- 172.21.15.29:0/2533263982 learned_addr learned my addr 172.21.15.29:0/2533263982 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03d7fff640 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b8280 0x560228b7f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03e4f68640 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287d8f60 0x560228b7ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03ddffb640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03cc0b6540 0x7f03cc0b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03dffff640 1 -- 172.21.15.29:0/2533263982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b8280 msgr2=0x560228b7f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03dffff640 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b8280 0x560228b7f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03dffff640 1 -- 172.21.15.29:0/2533263982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287d8f60 msgr2=0x560228b7ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03dffff640 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287d8f60 0x560228b7ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03dffff640 1 -- 172.21.15.29:0/2533263982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560228844540 con 0x5602287b00a0 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03e4f68640 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287d8f60 0x560228b7ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03e4f68640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03cc0b6540 0x7f03cc0b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.487+0000 7f03ddffb640 1 -- 172.21.15.29:0/909160009 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f03d00e71a0 con 0x7f03e0102a80 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.488+0000 7f03dffff640 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b00a0 0x560228b7f440 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f03d00a56f0 tx=0x7f03d0022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df3ca33a8f1d1e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.488+0000 7f03d67fc640 1 -- 172.21.15.29:0/2533263982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d00acd20 con 0x5602287b00a0 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.488+0000 7f03d67fc640 1 -- 172.21.15.29:0/2533263982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03d00a5500 con 0x5602287b00a0 2024-09-21T02:38:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.488+0000 7f03d67fc640 1 -- 172.21.15.29:0/2533263982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d00eacd0 con 0x5602287b00a0 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2533263982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b00a0 msgr2=0x560228b7f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03e4f68640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03cc0b6540 0x7f03cc0b8a50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f03d8005fd0 tx=0x7f03d8004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b00a0 0x560228b7f440 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f03d00a56f0 tx=0x7f03d0022000 comp rx=0 tx=0).stop 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2533263982 shutdown_connections 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b8280 0x560228b7f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b00a0 0x560228b7f440 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2533263982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287d8f60 0x560228b7ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2533263982 >> 172.21.15.29:0/2533263982 conn(0x5602287cd350 msgr2=0x560228b7cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2533263982 shutdown_connections 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2533263982 wait complete. 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 -- start start 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.489+0000 7f03ecf3f740 1 Processor -- start 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b8a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b8ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287d8f60 0x560228b969a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5602289475f0 con 0x5602287b8280 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560228a81010 con 0x5602287b00a0 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560228b8d680 con 0x5602287d8f60 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03ecf3f740 10 client.0 _ll_register_callbacks cb 0x560228ae0340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b8ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b8ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42624/0 (socket says 172.21.15.29:42624) 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 -- 172.21.15.29:0/2222528759 learned_addr learned my addr 172.21.15.29:0/2222528759 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03e4f68640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287d8f60 0x560228b969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03dffff640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b8a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 -- 172.21.15.29:0/2222528759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287d8f60 msgr2=0x560228b969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287d8f60 0x560228b969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 -- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 msgr2=0x560228b8a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b8a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 -- 172.21.15.29:0/2222528759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560228aaea00 con 0x5602287b8280 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03e4f68640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287d8f60 0x560228b969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03dffff640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b8a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03d7fff640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b8ad20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f03c8006010 tx=0x7f03c80053b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7a5e8823c19a09 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.490+0000 7f03bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03bf7fe640 1 -- 172.21.15.29:0/2222528759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8035040 con 0x5602287b8280 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03bf7fe640 1 -- 172.21.15.29:0/2222528759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f03c8004ad0 con 0x5602287b8280 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2222528759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560228af8b60 con 0x5602287b8280 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03bf7fe640 1 -- 172.21.15.29:0/2222528759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f03c80055e0 con 0x5602287b8280 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 10 client.15396 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2222528759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560228b99590 con 0x5602287b8280 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03bf7fe640 1 -- 172.21.15.29:0/2222528759 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x560228af8b60 con 0x5602287b8280 2024-09-21T02:38:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03bf7fe640 1 -- 172.21.15.29:0/2222528759 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f03c8045470 con 0x5602287b8280 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 10 client.15396 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 10 client.15396 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 20 client.15396 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x5602287978e0 0x560228b9cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 4 client.15396 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.491+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2222528759 --> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560228af4610 con 0x5602287978e0 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.492+0000 7f03dffff640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x5602287978e0 0x560228b9cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.492+0000 7f03dffff640 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x5602287978e0 0x560228b9cff0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ea34c0c1b1e67ece server_cookie=a99d4129ffd00f50 in_seq=0 out_seq=0 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.492+0000 7f03bf7fe640 10 client.15396 ms_handle_connect on v2:172.21.15.66:6836/3184195773 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03bf7fe640 1 -- 172.21.15.29:0/2222528759 <== mds.4 v2:172.21.15.66:6836/3184195773 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560228af4610 con 0x5602287978e0 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03bf7fe640 10 client.15396 handle_command_reply: tid=0 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 client.15396 shutdown 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2222528759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 msgr2=0x560228b8ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b8ad20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f03c8006010 tx=0x7f03c80053b0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2222528759 shutdown_connections 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x5602287978e0 0x560228b9cff0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287d8f60 0x560228b969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b8ad20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03bf7fe640 0 client.15396 ms_handle_reset on v2:172.21.15.66:6836/3184195773 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/2222528759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b8a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2222528759 >> 172.21.15.29:0/2222528759 conn(0x5602287cd350 msgr2=0x560228b82090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2222528759 shutdown_connections 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 1 -- 172.21.15.29:0/2222528759 wait complete. 2024-09-21T02:38:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.493+0000 7f03ecf3f740 20 client.15396 trim_cache size 0 max 16384 2024-09-21T02:38:51.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:51 smithi066 ceph-mon[33436]: pgmap v381: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:51.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03ecf3f740 1 Processor -- start 2024-09-21T02:38:51.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03ecf3f740 1 -- start start 2024-09-21T02:38:51.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287978e0 0x560228b96910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b96e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b7c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560228b9b330 con 0x5602287b8280 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560228b9ba00 con 0x5602287b00a0 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560228b9c0d0 con 0x5602287978e0 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03e4f68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b7c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03e4f68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b7c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42632/0 (socket says 172.21.15.29:42632) 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03e4f68640 1 -- 172.21.15.29:0/1546754757 learned_addr learned my addr 172.21.15.29:0/1546754757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03d7fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b96e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.512+0000 7f03dffff640 1 --2- 172.21.15.29:0/1546754757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287978e0 0x560228b96910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03e4f68640 1 -- 172.21.15.29:0/1546754757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287978e0 msgr2=0x560228b96910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03e4f68640 1 --2- 172.21.15.29:0/1546754757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287978e0 0x560228b96910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03e4f68640 1 -- 172.21.15.29:0/1546754757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 msgr2=0x560228b96e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03e4f68640 1 --2- 172.21.15.29:0/1546754757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b96e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03e4f68640 1 -- 172.21.15.29:0/1546754757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560228b811f0 con 0x5602287b8280 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03e4f68640 1 --2- 172.21.15.29:0/1546754757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b7c600 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f03d8011760 tx=0x7f03d8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527c3317660d2680 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03d5ffb640 1 -- 172.21.15.29:0/1546754757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d801c870 con 0x5602287b8280 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03d5ffb640 1 -- 172.21.15.29:0/1546754757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03d801a440 con 0x5602287b8280 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.513+0000 7f03d5ffb640 1 -- 172.21.15.29:0/1546754757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8010630 con 0x5602287b8280 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 -- 172.21.15.29:0/1546754757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 msgr2=0x560228b7c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/1546754757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b7c600 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f03d8011760 tx=0x7f03d8012000 comp rx=0 tx=0).stop 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 -- 172.21.15.29:0/1546754757 shutdown_connections 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/1546754757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287b8280 0x560228b7c600 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/1546754757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b00a0 0x560228b96e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/1546754757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287978e0 0x560228b96910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 -- 172.21.15.29:0/1546754757 >> 172.21.15.29:0/1546754757 conn(0x5602287cd350 msgr2=0x560228b89000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 -- 172.21.15.29:0/1546754757 shutdown_connections 2024-09-21T02:38:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 -- 172.21.15.29:0/1546754757 wait complete. 2024-09-21T02:38:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.514+0000 7f03ecf3f740 1 -- start start 2024-09-21T02:38:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03ecf3f740 1 Processor -- start 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287978e0 0x560228ba5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b00a0 0x560228b9f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03ecf3f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b8280 0x560228b9f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560228b9ce70 con 0x5602287978e0 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560228b9c7a0 con 0x5602287b8280 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03ecf3f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560228b9d540 con 0x5602287b00a0 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03ecf3f740 10 client.0 _ll_register_callbacks cb 0x7f03c8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287978e0 0x560228ba5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287978e0 0x560228ba5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42634/0 (socket says 172.21.15.29:42634) 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03dffff640 1 -- 172.21.15.29:0/3277739755 learned_addr learned my addr 172.21.15.29:0/3277739755 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03e4f68640 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b8280 0x560228b9f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03d7fff640 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b00a0 0x560228b9f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03dffff640 1 -- 172.21.15.29:0/3277739755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b00a0 msgr2=0x560228b9f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03dffff640 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b00a0 0x560228b9f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03dffff640 1 -- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b8280 msgr2=0x560228b9f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03dffff640 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b8280 0x560228b9f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03dffff640 1 -- 172.21.15.29:0/3277739755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560228ba4900 con 0x5602287978e0 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.515+0000 7f03e4f68640 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b8280 0x560228b9f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03dffff640 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287978e0 0x560228ba5cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f03d00aa1c0 tx=0x7f03d00d0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13065367bcae832 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03bf7fe640 1 -- 172.21.15.29:0/3277739755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0020040 con 0x5602287978e0 2024-09-21T02:38:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03bf7fe640 1 -- 172.21.15.29:0/3277739755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f03d00a91c0 con 0x5602287978e0 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03ecf3f740 1 -- 172.21.15.29:0/3277739755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d801a440 con 0x5602287978e0 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03bf7fe640 1 -- 172.21.15.29:0/3277739755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f03d00ac6c0 con 0x5602287978e0 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03ecf3f740 10 client.15402 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03ecf3f740 1 -- 172.21.15.29:0/3277739755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560228c08680 con 0x5602287978e0 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03bf7fe640 1 -- 172.21.15.29:0/3277739755 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03d801a440 con 0x5602287978e0 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03bf7fe640 1 -- 172.21.15.29:0/3277739755 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f03d00050f0 con 0x5602287978e0 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.516+0000 7f03ecf3f740 10 client.15402 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.517+0000 7f03ecf3f740 10 client.15402 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.517+0000 7f03ecf3f740 20 client.15402 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.517+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x5602287d8f60 0x560228c0bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.517+0000 7f03ecf3f740 4 client.15402 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.517+0000 7f03ecf3f740 1 -- 172.21.15.29:0/3277739755 --> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] -- command(tid 0: {"prefix": "status"}) -- 0x560228af1a40 con 0x5602287d8f60 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.517+0000 7f03d7fff640 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x5602287d8f60 0x560228c0bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.517+0000 7f03bf7fe640 10 client.15402 ms_handle_connect on v2:172.21.15.66:6836/3184195773 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.517+0000 7f03d7fff640 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x5602287d8f60 0x560228c0bb40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=65e048f2b82ade57 server_cookie=6227cbeebce1428 in_seq=0 out_seq=0 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03bf7fe640 1 -- 172.21.15.29:0/3277739755 <== mds.4 v2:172.21.15.66:6836/3184195773 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560228af1a40 con 0x5602287d8f60 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03bf7fe640 10 client.15402 handle_command_reply: tid=0 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 client.15402 shutdown 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 -- 172.21.15.29:0/3277739755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287978e0 msgr2=0x560228ba5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287978e0 0x560228ba5cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f03d00aa1c0 tx=0x7f03d00d0040 comp rx=0 tx=0).stop 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 -- 172.21.15.29:0/3277739755 shutdown_connections 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x5602287d8f60 0x560228c0bb40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5602287b8280 0x560228b9f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602287b00a0 0x560228b9f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 --2- 172.21.15.29:0/3277739755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602287978e0 0x560228ba5cc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 -- 172.21.15.29:0/3277739755 >> 172.21.15.29:0/3277739755 conn(0x5602287cd350 msgr2=0x560228b7ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03bf7fe640 0 client.15402 ms_handle_reset on v2:172.21.15.66:6836/3184195773 2024-09-21T02:38:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 -- 172.21.15.29:0/3277739755 shutdown_connections 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 1 -- 172.21.15.29:0/3277739755 wait complete. 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.518+0000 7f03ecf3f740 20 client.15402 trim_cache size 0 max 16384 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24599, 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 102.48728813699999, 2024-09-21T02:38:51.525 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:51.526 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:38:51.526 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:51.526 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 167.73858802999999 2024-09-21T02:38:51.526 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 -- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03cc0b6540 msgr2=0x7f03cc0b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03cc0b6540 0x7f03cc0b8a50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f03d8005fd0 tx=0x7f03d8004a40 comp rx=0 tx=0).stop 2024-09-21T02:38:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 -- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e0102a80 msgr2=0x7f03e01992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e0102a80 0x7f03e01992d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f03d000d770 tx=0x7f03d000dc30 comp rx=0 tx=0).stop 2024-09-21T02:38:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 -- 172.21.15.29:0/909160009 shutdown_connections 2024-09-21T02:38:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03cc0b6540 0x7f03cc0b8a50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03e0108d10 0x7f03e019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03e01083c0 0x7f03e0199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 --2- 172.21.15.29:0/909160009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e0102a80 0x7f03e01992d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 -- 172.21.15.29:0/909160009 >> 172.21.15.29:0/909160009 conn(0x7f03e0079740 msgr2=0x7f03e0104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 -- 172.21.15.29:0/909160009 shutdown_connections 2024-09-21T02:38:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:51.519+0000 7f03e69cb640 1 -- 172.21.15.29:0/909160009 wait complete. 2024-09-21T02:38:51.527 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:51 smithi190 ceph-mon[33871]: pgmap v381: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:52.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.f status 2024-09-21T02:38:52.584 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.128+0000 7fd7836db640 1 -- 172.21.15.29:0/3663875421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c103b60 msgr2=0x7fd77c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.128+0000 7fd7836db640 1 --2- 172.21.15.29:0/3663875421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c103b60 0x7fd77c103f70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd778009900 tx=0x7fd77802a410 comp rx=0 tx=0).stop 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.128+0000 7fd7836db640 1 -- 172.21.15.29:0/3663875421 shutdown_connections 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.128+0000 7fd7836db640 1 --2- 172.21.15.29:0/3663875421 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c109df0 0x7fd77c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.128+0000 7fd7836db640 1 --2- 172.21.15.29:0/3663875421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c1094a0 0x7fd77c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.128+0000 7fd7836db640 1 --2- 172.21.15.29:0/3663875421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c103b60 0x7fd77c103f70 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.128+0000 7fd7836db640 1 -- 172.21.15.29:0/3663875421 >> 172.21.15.29:0/3663875421 conn(0x7fd77c07b850 msgr2=0x7fd77c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.129+0000 7fd7836db640 1 -- 172.21.15.29:0/3663875421 shutdown_connections 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.129+0000 7fd7836db640 1 -- 172.21.15.29:0/3663875421 wait complete. 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.129+0000 7fd7836db640 1 Processor -- start 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd7836db640 1 -- start start 2024-09-21T02:38:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd7836db640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103b60 0x7fd77c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd7836db640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c1094a0 0x7fd77c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd7836db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c109df0 0x7fd77c19fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd7836db640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c113a20 con 0x7fd77c109df0 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd7836db640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1138a0 con 0x7fd77c103b60 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd7836db640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c113ba0 con 0x7fd77c1094a0 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781c78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c109df0 0x7fd77c19fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781c78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c109df0 0x7fd77c19fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42656/0 (socket says 172.21.15.29:42656) 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781c78640 1 -- 172.21.15.29:0/3603328870 learned_addr learned my addr 172.21.15.29:0/3603328870 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781477640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103b60 0x7fd77c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd780c76640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c1094a0 0x7fd77c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781c78640 1 -- 172.21.15.29:0/3603328870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c1094a0 msgr2=0x7fd77c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781c78640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c1094a0 0x7fd77c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781c78640 1 -- 172.21.15.29:0/3603328870 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103b60 msgr2=0x7fd77c19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781c78640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103b60 0x7fd77c19a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.130+0000 7fd781c78640 1 -- 172.21.15.29:0/3603328870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c1a04c0 con 0x7fd77c109df0 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.131+0000 7fd780c76640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c1094a0 0x7fd77c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.131+0000 7fd781c78640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c109df0 0x7fd77c19fd90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd77400b970 tx=0x7fd77400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.131+0000 7fd7727fc640 1 -- 172.21.15.29:0/3603328870 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774019070 con 0x7fd77c109df0 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.131+0000 7fd7727fc640 1 -- 172.21.15.29:0/3603328870 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd774014440 con 0x7fd77c109df0 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.131+0000 7fd7836db640 1 -- 172.21.15.29:0/3603328870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c1a07b0 con 0x7fd77c109df0 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.131+0000 7fd7727fc640 1 -- 172.21.15.29:0/3603328870 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7740075d0 con 0x7fd77c109df0 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.131+0000 7fd7836db640 1 -- 172.21.15.29:0/3603328870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c1a6f10 con 0x7fd77c109df0 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd789c4f740 1 Processor -- start 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd7727fc640 1 -- 172.21.15.29:0/3603328870 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7fd77401c070 con 0x7fd77c109df0 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd789c4f740 1 -- start start 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 0x564f4b7d4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7d5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd789c4f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564f4b74cc10 con 0x564f4b4060a0 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd789c4f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564f4b73c040 con 0x564f4b40e280 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd789c4f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564f4b73fe50 con 0x564f4b42ef60 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd781477640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7d5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd781477640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7d5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42668/0 (socket says 172.21.15.29:42668) 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd781477640 1 -- 172.21.15.29:0/1827043661 learned_addr learned my addr 172.21.15.29:0/1827043661 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.135+0000 7fd780c76640 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 0x564f4b7d4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd780c76640 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd781477640 1 -- 172.21.15.29:0/1827043661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 msgr2=0x564f4b7d4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd781477640 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 0x564f4b7d4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd7727fc640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7680b6590 0x7fd7680b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd781477640 1 -- 172.21.15.29:0/1827043661 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 msgr2=0x564f4b7d5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd781477640 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd781477640 1 -- 172.21.15.29:0/1827043661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f4b49a540 con 0x564f4b4060a0 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd780c76640 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd781c78640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7680b6590 0x7fd7680b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd781477640 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7d5440 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd778009900 tx=0x7fd778031d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f633a63198fd2d7c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd7727fc640 1 -- 172.21.15.29:0/3603328870 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd7740e6a70 con 0x7fd77c109df0 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd7677fe640 1 -- 172.21.15.29:0/1827043661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778035070 con 0x564f4b4060a0 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd7677fe640 1 -- 172.21.15.29:0/1827043661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd778040070 con 0x564f4b4060a0 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.136+0000 7fd7677fe640 1 -- 172.21.15.29:0/1827043661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778003b80 con 0x564f4b4060a0 2024-09-21T02:38:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 -- 172.21.15.29:0/1827043661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 msgr2=0x564f4b7d5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd781c78640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7680b6590 0x7fd7680b8aa0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd7740ac8c0 tx=0x7fd7740a11e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7d5440 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd778009900 tx=0x7fd778031d10 comp rx=0 tx=0).stop 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 -- 172.21.15.29:0/1827043661 shutdown_connections 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7d5440 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1827043661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 0x564f4b7d4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 -- 172.21.15.29:0/1827043661 >> 172.21.15.29:0/1827043661 conn(0x564f4b423350 msgr2=0x564f4b7d2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 -- 172.21.15.29:0/1827043661 shutdown_connections 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.137+0000 7fd789c4f740 1 -- 172.21.15.29:0/1827043661 wait complete. 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 1 -- start start 2024-09-21T02:38:53.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 1 Processor -- start 2024-09-21T02:38:53.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 0x564f4b7e4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7e5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564f4b59d5f0 con 0x564f4b40e280 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564f4b6d7010 con 0x564f4b42ef60 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564f4b7e79b0 con 0x564f4b4060a0 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd789c4f740 10 client.0 _ll_register_callbacks cb 0x564f4b736340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd781477640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd781477640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7e5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42684/0 (socket says 172.21.15.29:42684) 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.138+0000 7fd781477640 1 -- 172.21.15.29:0/1193622004 learned_addr learned my addr 172.21.15.29:0/1193622004 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd780c76640 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd780c76640 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 0x564f4b7e4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd781477640 1 -- 172.21.15.29:0/1193622004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 msgr2=0x564f4b7e4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd781477640 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 0x564f4b7e4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd781477640 1 -- 172.21.15.29:0/1193622004 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 msgr2=0x564f4b7f0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd781477640 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7f0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd781477640 1 -- 172.21.15.29:0/1193622004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f4b704a00 con 0x564f4b40e280 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd780c76640 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd781477640 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7e5160 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd77802a930 tx=0x7fd7780314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7d5983d50bc344 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd764ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd764ff9640 1 -- 172.21.15.29:0/1193622004 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778035040 con 0x564f4b40e280 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd764ff9640 1 -- 172.21.15.29:0/1193622004 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd778045460 con 0x564f4b40e280 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd789c4f740 1 -- 172.21.15.29:0/1193622004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd778040070 con 0x564f4b40e280 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.139+0000 7fd764ff9640 1 -- 172.21.15.29:0/1193622004 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd778045660 con 0x564f4b40e280 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd789c4f740 10 client.15414 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd789c4f740 1 -- 172.21.15.29:0/1193622004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f4b7f3880 con 0x564f4b40e280 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd764ff9640 1 -- 172.21.15.29:0/1193622004 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd778040070 con 0x564f4b40e280 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd764ff9640 1 -- 172.21.15.29:0/1193622004 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd77803c900 con 0x564f4b40e280 2024-09-21T02:38:53.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd789c4f740 10 client.15414 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd789c4f740 10 client.15414 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd789c4f740 20 client.15414 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x564f4b3ed8e0 0x564f4b7f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd789c4f740 4 client.15414 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd789c4f740 1 -- 172.21.15.29:0/1193622004 --> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f4b74eb60 con 0x564f4b3ed8e0 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.140+0000 7fd780c76640 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x564f4b3ed8e0 0x564f4b7f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.141+0000 7fd764ff9640 10 client.15414 ms_handle_connect on v2:172.21.15.190:6834/131555194 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.141+0000 7fd780c76640 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x564f4b3ed8e0 0x564f4b7f6e90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=59bac8748493b6d server_cookie=77d3bbc27c4a90e1 in_seq=0 out_seq=0 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd764ff9640 1 -- 172.21.15.29:0/1193622004 <== mds.3 v2:172.21.15.190:6834/131555194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564f4b74eb60 con 0x564f4b3ed8e0 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd764ff9640 10 client.15414 handle_command_reply: tid=0 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 client.15414 shutdown 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 -- 172.21.15.29:0/1193622004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 msgr2=0x564f4b7e5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7e5160 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd77802a930 tx=0x7fd7780314e0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 -- 172.21.15.29:0/1193622004 shutdown_connections 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x564f4b3ed8e0 0x564f4b7f6e90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7e5160 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd764ff9640 0 client.15414 ms_handle_reset on v2:172.21.15.190:6834/131555194 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 --2- 172.21.15.29:0/1193622004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 0x564f4b7e4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 -- 172.21.15.29:0/1193622004 >> 172.21.15.29:0/1193622004 conn(0x564f4b423350 msgr2=0x564f4b7d8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 -- 172.21.15.29:0/1193622004 shutdown_connections 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 1 -- 172.21.15.29:0/1193622004 wait complete. 2024-09-21T02:38:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.142+0000 7fd789c4f740 20 client.15414 trim_cache size 0 max 16384 2024-09-21T02:38:53.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd789c4f740 1 Processor -- start 2024-09-21T02:38:53.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd789c4f740 1 -- start start 2024-09-21T02:38:53.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 0x564f4b7d2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd789c4f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564f4b7f51d0 con 0x564f4b4060a0 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd789c4f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564f4b7f58a0 con 0x564f4b40e280 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd789c4f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564f4b7f5f70 con 0x564f4b42ef60 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd780c76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd780c76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7f0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42688/0 (socket says 172.21.15.29:42688) 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd780c76640 1 -- 172.21.15.29:0/715169456 learned_addr learned my addr 172.21.15.29:0/715169456 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd781477640 1 --2- 172.21.15.29:0/715169456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd780c76640 1 -- 172.21.15.29:0/715169456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 msgr2=0x564f4b7d2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:53.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd780c76640 1 --2- 172.21.15.29:0/715169456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 0x564f4b7d2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd780c76640 1 -- 172.21.15.29:0/715169456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 msgr2=0x564f4b7d2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:53.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd780c76640 1 --2- 172.21.15.29:0/715169456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.161+0000 7fd780c76640 1 -- 172.21.15.29:0/715169456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f4b7e5e40 con 0x564f4b4060a0 2024-09-21T02:38:53.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.162+0000 7fd780c76640 1 --2- 172.21.15.29:0/715169456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7f0ec0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00cf40 tx=0x7fd76c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78a12c9958943ca server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.162+0000 7fd766ffd640 1 -- 172.21.15.29:0/715169456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0155c0 con 0x564f4b4060a0 2024-09-21T02:38:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.162+0000 7fd766ffd640 1 -- 172.21.15.29:0/715169456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd76c015bf0 con 0x564f4b4060a0 2024-09-21T02:38:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.162+0000 7fd766ffd640 1 -- 172.21.15.29:0/715169456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c026b50 con 0x564f4b4060a0 2024-09-21T02:38:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 -- 172.21.15.29:0/715169456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 msgr2=0x564f4b7f0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 --2- 172.21.15.29:0/715169456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7f0ec0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00cf40 tx=0x7fd76c0047d0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 -- 172.21.15.29:0/715169456 shutdown_connections 2024-09-21T02:38:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 --2- 172.21.15.29:0/715169456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b42ef60 0x564f4b7d2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 --2- 172.21.15.29:0/715169456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b40e280 0x564f4b7d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 --2- 172.21.15.29:0/715169456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b4060a0 0x564f4b7f0ec0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 -- 172.21.15.29:0/715169456 >> 172.21.15.29:0/715169456 conn(0x564f4b423350 msgr2=0x564f4b7e3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 -- 172.21.15.29:0/715169456 shutdown_connections 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 -- 172.21.15.29:0/715169456 wait complete. 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 -- start start 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.163+0000 7fd789c4f740 1 Processor -- start 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 0x564f4b7f9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7f9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd789c4f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7fed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd789c4f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564f4b7f6d10 con 0x564f4b40e280 2024-09-21T02:38:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd789c4f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564f4b7f6640 con 0x564f4b42ef60 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd789c4f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564f4b7f73e0 con 0x564f4b4060a0 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd789c4f740 10 client.0 _ll_register_callbacks cb 0x7fd778009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7f9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7f9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42702/0 (socket says 172.21.15.29:42702) 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 -- 172.21.15.29:0/2768427337 learned_addr learned my addr 172.21.15.29:0/2768427337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd780c76640 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7fed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd780c76640 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 0x564f4b7f9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 -- 172.21.15.29:0/2768427337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 msgr2=0x564f4b7f9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 0x564f4b7f9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 -- 172.21.15.29:0/2768427337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 msgr2=0x564f4b7fed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7fed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 -- 172.21.15.29:0/2768427337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f4b7ff280 con 0x564f4b40e280 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd780c76640 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7fed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd781477640 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7f9d50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd7780057c0 tx=0x7fd778001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27dac48c77f9a52 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd764ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.164+0000 7fd764ff9640 1 -- 172.21.15.29:0/2768427337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778035070 con 0x564f4b40e280 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd764ff9640 1 -- 172.21.15.29:0/2768427337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd778045460 con 0x564f4b40e280 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 1 -- 172.21.15.29:0/2768427337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd76c015bf0 con 0x564f4b40e280 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd764ff9640 1 -- 172.21.15.29:0/2768427337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd778005850 con 0x564f4b40e280 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 10 client.15420 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 1 -- 172.21.15.29:0/2768427337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f4b862a80 con 0x564f4b40e280 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd764ff9640 1 -- 172.21.15.29:0/2768427337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd76c015bf0 con 0x564f4b40e280 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd764ff9640 1 -- 172.21.15.29:0/2768427337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd77805c7b0 con 0x564f4b40e280 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 10 client.15420 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 10 client.15420 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 20 client.15420 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x564f4b3c2c90 0x564f4b866150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 4 client.15420 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.165+0000 7fd789c4f740 1 -- 172.21.15.29:0/2768427337 --> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] -- command(tid 0: {"prefix": "status"}) -- 0x564f4b74a610 con 0x564f4b3c2c90 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.166+0000 7fd780c76640 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x564f4b3c2c90 0x564f4b866150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.166+0000 7fd764ff9640 10 client.15420 ms_handle_connect on v2:172.21.15.190:6834/131555194 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.166+0000 7fd780c76640 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x564f4b3c2c90 0x564f4b866150 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c82a79ea98ad9b18 server_cookie=890045533d8fdb10 in_seq=0 out_seq=0 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.166+0000 7fd764ff9640 1 -- 172.21.15.29:0/2768427337 <== mds.3 v2:172.21.15.190:6834/131555194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564f4b74a610 con 0x564f4b3c2c90 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.166+0000 7fd764ff9640 10 client.15420 handle_command_reply: tid=0 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.166+0000 7fd789c4f740 1 client.15420 shutdown 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 -- 172.21.15.29:0/2768427337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 msgr2=0x564f4b7f9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7f9d50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd7780057c0 tx=0x7fd778001e50 comp rx=0 tx=0).stop 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 -- 172.21.15.29:0/2768427337 shutdown_connections 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x564f4b3c2c90 0x564f4b866150 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564f4b42ef60 0x564f4b7fed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f4b40e280 0x564f4b7f9d50 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 --2- 172.21.15.29:0/2768427337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564f4b4060a0 0x564f4b7f9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 -- 172.21.15.29:0/2768427337 >> 172.21.15.29:0/2768427337 conn(0x564f4b423350 msgr2=0x564f4b7e4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd764ff9640 0 client.15420 ms_handle_reset on v2:172.21.15.190:6834/131555194 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 -- 172.21.15.29:0/2768427337 shutdown_connections 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 1 -- 172.21.15.29:0/2768427337 wait complete. 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd789c4f740 20 client.15420 trim_cache size 0 max 16384 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24613, 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 105.144837666, 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 174.52503691800001 2024-09-21T02:38:53.176 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd7836db640 1 -- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7680b6590 msgr2=0x7fd7680b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd7836db640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7680b6590 0x7fd7680b8aa0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd7740ac8c0 tx=0x7fd7740a11e0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd7836db640 1 -- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c109df0 msgr2=0x7fd77c19fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.167+0000 7fd7836db640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c109df0 0x7fd77c19fd90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd77400b970 tx=0x7fd77400be30 comp rx=0 tx=0).stop 2024-09-21T02:38:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.168+0000 7fd7836db640 1 -- 172.21.15.29:0/3603328870 shutdown_connections 2024-09-21T02:38:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.168+0000 7fd7836db640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7680b6590 0x7fd7680b8aa0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.168+0000 7fd7836db640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c109df0 0x7fd77c19fd90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.168+0000 7fd7836db640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c1094a0 0x7fd77c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.168+0000 7fd7836db640 1 --2- 172.21.15.29:0/3603328870 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103b60 0x7fd77c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.168+0000 7fd7836db640 1 -- 172.21.15.29:0/3603328870 >> 172.21.15.29:0/3603328870 conn(0x7fd77c07b850 msgr2=0x7fd77c1076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.168+0000 7fd7836db640 1 -- 172.21.15.29:0/3603328870 shutdown_connections 2024-09-21T02:38:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:53.168+0000 7fd7836db640 1 -- 172.21.15.29:0/3603328870 wait complete. 2024-09-21T02:38:53.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:53 smithi029 ceph-mon[31006]: pgmap v382: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 610 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:53.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:53 smithi066 ceph-mon[33436]: pgmap v382: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 610 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:53.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:53 smithi190 ceph-mon[33871]: pgmap v382: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 610 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:53.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.c status 2024-09-21T02:38:54.154 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 -- 172.21.15.29:0/2841651715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 msgr2=0x7f095010f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 --2- 172.21.15.29:0/2841651715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 0x7f095010f520 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0948009ca0 tx=0x7f094802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 -- 172.21.15.29:0/2841651715 shutdown_connections 2024-09-21T02:38:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 --2- 172.21.15.29:0/2841651715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 0x7f095010f520 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 --2- 172.21.15.29:0/2841651715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09501083a0 0x7f09501087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 --2- 172.21.15.29:0/2841651715 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0950102a60 0x7f0950102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 -- 172.21.15.29:0/2841651715 >> 172.21.15.29:0/2841651715 conn(0x7f0950079730 msgr2=0x7f0950079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 -- 172.21.15.29:0/2841651715 shutdown_connections 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.751+0000 7f0958645640 1 -- 172.21.15.29:0/2841651715 wait complete. 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0958645640 1 Processor -- start 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0958645640 1 -- start start 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0958645640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0950102a60 0x7f095019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0958645640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09501083a0 0x7f095019b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0958645640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 0x7f095019bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0958645640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0950112860 con 0x7f0950108cf0 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0958645640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f09501126e0 con 0x7f0950102a60 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0958645640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f09501129e0 con 0x7f09501083a0 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0956be2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 0x7f095019bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0956be2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 0x7f095019bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44472/0 (socket says 172.21.15.29:44472) 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f0956be2640 1 -- 172.21.15.29:0/2769826814 learned_addr learned my addr 172.21.15.29:0/2769826814 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.752+0000 7f09563e1640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0950102a60 0x7f095019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0955be0640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09501083a0 0x7f095019b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0956be2640 1 -- 172.21.15.29:0/2769826814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09501083a0 msgr2=0x7f095019b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0956be2640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09501083a0 0x7f095019b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0956be2640 1 -- 172.21.15.29:0/2769826814 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0950102a60 msgr2=0x7f095019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0956be2640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0950102a60 0x7f095019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0956be2640 1 -- 172.21.15.29:0/2769826814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09501a63a0 con 0x7f0950108cf0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0955be0640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09501083a0 0x7f095019b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0956be2640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 0x7f095019bb90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0948005e30 tx=0x7f09480093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f09477fe640 1 -- 172.21.15.29:0/2769826814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948032d10 con 0x7f0950108cf0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f09477fe640 1 -- 172.21.15.29:0/2769826814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0948032eb0 con 0x7f0950108cf0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0958645640 1 -- 172.21.15.29:0/2769826814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09501a6630 con 0x7f0950108cf0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f09477fe640 1 -- 172.21.15.29:0/2769826814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948030b20 con 0x7f0950108cf0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.753+0000 7f0958645640 1 -- 172.21.15.29:0/2769826814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09501a6b70 con 0x7f0950108cf0 2024-09-21T02:38:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.756+0000 7f095ebb9740 1 Processor -- start 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.756+0000 7f09477fe640 1 -- 172.21.15.29:0/2769826814 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f0948030cc0 con 0x7f0950108cf0 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.756+0000 7f095ebb9740 1 -- start start 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa57f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 0x56128fa58440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 0x56128fa58980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f095ebb9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56128f9cfc10 con 0x56128f6b1f60 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f095ebb9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56128f9bf040 con 0x56128f6890a0 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f095ebb9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56128f9c2e50 con 0x56128f691280 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0955be0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa57f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0955be0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa57f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44484/0 (socket says 172.21.15.29:44484) 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0955be0640 1 -- 172.21.15.29:0/3062924564 learned_addr learned my addr 172.21.15.29:0/3062924564 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f09563e1640 1 --2- 172.21.15.29:0/3062924564 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 0x56128fa58440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f09477fe640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f093c0b6540 0x7f093c0b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0955be0640 1 -- 172.21.15.29:0/3062924564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 msgr2=0x56128fa58980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0955be0640 1 --2- 172.21.15.29:0/3062924564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 0x56128fa58980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0955be0640 1 -- 172.21.15.29:0/3062924564 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 msgr2=0x56128fa58440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0955be0640 1 --2- 172.21.15.29:0/3062924564 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 0x56128fa58440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0955be0640 1 -- 172.21.15.29:0/3062924564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56128f71d540 con 0x56128f6b1f60 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.757+0000 7f0956be2640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f093c0b6540 0x7f093c0b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.758+0000 7f09477fe640 1 -- 172.21.15.29:0/2769826814 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0948108840 con 0x7f0950108cf0 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.758+0000 7f0955be0640 1 --2- 172.21.15.29:0/3062924564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa57f00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d80 tx=0x7f09400137d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75825422a8bbba40 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.758+0000 7f0944ff9640 1 -- 172.21.15.29:0/3062924564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094000f440 con 0x56128f6b1f60 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.758+0000 7f0944ff9640 1 -- 172.21.15.29:0/3062924564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0940017030 con 0x56128f6b1f60 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.758+0000 7f0944ff9640 1 -- 172.21.15.29:0/3062924564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094001b630 con 0x56128f6b1f60 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 -- 172.21.15.29:0/3062924564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 msgr2=0x56128fa57f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3062924564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa57f00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d80 tx=0x7f09400137d0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f0956be2640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f093c0b6540 0x7f093c0b8a50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f09480cac20 tx=0x7f09480972d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 -- 172.21.15.29:0/3062924564 shutdown_connections 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3062924564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 0x56128fa58980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3062924564 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 0x56128fa58440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3062924564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa57f00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 -- 172.21.15.29:0/3062924564 >> 172.21.15.29:0/3062924564 conn(0x56128f6a6350 msgr2=0x56128fa55df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 -- 172.21.15.29:0/3062924564 shutdown_connections 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 -- 172.21.15.29:0/3062924564 wait complete. 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 -- start start 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.759+0000 7f095ebb9740 1 Processor -- start 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 0x56128fa67df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 0x56128fa68330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa73d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f095ebb9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56128f8205f0 con 0x56128f6b1f60 2024-09-21T02:38:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f095ebb9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56128f95a010 con 0x56128f6890a0 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f095ebb9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56128fa6aa10 con 0x56128f691280 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f095ebb9740 10 client.0 _ll_register_callbacks cb 0x56128f9b9340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa73d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa73d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44498/0 (socket says 172.21.15.29:44498) 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 -- 172.21.15.29:0/3712232860 learned_addr learned my addr 172.21.15.29:0/3712232860 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f09563e1640 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 0x56128fa68330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 0x56128fa67df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 -- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 msgr2=0x56128fa68330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 0x56128fa68330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 -- 172.21.15.29:0/3712232860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 msgr2=0x56128fa67df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 0x56128fa67df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 -- 172.21.15.29:0/3712232860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56128f987a00 con 0x56128f6b1f60 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f09563e1640 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 0x56128fa68330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:54.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0955be0640 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa73d40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d50 tx=0x7f094000fde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28120c44d17f45b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0939ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.760+0000 7f0939ffb640 1 -- 172.21.15.29:0/3712232860 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09400099c0 con 0x56128f6b1f60 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f0939ffb640 1 -- 172.21.15.29:0/3712232860 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0940004260 con 0x56128f6b1f60 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 1 -- 172.21.15.29:0/3712232860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0940017030 con 0x56128f6b1f60 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f0939ffb640 1 -- 172.21.15.29:0/3712232860 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f094002b050 con 0x56128f6b1f60 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 10 client.15432 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 1 -- 172.21.15.29:0/3712232860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56128fa76990 con 0x56128f6b1f60 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f0939ffb640 1 -- 172.21.15.29:0/3712232860 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0940017030 con 0x56128f6b1f60 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f0939ffb640 1 -- 172.21.15.29:0/3712232860 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f09400047e0 con 0x56128f6b1f60 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 10 client.15432 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 10 client.15432 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 20 client.15432 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56128f6708e0 0x56128fa79f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 4 client.15432 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:54.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.761+0000 7f095ebb9740 1 -- 172.21.15.29:0/3712232860 --> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56128f9d1b60 con 0x56128f6708e0 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.762+0000 7f09563e1640 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56128f6708e0 0x56128fa79f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.762+0000 7f0939ffb640 10 client.15432 ms_handle_connect on v2:172.21.15.190:6836/3758896499 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.762+0000 7f09563e1640 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56128f6708e0 0x56128fa79f70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a95d19a7190c9a30 server_cookie=926c0b2f1cf6aa52 in_seq=0 out_seq=0 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f0939ffb640 1 -- 172.21.15.29:0/3712232860 <== mds.2 v2:172.21.15.190:6836/3758896499 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56128f9d1b60 con 0x56128f6708e0 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f0939ffb640 10 client.15432 handle_command_reply: tid=0 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 client.15432 shutdown 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 -- 172.21.15.29:0/3712232860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 msgr2=0x56128fa73d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa73d40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d50 tx=0x7f094000fde0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 -- 172.21.15.29:0/3712232860 shutdown_connections 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56128f6708e0 0x56128fa79f70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6b1f60 0x56128fa73d40 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f691280 0x56128fa68330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f0939ffb640 0 client.15432 ms_handle_reset on v2:172.21.15.190:6836/3758896499 2024-09-21T02:38:54.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 --2- 172.21.15.29:0/3712232860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6890a0 0x56128fa67df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.763+0000 7f095ebb9740 1 -- 172.21.15.29:0/3712232860 >> 172.21.15.29:0/3712232860 conn(0x56128f6a6350 msgr2=0x56128fa5b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:54.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.764+0000 7f095ebb9740 1 -- 172.21.15.29:0/3712232860 shutdown_connections 2024-09-21T02:38:54.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.764+0000 7f095ebb9740 1 -- 172.21.15.29:0/3712232860 wait complete. 2024-09-21T02:38:54.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.764+0000 7f095ebb9740 20 client.15432 trim_cache size 0 max 16384 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f095ebb9740 1 Processor -- start 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f095ebb9740 1 -- start start 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6890a0 0x56128fa62da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f691280 0x56128fa632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6b1f60 0x56128fa66470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f095ebb9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56128fa782b0 con 0x56128f691280 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f095ebb9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56128fa78980 con 0x56128f6b1f60 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f095ebb9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56128fa79050 con 0x56128f6890a0 2024-09-21T02:38:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f09563e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f691280 0x56128fa632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f09563e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f691280 0x56128fa632e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44500/0 (socket says 172.21.15.29:44500) 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.782+0000 7f09563e1640 1 -- 172.21.15.29:0/1952513282 learned_addr learned my addr 172.21.15.29:0/1952513282 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f0955be0640 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6b1f60 0x56128fa66470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f09563e1640 1 -- 172.21.15.29:0/1952513282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6890a0 msgr2=0x56128fa62da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f0955be0640 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6890a0 0x56128fa62da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f09563e1640 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6890a0 0x56128fa62da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f09563e1640 1 -- 172.21.15.29:0/1952513282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6b1f60 msgr2=0x56128fa66470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f09563e1640 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6b1f60 0x56128fa66470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f09563e1640 1 -- 172.21.15.29:0/1952513282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56128fa62870 con 0x56128f691280 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f09563e1640 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f691280 0x56128fa632e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f094c00ac40 tx=0x7f094c013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3715af2e0687ce server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f093bfff640 1 -- 172.21.15.29:0/1952513282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c013db0 con 0x56128f691280 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f093bfff640 1 -- 172.21.15.29:0/1952513282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f094c013110 con 0x56128f691280 2024-09-21T02:38:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.783+0000 7f093bfff640 1 -- 172.21.15.29:0/1952513282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c003cf0 con 0x56128f691280 2024-09-21T02:38:54.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 -- 172.21.15.29:0/1952513282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f691280 msgr2=0x56128fa632e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f691280 0x56128fa632e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f094c00ac40 tx=0x7f094c013830 comp rx=0 tx=0).stop 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 -- 172.21.15.29:0/1952513282 shutdown_connections 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f6b1f60 0x56128fa66470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f691280 0x56128fa632e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 --2- 172.21.15.29:0/1952513282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6890a0 0x56128fa62da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 -- 172.21.15.29:0/1952513282 >> 172.21.15.29:0/1952513282 conn(0x56128f6a6350 msgr2=0x56128fa50ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 -- 172.21.15.29:0/1952513282 shutdown_connections 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 -- 172.21.15.29:0/1952513282 wait complete. 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 -- start start 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.784+0000 7f095ebb9740 1 Processor -- start 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6890a0 0x56128fa7f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f691280 0x56128fa7fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f095ebb9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6b1f60 0x56128fa80170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f095ebb9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56128fa79df0 con 0x56128f6890a0 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f095ebb9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56128fa79720 con 0x56128f691280 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f095ebb9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56128fa7a4c0 con 0x56128f6b1f60 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f095ebb9740 10 client.0 _ll_register_callbacks cb 0x7f0940009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6890a0 0x56128fa7f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6890a0 0x56128fa7f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44504/0 (socket says 172.21.15.29:44504) 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 -- 172.21.15.29:0/580248267 learned_addr learned my addr 172.21.15.29:0/580248267 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 -- 172.21.15.29:0/580248267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6b1f60 msgr2=0x56128fa80170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6b1f60 0x56128fa80170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 -- 172.21.15.29:0/580248267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f691280 msgr2=0x56128fa7fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f691280 0x56128fa7fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 -- 172.21.15.29:0/580248267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56128fa7d670 con 0x56128f6890a0 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0955be0640 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6890a0 0x56128fa7f6f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d50 tx=0x7f0940004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f3dd4c80be860c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.785+0000 7f0939ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f0939ffb640 1 -- 172.21.15.29:0/580248267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094000f860 con 0x56128f6890a0 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f0939ffb640 1 -- 172.21.15.29:0/580248267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f094000fa00 con 0x56128f6890a0 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 1 -- 172.21.15.29:0/580248267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f094c013110 con 0x56128f6890a0 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f0939ffb640 1 -- 172.21.15.29:0/580248267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0940041050 con 0x56128f6890a0 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 10 client.15438 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 1 -- 172.21.15.29:0/580248267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56128fae5650 con 0x56128f6890a0 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f0939ffb640 1 -- 172.21.15.29:0/580248267 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f094c013110 con 0x56128f6890a0 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f0939ffb640 1 -- 172.21.15.29:0/580248267 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0940003700 con 0x56128f6890a0 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 10 client.15438 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 10 client.15438 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 20 client.15438 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56128f645c90 0x56128fae8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 4 client.15438 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:54.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.786+0000 7f095ebb9740 1 -- 172.21.15.29:0/580248267 --> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] -- command(tid 0: {"prefix": "status"}) -- 0x56128f9cd610 con 0x56128f645c90 2024-09-21T02:38:54.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.787+0000 7f09563e1640 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56128f645c90 0x56128fae8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.787+0000 7f09563e1640 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56128f645c90 0x56128fae8f00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c72a4f3db8cabd0f server_cookie=d4d8106016d94690 in_seq=0 out_seq=0 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.787+0000 7f0939ffb640 10 client.15438 ms_handle_connect on v2:172.21.15.190:6836/3758896499 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.787+0000 7f0939ffb640 1 -- 172.21.15.29:0/580248267 <== mds.2 v2:172.21.15.190:6836/3758896499 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x56128f9cd610 con 0x56128f645c90 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.787+0000 7f0939ffb640 10 client.15438 handle_command_reply: tid=0 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.787+0000 7f095ebb9740 1 client.15438 shutdown 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.787+0000 7f095ebb9740 1 -- 172.21.15.29:0/580248267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6890a0 msgr2=0x56128fa7f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.787+0000 7f095ebb9740 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6890a0 0x56128fa7f6f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d50 tx=0x7f0940004ee0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 1 -- 172.21.15.29:0/580248267 shutdown_connections 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56128f645c90 0x56128fae8f00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56128f6b1f60 0x56128fa80170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0939ffb640 0 client.15438 ms_handle_reset on v2:172.21.15.190:6836/3758896499 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56128f691280 0x56128fa7fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 1 --2- 172.21.15.29:0/580248267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56128f6890a0 0x56128fa7f6f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 1 -- 172.21.15.29:0/580248267 >> 172.21.15.29:0/580248267 conn(0x56128f6a6350 msgr2=0x56128fa50ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 1 -- 172.21.15.29:0/580248267 shutdown_connections 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 1 -- 172.21.15.29:0/580248267 wait complete. 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f095ebb9740 20 client.15438 trim_cache size 0 max 16384 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24625, 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:54.794 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:54.795 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 107.77452088, 2024-09-21T02:38:54.795 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:54.795 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:38:54.795 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:54.795 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 173.641683298 2024-09-21T02:38:54.795 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:54.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 -- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f093c0b6540 msgr2=0x7f093c0b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f093c0b6540 0x7f093c0b8a50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f09480cac20 tx=0x7f09480972d0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 -- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 msgr2=0x7f095019bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 0x7f095019bb90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0948005e30 tx=0x7f09480093d0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 -- 172.21.15.29:0/2769826814 shutdown_connections 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f093c0b6540 0x7f093c0b8a50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0950108cf0 0x7f095019bb90 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09501083a0 0x7f095019b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 --2- 172.21.15.29:0/2769826814 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0950102a60 0x7f095019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 -- 172.21.15.29:0/2769826814 >> 172.21.15.29:0/2769826814 conn(0x7f0950079730 msgr2=0x7f09501043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.788+0000 7f0958645640 1 -- 172.21.15.29:0/2769826814 shutdown_connections 2024-09-21T02:38:54.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:54.789+0000 7f0958645640 1 -- 172.21.15.29:0/2769826814 wait complete. 2024-09-21T02:38:55.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:55 smithi066 ceph-mon[33436]: pgmap v383: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:55.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:55 smithi029 ceph-mon[31006]: pgmap v383: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:55.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:55 smithi190 ceph-mon[33871]: pgmap v383: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:38:55.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T02:38:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/95454964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc105330 msgr2=0x7f7fdc10d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/95454964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc105330 0x7f7fdc10d880 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009900 tx=0x7f7fcc02a410 comp rx=0 tx=0).stop 2024-09-21T02:38:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/95454964 shutdown_connections 2024-09-21T02:38:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/95454964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fdc10de40 0x7f7fdc110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/95454964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc105330 0x7f7fdc10d880 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/95454964 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc1049e0 0x7f7fdc104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/95454964 >> 172.21.15.29:0/95454964 conn(0x7f7fdc079750 msgr2=0x7f7fdc079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/95454964 shutdown_connections 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.432+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/95454964 wait complete. 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.433+0000 7f7fe3fc3640 1 Processor -- start 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.433+0000 7f7fe3fc3640 1 -- start start 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.433+0000 7f7fe3fc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc1049e0 0x7f7fdc102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.433+0000 7f7fe3fc3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fdc105330 0x7f7fdc102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe3fc3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc10de40 0x7f7fdc1031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe3fc3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc113950 con 0x7f7fdc1049e0 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe3fc3640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc1137d0 con 0x7f7fdc10de40 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe3fc3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc113ad0 con 0x7f7fdc105330 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc1049e0 0x7f7fdc102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc1049e0 0x7f7fdc102720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44524/0 (socket says 172.21.15.29:44524) 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/11738386 learned_addr learned my addr 172.21.15.29:0/11738386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe155e640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fdc105330 0x7f7fdc102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/11738386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fdc105330 msgr2=0x7f7fdc102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fdc105330 0x7f7fdc102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/11738386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc10de40 msgr2=0x7f7fdc1031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe2560640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc10de40 0x7f7fdc1031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc10de40 0x7f7fdc1031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/11738386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fdc1a8630 con 0x7f7fdc1049e0 2024-09-21T02:38:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe2560640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc10de40 0x7f7fdc1031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc1049e0 0x7f7fdc102720 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c970 tx=0x7f7fd800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fd2ffd640 1 -- 172.21.15.29:0/11738386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8007c10 con 0x7f7fdc1049e0 2024-09-21T02:38:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fd2ffd640 1 -- 172.21.15.29:0/11738386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd8007db0 con 0x7f7fdc1049e0 2024-09-21T02:38:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fd2ffd640 1 -- 172.21.15.29:0/11738386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800f7d0 con 0x7f7fdc1049e0 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.434+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/11738386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fdc1a8920 con 0x7f7fdc1049e0 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.435+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/11738386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fdc1a8e30 con 0x7f7fdc1049e0 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.437+0000 7f7fea537740 1 Processor -- start 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.437+0000 7f7fea537740 1 -- start start 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.437+0000 7f7fea537740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 0x565011220f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.437+0000 7f7fea537740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e520a0 0x565011221440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fea537740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011221980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fea537740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x565011198c10 con 0x565010e520a0 2024-09-21T02:38:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fea537740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x565011188040 con 0x565010e7af60 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fea537740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56501118be50 con 0x565010e5a280 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fd2ffd640 1 -- 172.21.15.29:0/11738386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f7fd8018480 con 0x7f7fdc1049e0 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e520a0 0x565011221440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e520a0 0x565011221440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44536/0 (socket says 172.21.15.29:44536) 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/2409474998 learned_addr learned my addr 172.21.15.29:0/2409474998 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe155e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011221980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe2560640 1 --2- 172.21.15.29:0/2409474998 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 0x565011220f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/2409474998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 msgr2=0x565011221980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/2409474998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011221980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/2409474998 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 msgr2=0x565011220f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/2409474998 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 0x565011220f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/2409474998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565010ee6540 con 0x565010e520a0 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.438+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/2409474998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e520a0 0x565011221440 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80ebb80 tx=0x7f7fd80ebc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efeeecd19e43714e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.439+0000 7f7fc7fff640 1 -- 172.21.15.29:0/2409474998 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80f1040 con 0x565010e520a0 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.439+0000 7f7fc7fff640 1 -- 172.21.15.29:0/2409474998 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd80f0de0 con 0x565010e520a0 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.439+0000 7f7fd2ffd640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fc80b6510 0x7f7fc80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.439+0000 7f7fc7fff640 1 -- 172.21.15.29:0/2409474998 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80f5c00 con 0x565010e520a0 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.439+0000 7f7fe2560640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fc80b6510 0x7f7fc80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.439+0000 7f7fd2ffd640 1 -- 172.21.15.29:0/11738386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7fd80e7a70 con 0x7f7fdc1049e0 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 -- 172.21.15.29:0/2409474998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e520a0 msgr2=0x565011221440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fe2560640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fc80b6510 0x7f7fc80b8a20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4005fd0 tx=0x7f7fd4008630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 --2- 172.21.15.29:0/2409474998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e520a0 0x565011221440 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80ebb80 tx=0x7f7fd80ebc70 comp rx=0 tx=0).stop 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 -- 172.21.15.29:0/2409474998 shutdown_connections 2024-09-21T02:38:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 --2- 172.21.15.29:0/2409474998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011221980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 --2- 172.21.15.29:0/2409474998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e520a0 0x565011221440 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 --2- 172.21.15.29:0/2409474998 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 0x565011220f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 -- 172.21.15.29:0/2409474998 >> 172.21.15.29:0/2409474998 conn(0x565010e6f350 msgr2=0x56501121edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 -- 172.21.15.29:0/2409474998 shutdown_connections 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 -- 172.21.15.29:0/2409474998 wait complete. 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 -- start start 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.440+0000 7f7fea537740 1 Processor -- start 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fea537740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e520a0 0x56501122c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fea537740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e5a280 0x56501122cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fea537740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 0x565011238a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fea537740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x565010fe95f0 con 0x565010e5a280 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fea537740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x565011123010 con 0x565010e7af60 2024-09-21T02:38:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fea537740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56501122f6e0 con 0x565010e520a0 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fea537740 10 client.0 _ll_register_callbacks cb 0x565011182340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe155e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e5a280 0x56501122cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe155e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e5a280 0x56501122cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44548/0 (socket says 172.21.15.29:44548) 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe155e640 1 -- 172.21.15.29:0/4231572686 learned_addr learned my addr 172.21.15.29:0/4231572686 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e520a0 0x56501122c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe2560640 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 0x565011238a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe155e640 1 -- 172.21.15.29:0/4231572686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e520a0 msgr2=0x56501122c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe155e640 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e520a0 0x56501122c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe155e640 1 -- 172.21.15.29:0/4231572686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 msgr2=0x565011238a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe155e640 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 0x565011238a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe155e640 1 -- 172.21.15.29:0/4231572686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565011150a00 con 0x565010e5a280 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.441+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e520a0 0x56501122c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:56.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fe155e640 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e5a280 0x56501122cef0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc005940 tx=0x7f7fcc031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2555efdbc495518b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fc57fa640 1 -- 172.21.15.29:0/4231572686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc035040 con 0x565010e5a280 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fc57fa640 1 -- 172.21.15.29:0/4231572686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7fcc004240 con 0x565010e5a280 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fea537740 1 -- 172.21.15.29:0/4231572686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd80f0de0 con 0x565010e5a280 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fc57fa640 1 -- 172.21.15.29:0/4231572686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7fcc004440 con 0x565010e5a280 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fea537740 10 client.15450 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fea537740 1 -- 172.21.15.29:0/4231572686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56501123b5e0 con 0x565010e5a280 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fc57fa640 1 -- 172.21.15.29:0/4231572686 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd80f0de0 con 0x565010e5a280 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fc57fa640 1 -- 172.21.15.29:0/4231572686 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f7fcc03d500 con 0x565010e5a280 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fea537740 10 client.15450 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fea537740 10 client.15450 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.442+0000 7f7fea537740 20 client.15450 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.443+0000 7f7fea537740 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x565010e398e0 0x56501123f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.443+0000 7f7fea537740 4 client.15450 mds_command: new command op to 24632 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.443+0000 7f7fea537740 1 -- 172.21.15.29:0/4231572686 --> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56501119ab60 con 0x565010e398e0 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.443+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x565010e398e0 0x56501123f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.443+0000 7f7fc57fa640 10 client.15450 ms_handle_connect on v2:172.21.15.66:6838/3618856137 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.443+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x565010e398e0 0x56501123f060 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ba3b375a22854a6 server_cookie=14a07f59178fb189 in_seq=0 out_seq=0 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fc57fa640 1 -- 172.21.15.29:0/4231572686 <== mds.1 v2:172.21.15.66:6838/3618856137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56501119ab60 con 0x565010e398e0 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fc57fa640 10 client.15450 handle_command_reply: tid=0 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 client.15450 shutdown 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 -- 172.21.15.29:0/4231572686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e5a280 msgr2=0x56501122cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e5a280 0x56501122cef0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc005940 tx=0x7f7fcc031400 comp rx=0 tx=0).stop 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 -- 172.21.15.29:0/4231572686 shutdown_connections 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x565010e398e0 0x56501123f060 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e7af60 0x565011238a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e5a280 0x56501122cef0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 --2- 172.21.15.29:0/4231572686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e520a0 0x56501122c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fc57fa640 0 client.15450 ms_handle_reset on v2:172.21.15.66:6838/3618856137 2024-09-21T02:38:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 -- 172.21.15.29:0/4231572686 >> 172.21.15.29:0/4231572686 conn(0x565010e6f350 msgr2=0x565011224090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 -- 172.21.15.29:0/4231572686 shutdown_connections 2024-09-21T02:38:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.444+0000 7f7fea537740 1 -- 172.21.15.29:0/4231572686 wait complete. 2024-09-21T02:38:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.445+0000 7f7fea537740 20 client.15450 trim_cache size 0 max 16384 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.463+0000 7f7fea537740 1 Processor -- start 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fea537740 1 -- start start 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fea537740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x565011238970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fea537740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x565011238eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fea537740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011227960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fea537740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56501123d3a0 con 0x565010e398e0 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fea537740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56501123da70 con 0x565010e520a0 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fea537740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56501123e140 con 0x565010e5a280 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe1d5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x565011238970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe1d5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x565011238970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44552/0 (socket says 172.21.15.29:44552) 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/495447855 learned_addr learned my addr 172.21.15.29:0/495447855 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe2560640 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011227960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe155e640 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x565011238eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/495447855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 msgr2=0x565011227960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011227960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/495447855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 msgr2=0x565011238eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x565011238eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/495447855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565011223d60 con 0x565010e398e0 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe2560640 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011227960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.464+0000 7f7fe155e640 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x565011238eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.465+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x565011238970 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80e5240 tx=0x7f7fd8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d614d48d0acd03d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.465+0000 7f7fc77fe640 1 -- 172.21.15.29:0/495447855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80a97b0 con 0x565010e398e0 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.465+0000 7f7fc77fe640 1 -- 172.21.15.29:0/495447855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd80ad420 con 0x565010e398e0 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.465+0000 7f7fc77fe640 1 -- 172.21.15.29:0/495447855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80a9080 con 0x565010e398e0 2024-09-21T02:38:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 -- 172.21.15.29:0/495447855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 msgr2=0x565011238970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x565011238970 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80e5240 tx=0x7f7fd8000bc0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 -- 172.21.15.29:0/495447855 shutdown_connections 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011227960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x565011238eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 --2- 172.21.15.29:0/495447855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x565011238970 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 -- 172.21.15.29:0/495447855 >> 172.21.15.29:0/495447855 conn(0x565010e6f350 msgr2=0x56501122c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 -- 172.21.15.29:0/495447855 shutdown_connections 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 -- 172.21.15.29:0/495447855 wait complete. 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 -- start start 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.466+0000 7f7fea537740 1 Processor -- start 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fea537740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x5650112414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fea537740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x5650112419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fea537740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011246300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fea537740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56501123eee0 con 0x565010e398e0 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fea537740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56501123e810 con 0x565010e520a0 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fea537740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56501123f5b0 con 0x565010e5a280 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fea537740 10 client.0 _ll_register_callbacks cb 0x7f7fcc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe1d5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x5650112414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe1d5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x5650112414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44558/0 (socket says 172.21.15.29:44558) 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/757004708 learned_addr learned my addr 172.21.15.29:0/757004708 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe155e640 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x5650112419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe2560640 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011246300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/757004708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 msgr2=0x565011246300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011246300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/757004708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 msgr2=0x5650112419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x5650112419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe1d5f640 1 -- 172.21.15.29:0/757004708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650112468f0 con 0x565010e398e0 2024-09-21T02:38:56.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.467+0000 7f7fe2560640 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011246300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fe1d5f640 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x5650112414b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80ebad0 tx=0x7f7fd80a9100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f966b731a778bf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fc57fa640 1 -- 172.21.15.29:0/757004708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80ad080 con 0x565010e398e0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fc57fa640 1 -- 172.21.15.29:0/757004708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7fd8018ac0 con 0x565010e398e0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 1 -- 172.21.15.29:0/757004708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd80ad420 con 0x565010e398e0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fc57fa640 1 -- 172.21.15.29:0/757004708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7fd80ad260 con 0x565010e398e0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 10 client.15456 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 1 -- 172.21.15.29:0/757004708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565011247b50 con 0x565010e398e0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fc57fa640 1 -- 172.21.15.29:0/757004708 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd80ad420 con 0x565010e398e0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fc57fa640 1 -- 172.21.15.29:0/757004708 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f7fd80f4070 con 0x565010e398e0 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 10 client.15456 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 10 client.15456 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 20 client.15456 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x565010e7af60 0x5650112ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 4 client.15456 mds_command: new command op to 24632 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.468+0000 7f7fea537740 1 -- 172.21.15.29:0/757004708 --> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] -- command(tid 0: {"prefix": "status"}) -- 0x565011196610 con 0x565010e7af60 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.469+0000 7f7fe155e640 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x565010e7af60 0x5650112ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.469+0000 7f7fc57fa640 10 client.15456 ms_handle_connect on v2:172.21.15.66:6838/3618856137 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.469+0000 7f7fe155e640 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x565010e7af60 0x5650112ad0b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=debcf58d1449d85e server_cookie=7a2c8dad9d62c3dd in_seq=0 out_seq=0 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.469+0000 7f7fc57fa640 1 -- 172.21.15.29:0/757004708 <== mds.1 v2:172.21.15.66:6838/3618856137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x565011196610 con 0x565010e7af60 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.469+0000 7f7fc57fa640 10 client.15456 handle_command_reply: tid=0 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.469+0000 7f7fea537740 1 client.15456 shutdown 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 -- 172.21.15.29:0/757004708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 msgr2=0x5650112414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x5650112414b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80ebad0 tx=0x7f7fd80a9100 comp rx=0 tx=0).stop 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 -- 172.21.15.29:0/757004708 shutdown_connections 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x565010e7af60 0x5650112ad0b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565010e5a280 0x565011246300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565010e520a0 0x5650112419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 --2- 172.21.15.29:0/757004708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565010e398e0 0x5650112414b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fc57fa640 0 client.15456 ms_handle_reset on v2:172.21.15.66:6838/3618856137 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 -- 172.21.15.29:0/757004708 >> 172.21.15.29:0/757004708 conn(0x565010e6f350 msgr2=0x565011227960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 -- 172.21.15.29:0/757004708 shutdown_connections 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 1 -- 172.21.15.29:0/757004708 wait complete. 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fea537740 20 client.15456 trim_cache size 0 max 16384 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24632, 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:56.482 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:56.483 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:56.483 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 110.463511978, 2024-09-21T02:38:56.483 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:56.483 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:38:56.483 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:56.483 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 160.83502729200001 2024-09-21T02:38:56.483 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fc80b6510 msgr2=0x7f7fc80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fc80b6510 0x7f7fc80b8a20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4005fd0 tx=0x7f7fd4008630 comp rx=0 tx=0).stop 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc1049e0 msgr2=0x7f7fdc102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.470+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc1049e0 0x7f7fdc102720 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c970 tx=0x7f7fd800ce30 comp rx=0 tx=0).stop 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.471+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/11738386 shutdown_connections 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.471+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fc80b6510 0x7f7fc80b8a20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.471+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc10de40 0x7f7fdc1031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.471+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fdc105330 0x7f7fdc102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.471+0000 7f7fe3fc3640 1 --2- 172.21.15.29:0/11738386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fdc1049e0 0x7f7fdc102720 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.471+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/11738386 >> 172.21.15.29:0/11738386 conn(0x7f7fdc079750 msgr2=0x7f7fdc0ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.471+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/11738386 shutdown_connections 2024-09-21T02:38:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:56.471+0000 7f7fe3fc3640 1 -- 172.21.15.29:0/11738386 wait complete. 2024-09-21T02:38:57.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:57 smithi029 ceph-mon[31006]: pgmap v384: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:57.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T02:38:57.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:57 smithi066 ceph-mon[33436]: pgmap v384: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:57.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:57 smithi190 ceph-mon[33871]: pgmap v384: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 -- 172.21.15.29:0/201933543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 msgr2=0x7fef7010f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 --2- 172.21.15.29:0/201933543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 0x7fef7010f540 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fef68009ca0 tx=0x7fef6802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 -- 172.21.15.29:0/201933543 shutdown_connections 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 --2- 172.21.15.29:0/201933543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 0x7fef7010f540 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 --2- 172.21.15.29:0/201933543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef701083c0 0x7fef701087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 --2- 172.21.15.29:0/201933543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef70102a80 0x7fef70102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 -- 172.21.15.29:0/201933543 >> 172.21.15.29:0/201933543 conn(0x7fef70079740 msgr2=0x7fef70079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 -- 172.21.15.29:0/201933543 shutdown_connections 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 -- 172.21.15.29:0/201933543 wait complete. 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 Processor -- start 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.140+0000 7fef762fb640 1 -- start start 2024-09-21T02:38:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef762fb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef70102a80 0x7fef7019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef762fb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef701083c0 0x7fef7019a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef762fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 0x7fef7019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef762fb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef70113980 con 0x7fef70108d10 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef762fb640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fef70113800 con 0x7fef701083c0 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef762fb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fef70113b00 con 0x7fef70102a80 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef74898640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 0x7fef7019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef74898640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 0x7fef7019fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44562/0 (socket says 172.21.15.29:44562) 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef74898640 1 -- 172.21.15.29:0/2286843050 learned_addr learned my addr 172.21.15.29:0/2286843050 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef6ffff640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef70102a80 0x7fef7019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef701083c0 0x7fef7019a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef74898640 1 -- 172.21.15.29:0/2286843050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef70102a80 msgr2=0x7fef7019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef74898640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef70102a80 0x7fef7019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef74898640 1 -- 172.21.15.29:0/2286843050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef701083c0 msgr2=0x7fef7019a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef74898640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef701083c0 0x7fef7019a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef74898640 1 -- 172.21.15.29:0/2286843050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef701a0260 con 0x7fef70108d10 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.141+0000 7fef6ffff640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef70102a80 0x7fef7019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.142+0000 7fef74898640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 0x7fef7019fb30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fef68005cc0 tx=0x7fef68032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.142+0000 7fef6d7fa640 1 -- 172.21.15.29:0/2286843050 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68032e00 con 0x7fef70108d10 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.142+0000 7fef6d7fa640 1 -- 172.21.15.29:0/2286843050 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef680044d0 con 0x7fef70108d10 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.142+0000 7fef762fb640 1 -- 172.21.15.29:0/2286843050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef701a04f0 con 0x7fef70108d10 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.142+0000 7fef6d7fa640 1 -- 172.21.15.29:0/2286843050 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68030b70 con 0x7fef70108d10 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.142+0000 7fef762fb640 1 -- 172.21.15.29:0/2286843050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef701a6cb0 con 0x7fef70108d10 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6d7fa640 1 -- 172.21.15.29:0/2286843050 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7fef68004680 con 0x7fef70108d10 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef7c86f740 1 Processor -- start 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef7c86f740 1 -- start start 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 0x556bb4946f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4947440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4947980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef7c86f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556bb48bec10 con 0x556bb45780a0 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef7c86f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556bb48ae040 con 0x556bb4580280 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef7c86f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556bb48b1e50 con 0x556bb45a0f60 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4947440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4947440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44578/0 (socket says 172.21.15.29:44578) 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6ffff640 1 -- 172.21.15.29:0/3593683705 learned_addr learned my addr 172.21.15.29:0/3593683705 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 0x556bb4946f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4947980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6d7fa640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef580b6590 0x7fef580b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6ffff640 1 -- 172.21.15.29:0/3593683705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 msgr2=0x556bb4946f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6ffff640 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 0x556bb4946f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6ffff640 1 -- 172.21.15.29:0/3593683705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 msgr2=0x556bb4947980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6ffff640 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4947980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.145+0000 7fef6ffff640 1 -- 172.21.15.29:0/3593683705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bb460c540 con 0x556bb45780a0 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.146+0000 7fef74898640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef580b6590 0x7fef580b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.146+0000 7fef6ffff640 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4947440 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fef6000bd40 tx=0x7fef6000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7097e87f47e42f18 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.146+0000 7fef6d7fa640 1 -- 172.21.15.29:0/2286843050 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fef68108950 con 0x7fef70108d10 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.146+0000 7fef567fc640 1 -- 172.21.15.29:0/3593683705 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60015070 con 0x556bb45780a0 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.146+0000 7fef567fc640 1 -- 172.21.15.29:0/3593683705 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef6001a8f0 con 0x556bb45780a0 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.146+0000 7fef567fc640 1 -- 172.21.15.29:0/3593683705 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60011840 con 0x556bb45780a0 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 -- 172.21.15.29:0/3593683705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 msgr2=0x556bb4947440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4947440 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fef6000bd40 tx=0x7fef6000f730 comp rx=0 tx=0).stop 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef74898640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef580b6590 0x7fef580b8aa0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fef680cb9d0 tx=0x7fef6809bba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 -- 172.21.15.29:0/3593683705 shutdown_connections 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4947980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4947440 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 --2- 172.21.15.29:0/3593683705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 0x556bb4946f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 -- 172.21.15.29:0/3593683705 >> 172.21.15.29:0/3593683705 conn(0x556bb4595350 msgr2=0x556bb4944df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 -- 172.21.15.29:0/3593683705 shutdown_connections 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 -- 172.21.15.29:0/3593683705 wait complete. 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.147+0000 7fef7c86f740 1 -- start start 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef7c86f740 1 Processor -- start 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45780a0 0x556bb4956e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb49573c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb4962da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef7c86f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556bb470f5f0 con 0x556bb45a0f60 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef7c86f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556bb4849010 con 0x556bb4580280 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef7c86f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556bb4959a40 con 0x556bb45780a0 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef7c86f740 10 client.0 _ll_register_callbacks cb 0x556bb48a8340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb4962da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb4962da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44588/0 (socket says 172.21.15.29:44588) 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 -- 172.21.15.29:0/965263649 learned_addr learned my addr 172.21.15.29:0/965263649 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6ffff640 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb49573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45780a0 0x556bb4956e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 -- 172.21.15.29:0/965263649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45780a0 msgr2=0x556bb4956e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45780a0 0x556bb4956e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 -- 172.21.15.29:0/965263649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 msgr2=0x556bb49573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb49573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.148+0000 7fef6f7fe640 1 -- 172.21.15.29:0/965263649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bb4876a00 con 0x556bb45a0f60 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb4962da0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fef6400c960 tx=0x7fef6400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e91e6887f26eb27 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef4f7fe640 1 -- 172.21.15.29:0/965263649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64004030 con 0x556bb45a0f60 2024-09-21T02:38:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef4f7fe640 1 -- 172.21.15.29:0/965263649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fef640041d0 con 0x556bb45a0f60 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef7c86f740 1 -- 172.21.15.29:0/965263649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6001a8f0 con 0x556bb45a0f60 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef4f7fe640 1 -- 172.21.15.29:0/965263649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fef64028050 con 0x556bb45a0f60 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef7c86f740 10 client.15468 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef7c86f740 1 -- 172.21.15.29:0/965263649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556bb4965a40 con 0x556bb45a0f60 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef4f7fe640 1 -- 172.21.15.29:0/965263649 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef6001a8f0 con 0x556bb45a0f60 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef4f7fe640 1 -- 172.21.15.29:0/965263649 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fef64012450 con 0x556bb45a0f60 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef7c86f740 10 client.15468 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef7c86f740 10 client.15468 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef7c86f740 20 client.15468 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.149+0000 7fef7c86f740 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556bb455f8e0 0x556bb49694c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.150+0000 7fef7c86f740 4 client.15468 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.150+0000 7fef7c86f740 1 -- 172.21.15.29:0/965263649 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bb48c0b60 con 0x556bb455f8e0 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.150+0000 7fef6ffff640 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556bb455f8e0 0x556bb49694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.150+0000 7fef6ffff640 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556bb455f8e0 0x556bb49694c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89376f1787e17b96 server_cookie=4c6092857ee60008 in_seq=0 out_seq=0 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.150+0000 7fef4f7fe640 10 client.15468 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef4f7fe640 1 -- 172.21.15.29:0/965263649 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556bb48c0b60 con 0x556bb455f8e0 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef4f7fe640 10 client.15468 handle_command_reply: tid=0 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 client.15468 shutdown 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 -- 172.21.15.29:0/965263649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 msgr2=0x556bb4962da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb4962da0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fef6400c960 tx=0x7fef6400cb70 comp rx=0 tx=0).stop 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 -- 172.21.15.29:0/965263649 shutdown_connections 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556bb455f8e0 0x556bb49694c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb4962da0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb49573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 --2- 172.21.15.29:0/965263649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45780a0 0x556bb4956e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef4f7fe640 0 client.15468 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:38:58.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 -- 172.21.15.29:0/965263649 >> 172.21.15.29:0/965263649 conn(0x556bb4595350 msgr2=0x556bb49488f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:58.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 -- 172.21.15.29:0/965263649 shutdown_connections 2024-09-21T02:38:58.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.151+0000 7fef7c86f740 1 -- 172.21.15.29:0/965263649 wait complete. 2024-09-21T02:38:58.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.152+0000 7fef7c86f740 20 client.15468 trim_cache size 0 max 16384 2024-09-21T02:38:58.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef7c86f740 1 Processor -- start 2024-09-21T02:38:58.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef7c86f740 1 -- start start 2024-09-21T02:38:58.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4968050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4944730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 0x556bb4944c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef7c86f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556bb4967800 con 0x556bb45780a0 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef7c86f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556bb4967ed0 con 0x556bb4580280 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef7c86f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556bb49685a0 con 0x556bb45a0f60 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4968050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4968050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44596/0 (socket says 172.21.15.29:44596) 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 -- 172.21.15.29:0/1466579451 learned_addr learned my addr 172.21.15.29:0/1466579451 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 0x556bb4944c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6ffff640 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4944730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 -- 172.21.15.29:0/1466579451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 msgr2=0x556bb4944c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 0x556bb4944c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 -- 172.21.15.29:0/1466579451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 msgr2=0x556bb4944730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4944730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.170+0000 7fef6f7fe640 1 -- 172.21.15.29:0/1466579451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bb493f0f0 con 0x556bb45780a0 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.171+0000 7fef6ffff640 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4944730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.171+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4968050 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fef64000c00 tx=0x7fef64005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bf6f35153eed50 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.171+0000 7fef55ffb640 1 -- 172.21.15.29:0/1466579451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64032070 con 0x556bb45780a0 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.171+0000 7fef55ffb640 1 -- 172.21.15.29:0/1466579451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef64014300 con 0x556bb45780a0 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.171+0000 7fef55ffb640 1 -- 172.21.15.29:0/1466579451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64004030 con 0x556bb45780a0 2024-09-21T02:38:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 -- 172.21.15.29:0/1466579451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 msgr2=0x556bb4968050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4968050 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fef64000c00 tx=0x7fef64005fd0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 -- 172.21.15.29:0/1466579451 shutdown_connections 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb45a0f60 0x556bb4944c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb4580280 0x556bb4944730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 --2- 172.21.15.29:0/1466579451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45780a0 0x556bb4968050 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 -- 172.21.15.29:0/1466579451 >> 172.21.15.29:0/1466579451 conn(0x556bb4595350 msgr2=0x556bb494c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 -- 172.21.15.29:0/1466579451 shutdown_connections 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 -- 172.21.15.29:0/1466579451 wait complete. 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 -- start start 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.172+0000 7fef7c86f740 1 Processor -- start 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb45780a0 0x556bb496b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb4580280 0x556bb496beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef7c86f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb496c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef7c86f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556bb4969340 con 0x556bb45a0f60 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef7c86f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556bb4968c70 con 0x556bb45780a0 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef7c86f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556bb4969a10 con 0x556bb4580280 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef7c86f740 10 client.0 _ll_register_callbacks cb 0x7fef6400c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb496c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb496c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44602/0 (socket says 172.21.15.29:44602) 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 -- 172.21.15.29:0/710074017 learned_addr learned my addr 172.21.15.29:0/710074017 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6ffff640 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb4580280 0x556bb496beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb45780a0 0x556bb496b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 -- 172.21.15.29:0/710074017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb4580280 msgr2=0x556bb496beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb4580280 0x556bb496beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 -- 172.21.15.29:0/710074017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb45780a0 msgr2=0x556bb496b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb45780a0 0x556bb496b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 -- 172.21.15.29:0/710074017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bb4947d40 con 0x556bb45a0f60 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef6f7fe640 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb496c430 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fef64015ba0 tx=0x7fef64025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9199da27d14731e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.173+0000 7fef4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef4f7fe640 1 -- 172.21.15.29:0/710074017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64032070 con 0x556bb45a0f60 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef4f7fe640 1 -- 172.21.15.29:0/710074017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fef64015040 con 0x556bb45a0f60 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 1 -- 172.21.15.29:0/710074017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef64014300 con 0x556bb45a0f60 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef4f7fe640 1 -- 172.21.15.29:0/710074017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fef64015240 con 0x556bb45a0f60 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 10 client.15474 fetch_fsmap learned FSMap version 28 2024-09-21T02:38:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 1 -- 172.21.15.29:0/710074017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556bb49d4bf0 con 0x556bb45a0f60 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef4f7fe640 1 -- 172.21.15.29:0/710074017 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef64014300 con 0x556bb45a0f60 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef4f7fe640 1 -- 172.21.15.29:0/710074017 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fef64003780 con 0x556bb45a0f60 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 10 client.15474 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 10 client.15474 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 20 client.15474 populate_metadata read hostname 'smithi029' 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556bb4534c90 0x556bb49d82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 4 client.15474 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.174+0000 7fef7c86f740 1 -- 172.21.15.29:0/710074017 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "status"}) -- 0x556bb48bc610 con 0x556bb4534c90 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.175+0000 7fef6ffff640 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556bb4534c90 0x556bb49d82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.175+0000 7fef4f7fe640 10 client.15474 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.175+0000 7fef6ffff640 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556bb4534c90 0x556bb49d82b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=829ecbbf842c3562 server_cookie=28e1c5f55990450d in_seq=0 out_seq=0 2024-09-21T02:38:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.175+0000 7fef4f7fe640 1 -- 172.21.15.29:0/710074017 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556bb48bc610 con 0x556bb4534c90 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.175+0000 7fef4f7fe640 10 client.15474 handle_command_reply: tid=0 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.175+0000 7fef7c86f740 1 client.15474 shutdown 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.175+0000 7fef7c86f740 1 -- 172.21.15.29:0/710074017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 msgr2=0x556bb496c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.175+0000 7fef7c86f740 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb496c430 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fef64015ba0 tx=0x7fef64025040 comp rx=0 tx=0).stop 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 1 -- 172.21.15.29:0/710074017 shutdown_connections 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556bb4534c90 0x556bb49d82b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556bb45a0f60 0x556bb496c430 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bb4580280 0x556bb496beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef4f7fe640 0 client.15474 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 1 --2- 172.21.15.29:0/710074017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556bb45780a0 0x556bb496b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 1 -- 172.21.15.29:0/710074017 >> 172.21.15.29:0/710074017 conn(0x556bb4595350 msgr2=0x556bb49446b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 1 -- 172.21.15.29:0/710074017 shutdown_connections 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 1 -- 172.21.15.29:0/710074017 wait complete. 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef7c86f740 20 client.15474 trim_cache size 0 max 16384 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24637, 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:38:58.189 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:38:58.190 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 151.442149739, 2024-09-21T02:38:58.190 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-09-21T02:38:58.190 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:38:58.190 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:38:58.190 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 171.88241959499999 2024-09-21T02:38:58.190 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 -- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef580b6590 msgr2=0x7fef580b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef580b6590 0x7fef580b8aa0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fef680cb9d0 tx=0x7fef6809bba0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 -- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 msgr2=0x7fef7019fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 0x7fef7019fb30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fef68005cc0 tx=0x7fef68032a10 comp rx=0 tx=0).stop 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 -- 172.21.15.29:0/2286843050 shutdown_connections 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef580b6590 0x7fef580b8aa0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef70108d10 0x7fef7019fb30 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef701083c0 0x7fef7019a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 --2- 172.21.15.29:0/2286843050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef70102a80 0x7fef7019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.176+0000 7fef762fb640 1 -- 172.21.15.29:0/2286843050 >> 172.21.15.29:0/2286843050 conn(0x7fef70079740 msgr2=0x7fef701043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.177+0000 7fef762fb640 1 -- 172.21.15.29:0/2286843050 shutdown_connections 2024-09-21T02:38:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:38:58.177+0000 7fef762fb640 1 -- 172.21.15.29:0/2286843050 wait complete. 2024-09-21T02:38:59.010 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-21T02:38:59.021 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-21T02:38:59.021 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-09-21T02:38:59.021 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-09-21T02:38:59.021 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T02:38:59.021 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T02:38:59.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:38:59.021 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-09-21T02:38:59.039 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-21T02:38:59.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:38:59.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:38:59.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:38:59.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-09-21T02:38:59.201 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-21T02:38:59.202 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-21T02:38:59.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T02:38:59.462 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:38:59.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:38:59 smithi029 ceph-mon[31006]: pgmap v385: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:59.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:38:59 smithi190 ceph-mon[33871]: pgmap v385: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:38:59.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:38:59 smithi066 ceph-mon[33436]: pgmap v385: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:39:00.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 -- 172.21.15.29:0/1386022108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c103b80 msgr2=0x7f9c3c103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:00.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 --2- 172.21.15.29:0/1386022108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c103b80 0x7f9c3c103f90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009900 tx=0x7f9c3402a410 comp rx=0 tx=0).stop 2024-09-21T02:39:00.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 -- 172.21.15.29:0/1386022108 shutdown_connections 2024-09-21T02:39:00.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 --2- 172.21.15.29:0/1386022108 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c3c109e10 0x7f9c3c110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 --2- 172.21.15.29:0/1386022108 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c3c1094c0 0x7f9c3c1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 --2- 172.21.15.29:0/1386022108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c103b80 0x7f9c3c103f90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 -- 172.21.15.29:0/1386022108 >> 172.21.15.29:0/1386022108 conn(0x7f9c3c0ff610 msgr2=0x7f9c3c101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 -- 172.21.15.29:0/1386022108 shutdown_connections 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.023+0000 7f9c40a35640 1 -- 172.21.15.29:0/1386022108 wait complete. 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.024+0000 7f9c40a35640 1 Processor -- start 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.024+0000 7f9c40a35640 1 -- start start 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c40a35640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c3c103b80 0x7f9c3c19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c40a35640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c3c1094c0 0x7f9c3c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c40a35640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c109e10 0x7f9c3c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c40a35640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c113980 con 0x7f9c3c109e10 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c40a35640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c113800 con 0x7f9c3c1094c0 2024-09-21T02:39:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c40a35640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c113b00 con 0x7f9c3c103b80 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c109e10 0x7f9c3c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c109e10 0x7f9c3c19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44628/0 (socket says 172.21.15.29:44628) 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 -- 172.21.15.29:0/3171894366 learned_addr learned my addr 172.21.15.29:0/3171894366 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3a59c640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c3c103b80 0x7f9c3c19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c39d9b640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c3c1094c0 0x7f9c3c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 -- 172.21.15.29:0/3171894366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c3c103b80 msgr2=0x7f9c3c19a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c3c103b80 0x7f9c3c19a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 -- 172.21.15.29:0/3171894366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c3c1094c0 msgr2=0x7f9c3c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c3c1094c0 0x7f9c3c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 -- 172.21.15.29:0/3171894366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c1a0340 con 0x7f9c3c109e10 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c39d9b640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c3c1094c0 0x7f9c3c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.025+0000 7f9c3ad9d640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c109e10 0x7f9c3c19fc10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00eaa0 tx=0x7f9c2c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.026+0000 7f9c337fe640 1 -- 172.21.15.29:0/3171894366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c00cdb0 con 0x7f9c3c109e10 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.026+0000 7f9c337fe640 1 -- 172.21.15.29:0/3171894366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c2c005020 con 0x7f9c3c109e10 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.026+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c1a0630 con 0x7f9c3c109e10 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.026+0000 7f9c337fe640 1 -- 172.21.15.29:0/3171894366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0116d0 con 0x7f9c3c109e10 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.026+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3c1a6d90 con 0x7f9c3c109e10 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.027+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3c104e00 con 0x7f9c3c109e10 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.028+0000 7f9c337fe640 1 -- 172.21.15.29:0/3171894366 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f9c2c0051d0 con 0x7f9c3c109e10 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.029+0000 7f9c337fe640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9c280b6510 0x7f9c280b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.029+0000 7f9c3a59c640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9c280b6510 0x7f9c280b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.030+0000 7f9c337fe640 1 -- 172.21.15.29:0/3171894366 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9c2c0e7d00 con 0x7f9c3c109e10 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.030+0000 7f9c3a59c640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9c280b6510 0x7f9c280b8a20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c34000c00 tx=0x7f9c34009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.032+0000 7f9c337fe640 1 -- 172.21.15.29:0/3171894366 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9c2c016030 con 0x7f9c3c109e10 2024-09-21T02:39:00.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.260+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9c3c10b650 con 0x7f9c3c109e10 2024-09-21T02:39:00.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.262+0000 7f9c337fe640 1 -- 172.21.15.29:0/3171894366 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f9c2c0a9030 con 0x7f9c3c109e10 2024-09-21T02:39:00.263 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-21T02:39:00.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9c280b6510 msgr2=0x7f9c280b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:00.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9c280b6510 0x7f9c280b8a20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c34000c00 tx=0x7f9c34009680 comp rx=0 tx=0).stop 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c109e10 msgr2=0x7f9c3c19fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c109e10 0x7f9c3c19fc10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00eaa0 tx=0x7f9c2c00ef60 comp rx=0 tx=0).stop 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 shutdown_connections 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9c280b6510 0x7f9c280b8a20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c109e10 0x7f9c3c19fc10 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c3c1094c0 0x7f9c3c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 --2- 172.21.15.29:0/3171894366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c3c103b80 0x7f9c3c19a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 >> 172.21.15.29:0/3171894366 conn(0x7f9c3c0ff610 msgr2=0x7f9c3c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 shutdown_connections 2024-09-21T02:39:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:00.264+0000 7f9c40a35640 1 -- 172.21.15.29:0/3171894366 wait complete. 2024-09-21T02:39:01.028 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-21T02:39:01.028 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T02:39:01.028 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-09-21T02:39:01.029 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-21T02:39:01.029 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:01.029 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-21T02:39:01.029 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T02:39:01.029 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T02:39:01.029 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-21T02:39:01.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:01.029 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-09-21T02:39:01.047 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T02:39:01.047 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T02:39:01.047 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-09-21T02:39:01.048 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-21T02:39:01.049 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42027sec preferred_lft 42027sec 2024-09-21T02:39:01.049 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-09-21T02:39:01.049 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:39:01.049 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:39:01.049 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-09-21T02:39:01.049 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-09-21T02:39:01.049 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T02:39:01.049 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:39:01.049 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:39:01.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-09-21T02:39:01.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-09-21T02:39:01.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-09-21T02:39:01.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T02:39:01.050 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:39:01.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:39:01.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:39:01.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:01.196 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T02:39:01.226 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-09-21T02:39:01.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:01.229 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-09-21T02:39:01.290 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-09-21T02:39:01.290 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T02:39:01.290 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-21T02:39:01.290 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-21T02:39:01.291 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T02:39:01.291 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:39:01.291 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:39:01.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-21T02:39:01.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-21T02:39:01.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-21T02:39:01.291 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:39:01.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:39:01.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:39:01.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:01.422 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-09-21T02:39:01.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:01.440 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-09-21T02:39:01.498 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:39:01.498 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:39:01.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:39:01.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-21T02:39:01.498 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:39:01.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:39:01.597 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:01 smithi029 ceph-mon[31006]: pgmap v386: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:39:01.597 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:01 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3171894366' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:39:01.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:39:01.619 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-21T02:39:01.620 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:39:01.620 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:39:01.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-21T02:39:01.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T02:39:01.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-21T02:39:01.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-21T02:39:01.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-21T02:39:01.621 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:39:01.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:01 smithi190 ceph-mon[33871]: pgmap v386: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:39:01.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:01 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3171894366' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:39:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:39:01.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:39:01.739 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:39:01.739 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-09-21T02:39:01.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-09-21T02:39:01.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T02:39:01.739 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-09-21T02:39:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:01 smithi066 ceph-mon[33436]: pgmap v386: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-21T02:39:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:01 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3171894366' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:39:01.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:39:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:39:01.818 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-09-21T02:39:01.818 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T02:39:01.818 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:01.834 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-21T02:39:01.834 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T02:39:01.834 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:01.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-09-21T02:39:01.958 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-09-21T02:39:02.015 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:39:02.016 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/65e94114351b2767a54725ea472225cf3493330a42fa5f3c5371e4195d48759f/merged 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9e18fad01859c31c997a0b7a85bd005aecfa2b6c148bf35f11fb68fe505072b3/merged 2024-09-21T02:39:02.017 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae089362fa72be0b3ed38359504dfbbac148a50d357690d53ba08ac2d4afa09f/merged 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae9c5dd64f8d38c950d8e898c4f62b3c1f190b52b226a3e4ecd6dfbadfbe5a71/merged 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/8b2d054545a75eb0f5b709ecf46b8aa54abe476f2b6c79737325518cb5a40565/merged 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/380069ce9b157bf5c5e4b90386b11fcf43e7f52a57f8e087c88af902e8a81235/merged 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b966f50adb464157d55eca43102cf7570966d672cdf1c05427ac3e97c5702866/merged 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/5fde1bef43735c21d5b405c73400f7aeeadbb1648b9d09d11052e12b8473fe7b/merged 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b2ce62941dd7f4417fd6703a397ae2a8497e5ebbac5686951e2e8c8bed23ed87/merged 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ec963248cbfe4528a5340d7a3068cb9447df56f02903ca4cf26504d73424e9c7/merged 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-09-21T02:39:02.018 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:39:02.019 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:39:02.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:02.019 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-09-21T02:39:02.071 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-21T02:39:02.071 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/2a51c03c-c7d2-43c6-8134-7298b740d3a8 --client_fs=cephfs) 2024-09-21T02:39:02.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-09-21T02:39:02.139 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:39:02.197 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-09-21T02:39:02.197 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-09-21T02:39:02.197 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-09-21T02:39:02.197 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-09-21T02:39:02.197 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-09-21T02:39:02.198 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/65e94114351b2767a54725ea472225cf3493330a42fa5f3c5371e4195d48759f/merged 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9e18fad01859c31c997a0b7a85bd005aecfa2b6c148bf35f11fb68fe505072b3/merged 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-09-21T02:39:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae089362fa72be0b3ed38359504dfbbac148a50d357690d53ba08ac2d4afa09f/merged 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae9c5dd64f8d38c950d8e898c4f62b3c1f190b52b226a3e4ecd6dfbadfbe5a71/merged 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/8b2d054545a75eb0f5b709ecf46b8aa54abe476f2b6c79737325518cb5a40565/merged 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/380069ce9b157bf5c5e4b90386b11fcf43e7f52a57f8e087c88af902e8a81235/merged 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b966f50adb464157d55eca43102cf7570966d672cdf1c05427ac3e97c5702866/merged 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/5fde1bef43735c21d5b405c73400f7aeeadbb1648b9d09d11052e12b8473fe7b/merged 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b2ce62941dd7f4417fd6703a397ae2a8497e5ebbac5686951e2e8c8bed23ed87/merged 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ec963248cbfe4528a5340d7a3068cb9447df56f02903ca4cf26504d73424e9c7/merged 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:39:02.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:02.200 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-09-21T02:39:02.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi029.stderr:2024-09-21T02:39:02.207+0000 7f55b9d1f480 -1 init, newargv = 0x7f559c0046a0 newargc=15 2024-09-21T02:39:02.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi029.stderr:ceph-fuse[85391]: starting ceph client 2024-09-21T02:39:02.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi029.stderr:ceph-fuse[85391]: starting fuse 2024-09-21T02:39:03.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-09-21T02:39:03.358 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:39:03.417 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-09-21T02:39:03.417 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-09-21T02:39:03.417 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:39:03.418 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/65e94114351b2767a54725ea472225cf3493330a42fa5f3c5371e4195d48759f/merged 2024-09-21T02:39:03.419 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9e18fad01859c31c997a0b7a85bd005aecfa2b6c148bf35f11fb68fe505072b3/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae089362fa72be0b3ed38359504dfbbac148a50d357690d53ba08ac2d4afa09f/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ae9c5dd64f8d38c950d8e898c4f62b3c1f190b52b226a3e4ecd6dfbadfbe5a71/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/8b2d054545a75eb0f5b709ecf46b8aa54abe476f2b6c79737325518cb5a40565/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/380069ce9b157bf5c5e4b90386b11fcf43e7f52a57f8e087c88af902e8a81235/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b966f50adb464157d55eca43102cf7570966d672cdf1c05427ac3e97c5702866/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/5fde1bef43735c21d5b405c73400f7aeeadbb1648b9d09d11052e12b8473fe7b/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/b2ce62941dd7f4417fd6703a397ae2a8497e5ebbac5686951e2e8c8bed23ed87/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/ec963248cbfe4528a5340d7a3068cb9447df56f02903ca4cf26504d73424e9c7/merged 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-09-21T02:39:03.420 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:39:03.421 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:39:03.421 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:03.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:03.421 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-09-21T02:39:03.474 INFO:teuthology.orchestra.run.smithi029.stdout:120 2024-09-21T02:39:03.474 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo stdin-killer -- python3 -c ' 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> import glob 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> import re 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> import os 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> import subprocess 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> def _find_admin_socket(client_name): 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> files = glob.glob(asok_path) 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> 2024-09-21T02:39:03.475 DEBUG:teuthology.orchestra.run.smithi029:> # Given a non-glob path, it better be there 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> if "*" not in asok_path: 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> assert(len(files) == 1) 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> return files[0] 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> for f in files: 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> contents = proc_f.read() 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> if mountpoint in contents: 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> return f 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> print(_find_admin_socket("client.0")) 2024-09-21T02:39:03.476 DEBUG:teuthology.orchestra.run.smithi029:> ' 2024-09-21T02:39:03.538 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:03 smithi029 ceph-mon[31006]: pgmap v387: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:39:03.597 INFO:teuthology.orchestra.run.smithi029.stdout:/var/run/ceph/ceph-client.0.85391.asok 2024-09-21T02:39:03.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:39:03.608 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.85391.asok 2024-09-21T02:39:03.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:03.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.85391.asok status 2024-09-21T02:39:03.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:03 smithi190 ceph-mon[33871]: pgmap v387: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:39:03.757 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:39:03.757 INFO:teuthology.orchestra.run.smithi029.stdout: "metadata": { 2024-09-21T02:39:03.757 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_sha1": "1843221895adccfb3b692060251dd6a0183bc410", 2024-09-21T02:39:03.757 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T02:39:03.757 INFO:teuthology.orchestra.run.smithi029.stdout: "entity_id": "0", 2024-09-21T02:39:03.757 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": "smithi029", 2024-09-21T02:39:03.757 INFO:teuthology.orchestra.run.smithi029.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "pid": "85391", 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "root": "/volumes/qa/sv_0/2a51c03c-c7d2-43c6-8134-7298b740d3a8" 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_count": 0, 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_pinned_count": 0, 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 15486, 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "inst": { 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "name": { 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "client", 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "num": 15486 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-09-21T02:39:03.758 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "192.168.0.1:0", 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3610304640 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "192.168.0.1:0", 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3610304640 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "inst_str": "client.15486 192.168.0.1:0/3610304640", 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "addr_str": "192.168.0.1:0/3610304640", 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "inode_count": 5, 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_epoch": 28, 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch": 85, 2024-09-21T02:39:03.759 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch_barrier": 0, 2024-09-21T02:39:03.760 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklisted": false, 2024-09-21T02:39:03.760 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs" 2024-09-21T02:39:03.760 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:39:03.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:03 smithi066 ceph-mon[33436]: pgmap v387: 129 pgs: 129 active+clean; 650 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:39:03.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:03.766 DEBUG:teuthology.orchestra.run.smithi029:> stat --file-system '--printf=%T 2024-09-21T02:39:03.766 DEBUG:teuthology.orchestra.run.smithi029:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:03.783 INFO:teuthology.orchestra.run.smithi029.stdout:fuseblk 2024-09-21T02:39:03.783 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:03.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:39:03.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:03.854 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T02:39:03.875 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T02:39:03.875 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T02:39:03.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-09-21T02:39:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 -- 172.21.15.29:0/2216085671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103620 msgr2=0x7f41dc103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 --2- 172.21.15.29:0/2216085671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103620 0x7f41dc103a30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f41cc009900 tx=0x7f41cc02a410 comp rx=0 tx=0).stop 2024-09-21T02:39:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 -- 172.21.15.29:0/2216085671 shutdown_connections 2024-09-21T02:39:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 --2- 172.21.15.29:0/2216085671 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41dc103fc0 0x7f41dc10a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 --2- 172.21.15.29:0/2216085671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103620 0x7f41dc103a30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 --2- 172.21.15.29:0/2216085671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41dc1025a0 0x7f41dc102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 -- 172.21.15.29:0/2216085671 >> 172.21.15.29:0/2216085671 conn(0x7f41dc0fdee0 msgr2=0x7f41dc100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 -- 172.21.15.29:0/2216085671 shutdown_connections 2024-09-21T02:39:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.008+0000 7f41e10f4640 1 -- 172.21.15.29:0/2216085671 wait complete. 2024-09-21T02:39:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.009+0000 7f41e10f4640 1 Processor -- start 2024-09-21T02:39:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.009+0000 7f41e10f4640 1 -- start start 2024-09-21T02:39:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.009+0000 7f41e10f4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41dc1025a0 0x7f41dc199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.009+0000 7f41e10f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41dc103620 0x7f41dc19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.009+0000 7f41e10f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103fc0 0x7f41dc19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41e10f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc10e090 con 0x7f41dc103fc0 2024-09-21T02:39:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41e10f4640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc10df10 con 0x7f41dc1025a0 2024-09-21T02:39:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41e10f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc10e210 con 0x7f41dc103620 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103fc0 0x7f41dc19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41dad9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41dc1025a0 0x7f41dc199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103fc0 0x7f41dc19b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44640/0 (socket says 172.21.15.29:44640) 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 -- 172.21.15.29:0/340406540 learned_addr learned my addr 172.21.15.29:0/340406540 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41da59c640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41dc103620 0x7f41dc19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 -- 172.21.15.29:0/340406540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41dc103620 msgr2=0x7f41dc19a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41dc103620 0x7f41dc19a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 -- 172.21.15.29:0/340406540 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41dc1025a0 msgr2=0x7f41dc199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41dc1025a0 0x7f41dc199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 -- 172.21.15.29:0/340406540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc19b850 con 0x7f41dc103fc0 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41dad9d640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41dc1025a0 0x7f41dc199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41db59e640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103fc0 0x7f41dc19b0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f41d400ccb0 tx=0x7f41d400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.010+0000 7f41c3fff640 1 -- 172.21.15.29:0/340406540 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4004b90 con 0x7f41dc103fc0 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.011+0000 7f41c3fff640 1 -- 172.21.15.29:0/340406540 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f41d4004d30 con 0x7f41dc103fc0 2024-09-21T02:39:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.011+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc19bb40 con 0x7f41dc103fc0 2024-09-21T02:39:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.011+0000 7f41c3fff640 1 -- 172.21.15.29:0/340406540 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4005290 con 0x7f41dc103fc0 2024-09-21T02:39:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.011+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc1a22c0 con 0x7f41dc103fc0 2024-09-21T02:39:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.012+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41dc1a2490 con 0x7f41dc103fc0 2024-09-21T02:39:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.014+0000 7f41c3fff640 1 -- 172.21.15.29:0/340406540 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f41d400f450 con 0x7f41dc103fc0 2024-09-21T02:39:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.014+0000 7f41c3fff640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41c40b6510 0x7f41c40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.014+0000 7f41c3fff640 1 -- 172.21.15.29:0/340406540 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f41d40e7a10 con 0x7f41dc103fc0 2024-09-21T02:39:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.015+0000 7f41dad9d640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41c40b6510 0x7f41c40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:04.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.015+0000 7f41dad9d640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41c40b6510 0x7f41c40b8a20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f41d000bfd0 tx=0x7f41d000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.017+0000 7f41c3fff640 1 -- 172.21.15.29:0/340406540 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f41d4014070 con 0x7f41dc103fc0 2024-09-21T02:39:04.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.251+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f41dc1a2660 con 0x7f41dc103fc0 2024-09-21T02:39:04.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.276+0000 7f41c3fff640 1 -- 172.21.15.29:0/340406540 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7f41d40b2300 con 0x7f41dc103fc0 2024-09-21T02:39:04.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41c40b6510 msgr2=0x7f41c40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41c40b6510 0x7f41c40b8a20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f41d000bfd0 tx=0x7f41d000b390 comp rx=0 tx=0).stop 2024-09-21T02:39:04.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103fc0 msgr2=0x7f41dc19b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103fc0 0x7f41dc19b0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f41d400ccb0 tx=0x7f41d400a620 comp rx=0 tx=0).stop 2024-09-21T02:39:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 shutdown_connections 2024-09-21T02:39:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41c40b6510 0x7f41c40b8a20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41dc103fc0 0x7f41dc19b0c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41dc103620 0x7f41dc19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 --2- 172.21.15.29:0/340406540 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41dc1025a0 0x7f41dc199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 >> 172.21.15.29:0/340406540 conn(0x7f41dc0fdee0 msgr2=0x7f41dc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 shutdown_connections 2024-09-21T02:39:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.279+0000 7f41e10f4640 1 -- 172.21.15.29:0/340406540 wait complete. 2024-09-21T02:39:04.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.406+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2591394240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 msgr2=0x7fc3f01063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.406+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2591394240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 0x7fc3f01063a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009900 tx=0x7fc3ec02a410 comp rx=0 tx=0).stop 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.406+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2591394240 shutdown_connections 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.406+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2591394240 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3f01093f0 0x7fc3f010b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.406+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2591394240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3f01068e0 0x7fc3f0108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.406+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2591394240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 0x7fc3f01063a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.406+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2591394240 >> 172.21.15.29:0/2591394240 conn(0x7fc3f00fdee0 msgr2=0x7fc3f0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.407+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2591394240 shutdown_connections 2024-09-21T02:39:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.407+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2591394240 wait complete. 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.407+0000 7fc3f7bc3640 1 Processor -- start 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f7bc3640 1 -- start start 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f7bc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 0x7fc3f019c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f7bc3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3f01068e0 0x7fc3f019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f7bc3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3f01093f0 0x7fc3f019d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f7bc3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f010f180 con 0x7fc3f006b650 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f7bc3640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f010f000 con 0x7fc3f01068e0 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f7bc3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f010f300 con 0x7fc3f01093f0 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f595f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 0x7fc3f019c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f595f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 0x7fc3f019c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47934/0 (socket says 172.21.15.29:47934) 2024-09-21T02:39:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f595f640 1 -- 172.21.15.29:0/2444572166 learned_addr learned my addr 172.21.15.29:0/2444572166 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f6160640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3f01093f0 0x7fc3f019d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f515e640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3f01068e0 0x7fc3f019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f595f640 1 -- 172.21.15.29:0/2444572166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3f01093f0 msgr2=0x7fc3f019d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f595f640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3f01093f0 0x7fc3f019d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f595f640 1 -- 172.21.15.29:0/2444572166 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3f01068e0 msgr2=0x7fc3f019c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f595f640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3f01068e0 0x7fc3f019c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f595f640 1 -- 172.21.15.29:0/2444572166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f019dae0 con 0x7fc3f006b650 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.408+0000 7fc3f515e640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3f01068e0 0x7fc3f019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.409+0000 7fc3f595f640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 0x7fc3f019c0a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009230 tx=0x7fc3ec0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.409+0000 7fc3e2ffd640 1 -- 172.21.15.29:0/2444572166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec03f030 con 0x7fc3f006b650 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.409+0000 7fc3e2ffd640 1 -- 172.21.15.29:0/2444572166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3ec0044c0 con 0x7fc3f006b650 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.409+0000 7fc3e2ffd640 1 -- 172.21.15.29:0/2444572166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec035070 con 0x7fc3f006b650 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.409+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f019dd70 con 0x7fc3f006b650 2024-09-21T02:39:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.409+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f01a44c0 con 0x7fc3f006b650 2024-09-21T02:39:04.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.410+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f0106cf0 con 0x7fc3f006b650 2024-09-21T02:39:04.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.411+0000 7fc3e2ffd640 1 -- 172.21.15.29:0/2444572166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7fc3ec042030 con 0x7fc3f006b650 2024-09-21T02:39:04.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.413+0000 7fc3e2ffd640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3dc0b6590 0x7fc3dc0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:04.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.413+0000 7fc3f515e640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3dc0b6590 0x7fc3dc0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:04.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.413+0000 7fc3e2ffd640 1 -- 172.21.15.29:0/2444572166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc3ec107ba0 con 0x7fc3f006b650 2024-09-21T02:39:04.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.413+0000 7fc3f515e640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3dc0b6590 0x7fc3dc0b8aa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009b50 tx=0x7fc3e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.417+0000 7fc3e2ffd640 1 -- 172.21.15.29:0/2444572166 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc3ec0ce0b0 con 0x7fc3f006b650 2024-09-21T02:39:04.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:04 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/340406540' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} : dispatch 2024-09-21T02:39:04.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:04 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/340406540' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} : dispatch 2024-09-21T02:39:04.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.640+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fc3f010b350 con 0x7fc3dc0b6590 2024-09-21T02:39:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.659+0000 7fc3e2ffd640 1 -- 172.21.15.29:0/2444572166 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fc3f010b350 con 0x7fc3dc0b6590 2024-09-21T02:39:04.660 INFO:teuthology.orchestra.run.smithi029.stdout:{} 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3dc0b6590 msgr2=0x7fc3dc0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3dc0b6590 0x7fc3dc0b8aa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009b50 tx=0x7fc3e4005f90 comp rx=0 tx=0).stop 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 msgr2=0x7fc3f019c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 0x7fc3f019c0a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009230 tx=0x7fc3ec0040c0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 shutdown_connections 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc3dc0b6590 0x7fc3dc0b8aa0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3f01093f0 0x7fc3f019d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3f01068e0 0x7fc3f019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 --2- 172.21.15.29:0/2444572166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006b650 0x7fc3f019c0a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:04.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 >> 172.21.15.29:0/2444572166 conn(0x7fc3f00fdee0 msgr2=0x7fc3f0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:04.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 shutdown_connections 2024-09-21T02:39:04.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:04.661+0000 7fc3f7bc3640 1 -- 172.21.15.29:0/2444572166 wait complete. 2024-09-21T02:39:04.674 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-21T02:39:04.685 INFO:teuthology.run_tasks:Running task background_exec... 2024-09-21T02:39:04.741 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T02:39:04.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-09-21T02:39:04.744 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-21T02:39:04.756 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T02:39:04.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:39:04.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:04 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/340406540' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} : dispatch 2024-09-21T02:39:04.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:39:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:39:04.645+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:39:04.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:39:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:39:04.645+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:39:04.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:39:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:39:04.645+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:39:04.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:39:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:39:04.645+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:39:04.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 02:39:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T02:39:04.645+0000 7fab698dc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:39:05.018 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:05 smithi029 ceph-mon[31006]: pgmap v388: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-21T02:39:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:05 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/340406540' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-21T02:39:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:05 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:39:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:05 smithi029 ceph-mon[31006]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 -- 172.21.15.29:0/1500688935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 msgr2=0x7f2814102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 --2- 172.21.15.29:0/1500688935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 0x7f2814102e90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2804009900 tx=0x7f280402a410 comp rx=0 tx=0).stop 2024-09-21T02:39:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 -- 172.21.15.29:0/1500688935 shutdown_connections 2024-09-21T02:39:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 --2- 172.21.15.29:0/1500688935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2814108d10 0x7f281410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 --2- 172.21.15.29:0/1500688935 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28141083c0 0x7f28141087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 --2- 172.21.15.29:0/1500688935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 0x7f2814102e90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 -- 172.21.15.29:0/1500688935 >> 172.21.15.29:0/1500688935 conn(0x7f2814079740 msgr2=0x7f2814079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 -- 172.21.15.29:0/1500688935 shutdown_connections 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 -- 172.21.15.29:0/1500688935 wait complete. 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 Processor -- start 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.555+0000 7f281a3ed640 1 -- start start 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f281a3ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 0x7f281419b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f281a3ed640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28141083c0 0x7f281419b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f281a3ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2814108d10 0x7f281419bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f281a3ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2814112830 con 0x7f2814102a80 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f281a3ed640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f28141126b0 con 0x7f28141083c0 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f281a3ed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f28141129b0 con 0x7f2814108d10 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f2813fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 0x7f281419b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f2813fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 0x7f281419b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47962/0 (socket says 172.21.15.29:47962) 2024-09-21T02:39:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f2813fff640 1 -- 172.21.15.29:0/3116265488 learned_addr learned my addr 172.21.15.29:0/3116265488 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f28137fe640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28141083c0 0x7f281419b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f281898a640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2814108d10 0x7f281419bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f2813fff640 1 -- 172.21.15.29:0/3116265488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2814108d10 msgr2=0x7f281419bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f2813fff640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2814108d10 0x7f281419bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f2813fff640 1 -- 172.21.15.29:0/3116265488 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28141083c0 msgr2=0x7f281419b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f2813fff640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28141083c0 0x7f281419b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f2813fff640 1 -- 172.21.15.29:0/3116265488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28141a63f0 con 0x7f2814102a80 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.556+0000 7f281898a640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2814108d10 0x7f281419bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.557+0000 7f2813fff640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 0x7f281419b160 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2804009230 tx=0x7f28040319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.557+0000 7f28117fa640 1 -- 172.21.15.29:0/3116265488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280403f030 con 0x7f2814102a80 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.557+0000 7f28117fa640 1 -- 172.21.15.29:0/3116265488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28040043d0 con 0x7f2814102a80 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.557+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28141a6680 con 0x7f2814102a80 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.557+0000 7f28117fa640 1 -- 172.21.15.29:0/3116265488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2804035070 con 0x7f2814102a80 2024-09-21T02:39:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.557+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28141a6b60 con 0x7f2814102a80 2024-09-21T02:39:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.558+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2814103d00 con 0x7f2814102a80 2024-09-21T02:39:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.560+0000 7f28117fa640 1 -- 172.21.15.29:0/3116265488 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 154512+0+0 (secure 0 0 0) 0x7f2804042030 con 0x7f2814102a80 2024-09-21T02:39:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.561+0000 7f28117fa640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27fc0b6540 0x7f27fc0b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.561+0000 7f28137fe640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27fc0b6540 0x7f27fc0b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.561+0000 7f28117fa640 1 -- 172.21.15.29:0/3116265488 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2804107ad0 con 0x7f2814102a80 2024-09-21T02:39:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.561+0000 7f28137fe640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27fc0b6540 0x7f27fc0b8a50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2814100d40 tx=0x7f2808005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.565+0000 7f28117fa640 1 -- 172.21.15.29:0/3116265488 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f28040cdfe0 con 0x7f2814102a80 2024-09-21T02:39:05.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:05 smithi190 ceph-mon[33871]: pgmap v388: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-21T02:39:05.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:05 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/340406540' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-21T02:39:05.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:05 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:39:05.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:05 smithi190 ceph-mon[33871]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.754+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2814063800 con 0x7f27fc0b6540 2024-09-21T02:39:05.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:05 smithi066 ceph-mon[33436]: pgmap v388: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-21T02:39:05.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:05 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/340406540' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-21T02:39:05.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:05 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:39:05.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:05 smithi066 ceph-mon[33436]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.761+0000 7f28117fa640 1 -- 172.21.15.29:0/3116265488 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340777 (secure 0 0 0) 0x7f2814063800 con 0x7f27fc0b6540 2024-09-21T02:39:05.766 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:05.768 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T02:39:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27fc0b6540 msgr2=0x7f27fc0b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27fc0b6540 0x7f27fc0b8a50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2814100d40 tx=0x7f2808005f90 comp rx=0 tx=0).stop 2024-09-21T02:39:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 msgr2=0x7f281419b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 0x7f281419b160 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2804009230 tx=0x7f28040319a0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 shutdown_connections 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27fc0b6540 0x7f27fc0b8a50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2814108d10 0x7f281419bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28141083c0 0x7f281419b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 --2- 172.21.15.29:0/3116265488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2814102a80 0x7f281419b160 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 >> 172.21.15.29:0/3116265488 conn(0x7f2814079740 msgr2=0x7f28141014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 shutdown_connections 2024-09-21T02:39:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:05.764+0000 7f281a3ed640 1 -- 172.21.15.29:0/3116265488 wait complete. 2024-09-21T02:39:06.581 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-21T02:39:03.974579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666943,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1240,"num_read_kb":730,"num_write":394,"num_write_kb":2314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2952069,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":824,"ondisk_log_size":824,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369220,"kb_used_data":11600,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572604,"statfs":{"total":1151940427776,"available":1151562346496,"internally_reserved":0,"allocated":11878400,"data_stored":7791358,"data_compressed":463314,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":986,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":71,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003672"},"pg_stats":[{"pgid":"2.1f","version":"85'14","reported_seq":168,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.209564+0000","last_change":"2024-09-21T02:36:23.729337+0000","last_active":"2024-09-21T02:39:02.209564+0000","last_peered":"2024-09-21T02:39:02.209564+0000","last_clean":"2024-09-21T02:39:02.209564+0000","last_became_active":"2024-09-21T02:36:23.728576+0000","last_became_peered":"2024-09-21T02:36:23.728576+0000","last_unstale":"2024-09-21T02:39:02.209564+0000","last_undegraded":"2024-09-21T02:39:02.209564+0000","last_fullsized":"2024-09-21T02:39:02.209564+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:31.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294323+0000","last_change":"2024-09-21T02:36:25.748925+0000","last_active":"2024-09-21T02:38:40.294323+0000","last_peered":"2024-09-21T02:38:40.294323+0000","last_clean":"2024-09-21T02:38:40.294323+0000","last_became_active":"2024-09-21T02:36:25.748333+0000","last_became_peered":"2024-09-21T02:36:25.748333+0000","last_unstale":"2024-09-21T02:38:40.294323+0000","last_undegraded":"2024-09-21T02:38:40.294323+0000","last_fullsized":"2024-09-21T02:38:40.294323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:46.889497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224117+0000","last_change":"2024-09-21T02:36:32.807771+0000","last_active":"2024-09-21T02:38:40.224117+0000","last_peered":"2024-09-21T02:38:40.224117+0000","last_clean":"2024-09-21T02:38:40.224117+0000","last_became_active":"2024-09-21T02:36:32.807291+0000","last_became_peered":"2024-09-21T02:36:32.807291+0000","last_unstale":"2024-09-21T02:38:40.224117+0000","last_undegraded":"2024-09-21T02:38:40.224117+0000","last_fullsized":"2024-09-21T02:38:40.224117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:41.468318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294537+0000","last_change":"2024-09-21T02:36:23.800779+0000","last_active":"2024-09-21T02:38:40.294537+0000","last_peered":"2024-09-21T02:38:40.294537+0000","last_clean":"2024-09-21T02:38:40.294537+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:38:40.294537+0000","last_undegraded":"2024-09-21T02:38:40.294537+0000","last_fullsized":"2024-09-21T02:38:40.294537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:24.843233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232565+0000","last_change":"2024-09-21T02:36:25.745277+0000","last_active":"2024-09-21T02:38:40.232565+0000","last_peered":"2024-09-21T02:38:40.232565+0000","last_clean":"2024-09-21T02:38:40.232565+0000","last_became_active":"2024-09-21T02:36:25.744501+0000","last_became_peered":"2024-09-21T02:36:25.744501+0000","last_unstale":"2024-09-21T02:38:40.232565+0000","last_undegraded":"2024-09-21T02:38:40.232565+0000","last_fullsized":"2024-09-21T02:38:40.232565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:14.776250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502659+0000","last_change":"2024-09-21T02:36:32.801978+0000","last_active":"2024-09-21T02:38:40.502659+0000","last_peered":"2024-09-21T02:38:40.502659+0000","last_clean":"2024-09-21T02:38:40.502659+0000","last_became_active":"2024-09-21T02:36:32.801436+0000","last_became_peered":"2024-09-21T02:36:32.801436+0000","last_unstale":"2024-09-21T02:38:40.502659+0000","last_undegraded":"2024-09-21T02:38:40.502659+0000","last_fullsized":"2024-09-21T02:38:40.502659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:37.380417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503616+0000","last_change":"2024-09-21T02:36:23.729336+0000","last_active":"2024-09-21T02:38:40.503616+0000","last_peered":"2024-09-21T02:38:40.503616+0000","last_clean":"2024-09-21T02:38:40.503616+0000","last_became_active":"2024-09-21T02:36:23.728724+0000","last_became_peered":"2024-09-21T02:36:23.728724+0000","last_unstale":"2024-09-21T02:38:40.503616+0000","last_undegraded":"2024-09-21T02:38:40.503616+0000","last_fullsized":"2024-09-21T02:38:40.503616+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:31:52.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228248+0000","last_change":"2024-09-21T02:36:25.737577+0000","last_active":"2024-09-21T02:38:40.228248+0000","last_peered":"2024-09-21T02:38:40.228248+0000","last_clean":"2024-09-21T02:38:40.228248+0000","last_became_active":"2024-09-21T02:36:25.737221+0000","last_became_peered":"2024-09-21T02:36:25.737221+0000","last_unstale":"2024-09-21T02:38:40.228248+0000","last_undegraded":"2024-09-21T02:38:40.228248+0000","last_fullsized":"2024-09-21T02:38:40.228248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:42:54.786464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504151+0000","last_change":"2024-09-21T02:36:32.802507+0000","last_active":"2024-09-21T02:38:40.504151+0000","last_peered":"2024-09-21T02:38:40.504151+0000","last_clean":"2024-09-21T02:38:40.504151+0000","last_became_active":"2024-09-21T02:36:32.801879+0000","last_became_peered":"2024-09-21T02:36:32.801879+0000","last_unstale":"2024-09-21T02:38:40.504151+0000","last_undegraded":"2024-09-21T02:38:40.504151+0000","last_fullsized":"2024-09-21T02:38:40.504151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:09.873825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503447+0000","last_change":"2024-09-21T02:36:23.730889+0000","last_active":"2024-09-21T02:38:40.503447+0000","last_peered":"2024-09-21T02:38:40.503447+0000","last_clean":"2024-09-21T02:38:40.503447+0000","last_became_active":"2024-09-21T02:36:23.730551+0000","last_became_peered":"2024-09-21T02:36:23.730551+0000","last_unstale":"2024-09-21T02:38:40.503447+0000","last_undegraded":"2024-09-21T02:38:40.503447+0000","last_fullsized":"2024-09-21T02:38:40.503447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:17.178848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504430+0000","last_change":"2024-09-21T02:37:39.869272+0000","last_active":"2024-09-21T02:38:40.504430+0000","last_peered":"2024-09-21T02:38:40.504430+0000","last_clean":"2024-09-21T02:38:40.504430+0000","last_became_active":"2024-09-21T02:37:39.868824+0000","last_became_peered":"2024-09-21T02:37:39.868824+0000","last_unstale":"2024-09-21T02:38:40.504430+0000","last_undegraded":"2024-09-21T02:38:40.504430+0000","last_fullsized":"2024-09-21T02:38:40.504430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:05:31.503359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503343+0000","last_change":"2024-09-21T02:36:32.800489+0000","last_active":"2024-09-21T02:38:40.503343+0000","last_peered":"2024-09-21T02:38:40.503343+0000","last_clean":"2024-09-21T02:38:40.503343+0000","last_became_active":"2024-09-21T02:36:32.799876+0000","last_became_peered":"2024-09-21T02:36:32.799876+0000","last_unstale":"2024-09-21T02:38:40.503343+0000","last_undegraded":"2024-09-21T02:38:40.503343+0000","last_fullsized":"2024-09-21T02:38:40.503343+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:02:56.711257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502829+0000","last_change":"2024-09-21T02:36:25.744561+0000","last_active":"2024-09-21T02:38:40.502829+0000","last_peered":"2024-09-21T02:38:40.502829+0000","last_clean":"2024-09-21T02:38:40.502829+0000","last_became_active":"2024-09-21T02:36:25.743757+0000","last_became_peered":"2024-09-21T02:36:25.743757+0000","last_unstale":"2024-09-21T02:38:40.502829+0000","last_undegraded":"2024-09-21T02:38:40.502829+0000","last_fullsized":"2024-09-21T02:38:40.502829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:30.235517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503205+0000","last_change":"2024-09-21T02:37:39.616628+0000","last_active":"2024-09-21T02:38:40.503205+0000","last_peered":"2024-09-21T02:38:40.503205+0000","last_clean":"2024-09-21T02:38:40.503205+0000","last_became_active":"2024-09-21T02:37:39.609108+0000","last_became_peered":"2024-09-21T02:37:39.609108+0000","last_unstale":"2024-09-21T02:38:40.503205+0000","last_undegraded":"2024-09-21T02:38:40.503205+0000","last_fullsized":"2024-09-21T02:38:40.503205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:11:28.944244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504031+0000","last_change":"2024-09-21T02:36:33.003670+0000","last_active":"2024-09-21T02:38:40.504031+0000","last_peered":"2024-09-21T02:38:40.504031+0000","last_clean":"2024-09-21T02:38:40.504031+0000","last_became_active":"2024-09-21T02:36:33.003022+0000","last_became_peered":"2024-09-21T02:36:33.003022+0000","last_unstale":"2024-09-21T02:38:40.504031+0000","last_undegraded":"2024-09-21T02:38:40.504031+0000","last_fullsized":"2024-09-21T02:38:40.504031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:12.753375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294239+0000","last_change":"2024-09-21T02:36:25.749404+0000","last_active":"2024-09-21T02:38:40.294239+0000","last_peered":"2024-09-21T02:38:40.294239+0000","last_clean":"2024-09-21T02:38:40.294239+0000","last_became_active":"2024-09-21T02:36:25.747854+0000","last_became_peered":"2024-09-21T02:36:25.747854+0000","last_unstale":"2024-09-21T02:38:40.294239+0000","last_undegraded":"2024-09-21T02:38:40.294239+0000","last_fullsized":"2024-09-21T02:38:40.294239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:41.224740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224160+0000","last_change":"2024-09-21T02:36:23.726689+0000","last_active":"2024-09-21T02:38:40.224160+0000","last_peered":"2024-09-21T02:38:40.224160+0000","last_clean":"2024-09-21T02:38:40.224160+0000","last_became_active":"2024-09-21T02:36:23.726165+0000","last_became_peered":"2024-09-21T02:36:23.726165+0000","last_unstale":"2024-09-21T02:38:40.224160+0000","last_undegraded":"2024-09-21T02:38:40.224160+0000","last_fullsized":"2024-09-21T02:38:40.224160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:58.166457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503916+0000","last_change":"2024-09-21T02:36:32.809594+0000","last_active":"2024-09-21T02:38:40.503916+0000","last_peered":"2024-09-21T02:38:40.503916+0000","last_clean":"2024-09-21T02:38:40.503916+0000","last_became_active":"2024-09-21T02:36:32.808893+0000","last_became_peered":"2024-09-21T02:36:32.808893+0000","last_unstale":"2024-09-21T02:38:40.503916+0000","last_undegraded":"2024-09-21T02:38:40.503916+0000","last_fullsized":"2024-09-21T02:38:40.503916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:55.187241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331726+0000","last_change":"2024-09-21T02:36:25.755955+0000","last_active":"2024-09-21T02:38:40.331726+0000","last_peered":"2024-09-21T02:38:40.331726+0000","last_clean":"2024-09-21T02:38:40.331726+0000","last_became_active":"2024-09-21T02:36:25.755577+0000","last_became_peered":"2024-09-21T02:36:25.755577+0000","last_unstale":"2024-09-21T02:38:40.331726+0000","last_undegraded":"2024-09-21T02:38:40.331726+0000","last_fullsized":"2024-09-21T02:38:40.331726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:48.447758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294101+0000","last_change":"2024-09-21T02:36:23.730043+0000","last_active":"2024-09-21T02:38:40.294101+0000","last_peered":"2024-09-21T02:38:40.294101+0000","last_clean":"2024-09-21T02:38:40.294101+0000","last_became_active":"2024-09-21T02:36:23.729576+0000","last_became_peered":"2024-09-21T02:36:23.729576+0000","last_unstale":"2024-09-21T02:38:40.294101+0000","last_undegraded":"2024-09-21T02:38:40.294101+0000","last_fullsized":"2024-09-21T02:38:40.294101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:48:09.530913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503449+0000","last_change":"2024-09-21T02:36:32.806923+0000","last_active":"2024-09-21T02:38:40.503449+0000","last_peered":"2024-09-21T02:38:40.503449+0000","last_clean":"2024-09-21T02:38:40.503449+0000","last_became_active":"2024-09-21T02:36:32.806176+0000","last_became_peered":"2024-09-21T02:36:32.806176+0000","last_unstale":"2024-09-21T02:38:40.503449+0000","last_undegraded":"2024-09-21T02:38:40.503449+0000","last_fullsized":"2024-09-21T02:38:40.503449+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.165638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502828+0000","last_change":"2024-09-21T02:36:25.747501+0000","last_active":"2024-09-21T02:38:40.502828+0000","last_peered":"2024-09-21T02:38:40.502828+0000","last_clean":"2024-09-21T02:38:40.502828+0000","last_became_active":"2024-09-21T02:36:25.747111+0000","last_became_peered":"2024-09-21T02:36:25.747111+0000","last_unstale":"2024-09-21T02:38:40.502828+0000","last_undegraded":"2024-09-21T02:38:40.502828+0000","last_fullsized":"2024-09-21T02:38:40.502828+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:03:04.061537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.335704+0000","last_change":"2024-09-21T02:38:40.335626+0000","last_active":"2024-09-21T02:38:40.335704+0000","last_peered":"2024-09-21T02:38:40.335704+0000","last_clean":"2024-09-21T02:38:40.335704+0000","last_became_active":"2024-09-21T02:38:40.332735+0000","last_became_peered":"2024-09-21T02:38:40.332735+0000","last_unstale":"2024-09-21T02:38:40.335704+0000","last_undegraded":"2024-09-21T02:38:40.335704+0000","last_fullsized":"2024-09-21T02:38:40.335704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:42:37.763918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504172+0000","last_change":"2024-09-21T02:37:39.870041+0000","last_active":"2024-09-21T02:38:40.504172+0000","last_peered":"2024-09-21T02:38:40.504172+0000","last_clean":"2024-09-21T02:38:40.504172+0000","last_became_active":"2024-09-21T02:37:39.869518+0000","last_became_peered":"2024-09-21T02:37:39.869518+0000","last_unstale":"2024-09-21T02:38:40.504172+0000","last_undegraded":"2024-09-21T02:38:40.504172+0000","last_fullsized":"2024-09-21T02:38:40.504172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:20.276471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233047+0000","last_change":"2024-09-21T02:36:25.746139+0000","last_active":"2024-09-21T02:38:40.233047+0000","last_peered":"2024-09-21T02:38:40.233047+0000","last_clean":"2024-09-21T02:38:40.233047+0000","last_became_active":"2024-09-21T02:36:25.744418+0000","last_became_peered":"2024-09-21T02:36:25.744418+0000","last_unstale":"2024-09-21T02:38:40.233047+0000","last_undegraded":"2024-09-21T02:38:40.233047+0000","last_fullsized":"2024-09-21T02:38:40.233047+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:21.706782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503137+0000","last_change":"2024-09-21T02:36:24.261945+0000","last_active":"2024-09-21T02:38:40.503137+0000","last_peered":"2024-09-21T02:38:40.503137+0000","last_clean":"2024-09-21T02:38:40.503137+0000","last_became_active":"2024-09-21T02:36:24.261486+0000","last_became_peered":"2024-09-21T02:36:24.261486+0000","last_unstale":"2024-09-21T02:38:40.503137+0000","last_undegraded":"2024-09-21T02:38:40.503137+0000","last_fullsized":"2024-09-21T02:38:40.503137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:45:18.331035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224142+0000","last_change":"2024-09-21T02:36:32.799464+0000","last_active":"2024-09-21T02:38:40.224142+0000","last_peered":"2024-09-21T02:38:40.224142+0000","last_clean":"2024-09-21T02:38:40.224142+0000","last_became_active":"2024-09-21T02:36:32.799000+0000","last_became_peered":"2024-09-21T02:36:32.799000+0000","last_unstale":"2024-09-21T02:38:40.224142+0000","last_undegraded":"2024-09-21T02:38:40.224142+0000","last_fullsized":"2024-09-21T02:38:40.224142+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:56:36.271172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294464+0000","last_change":"2024-09-21T02:36:25.741151+0000","last_active":"2024-09-21T02:38:40.294464+0000","last_peered":"2024-09-21T02:38:40.294464+0000","last_clean":"2024-09-21T02:38:40.294464+0000","last_became_active":"2024-09-21T02:36:25.740803+0000","last_became_peered":"2024-09-21T02:36:25.740803+0000","last_unstale":"2024-09-21T02:38:40.294464+0000","last_undegraded":"2024-09-21T02:38:40.294464+0000","last_fullsized":"2024-09-21T02:38:40.294464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:44.585136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'18","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503589+0000","last_change":"2024-09-21T02:36:23.725155+0000","last_active":"2024-09-21T02:38:40.503589+0000","last_peered":"2024-09-21T02:38:40.503589+0000","last_clean":"2024-09-21T02:38:40.503589+0000","last_became_active":"2024-09-21T02:36:23.724791+0000","last_became_peered":"2024-09-21T02:36:23.724791+0000","last_unstale":"2024-09-21T02:38:40.503589+0000","last_undegraded":"2024-09-21T02:38:40.503589+0000","last_fullsized":"2024-09-21T02:38:40.503589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:06.520465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293747+0000","last_change":"2024-09-21T02:36:33.003401+0000","last_active":"2024-09-21T02:38:40.293747+0000","last_peered":"2024-09-21T02:38:40.293747+0000","last_clean":"2024-09-21T02:38:40.293747+0000","last_became_active":"2024-09-21T02:36:33.002930+0000","last_became_peered":"2024-09-21T02:36:33.002930+0000","last_unstale":"2024-09-21T02:38:40.293747+0000","last_undegraded":"2024-09-21T02:38:40.293747+0000","last_fullsized":"2024-09-21T02:38:40.293747+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:40:23.629763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222161+0000","last_change":"2024-09-21T02:36:25.735501+0000","last_active":"2024-09-21T02:38:40.222161+0000","last_peered":"2024-09-21T02:38:40.222161+0000","last_clean":"2024-09-21T02:38:40.222161+0000","last_became_active":"2024-09-21T02:36:25.734453+0000","last_became_peered":"2024-09-21T02:36:25.734453+0000","last_unstale":"2024-09-21T02:38:40.222161+0000","last_undegraded":"2024-09-21T02:38:40.222161+0000","last_fullsized":"2024-09-21T02:38:40.222161+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:50.238529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503639+0000","last_change":"2024-09-21T02:36:23.800338+0000","last_active":"2024-09-21T02:38:40.503639+0000","last_peered":"2024-09-21T02:38:40.503639+0000","last_clean":"2024-09-21T02:38:40.503639+0000","last_became_active":"2024-09-21T02:36:23.799796+0000","last_became_peered":"2024-09-21T02:36:23.799796+0000","last_unstale":"2024-09-21T02:38:40.503639+0000","last_undegraded":"2024-09-21T02:38:40.503639+0000","last_fullsized":"2024-09-21T02:38:40.503639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:00:18.459515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331347+0000","last_change":"2024-09-21T02:37:39.611513+0000","last_active":"2024-09-21T02:38:40.331347+0000","last_peered":"2024-09-21T02:38:40.331347+0000","last_clean":"2024-09-21T02:38:40.331347+0000","last_became_active":"2024-09-21T02:37:39.610843+0000","last_became_peered":"2024-09-21T02:37:39.610843+0000","last_unstale":"2024-09-21T02:38:40.331347+0000","last_undegraded":"2024-09-21T02:38:40.331347+0000","last_fullsized":"2024-09-21T02:38:40.331347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:51.911830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223967+0000","last_change":"2024-09-21T02:36:25.745497+0000","last_active":"2024-09-21T02:38:40.223967+0000","last_peered":"2024-09-21T02:38:40.223967+0000","last_clean":"2024-09-21T02:38:40.223967+0000","last_became_active":"2024-09-21T02:36:25.745126+0000","last_became_peered":"2024-09-21T02:36:25.745126+0000","last_unstale":"2024-09-21T02:38:40.223967+0000","last_undegraded":"2024-09-21T02:38:40.223967+0000","last_fullsized":"2024-09-21T02:38:40.223967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:42:17.835784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503417+0000","last_change":"2024-09-21T02:36:23.802440+0000","last_active":"2024-09-21T02:38:40.503417+0000","last_peered":"2024-09-21T02:38:40.503417+0000","last_clean":"2024-09-21T02:38:40.503417+0000","last_became_active":"2024-09-21T02:36:23.802065+0000","last_became_peered":"2024-09-21T02:36:23.802065+0000","last_unstale":"2024-09-21T02:38:40.503417+0000","last_undegraded":"2024-09-21T02:38:40.503417+0000","last_fullsized":"2024-09-21T02:38:40.503417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:22:20.647476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502936+0000","last_change":"2024-09-21T02:36:33.003465+0000","last_active":"2024-09-21T02:38:40.502936+0000","last_peered":"2024-09-21T02:38:40.502936+0000","last_clean":"2024-09-21T02:38:40.502936+0000","last_became_active":"2024-09-21T02:36:33.002874+0000","last_became_peered":"2024-09-21T02:36:33.002874+0000","last_unstale":"2024-09-21T02:38:40.502936+0000","last_undegraded":"2024-09-21T02:38:40.502936+0000","last_fullsized":"2024-09-21T02:38:40.502936+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:02.388770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224086+0000","last_change":"2024-09-21T02:36:25.746092+0000","last_active":"2024-09-21T02:38:40.224086+0000","last_peered":"2024-09-21T02:38:40.224086+0000","last_clean":"2024-09-21T02:38:40.224086+0000","last_became_active":"2024-09-21T02:36:25.745730+0000","last_became_peered":"2024-09-21T02:36:25.745730+0000","last_unstale":"2024-09-21T02:38:40.224086+0000","last_undegraded":"2024-09-21T02:38:40.224086+0000","last_fullsized":"2024-09-21T02:38:40.224086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:24:13.212450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":156,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:58.204241+0000","last_change":"2024-09-21T02:36:23.732090+0000","last_active":"2024-09-21T02:38:58.204241+0000","last_peered":"2024-09-21T02:38:58.204241+0000","last_clean":"2024-09-21T02:38:58.204241+0000","last_became_active":"2024-09-21T02:36:23.731653+0000","last_became_peered":"2024-09-21T02:36:23.731653+0000","last_unstale":"2024-09-21T02:38:58.204241+0000","last_undegraded":"2024-09-21T02:38:58.204241+0000","last_fullsized":"2024-09-21T02:38:58.204241+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:02:42.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222021+0000","last_change":"2024-09-21T02:36:32.794893+0000","last_active":"2024-09-21T02:38:40.222021+0000","last_peered":"2024-09-21T02:38:40.222021+0000","last_clean":"2024-09-21T02:38:40.222021+0000","last_became_active":"2024-09-21T02:36:32.794406+0000","last_became_peered":"2024-09-21T02:36:32.794406+0000","last_unstale":"2024-09-21T02:38:40.222021+0000","last_undegraded":"2024-09-21T02:38:40.222021+0000","last_fullsized":"2024-09-21T02:38:40.222021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:40:39.621502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504064+0000","last_change":"2024-09-21T02:36:25.748220+0000","last_active":"2024-09-21T02:38:40.504064+0000","last_peered":"2024-09-21T02:38:40.504064+0000","last_clean":"2024-09-21T02:38:40.504064+0000","last_became_active":"2024-09-21T02:36:25.747553+0000","last_became_peered":"2024-09-21T02:36:25.747553+0000","last_unstale":"2024-09-21T02:38:40.504064+0000","last_undegraded":"2024-09-21T02:38:40.504064+0000","last_fullsized":"2024-09-21T02:38:40.504064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:28.757981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:57.202092+0000","last_change":"2024-09-21T02:36:23.801087+0000","last_active":"2024-09-21T02:38:57.202092+0000","last_peered":"2024-09-21T02:38:57.202092+0000","last_clean":"2024-09-21T02:38:57.202092+0000","last_became_active":"2024-09-21T02:36:23.800077+0000","last_became_peered":"2024-09-21T02:36:23.800077+0000","last_unstale":"2024-09-21T02:38:57.202092+0000","last_undegraded":"2024-09-21T02:38:57.202092+0000","last_fullsized":"2024-09-21T02:38:57.202092+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:50.074650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222239+0000","last_change":"2024-09-21T02:36:33.003354+0000","last_active":"2024-09-21T02:38:40.222239+0000","last_peered":"2024-09-21T02:38:40.222239+0000","last_clean":"2024-09-21T02:38:40.222239+0000","last_became_active":"2024-09-21T02:36:33.002912+0000","last_became_peered":"2024-09-21T02:36:33.002912+0000","last_unstale":"2024-09-21T02:38:40.222239+0000","last_undegraded":"2024-09-21T02:38:40.222239+0000","last_fullsized":"2024-09-21T02:38:40.222239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:21.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221848+0000","last_change":"2024-09-21T02:36:25.741701+0000","last_active":"2024-09-21T02:38:40.221848+0000","last_peered":"2024-09-21T02:38:40.221848+0000","last_clean":"2024-09-21T02:38:40.221848+0000","last_became_active":"2024-09-21T02:36:25.741222+0000","last_became_peered":"2024-09-21T02:36:25.741222+0000","last_unstale":"2024-09-21T02:38:40.221848+0000","last_undegraded":"2024-09-21T02:38:40.221848+0000","last_fullsized":"2024-09-21T02:38:40.221848+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:35:39.896852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502899+0000","last_change":"2024-09-21T02:36:23.728793+0000","last_active":"2024-09-21T02:38:40.502899+0000","last_peered":"2024-09-21T02:38:40.502899+0000","last_clean":"2024-09-21T02:38:40.502899+0000","last_became_active":"2024-09-21T02:36:23.728362+0000","last_became_peered":"2024-09-21T02:36:23.728362+0000","last_unstale":"2024-09-21T02:38:40.502899+0000","last_undegraded":"2024-09-21T02:38:40.502899+0000","last_fullsized":"2024-09-21T02:38:40.502899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:28:15.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504244+0000","last_change":"2024-09-21T02:36:33.005225+0000","last_active":"2024-09-21T02:38:40.504244+0000","last_peered":"2024-09-21T02:38:40.504244+0000","last_clean":"2024-09-21T02:38:40.504244+0000","last_became_active":"2024-09-21T02:36:33.004756+0000","last_became_peered":"2024-09-21T02:36:33.004756+0000","last_unstale":"2024-09-21T02:38:40.504244+0000","last_undegraded":"2024-09-21T02:38:40.504244+0000","last_fullsized":"2024-09-21T02:38:40.504244+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:02:16.686284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294112+0000","last_change":"2024-09-21T02:36:25.745639+0000","last_active":"2024-09-21T02:38:40.294112+0000","last_peered":"2024-09-21T02:38:40.294112+0000","last_clean":"2024-09-21T02:38:40.294112+0000","last_became_active":"2024-09-21T02:36:25.745202+0000","last_became_peered":"2024-09-21T02:36:25.745202+0000","last_unstale":"2024-09-21T02:38:40.294112+0000","last_undegraded":"2024-09-21T02:38:40.294112+0000","last_fullsized":"2024-09-21T02:38:40.294112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:08.712477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504086+0000","last_change":"2024-09-21T02:36:23.730710+0000","last_active":"2024-09-21T02:38:40.504086+0000","last_peered":"2024-09-21T02:38:40.504086+0000","last_clean":"2024-09-21T02:38:40.504086+0000","last_became_active":"2024-09-21T02:36:23.730247+0000","last_became_peered":"2024-09-21T02:36:23.730247+0000","last_unstale":"2024-09-21T02:38:40.504086+0000","last_undegraded":"2024-09-21T02:38:40.504086+0000","last_fullsized":"2024-09-21T02:38:40.504086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:43.863444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":113,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503097+0000","last_change":"2024-09-21T02:35:09.949841+0000","last_active":"2024-09-21T02:38:40.503097+0000","last_peered":"2024-09-21T02:38:40.503097+0000","last_clean":"2024-09-21T02:38:40.503097+0000","last_became_active":"2024-09-21T02:35:09.944224+0000","last_became_peered":"2024-09-21T02:35:09.944224+0000","last_unstale":"2024-09-21T02:38:40.503097+0000","last_undegraded":"2024-09-21T02:38:40.503097+0000","last_fullsized":"2024-09-21T02:38:40.503097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_clean_scrub_stamp":"2024-09-21T02:31:38.930125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:56.467576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503366+0000","last_change":"2024-09-21T02:37:39.874486+0000","last_active":"2024-09-21T02:38:40.503366+0000","last_peered":"2024-09-21T02:38:40.503366+0000","last_clean":"2024-09-21T02:38:40.503366+0000","last_became_active":"2024-09-21T02:37:39.869523+0000","last_became_peered":"2024-09-21T02:37:39.869523+0000","last_unstale":"2024-09-21T02:38:40.503366+0000","last_undegraded":"2024-09-21T02:38:40.503366+0000","last_fullsized":"2024-09-21T02:38:40.503366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:21:33.524428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503759+0000","last_change":"2024-09-21T02:36:25.741176+0000","last_active":"2024-09-21T02:38:40.503759+0000","last_peered":"2024-09-21T02:38:40.503759+0000","last_clean":"2024-09-21T02:38:40.503759+0000","last_became_active":"2024-09-21T02:36:25.740355+0000","last_became_peered":"2024-09-21T02:36:25.740355+0000","last_unstale":"2024-09-21T02:38:40.503759+0000","last_undegraded":"2024-09-21T02:38:40.503759+0000","last_fullsized":"2024-09-21T02:38:40.503759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:47:48.204768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'8","reported_seq":166,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.214476+0000","last_change":"2024-09-21T02:36:24.261500+0000","last_active":"2024-09-21T02:39:02.214476+0000","last_peered":"2024-09-21T02:39:02.214476+0000","last_clean":"2024-09-21T02:39:02.214476+0000","last_became_active":"2024-09-21T02:36:24.261089+0000","last_became_peered":"2024-09-21T02:36:24.261089+0000","last_unstale":"2024-09-21T02:39:02.214476+0000","last_undegraded":"2024-09-21T02:39:02.214476+0000","last_fullsized":"2024-09-21T02:39:02.214476+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:33:33.158177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2195,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227148+0000","last_change":"2024-09-21T02:36:32.796980+0000","last_active":"2024-09-21T02:38:40.227148+0000","last_peered":"2024-09-21T02:38:40.227148+0000","last_clean":"2024-09-21T02:38:40.227148+0000","last_became_active":"2024-09-21T02:36:32.796517+0000","last_became_peered":"2024-09-21T02:36:32.796517+0000","last_unstale":"2024-09-21T02:38:40.227148+0000","last_undegraded":"2024-09-21T02:38:40.227148+0000","last_fullsized":"2024-09-21T02:38:40.227148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:07.230177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502998+0000","last_change":"2024-09-21T02:36:25.750931+0000","last_active":"2024-09-21T02:38:40.502998+0000","last_peered":"2024-09-21T02:38:40.502998+0000","last_clean":"2024-09-21T02:38:40.502998+0000","last_became_active":"2024-09-21T02:36:25.750604+0000","last_became_peered":"2024-09-21T02:36:25.750604+0000","last_unstale":"2024-09-21T02:38:40.502998+0000","last_undegraded":"2024-09-21T02:38:40.502998+0000","last_fullsized":"2024-09-21T02:38:40.502998+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:21.325095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504562+0000","last_change":"2024-09-21T02:36:23.730700+0000","last_active":"2024-09-21T02:38:40.504562+0000","last_peered":"2024-09-21T02:38:40.504562+0000","last_clean":"2024-09-21T02:38:40.504562+0000","last_became_active":"2024-09-21T02:36:23.730367+0000","last_became_peered":"2024-09-21T02:36:23.730367+0000","last_unstale":"2024-09-21T02:38:40.504562+0000","last_undegraded":"2024-09-21T02:38:40.504562+0000","last_fullsized":"2024-09-21T02:38:40.504562+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:27:57.373077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233491+0000","last_change":"2024-09-21T02:36:32.805885+0000","last_active":"2024-09-21T02:38:40.233491+0000","last_peered":"2024-09-21T02:38:40.233491+0000","last_clean":"2024-09-21T02:38:40.233491+0000","last_became_active":"2024-09-21T02:36:32.805479+0000","last_became_peered":"2024-09-21T02:36:32.805479+0000","last_unstale":"2024-09-21T02:38:40.233491+0000","last_undegraded":"2024-09-21T02:38:40.233491+0000","last_fullsized":"2024-09-21T02:38:40.233491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:11:29.910642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503631+0000","last_change":"2024-09-21T02:36:25.742542+0000","last_active":"2024-09-21T02:38:40.503631+0000","last_peered":"2024-09-21T02:38:40.503631+0000","last_clean":"2024-09-21T02:38:40.503631+0000","last_became_active":"2024-09-21T02:36:25.742000+0000","last_became_peered":"2024-09-21T02:36:25.742000+0000","last_unstale":"2024-09-21T02:38:40.503631+0000","last_undegraded":"2024-09-21T02:38:40.503631+0000","last_fullsized":"2024-09-21T02:38:40.503631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:52.300212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222446+0000","last_change":"2024-09-21T02:36:23.723896+0000","last_active":"2024-09-21T02:38:40.222446+0000","last_peered":"2024-09-21T02:38:40.222446+0000","last_clean":"2024-09-21T02:38:40.222446+0000","last_became_active":"2024-09-21T02:36:23.723518+0000","last_became_peered":"2024-09-21T02:36:23.723518+0000","last_unstale":"2024-09-21T02:38:40.222446+0000","last_undegraded":"2024-09-21T02:38:40.222446+0000","last_fullsized":"2024-09-21T02:38:40.222446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:37.250247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503797+0000","last_change":"2024-09-21T02:36:32.799924+0000","last_active":"2024-09-21T02:38:40.503797+0000","last_peered":"2024-09-21T02:38:40.503797+0000","last_clean":"2024-09-21T02:38:40.503797+0000","last_became_active":"2024-09-21T02:36:32.799437+0000","last_became_peered":"2024-09-21T02:36:32.799437+0000","last_unstale":"2024-09-21T02:38:40.503797+0000","last_undegraded":"2024-09-21T02:38:40.503797+0000","last_fullsized":"2024-09-21T02:38:40.503797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:19.258293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331925+0000","last_change":"2024-09-21T02:36:25.747321+0000","last_active":"2024-09-21T02:38:40.331925+0000","last_peered":"2024-09-21T02:38:40.331925+0000","last_clean":"2024-09-21T02:38:40.331925+0000","last_became_active":"2024-09-21T02:36:25.746955+0000","last_became_peered":"2024-09-21T02:36:25.746955+0000","last_unstale":"2024-09-21T02:38:40.331925+0000","last_undegraded":"2024-09-21T02:38:40.331925+0000","last_fullsized":"2024-09-21T02:38:40.331925+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:25.148300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503704+0000","last_change":"2024-09-21T02:36:23.725350+0000","last_active":"2024-09-21T02:38:40.503704+0000","last_peered":"2024-09-21T02:38:40.503704+0000","last_clean":"2024-09-21T02:38:40.503704+0000","last_became_active":"2024-09-21T02:36:23.725090+0000","last_became_peered":"2024-09-21T02:36:23.725090+0000","last_unstale":"2024-09-21T02:38:40.503704+0000","last_undegraded":"2024-09-21T02:38:40.503704+0000","last_fullsized":"2024-09-21T02:38:40.503704+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:52.130345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503689+0000","last_change":"2024-09-21T02:36:32.801355+0000","last_active":"2024-09-21T02:38:40.503689+0000","last_peered":"2024-09-21T02:38:40.503689+0000","last_clean":"2024-09-21T02:38:40.503689+0000","last_became_active":"2024-09-21T02:36:32.800803+0000","last_became_peered":"2024-09-21T02:36:32.800803+0000","last_unstale":"2024-09-21T02:38:40.503689+0000","last_undegraded":"2024-09-21T02:38:40.503689+0000","last_fullsized":"2024-09-21T02:38:40.503689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:54.755733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504077+0000","last_change":"2024-09-21T02:36:25.743433+0000","last_active":"2024-09-21T02:38:40.504077+0000","last_peered":"2024-09-21T02:38:40.504077+0000","last_clean":"2024-09-21T02:38:40.504077+0000","last_became_active":"2024-09-21T02:36:25.742643+0000","last_became_peered":"2024-09-21T02:36:25.742643+0000","last_unstale":"2024-09-21T02:38:40.504077+0000","last_undegraded":"2024-09-21T02:38:40.504077+0000","last_fullsized":"2024-09-21T02:38:40.504077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:20:09.849806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331234+0000","last_change":"2024-09-21T02:37:39.625360+0000","last_active":"2024-09-21T02:38:40.331234+0000","last_peered":"2024-09-21T02:38:40.331234+0000","last_clean":"2024-09-21T02:38:40.331234+0000","last_became_active":"2024-09-21T02:37:39.611781+0000","last_became_peered":"2024-09-21T02:37:39.611781+0000","last_unstale":"2024-09-21T02:38:40.331234+0000","last_undegraded":"2024-09-21T02:38:40.331234+0000","last_fullsized":"2024-09-21T02:38:40.331234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:11.846383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222385+0000","last_change":"2024-09-21T02:36:32.799728+0000","last_active":"2024-09-21T02:38:40.222385+0000","last_peered":"2024-09-21T02:38:40.222385+0000","last_clean":"2024-09-21T02:38:40.222385+0000","last_became_active":"2024-09-21T02:36:32.798419+0000","last_became_peered":"2024-09-21T02:36:32.798419+0000","last_unstale":"2024-09-21T02:38:40.222385+0000","last_undegraded":"2024-09-21T02:38:40.222385+0000","last_fullsized":"2024-09-21T02:38:40.222385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:53.681473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502716+0000","last_change":"2024-09-21T02:36:25.747712+0000","last_active":"2024-09-21T02:38:40.502716+0000","last_peered":"2024-09-21T02:38:40.502716+0000","last_clean":"2024-09-21T02:38:40.502716+0000","last_became_active":"2024-09-21T02:36:25.747197+0000","last_became_peered":"2024-09-21T02:36:25.747197+0000","last_unstale":"2024-09-21T02:38:40.502716+0000","last_undegraded":"2024-09-21T02:38:40.502716+0000","last_fullsized":"2024-09-21T02:38:40.502716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:03.900360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":158,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:59.205933+0000","last_change":"2024-09-21T02:36:23.732156+0000","last_active":"2024-09-21T02:38:59.205933+0000","last_peered":"2024-09-21T02:38:59.205933+0000","last_clean":"2024-09-21T02:38:59.205933+0000","last_became_active":"2024-09-21T02:36:23.730238+0000","last_became_peered":"2024-09-21T02:36:23.730238+0000","last_unstale":"2024-09-21T02:38:59.205933+0000","last_undegraded":"2024-09-21T02:38:59.205933+0000","last_fullsized":"2024-09-21T02:38:59.205933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:18.053890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503510+0000","last_change":"2024-09-21T02:36:32.806674+0000","last_active":"2024-09-21T02:38:40.503510+0000","last_peered":"2024-09-21T02:38:40.503510+0000","last_clean":"2024-09-21T02:38:40.503510+0000","last_became_active":"2024-09-21T02:36:32.805373+0000","last_became_peered":"2024-09-21T02:36:32.805373+0000","last_unstale":"2024-09-21T02:38:40.503510+0000","last_undegraded":"2024-09-21T02:38:40.503510+0000","last_fullsized":"2024-09-21T02:38:40.503510+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:29.249243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503029+0000","last_change":"2024-09-21T02:36:25.748703+0000","last_active":"2024-09-21T02:38:40.503029+0000","last_peered":"2024-09-21T02:38:40.503029+0000","last_clean":"2024-09-21T02:38:40.503029+0000","last_became_active":"2024-09-21T02:36:25.748340+0000","last_became_peered":"2024-09-21T02:36:25.748340+0000","last_unstale":"2024-09-21T02:38:40.503029+0000","last_undegraded":"2024-09-21T02:38:40.503029+0000","last_fullsized":"2024-09-21T02:38:40.503029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:07.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504103+0000","last_change":"2024-09-21T02:36:23.732332+0000","last_active":"2024-09-21T02:38:40.504103+0000","last_peered":"2024-09-21T02:38:40.504103+0000","last_clean":"2024-09-21T02:38:40.504103+0000","last_became_active":"2024-09-21T02:36:23.731974+0000","last_became_peered":"2024-09-21T02:36:23.731974+0000","last_unstale":"2024-09-21T02:38:40.504103+0000","last_undegraded":"2024-09-21T02:38:40.504103+0000","last_fullsized":"2024-09-21T02:38:40.504103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:52.284192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331545+0000","last_change":"2024-09-21T02:37:39.610698+0000","last_active":"2024-09-21T02:38:40.331545+0000","last_peered":"2024-09-21T02:38:40.331545+0000","last_clean":"2024-09-21T02:38:40.331545+0000","last_became_active":"2024-09-21T02:37:39.610185+0000","last_became_peered":"2024-09-21T02:37:39.610185+0000","last_unstale":"2024-09-21T02:38:40.331545+0000","last_undegraded":"2024-09-21T02:38:40.331545+0000","last_fullsized":"2024-09-21T02:38:40.331545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:54:51.050463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221965+0000","last_change":"2024-09-21T02:36:25.751720+0000","last_active":"2024-09-21T02:38:40.221965+0000","last_peered":"2024-09-21T02:38:40.221965+0000","last_clean":"2024-09-21T02:38:40.221965+0000","last_became_active":"2024-09-21T02:36:25.751351+0000","last_became_peered":"2024-09-21T02:36:25.751351+0000","last_unstale":"2024-09-21T02:38:40.221965+0000","last_undegraded":"2024-09-21T02:38:40.221965+0000","last_fullsized":"2024-09-21T02:38:40.221965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:01.201453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224603+0000","last_change":"2024-09-21T02:36:23.729981+0000","last_active":"2024-09-21T02:38:40.224603+0000","last_peered":"2024-09-21T02:38:40.224603+0000","last_clean":"2024-09-21T02:38:40.224603+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:38:40.224603+0000","last_undegraded":"2024-09-21T02:38:40.224603+0000","last_fullsized":"2024-09-21T02:38:40.224603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:03.115712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504439+0000","last_change":"2024-09-21T02:36:32.805286+0000","last_active":"2024-09-21T02:38:40.504439+0000","last_peered":"2024-09-21T02:38:40.504439+0000","last_clean":"2024-09-21T02:38:40.504439+0000","last_became_active":"2024-09-21T02:36:32.804770+0000","last_became_peered":"2024-09-21T02:36:32.804770+0000","last_unstale":"2024-09-21T02:38:40.504439+0000","last_undegraded":"2024-09-21T02:38:40.504439+0000","last_fullsized":"2024-09-21T02:38:40.504439+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:18:34.155892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503496+0000","last_change":"2024-09-21T02:36:25.747812+0000","last_active":"2024-09-21T02:38:40.503496+0000","last_peered":"2024-09-21T02:38:40.503496+0000","last_clean":"2024-09-21T02:38:40.503496+0000","last_became_active":"2024-09-21T02:36:25.747187+0000","last_became_peered":"2024-09-21T02:36:25.747187+0000","last_unstale":"2024-09-21T02:38:40.503496+0000","last_undegraded":"2024-09-21T02:38:40.503496+0000","last_fullsized":"2024-09-21T02:38:40.503496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:51.395869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":316,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.223137+0000","last_change":"2024-09-21T02:36:23.735676+0000","last_active":"2024-09-21T02:39:02.223137+0000","last_peered":"2024-09-21T02:39:02.223137+0000","last_clean":"2024-09-21T02:39:02.223137+0000","last_became_active":"2024-09-21T02:36:23.735281+0000","last_became_peered":"2024-09-21T02:36:23.735281+0000","last_unstale":"2024-09-21T02:39:02.223137+0000","last_undegraded":"2024-09-21T02:39:02.223137+0000","last_fullsized":"2024-09-21T02:39:02.223137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:28:01.547245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68362,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":236,"num_read_kb":184,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503129+0000","last_change":"2024-09-21T02:36:32.806585+0000","last_active":"2024-09-21T02:38:40.503129+0000","last_peered":"2024-09-21T02:38:40.503129+0000","last_clean":"2024-09-21T02:38:40.503129+0000","last_became_active":"2024-09-21T02:36:32.805913+0000","last_became_peered":"2024-09-21T02:36:32.805913+0000","last_unstale":"2024-09-21T02:38:40.503129+0000","last_undegraded":"2024-09-21T02:38:40.503129+0000","last_fullsized":"2024-09-21T02:38:40.503129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:59:31.888645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228365+0000","last_change":"2024-09-21T02:36:25.742208+0000","last_active":"2024-09-21T02:38:40.228365+0000","last_peered":"2024-09-21T02:38:40.228365+0000","last_clean":"2024-09-21T02:38:40.228365+0000","last_became_active":"2024-09-21T02:36:25.741849+0000","last_became_peered":"2024-09-21T02:36:25.741849+0000","last_unstale":"2024-09-21T02:38:40.228365+0000","last_undegraded":"2024-09-21T02:38:40.228365+0000","last_fullsized":"2024-09-21T02:38:40.228365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:21:43.720481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504523+0000","last_change":"2024-09-21T02:36:23.800751+0000","last_active":"2024-09-21T02:38:40.504523+0000","last_peered":"2024-09-21T02:38:40.504523+0000","last_clean":"2024-09-21T02:38:40.504523+0000","last_became_active":"2024-09-21T02:36:23.800341+0000","last_became_peered":"2024-09-21T02:36:23.800341+0000","last_unstale":"2024-09-21T02:38:40.504523+0000","last_undegraded":"2024-09-21T02:38:40.504523+0000","last_fullsized":"2024-09-21T02:38:40.504523+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:08:02.556549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503710+0000","last_change":"2024-09-21T02:36:32.802879+0000","last_active":"2024-09-21T02:38:40.503710+0000","last_peered":"2024-09-21T02:38:40.503710+0000","last_clean":"2024-09-21T02:38:40.503710+0000","last_became_active":"2024-09-21T02:36:32.802301+0000","last_became_peered":"2024-09-21T02:36:32.802301+0000","last_unstale":"2024-09-21T02:38:40.503710+0000","last_undegraded":"2024-09-21T02:38:40.503710+0000","last_fullsized":"2024-09-21T02:38:40.503710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:08.135814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232720+0000","last_change":"2024-09-21T02:36:25.744647+0000","last_active":"2024-09-21T02:38:40.232720+0000","last_peered":"2024-09-21T02:38:40.232720+0000","last_clean":"2024-09-21T02:38:40.232720+0000","last_became_active":"2024-09-21T02:36:25.744049+0000","last_became_peered":"2024-09-21T02:36:25.744049+0000","last_unstale":"2024-09-21T02:38:40.232720+0000","last_undegraded":"2024-09-21T02:38:40.232720+0000","last_fullsized":"2024-09-21T02:38:40.232720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:52:02.210380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503652+0000","last_change":"2024-09-21T02:36:23.733590+0000","last_active":"2024-09-21T02:38:40.503652+0000","last_peered":"2024-09-21T02:38:40.503652+0000","last_clean":"2024-09-21T02:38:40.503652+0000","last_became_active":"2024-09-21T02:36:23.732922+0000","last_became_peered":"2024-09-21T02:36:23.732922+0000","last_unstale":"2024-09-21T02:38:40.503652+0000","last_undegraded":"2024-09-21T02:38:40.503652+0000","last_fullsized":"2024-09-21T02:38:40.503652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503016+0000","last_change":"2024-09-21T02:36:32.806540+0000","last_active":"2024-09-21T02:38:40.503016+0000","last_peered":"2024-09-21T02:38:40.503016+0000","last_clean":"2024-09-21T02:38:40.503016+0000","last_became_active":"2024-09-21T02:36:32.805612+0000","last_became_peered":"2024-09-21T02:36:32.805612+0000","last_unstale":"2024-09-21T02:38:40.503016+0000","last_undegraded":"2024-09-21T02:38:40.503016+0000","last_fullsized":"2024-09-21T02:38:40.503016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:23:24.686383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227650+0000","last_change":"2024-09-21T02:36:25.746229+0000","last_active":"2024-09-21T02:38:40.227650+0000","last_peered":"2024-09-21T02:38:40.227650+0000","last_clean":"2024-09-21T02:38:40.227650+0000","last_became_active":"2024-09-21T02:36:25.745892+0000","last_became_peered":"2024-09-21T02:36:25.745892+0000","last_unstale":"2024-09-21T02:38:40.227650+0000","last_undegraded":"2024-09-21T02:38:40.227650+0000","last_fullsized":"2024-09-21T02:38:40.227650+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:55.154499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"85'8","reported_seq":274,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.213998+0000","last_change":"2024-09-21T02:36:23.730987+0000","last_active":"2024-09-21T02:39:02.213998+0000","last_peered":"2024-09-21T02:39:02.213998+0000","last_clean":"2024-09-21T02:39:02.213998+0000","last_became_active":"2024-09-21T02:36:23.730566+0000","last_became_peered":"2024-09-21T02:36:23.730566+0000","last_unstale":"2024-09-21T02:39:02.213998+0000","last_undegraded":"2024-09-21T02:39:02.213998+0000","last_fullsized":"2024-09-21T02:39:02.213998+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:18:12.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293972+0000","last_change":"2024-09-21T02:36:32.804608+0000","last_active":"2024-09-21T02:38:40.293972+0000","last_peered":"2024-09-21T02:38:40.293972+0000","last_clean":"2024-09-21T02:38:40.293972+0000","last_became_active":"2024-09-21T02:36:32.803567+0000","last_became_peered":"2024-09-21T02:36:32.803567+0000","last_unstale":"2024-09-21T02:38:40.293972+0000","last_undegraded":"2024-09-21T02:38:40.293972+0000","last_fullsized":"2024-09-21T02:38:40.293972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:41.692644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232808+0000","last_change":"2024-09-21T02:36:32.804687+0000","last_active":"2024-09-21T02:38:40.232808+0000","last_peered":"2024-09-21T02:38:40.232808+0000","last_clean":"2024-09-21T02:38:40.232808+0000","last_became_active":"2024-09-21T02:36:32.804232+0000","last_became_peered":"2024-09-21T02:36:32.804232+0000","last_unstale":"2024-09-21T02:38:40.232808+0000","last_undegraded":"2024-09-21T02:38:40.232808+0000","last_fullsized":"2024-09-21T02:38:40.232808+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:43.899924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503127+0000","last_change":"2024-09-21T02:36:25.749114+0000","last_active":"2024-09-21T02:38:40.503127+0000","last_peered":"2024-09-21T02:38:40.503127+0000","last_clean":"2024-09-21T02:38:40.503127+0000","last_became_active":"2024-09-21T02:36:25.748758+0000","last_became_peered":"2024-09-21T02:36:25.748758+0000","last_unstale":"2024-09-21T02:38:40.503127+0000","last_undegraded":"2024-09-21T02:38:40.503127+0000","last_fullsized":"2024-09-21T02:38:40.503127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:57.958533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504282+0000","last_change":"2024-09-21T02:36:23.731473+0000","last_active":"2024-09-21T02:38:40.504282+0000","last_peered":"2024-09-21T02:38:40.504282+0000","last_clean":"2024-09-21T02:38:40.504282+0000","last_became_active":"2024-09-21T02:36:23.731134+0000","last_became_peered":"2024-09-21T02:36:23.731134+0000","last_unstale":"2024-09-21T02:38:40.504282+0000","last_undegraded":"2024-09-21T02:38:40.504282+0000","last_fullsized":"2024-09-21T02:38:40.504282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:05.368941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331458+0000","last_change":"2024-09-21T02:36:32.809630+0000","last_active":"2024-09-21T02:38:40.331458+0000","last_peered":"2024-09-21T02:38:40.331458+0000","last_clean":"2024-09-21T02:38:40.331458+0000","last_became_active":"2024-09-21T02:36:32.809030+0000","last_became_peered":"2024-09-21T02:36:32.809030+0000","last_unstale":"2024-09-21T02:38:40.331458+0000","last_undegraded":"2024-09-21T02:38:40.331458+0000","last_fullsized":"2024-09-21T02:38:40.331458+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:57:42.436590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293966+0000","last_change":"2024-09-21T02:36:25.742558+0000","last_active":"2024-09-21T02:38:40.293966+0000","last_peered":"2024-09-21T02:38:40.293966+0000","last_clean":"2024-09-21T02:38:40.293966+0000","last_became_active":"2024-09-21T02:36:25.742045+0000","last_became_peered":"2024-09-21T02:36:25.742045+0000","last_unstale":"2024-09-21T02:38:40.293966+0000","last_undegraded":"2024-09-21T02:38:40.293966+0000","last_fullsized":"2024-09-21T02:38:40.293966+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:57:06.705561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221778+0000","last_change":"2024-09-21T02:36:23.724345+0000","last_active":"2024-09-21T02:38:40.221778+0000","last_peered":"2024-09-21T02:38:40.221778+0000","last_clean":"2024-09-21T02:38:40.221778+0000","last_became_active":"2024-09-21T02:36:23.723635+0000","last_became_peered":"2024-09-21T02:36:23.723635+0000","last_unstale":"2024-09-21T02:38:40.221778+0000","last_undegraded":"2024-09-21T02:38:40.221778+0000","last_fullsized":"2024-09-21T02:38:40.221778+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:18.812176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223962+0000","last_change":"2024-09-21T02:36:23.732897+0000","last_active":"2024-09-21T02:38:40.223962+0000","last_peered":"2024-09-21T02:38:40.223962+0000","last_clean":"2024-09-21T02:38:40.223962+0000","last_became_active":"2024-09-21T02:36:23.732509+0000","last_became_peered":"2024-09-21T02:36:23.732509+0000","last_unstale":"2024-09-21T02:38:40.223962+0000","last_undegraded":"2024-09-21T02:38:40.223962+0000","last_fullsized":"2024-09-21T02:38:40.223962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:12:27.117962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504315+0000","last_change":"2024-09-21T02:36:25.748929+0000","last_active":"2024-09-21T02:38:40.504315+0000","last_peered":"2024-09-21T02:38:40.504315+0000","last_clean":"2024-09-21T02:38:40.504315+0000","last_became_active":"2024-09-21T02:36:25.748588+0000","last_became_peered":"2024-09-21T02:36:25.748588+0000","last_unstale":"2024-09-21T02:38:40.504315+0000","last_undegraded":"2024-09-21T02:38:40.504315+0000","last_fullsized":"2024-09-21T02:38:40.504315+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.086472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224498+0000","last_change":"2024-09-21T02:36:32.807655+0000","last_active":"2024-09-21T02:38:40.224498+0000","last_peered":"2024-09-21T02:38:40.224498+0000","last_clean":"2024-09-21T02:38:40.224498+0000","last_became_active":"2024-09-21T02:36:32.806995+0000","last_became_peered":"2024-09-21T02:36:32.806995+0000","last_unstale":"2024-09-21T02:38:40.224498+0000","last_undegraded":"2024-09-21T02:38:40.224498+0000","last_fullsized":"2024-09-21T02:38:40.224498+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:28:20.420591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'7","reported_seq":163,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:00.209025+0000","last_change":"2024-09-21T02:36:23.729427+0000","last_active":"2024-09-21T02:39:00.209025+0000","last_peered":"2024-09-21T02:39:00.209025+0000","last_clean":"2024-09-21T02:39:00.209025+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:39:00.209025+0000","last_undegraded":"2024-09-21T02:39:00.209025+0000","last_fullsized":"2024-09-21T02:39:00.209025+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:10:41.251665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502991+0000","last_change":"2024-09-21T02:36:25.743212+0000","last_active":"2024-09-21T02:38:40.502991+0000","last_peered":"2024-09-21T02:38:40.502991+0000","last_clean":"2024-09-21T02:38:40.502991+0000","last_became_active":"2024-09-21T02:36:25.742777+0000","last_became_peered":"2024-09-21T02:36:25.742777+0000","last_unstale":"2024-09-21T02:38:40.502991+0000","last_undegraded":"2024-09-21T02:38:40.502991+0000","last_fullsized":"2024-09-21T02:38:40.502991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:12:19.917214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224485+0000","last_change":"2024-09-21T02:36:32.801880+0000","last_active":"2024-09-21T02:38:40.224485+0000","last_peered":"2024-09-21T02:38:40.224485+0000","last_clean":"2024-09-21T02:38:40.224485+0000","last_became_active":"2024-09-21T02:36:32.801389+0000","last_became_peered":"2024-09-21T02:36:32.801389+0000","last_unstale":"2024-09-21T02:38:40.224485+0000","last_undegraded":"2024-09-21T02:38:40.224485+0000","last_fullsized":"2024-09-21T02:38:40.224485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:11:44.877062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331561+0000","last_change":"2024-09-21T02:36:25.754014+0000","last_active":"2024-09-21T02:38:40.331561+0000","last_peered":"2024-09-21T02:38:40.331561+0000","last_clean":"2024-09-21T02:38:40.331561+0000","last_became_active":"2024-09-21T02:36:25.753548+0000","last_became_peered":"2024-09-21T02:36:25.753548+0000","last_unstale":"2024-09-21T02:38:40.331561+0000","last_undegraded":"2024-09-21T02:38:40.331561+0000","last_fullsized":"2024-09-21T02:38:40.331561+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:03.850625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224637+0000","last_change":"2024-09-21T02:37:39.610894+0000","last_active":"2024-09-21T02:38:40.224637+0000","last_peered":"2024-09-21T02:38:40.224637+0000","last_clean":"2024-09-21T02:38:40.224637+0000","last_became_active":"2024-09-21T02:37:39.610404+0000","last_became_peered":"2024-09-21T02:37:39.610404+0000","last_unstale":"2024-09-21T02:38:40.224637+0000","last_undegraded":"2024-09-21T02:38:40.224637+0000","last_fullsized":"2024-09-21T02:38:40.224637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:54.347708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224163+0000","last_change":"2024-09-21T02:36:25.752839+0000","last_active":"2024-09-21T02:38:40.224163+0000","last_peered":"2024-09-21T02:38:40.224163+0000","last_clean":"2024-09-21T02:38:40.224163+0000","last_became_active":"2024-09-21T02:36:25.750882+0000","last_became_peered":"2024-09-21T02:36:25.750882+0000","last_unstale":"2024-09-21T02:38:40.224163+0000","last_undegraded":"2024-09-21T02:38:40.224163+0000","last_fullsized":"2024-09-21T02:38:40.224163+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:53.028611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233548+0000","last_change":"2024-09-21T02:36:25.744624+0000","last_active":"2024-09-21T02:38:40.233548+0000","last_peered":"2024-09-21T02:38:40.233548+0000","last_clean":"2024-09-21T02:38:40.233548+0000","last_became_active":"2024-09-21T02:36:25.744173+0000","last_became_peered":"2024-09-21T02:36:25.744173+0000","last_unstale":"2024-09-21T02:38:40.233548+0000","last_undegraded":"2024-09-21T02:38:40.233548+0000","last_fullsized":"2024-09-21T02:38:40.233548+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:55.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222157+0000","last_change":"2024-09-21T02:37:39.609050+0000","last_active":"2024-09-21T02:38:40.222157+0000","last_peered":"2024-09-21T02:38:40.222157+0000","last_clean":"2024-09-21T02:38:40.222157+0000","last_became_active":"2024-09-21T02:37:39.608595+0000","last_became_peered":"2024-09-21T02:37:39.608595+0000","last_unstale":"2024-09-21T02:38:40.222157+0000","last_undegraded":"2024-09-21T02:38:40.222157+0000","last_fullsized":"2024-09-21T02:38:40.222157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:26.831121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331839+0000","last_change":"2024-09-21T02:36:25.743476+0000","last_active":"2024-09-21T02:38:40.331839+0000","last_peered":"2024-09-21T02:38:40.331839+0000","last_clean":"2024-09-21T02:38:40.331839+0000","last_became_active":"2024-09-21T02:36:25.743117+0000","last_became_peered":"2024-09-21T02:36:25.743117+0000","last_unstale":"2024-09-21T02:38:40.331839+0000","last_undegraded":"2024-09-21T02:38:40.331839+0000","last_fullsized":"2024-09-21T02:38:40.331839+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:08.084109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224831+0000","last_change":"2024-09-21T02:36:25.753014+0000","last_active":"2024-09-21T02:38:40.224831+0000","last_peered":"2024-09-21T02:38:40.224831+0000","last_clean":"2024-09-21T02:38:40.224831+0000","last_became_active":"2024-09-21T02:36:25.751232+0000","last_became_peered":"2024-09-21T02:36:25.751232+0000","last_unstale":"2024-09-21T02:38:40.224831+0000","last_undegraded":"2024-09-21T02:38:40.224831+0000","last_fullsized":"2024-09-21T02:38:40.224831+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:03.052737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503339+0000","last_change":"2024-09-21T02:37:39.606525+0000","last_active":"2024-09-21T02:38:40.503339+0000","last_peered":"2024-09-21T02:38:40.503339+0000","last_clean":"2024-09-21T02:38:40.503339+0000","last_became_active":"2024-09-21T02:37:39.606078+0000","last_became_peered":"2024-09-21T02:37:39.606078+0000","last_unstale":"2024-09-21T02:38:40.503339+0000","last_undegraded":"2024-09-21T02:38:40.503339+0000","last_fullsized":"2024-09-21T02:38:40.503339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:22.423457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503992+0000","last_change":"2024-09-21T02:36:25.749605+0000","last_active":"2024-09-21T02:38:40.503992+0000","last_peered":"2024-09-21T02:38:40.503992+0000","last_clean":"2024-09-21T02:38:40.503992+0000","last_became_active":"2024-09-21T02:36:25.749165+0000","last_became_peered":"2024-09-21T02:36:25.749165+0000","last_unstale":"2024-09-21T02:38:40.503992+0000","last_undegraded":"2024-09-21T02:38:40.503992+0000","last_fullsized":"2024-09-21T02:38:40.503992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:15:51.412761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294064+0000","last_change":"2024-09-21T02:36:25.747494+0000","last_active":"2024-09-21T02:38:40.294064+0000","last_peered":"2024-09-21T02:38:40.294064+0000","last_clean":"2024-09-21T02:38:40.294064+0000","last_became_active":"2024-09-21T02:36:25.746965+0000","last_became_peered":"2024-09-21T02:36:25.746965+0000","last_unstale":"2024-09-21T02:38:40.294064+0000","last_undegraded":"2024-09-21T02:38:40.294064+0000","last_fullsized":"2024-09-21T02:38:40.294064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:51.309307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227481+0000","last_change":"2024-09-21T02:36:25.749517+0000","last_active":"2024-09-21T02:38:40.227481+0000","last_peered":"2024-09-21T02:38:40.227481+0000","last_clean":"2024-09-21T02:38:40.227481+0000","last_became_active":"2024-09-21T02:36:25.749117+0000","last_became_peered":"2024-09-21T02:36:25.749117+0000","last_unstale":"2024-09-21T02:38:40.227481+0000","last_undegraded":"2024-09-21T02:38:40.227481+0000","last_fullsized":"2024-09-21T02:38:40.227481+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:38.758019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222549+0000","last_change":"2024-09-21T02:36:25.736067+0000","last_active":"2024-09-21T02:38:40.222549+0000","last_peered":"2024-09-21T02:38:40.222549+0000","last_clean":"2024-09-21T02:38:40.222549+0000","last_became_active":"2024-09-21T02:36:25.735710+0000","last_became_peered":"2024-09-21T02:36:25.735710+0000","last_unstale":"2024-09-21T02:38:40.222549+0000","last_undegraded":"2024-09-21T02:38:40.222549+0000","last_fullsized":"2024-09-21T02:38:40.222549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.559448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331864+0000","last_change":"2024-09-21T02:36:25.746307+0000","last_active":"2024-09-21T02:38:40.331864+0000","last_peered":"2024-09-21T02:38:40.331864+0000","last_clean":"2024-09-21T02:38:40.331864+0000","last_became_active":"2024-09-21T02:36:25.745872+0000","last_became_peered":"2024-09-21T02:36:25.745872+0000","last_unstale":"2024-09-21T02:38:40.331864+0000","last_undegraded":"2024-09-21T02:38:40.331864+0000","last_fullsized":"2024-09-21T02:38:40.331864+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:58:53.074559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224609+0000","last_change":"2024-09-21T02:36:25.753393+0000","last_active":"2024-09-21T02:38:40.224609+0000","last_peered":"2024-09-21T02:38:40.224609+0000","last_clean":"2024-09-21T02:38:40.224609+0000","last_became_active":"2024-09-21T02:36:25.751871+0000","last_became_peered":"2024-09-21T02:36:25.751871+0000","last_unstale":"2024-09-21T02:38:40.224609+0000","last_undegraded":"2024-09-21T02:38:40.224609+0000","last_fullsized":"2024-09-21T02:38:40.224609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:28.199451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503585+0000","last_change":"2024-09-21T02:36:25.744845+0000","last_active":"2024-09-21T02:38:40.503585+0000","last_peered":"2024-09-21T02:38:40.503585+0000","last_clean":"2024-09-21T02:38:40.503585+0000","last_became_active":"2024-09-21T02:36:25.744492+0000","last_became_peered":"2024-09-21T02:36:25.744492+0000","last_unstale":"2024-09-21T02:38:40.503585+0000","last_undegraded":"2024-09-21T02:38:40.503585+0000","last_fullsized":"2024-09-21T02:38:40.503585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:08.524871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331829+0000","last_change":"2024-09-21T02:36:25.751187+0000","last_active":"2024-09-21T02:38:40.331829+0000","last_peered":"2024-09-21T02:38:40.331829+0000","last_clean":"2024-09-21T02:38:40.331829+0000","last_became_active":"2024-09-21T02:36:25.750813+0000","last_became_peered":"2024-09-21T02:36:25.750813+0000","last_unstale":"2024-09-21T02:38:40.331829+0000","last_undegraded":"2024-09-21T02:38:40.331829+0000","last_fullsized":"2024-09-21T02:38:40.331829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.590210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.226878+0000","last_change":"2024-09-21T02:36:25.744046+0000","last_active":"2024-09-21T02:38:40.226878+0000","last_peered":"2024-09-21T02:38:40.226878+0000","last_clean":"2024-09-21T02:38:40.226878+0000","last_became_active":"2024-09-21T02:36:25.743695+0000","last_became_peered":"2024-09-21T02:36:25.743695+0000","last_unstale":"2024-09-21T02:38:40.226878+0000","last_undegraded":"2024-09-21T02:38:40.226878+0000","last_fullsized":"2024-09-21T02:38:40.226878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:12.092940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.231911+0000","last_change":"2024-09-21T02:36:25.749113+0000","last_active":"2024-09-21T02:38:40.231911+0000","last_peered":"2024-09-21T02:38:40.231911+0000","last_clean":"2024-09-21T02:38:40.231911+0000","last_became_active":"2024-09-21T02:36:25.748759+0000","last_became_peered":"2024-09-21T02:36:25.748759+0000","last_unstale":"2024-09-21T02:38:40.231911+0000","last_undegraded":"2024-09-21T02:38:40.231911+0000","last_fullsized":"2024-09-21T02:38:40.231911+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:22.785273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232166+0000","last_change":"2024-09-21T02:36:25.745454+0000","last_active":"2024-09-21T02:38:40.232166+0000","last_peered":"2024-09-21T02:38:40.232166+0000","last_clean":"2024-09-21T02:38:40.232166+0000","last_became_active":"2024-09-21T02:36:25.745108+0000","last_became_peered":"2024-09-21T02:36:25.745108+0000","last_unstale":"2024-09-21T02:38:40.232166+0000","last_undegraded":"2024-09-21T02:38:40.232166+0000","last_fullsized":"2024-09-21T02:38:40.232166+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:36.710456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222535+0000","last_change":"2024-09-21T02:36:25.737213+0000","last_active":"2024-09-21T02:38:40.222535+0000","last_peered":"2024-09-21T02:38:40.222535+0000","last_clean":"2024-09-21T02:38:40.222535+0000","last_became_active":"2024-09-21T02:36:25.736641+0000","last_became_peered":"2024-09-21T02:36:25.736641+0000","last_unstale":"2024-09-21T02:38:40.222535+0000","last_undegraded":"2024-09-21T02:38:40.222535+0000","last_fullsized":"2024-09-21T02:38:40.222535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:47:50.258805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504287+0000","last_change":"2024-09-21T02:36:25.748230+0000","last_active":"2024-09-21T02:38:40.504287+0000","last_peered":"2024-09-21T02:38:40.504287+0000","last_clean":"2024-09-21T02:38:40.504287+0000","last_became_active":"2024-09-21T02:36:25.747859+0000","last_became_peered":"2024-09-21T02:36:25.747859+0000","last_unstale":"2024-09-21T02:38:40.504287+0000","last_undegraded":"2024-09-21T02:38:40.504287+0000","last_fullsized":"2024-09-21T02:38:40.504287+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:15:21.244825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503711+0000","last_change":"2024-09-21T02:36:25.744379+0000","last_active":"2024-09-21T02:38:40.503711+0000","last_peered":"2024-09-21T02:38:40.503711+0000","last_clean":"2024-09-21T02:38:40.503711+0000","last_became_active":"2024-09-21T02:36:25.744003+0000","last_became_peered":"2024-09-21T02:36:25.744003+0000","last_unstale":"2024-09-21T02:38:40.503711+0000","last_undegraded":"2024-09-21T02:38:40.503711+0000","last_fullsized":"2024-09-21T02:38:40.503711+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:59:59.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224543+0000","last_change":"2024-09-21T02:36:25.755616+0000","last_active":"2024-09-21T02:38:40.224543+0000","last_peered":"2024-09-21T02:38:40.224543+0000","last_clean":"2024-09-21T02:38:40.224543+0000","last_became_active":"2024-09-21T02:36:25.755304+0000","last_became_peered":"2024-09-21T02:36:25.755304+0000","last_unstale":"2024-09-21T02:38:40.224543+0000","last_undegraded":"2024-09-21T02:38:40.224543+0000","last_fullsized":"2024-09-21T02:38:40.224543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:24.813780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228147+0000","last_change":"2024-09-21T02:36:25.739747+0000","last_active":"2024-09-21T02:38:40.228147+0000","last_peered":"2024-09-21T02:38:40.228147+0000","last_clean":"2024-09-21T02:38:40.228147+0000","last_became_active":"2024-09-21T02:36:25.739404+0000","last_became_peered":"2024-09-21T02:36:25.739404+0000","last_unstale":"2024-09-21T02:38:40.228147+0000","last_undegraded":"2024-09-21T02:38:40.228147+0000","last_fullsized":"2024-09-21T02:38:40.228147+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:17.975419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293973+0000","last_change":"2024-09-21T02:36:25.747723+0000","last_active":"2024-09-21T02:38:40.293973+0000","last_peered":"2024-09-21T02:38:40.293973+0000","last_clean":"2024-09-21T02:38:40.293973+0000","last_became_active":"2024-09-21T02:36:25.747368+0000","last_became_peered":"2024-09-21T02:36:25.747368+0000","last_unstale":"2024-09-21T02:38:40.293973+0000","last_undegraded":"2024-09-21T02:38:40.293973+0000","last_fullsized":"2024-09-21T02:38:40.293973+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.319756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503361+0000","last_change":"2024-09-21T02:36:25.743772+0000","last_active":"2024-09-21T02:38:40.503361+0000","last_peered":"2024-09-21T02:38:40.503361+0000","last_clean":"2024-09-21T02:38:40.503361+0000","last_became_active":"2024-09-21T02:36:25.743403+0000","last_became_peered":"2024-09-21T02:36:25.743403+0000","last_unstale":"2024-09-21T02:38:40.503361+0000","last_undegraded":"2024-09-21T02:38:40.503361+0000","last_fullsized":"2024-09-21T02:38:40.503361+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:11:29.930248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223893+0000","last_change":"2024-09-21T02:36:25.746911+0000","last_active":"2024-09-21T02:38:40.223893+0000","last_peered":"2024-09-21T02:38:40.223893+0000","last_clean":"2024-09-21T02:38:40.223893+0000","last_became_active":"2024-09-21T02:36:25.746426+0000","last_became_peered":"2024-09-21T02:36:25.746426+0000","last_unstale":"2024-09-21T02:38:40.223893+0000","last_undegraded":"2024-09-21T02:38:40.223893+0000","last_fullsized":"2024-09-21T02:38:40.223893+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:12.835851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503097+0000","last_change":"2024-09-21T02:36:25.751278+0000","last_active":"2024-09-21T02:38:40.503097+0000","last_peered":"2024-09-21T02:38:40.503097+0000","last_clean":"2024-09-21T02:38:40.503097+0000","last_became_active":"2024-09-21T02:36:25.750414+0000","last_became_peered":"2024-09-21T02:36:25.750414+0000","last_unstale":"2024-09-21T02:38:40.503097+0000","last_undegraded":"2024-09-21T02:38:40.503097+0000","last_fullsized":"2024-09-21T02:38:40.503097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:13:56.080122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504014+0000","last_change":"2024-09-21T02:36:25.752852+0000","last_active":"2024-09-21T02:38:40.504014+0000","last_peered":"2024-09-21T02:38:40.504014+0000","last_clean":"2024-09-21T02:38:40.504014+0000","last_became_active":"2024-09-21T02:36:25.752514+0000","last_became_peered":"2024-09-21T02:36:25.752514+0000","last_unstale":"2024-09-21T02:38:40.504014+0000","last_undegraded":"2024-09-21T02:38:40.504014+0000","last_fullsized":"2024-09-21T02:38:40.504014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:55.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.231464+0000","last_change":"2024-09-21T02:36:25.750081+0000","last_active":"2024-09-21T02:38:40.231464+0000","last_peered":"2024-09-21T02:38:40.231464+0000","last_clean":"2024-09-21T02:38:40.231464+0000","last_became_active":"2024-09-21T02:36:25.749602+0000","last_became_peered":"2024-09-21T02:36:25.749602+0000","last_unstale":"2024-09-21T02:38:40.231464+0000","last_undegraded":"2024-09-21T02:38:40.231464+0000","last_fullsized":"2024-09-21T02:38:40.231464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:06.785998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232934+0000","last_change":"2024-09-21T02:36:25.746897+0000","last_active":"2024-09-21T02:38:40.232934+0000","last_peered":"2024-09-21T02:38:40.232934+0000","last_clean":"2024-09-21T02:38:40.232934+0000","last_became_active":"2024-09-21T02:36:25.746477+0000","last_became_peered":"2024-09-21T02:36:25.746477+0000","last_unstale":"2024-09-21T02:38:40.232934+0000","last_undegraded":"2024-09-21T02:38:40.232934+0000","last_fullsized":"2024-09-21T02:38:40.232934+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:34:00.739410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331749+0000","last_change":"2024-09-21T02:36:25.745233+0000","last_active":"2024-09-21T02:38:40.331749+0000","last_peered":"2024-09-21T02:38:40.331749+0000","last_clean":"2024-09-21T02:38:40.331749+0000","last_became_active":"2024-09-21T02:36:25.744637+0000","last_became_peered":"2024-09-21T02:36:25.744637+0000","last_unstale":"2024-09-21T02:38:40.331749+0000","last_undegraded":"2024-09-21T02:38:40.331749+0000","last_fullsized":"2024-09-21T02:38:40.331749+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:34:59.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76341,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1164,"num_read_kb":666,"num_write":269,"num_write_kb":162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":229516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249064,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":69,"seq":296352743468,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":950272,"data_stored":485845,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906994,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1144968,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070520,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":485131,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266749,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":954368,"data_stored":485397,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462978,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1015808,"data_stored":1082147,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659207,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":942080,"data_stored":480635,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855437,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1024000,"data_stored":1077729,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018961,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":950272,"data_stored":484709,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215190,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":480343,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":552297,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1015808,"data_stored":549534,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":966656,"data_stored":482623,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":70591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:39:06.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:39:06.846 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/503672493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 msgr2=0x7fc2d01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/503672493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 0x7fc2d01098b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009900 tx=0x7fc2c402a410 comp rx=0 tx=0).stop 2024-09-21T02:39:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/503672493 shutdown_connections 2024-09-21T02:39:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/503672493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d0109df0 0x7fc2d0110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/503672493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 0x7fc2d01098b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/503672493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2d0103b60 0x7fc2d0103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/503672493 >> 172.21.15.29:0/503672493 conn(0x7fc2d007b850 msgr2=0x7fc2d007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/503672493 shutdown_connections 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.409+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/503672493 wait complete. 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2d4fc3640 1 Processor -- start 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2d4fc3640 1 -- start start 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2d4fc3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2d0103b60 0x7fc2d019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2d4fc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 0x7fc2d019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2d4fc3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d0109df0 0x7fc2d019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2d4fc3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0113960 con 0x7fc2d01094a0 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2d4fc3640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01137e0 con 0x7fc2d0103b60 2024-09-21T02:39:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2d4fc3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0113ae0 con 0x7fc2d0109df0 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2cdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 0x7fc2d019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2cdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 0x7fc2d019a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47988/0 (socket says 172.21.15.29:47988) 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2cdd9b640 1 -- 172.21.15.29:0/3134266815 learned_addr learned my addr 172.21.15.29:0/3134266815 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2ced9d640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d0109df0 0x7fc2d019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.410+0000 7fc2ce59c640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2d0103b60 0x7fc2d019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2cdd9b640 1 -- 172.21.15.29:0/3134266815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d0109df0 msgr2=0x7fc2d019fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2cdd9b640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d0109df0 0x7fc2d019fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2cdd9b640 1 -- 172.21.15.29:0/3134266815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2d0103b60 msgr2=0x7fc2d019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2cdd9b640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2d0103b60 0x7fc2d019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2cdd9b640 1 -- 172.21.15.29:0/3134266815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d01a0380 con 0x7fc2d01094a0 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2ce59c640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2d0103b60 0x7fc2d019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2cdd9b640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 0x7fc2d019a9b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005940 tx=0x7fc2c40040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3134266815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c403f030 con 0x7fc2d01094a0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3134266815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2c4003a80 con 0x7fc2d01094a0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d01a0610 con 0x7fc2d01094a0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3134266815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4035070 con 0x7fc2d01094a0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.411+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d01a6dd0 con 0x7fc2d01094a0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.412+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d0103f70 con 0x7fc2d01094a0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.414+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3134266815 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7fc2c4042030 con 0x7fc2d01094a0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.415+0000 7fc2bf7fe640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2b80b6620 0x7fc2b80b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.415+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3134266815 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc2c4108120 con 0x7fc2d01094a0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.415+0000 7fc2ce59c640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2b80b6620 0x7fc2b80b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.415+0000 7fc2ce59c640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2b80b6620 0x7fc2b80b8b30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0005fd0 tx=0x7fc2c0009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.418+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3134266815 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc2c40ce4d0 con 0x7fc2d01094a0 2024-09-21T02:39:07.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.603+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2d010bc40 con 0x7fc2b80b6620 2024-09-21T02:39:07.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:07 smithi029 ceph-mon[31006]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:07.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:07 smithi029 ceph-mon[31006]: pgmap v389: 129 pgs: 129 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:39:07.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:07 smithi029 ceph-mon[31006]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-09-21T02:39:07.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.609+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3134266815 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340784 (secure 0 0 0) 0x7fc2d010bc40 con 0x7fc2b80b6620 2024-09-21T02:39:07.613 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:07.615 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2b80b6620 msgr2=0x7fc2b80b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2b80b6620 0x7fc2b80b8b30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0005fd0 tx=0x7fc2c0009520 comp rx=0 tx=0).stop 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 msgr2=0x7fc2d019a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 0x7fc2d019a9b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005940 tx=0x7fc2c40040e0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 shutdown_connections 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2b80b6620 0x7fc2b80b8b30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d0109df0 0x7fc2d019fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d01094a0 0x7fc2d019a9b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 --2- 172.21.15.29:0/3134266815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2d0103b60 0x7fc2d019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 >> 172.21.15.29:0/3134266815 conn(0x7fc2d007b850 msgr2=0x7fc2d01075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 shutdown_connections 2024-09-21T02:39:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:07.613+0000 7fc2d4fc3640 1 -- 172.21.15.29:0/3134266815 wait complete. 2024-09-21T02:39:07.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:07 smithi190 ceph-mon[33871]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:07.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:07 smithi190 ceph-mon[33871]: pgmap v389: 129 pgs: 129 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:39:07.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:07 smithi190 ceph-mon[33871]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-09-21T02:39:07.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:07 smithi066 ceph-mon[33436]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:07.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:07 smithi066 ceph-mon[33436]: pgmap v389: 129 pgs: 129 active+clean; 652 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:39:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:07 smithi066 ceph-mon[33436]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-09-21T02:39:08.454 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-21T02:39:05.975337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667929,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1257,"num_read_kb":737,"num_write":397,"num_write_kb":2317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2952069,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":827,"ondisk_log_size":827,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369200,"kb_used_data":11600,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572624,"statfs":{"total":1151940427776,"available":1151562366976,"internally_reserved":0,"allocated":11878400,"data_stored":7794316,"data_compressed":463314,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1972,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":58,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003858"},"pg_stats":[{"pgid":"2.1f","version":"85'14","reported_seq":168,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.209564+0000","last_change":"2024-09-21T02:36:23.729337+0000","last_active":"2024-09-21T02:39:02.209564+0000","last_peered":"2024-09-21T02:39:02.209564+0000","last_clean":"2024-09-21T02:39:02.209564+0000","last_became_active":"2024-09-21T02:36:23.728576+0000","last_became_peered":"2024-09-21T02:36:23.728576+0000","last_unstale":"2024-09-21T02:39:02.209564+0000","last_undegraded":"2024-09-21T02:39:02.209564+0000","last_fullsized":"2024-09-21T02:39:02.209564+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:31.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294323+0000","last_change":"2024-09-21T02:36:25.748925+0000","last_active":"2024-09-21T02:38:40.294323+0000","last_peered":"2024-09-21T02:38:40.294323+0000","last_clean":"2024-09-21T02:38:40.294323+0000","last_became_active":"2024-09-21T02:36:25.748333+0000","last_became_peered":"2024-09-21T02:36:25.748333+0000","last_unstale":"2024-09-21T02:38:40.294323+0000","last_undegraded":"2024-09-21T02:38:40.294323+0000","last_fullsized":"2024-09-21T02:38:40.294323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:46.889497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224117+0000","last_change":"2024-09-21T02:36:32.807771+0000","last_active":"2024-09-21T02:38:40.224117+0000","last_peered":"2024-09-21T02:38:40.224117+0000","last_clean":"2024-09-21T02:38:40.224117+0000","last_became_active":"2024-09-21T02:36:32.807291+0000","last_became_peered":"2024-09-21T02:36:32.807291+0000","last_unstale":"2024-09-21T02:38:40.224117+0000","last_undegraded":"2024-09-21T02:38:40.224117+0000","last_fullsized":"2024-09-21T02:38:40.224117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:41.468318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294537+0000","last_change":"2024-09-21T02:36:23.800779+0000","last_active":"2024-09-21T02:38:40.294537+0000","last_peered":"2024-09-21T02:38:40.294537+0000","last_clean":"2024-09-21T02:38:40.294537+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:38:40.294537+0000","last_undegraded":"2024-09-21T02:38:40.294537+0000","last_fullsized":"2024-09-21T02:38:40.294537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:24.843233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232565+0000","last_change":"2024-09-21T02:36:25.745277+0000","last_active":"2024-09-21T02:38:40.232565+0000","last_peered":"2024-09-21T02:38:40.232565+0000","last_clean":"2024-09-21T02:38:40.232565+0000","last_became_active":"2024-09-21T02:36:25.744501+0000","last_became_peered":"2024-09-21T02:36:25.744501+0000","last_unstale":"2024-09-21T02:38:40.232565+0000","last_undegraded":"2024-09-21T02:38:40.232565+0000","last_fullsized":"2024-09-21T02:38:40.232565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:14.776250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502659+0000","last_change":"2024-09-21T02:36:32.801978+0000","last_active":"2024-09-21T02:38:40.502659+0000","last_peered":"2024-09-21T02:38:40.502659+0000","last_clean":"2024-09-21T02:38:40.502659+0000","last_became_active":"2024-09-21T02:36:32.801436+0000","last_became_peered":"2024-09-21T02:36:32.801436+0000","last_unstale":"2024-09-21T02:38:40.502659+0000","last_undegraded":"2024-09-21T02:38:40.502659+0000","last_fullsized":"2024-09-21T02:38:40.502659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:37.380417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503616+0000","last_change":"2024-09-21T02:36:23.729336+0000","last_active":"2024-09-21T02:38:40.503616+0000","last_peered":"2024-09-21T02:38:40.503616+0000","last_clean":"2024-09-21T02:38:40.503616+0000","last_became_active":"2024-09-21T02:36:23.728724+0000","last_became_peered":"2024-09-21T02:36:23.728724+0000","last_unstale":"2024-09-21T02:38:40.503616+0000","last_undegraded":"2024-09-21T02:38:40.503616+0000","last_fullsized":"2024-09-21T02:38:40.503616+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:31:52.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228248+0000","last_change":"2024-09-21T02:36:25.737577+0000","last_active":"2024-09-21T02:38:40.228248+0000","last_peered":"2024-09-21T02:38:40.228248+0000","last_clean":"2024-09-21T02:38:40.228248+0000","last_became_active":"2024-09-21T02:36:25.737221+0000","last_became_peered":"2024-09-21T02:36:25.737221+0000","last_unstale":"2024-09-21T02:38:40.228248+0000","last_undegraded":"2024-09-21T02:38:40.228248+0000","last_fullsized":"2024-09-21T02:38:40.228248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:42:54.786464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504222+0000","last_change":"2024-09-21T02:36:32.802507+0000","last_active":"2024-09-21T02:38:40.504222+0000","last_peered":"2024-09-21T02:38:40.504222+0000","last_clean":"2024-09-21T02:38:40.504222+0000","last_became_active":"2024-09-21T02:36:32.801879+0000","last_became_peered":"2024-09-21T02:36:32.801879+0000","last_unstale":"2024-09-21T02:38:40.504222+0000","last_undegraded":"2024-09-21T02:38:40.504222+0000","last_fullsized":"2024-09-21T02:38:40.504222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:09.873825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503447+0000","last_change":"2024-09-21T02:36:23.730889+0000","last_active":"2024-09-21T02:38:40.503447+0000","last_peered":"2024-09-21T02:38:40.503447+0000","last_clean":"2024-09-21T02:38:40.503447+0000","last_became_active":"2024-09-21T02:36:23.730551+0000","last_became_peered":"2024-09-21T02:36:23.730551+0000","last_unstale":"2024-09-21T02:38:40.503447+0000","last_undegraded":"2024-09-21T02:38:40.503447+0000","last_fullsized":"2024-09-21T02:38:40.503447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:17.178848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504430+0000","last_change":"2024-09-21T02:37:39.869272+0000","last_active":"2024-09-21T02:38:40.504430+0000","last_peered":"2024-09-21T02:38:40.504430+0000","last_clean":"2024-09-21T02:38:40.504430+0000","last_became_active":"2024-09-21T02:37:39.868824+0000","last_became_peered":"2024-09-21T02:37:39.868824+0000","last_unstale":"2024-09-21T02:38:40.504430+0000","last_undegraded":"2024-09-21T02:38:40.504430+0000","last_fullsized":"2024-09-21T02:38:40.504430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:05:31.503359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503414+0000","last_change":"2024-09-21T02:36:32.800489+0000","last_active":"2024-09-21T02:38:40.503414+0000","last_peered":"2024-09-21T02:38:40.503414+0000","last_clean":"2024-09-21T02:38:40.503414+0000","last_became_active":"2024-09-21T02:36:32.799876+0000","last_became_peered":"2024-09-21T02:36:32.799876+0000","last_unstale":"2024-09-21T02:38:40.503414+0000","last_undegraded":"2024-09-21T02:38:40.503414+0000","last_fullsized":"2024-09-21T02:38:40.503414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:02:56.711257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502829+0000","last_change":"2024-09-21T02:36:25.744561+0000","last_active":"2024-09-21T02:38:40.502829+0000","last_peered":"2024-09-21T02:38:40.502829+0000","last_clean":"2024-09-21T02:38:40.502829+0000","last_became_active":"2024-09-21T02:36:25.743757+0000","last_became_peered":"2024-09-21T02:36:25.743757+0000","last_unstale":"2024-09-21T02:38:40.502829+0000","last_undegraded":"2024-09-21T02:38:40.502829+0000","last_fullsized":"2024-09-21T02:38:40.502829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:30.235517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503205+0000","last_change":"2024-09-21T02:37:39.616628+0000","last_active":"2024-09-21T02:38:40.503205+0000","last_peered":"2024-09-21T02:38:40.503205+0000","last_clean":"2024-09-21T02:38:40.503205+0000","last_became_active":"2024-09-21T02:37:39.609108+0000","last_became_peered":"2024-09-21T02:37:39.609108+0000","last_unstale":"2024-09-21T02:38:40.503205+0000","last_undegraded":"2024-09-21T02:38:40.503205+0000","last_fullsized":"2024-09-21T02:38:40.503205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:11:28.944244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504031+0000","last_change":"2024-09-21T02:36:33.003670+0000","last_active":"2024-09-21T02:38:40.504031+0000","last_peered":"2024-09-21T02:38:40.504031+0000","last_clean":"2024-09-21T02:38:40.504031+0000","last_became_active":"2024-09-21T02:36:33.003022+0000","last_became_peered":"2024-09-21T02:36:33.003022+0000","last_unstale":"2024-09-21T02:38:40.504031+0000","last_undegraded":"2024-09-21T02:38:40.504031+0000","last_fullsized":"2024-09-21T02:38:40.504031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:12.753375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294239+0000","last_change":"2024-09-21T02:36:25.749404+0000","last_active":"2024-09-21T02:38:40.294239+0000","last_peered":"2024-09-21T02:38:40.294239+0000","last_clean":"2024-09-21T02:38:40.294239+0000","last_became_active":"2024-09-21T02:36:25.747854+0000","last_became_peered":"2024-09-21T02:36:25.747854+0000","last_unstale":"2024-09-21T02:38:40.294239+0000","last_undegraded":"2024-09-21T02:38:40.294239+0000","last_fullsized":"2024-09-21T02:38:40.294239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:41.224740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224160+0000","last_change":"2024-09-21T02:36:23.726689+0000","last_active":"2024-09-21T02:38:40.224160+0000","last_peered":"2024-09-21T02:38:40.224160+0000","last_clean":"2024-09-21T02:38:40.224160+0000","last_became_active":"2024-09-21T02:36:23.726165+0000","last_became_peered":"2024-09-21T02:36:23.726165+0000","last_unstale":"2024-09-21T02:38:40.224160+0000","last_undegraded":"2024-09-21T02:38:40.224160+0000","last_fullsized":"2024-09-21T02:38:40.224160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:58.166457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503916+0000","last_change":"2024-09-21T02:36:32.809594+0000","last_active":"2024-09-21T02:38:40.503916+0000","last_peered":"2024-09-21T02:38:40.503916+0000","last_clean":"2024-09-21T02:38:40.503916+0000","last_became_active":"2024-09-21T02:36:32.808893+0000","last_became_peered":"2024-09-21T02:36:32.808893+0000","last_unstale":"2024-09-21T02:38:40.503916+0000","last_undegraded":"2024-09-21T02:38:40.503916+0000","last_fullsized":"2024-09-21T02:38:40.503916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:55.187241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331726+0000","last_change":"2024-09-21T02:36:25.755955+0000","last_active":"2024-09-21T02:38:40.331726+0000","last_peered":"2024-09-21T02:38:40.331726+0000","last_clean":"2024-09-21T02:38:40.331726+0000","last_became_active":"2024-09-21T02:36:25.755577+0000","last_became_peered":"2024-09-21T02:36:25.755577+0000","last_unstale":"2024-09-21T02:38:40.331726+0000","last_undegraded":"2024-09-21T02:38:40.331726+0000","last_fullsized":"2024-09-21T02:38:40.331726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:48.447758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294101+0000","last_change":"2024-09-21T02:36:23.730043+0000","last_active":"2024-09-21T02:38:40.294101+0000","last_peered":"2024-09-21T02:38:40.294101+0000","last_clean":"2024-09-21T02:38:40.294101+0000","last_became_active":"2024-09-21T02:36:23.729576+0000","last_became_peered":"2024-09-21T02:36:23.729576+0000","last_unstale":"2024-09-21T02:38:40.294101+0000","last_undegraded":"2024-09-21T02:38:40.294101+0000","last_fullsized":"2024-09-21T02:38:40.294101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:48:09.530913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503449+0000","last_change":"2024-09-21T02:36:32.806923+0000","last_active":"2024-09-21T02:38:40.503449+0000","last_peered":"2024-09-21T02:38:40.503449+0000","last_clean":"2024-09-21T02:38:40.503449+0000","last_became_active":"2024-09-21T02:36:32.806176+0000","last_became_peered":"2024-09-21T02:36:32.806176+0000","last_unstale":"2024-09-21T02:38:40.503449+0000","last_undegraded":"2024-09-21T02:38:40.503449+0000","last_fullsized":"2024-09-21T02:38:40.503449+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.165638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502828+0000","last_change":"2024-09-21T02:36:25.747501+0000","last_active":"2024-09-21T02:38:40.502828+0000","last_peered":"2024-09-21T02:38:40.502828+0000","last_clean":"2024-09-21T02:38:40.502828+0000","last_became_active":"2024-09-21T02:36:25.747111+0000","last_became_peered":"2024-09-21T02:36:25.747111+0000","last_unstale":"2024-09-21T02:38:40.502828+0000","last_undegraded":"2024-09-21T02:38:40.502828+0000","last_fullsized":"2024-09-21T02:38:40.502828+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:03:04.061537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.335704+0000","last_change":"2024-09-21T02:38:40.335626+0000","last_active":"2024-09-21T02:38:40.335704+0000","last_peered":"2024-09-21T02:38:40.335704+0000","last_clean":"2024-09-21T02:38:40.335704+0000","last_became_active":"2024-09-21T02:38:40.332735+0000","last_became_peered":"2024-09-21T02:38:40.332735+0000","last_unstale":"2024-09-21T02:38:40.335704+0000","last_undegraded":"2024-09-21T02:38:40.335704+0000","last_fullsized":"2024-09-21T02:38:40.335704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:42:37.763918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504172+0000","last_change":"2024-09-21T02:37:39.870041+0000","last_active":"2024-09-21T02:38:40.504172+0000","last_peered":"2024-09-21T02:38:40.504172+0000","last_clean":"2024-09-21T02:38:40.504172+0000","last_became_active":"2024-09-21T02:37:39.869518+0000","last_became_peered":"2024-09-21T02:37:39.869518+0000","last_unstale":"2024-09-21T02:38:40.504172+0000","last_undegraded":"2024-09-21T02:38:40.504172+0000","last_fullsized":"2024-09-21T02:38:40.504172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:20.276471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233047+0000","last_change":"2024-09-21T02:36:25.746139+0000","last_active":"2024-09-21T02:38:40.233047+0000","last_peered":"2024-09-21T02:38:40.233047+0000","last_clean":"2024-09-21T02:38:40.233047+0000","last_became_active":"2024-09-21T02:36:25.744418+0000","last_became_peered":"2024-09-21T02:36:25.744418+0000","last_unstale":"2024-09-21T02:38:40.233047+0000","last_undegraded":"2024-09-21T02:38:40.233047+0000","last_fullsized":"2024-09-21T02:38:40.233047+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:21.706782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503137+0000","last_change":"2024-09-21T02:36:24.261945+0000","last_active":"2024-09-21T02:38:40.503137+0000","last_peered":"2024-09-21T02:38:40.503137+0000","last_clean":"2024-09-21T02:38:40.503137+0000","last_became_active":"2024-09-21T02:36:24.261486+0000","last_became_peered":"2024-09-21T02:36:24.261486+0000","last_unstale":"2024-09-21T02:38:40.503137+0000","last_undegraded":"2024-09-21T02:38:40.503137+0000","last_fullsized":"2024-09-21T02:38:40.503137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:45:18.331035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224142+0000","last_change":"2024-09-21T02:36:32.799464+0000","last_active":"2024-09-21T02:38:40.224142+0000","last_peered":"2024-09-21T02:38:40.224142+0000","last_clean":"2024-09-21T02:38:40.224142+0000","last_became_active":"2024-09-21T02:36:32.799000+0000","last_became_peered":"2024-09-21T02:36:32.799000+0000","last_unstale":"2024-09-21T02:38:40.224142+0000","last_undegraded":"2024-09-21T02:38:40.224142+0000","last_fullsized":"2024-09-21T02:38:40.224142+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:56:36.271172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294464+0000","last_change":"2024-09-21T02:36:25.741151+0000","last_active":"2024-09-21T02:38:40.294464+0000","last_peered":"2024-09-21T02:38:40.294464+0000","last_clean":"2024-09-21T02:38:40.294464+0000","last_became_active":"2024-09-21T02:36:25.740803+0000","last_became_peered":"2024-09-21T02:36:25.740803+0000","last_unstale":"2024-09-21T02:38:40.294464+0000","last_undegraded":"2024-09-21T02:38:40.294464+0000","last_fullsized":"2024-09-21T02:38:40.294464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:44.585136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'18","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503589+0000","last_change":"2024-09-21T02:36:23.725155+0000","last_active":"2024-09-21T02:38:40.503589+0000","last_peered":"2024-09-21T02:38:40.503589+0000","last_clean":"2024-09-21T02:38:40.503589+0000","last_became_active":"2024-09-21T02:36:23.724791+0000","last_became_peered":"2024-09-21T02:36:23.724791+0000","last_unstale":"2024-09-21T02:38:40.503589+0000","last_undegraded":"2024-09-21T02:38:40.503589+0000","last_fullsized":"2024-09-21T02:38:40.503589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:06.520465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293747+0000","last_change":"2024-09-21T02:36:33.003401+0000","last_active":"2024-09-21T02:38:40.293747+0000","last_peered":"2024-09-21T02:38:40.293747+0000","last_clean":"2024-09-21T02:38:40.293747+0000","last_became_active":"2024-09-21T02:36:33.002930+0000","last_became_peered":"2024-09-21T02:36:33.002930+0000","last_unstale":"2024-09-21T02:38:40.293747+0000","last_undegraded":"2024-09-21T02:38:40.293747+0000","last_fullsized":"2024-09-21T02:38:40.293747+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:40:23.629763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222161+0000","last_change":"2024-09-21T02:36:25.735501+0000","last_active":"2024-09-21T02:38:40.222161+0000","last_peered":"2024-09-21T02:38:40.222161+0000","last_clean":"2024-09-21T02:38:40.222161+0000","last_became_active":"2024-09-21T02:36:25.734453+0000","last_became_peered":"2024-09-21T02:36:25.734453+0000","last_unstale":"2024-09-21T02:38:40.222161+0000","last_undegraded":"2024-09-21T02:38:40.222161+0000","last_fullsized":"2024-09-21T02:38:40.222161+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:50.238529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503639+0000","last_change":"2024-09-21T02:36:23.800338+0000","last_active":"2024-09-21T02:38:40.503639+0000","last_peered":"2024-09-21T02:38:40.503639+0000","last_clean":"2024-09-21T02:38:40.503639+0000","last_became_active":"2024-09-21T02:36:23.799796+0000","last_became_peered":"2024-09-21T02:36:23.799796+0000","last_unstale":"2024-09-21T02:38:40.503639+0000","last_undegraded":"2024-09-21T02:38:40.503639+0000","last_fullsized":"2024-09-21T02:38:40.503639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:00:18.459515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331347+0000","last_change":"2024-09-21T02:37:39.611513+0000","last_active":"2024-09-21T02:38:40.331347+0000","last_peered":"2024-09-21T02:38:40.331347+0000","last_clean":"2024-09-21T02:38:40.331347+0000","last_became_active":"2024-09-21T02:37:39.610843+0000","last_became_peered":"2024-09-21T02:37:39.610843+0000","last_unstale":"2024-09-21T02:38:40.331347+0000","last_undegraded":"2024-09-21T02:38:40.331347+0000","last_fullsized":"2024-09-21T02:38:40.331347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:51.911830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223967+0000","last_change":"2024-09-21T02:36:25.745497+0000","last_active":"2024-09-21T02:38:40.223967+0000","last_peered":"2024-09-21T02:38:40.223967+0000","last_clean":"2024-09-21T02:38:40.223967+0000","last_became_active":"2024-09-21T02:36:25.745126+0000","last_became_peered":"2024-09-21T02:36:25.745126+0000","last_unstale":"2024-09-21T02:38:40.223967+0000","last_undegraded":"2024-09-21T02:38:40.223967+0000","last_fullsized":"2024-09-21T02:38:40.223967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:42:17.835784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503417+0000","last_change":"2024-09-21T02:36:23.802440+0000","last_active":"2024-09-21T02:38:40.503417+0000","last_peered":"2024-09-21T02:38:40.503417+0000","last_clean":"2024-09-21T02:38:40.503417+0000","last_became_active":"2024-09-21T02:36:23.802065+0000","last_became_peered":"2024-09-21T02:36:23.802065+0000","last_unstale":"2024-09-21T02:38:40.503417+0000","last_undegraded":"2024-09-21T02:38:40.503417+0000","last_fullsized":"2024-09-21T02:38:40.503417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:22:20.647476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502936+0000","last_change":"2024-09-21T02:36:33.003465+0000","last_active":"2024-09-21T02:38:40.502936+0000","last_peered":"2024-09-21T02:38:40.502936+0000","last_clean":"2024-09-21T02:38:40.502936+0000","last_became_active":"2024-09-21T02:36:33.002874+0000","last_became_peered":"2024-09-21T02:36:33.002874+0000","last_unstale":"2024-09-21T02:38:40.502936+0000","last_undegraded":"2024-09-21T02:38:40.502936+0000","last_fullsized":"2024-09-21T02:38:40.502936+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:02.388770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224086+0000","last_change":"2024-09-21T02:36:25.746092+0000","last_active":"2024-09-21T02:38:40.224086+0000","last_peered":"2024-09-21T02:38:40.224086+0000","last_clean":"2024-09-21T02:38:40.224086+0000","last_became_active":"2024-09-21T02:36:25.745730+0000","last_became_peered":"2024-09-21T02:36:25.745730+0000","last_unstale":"2024-09-21T02:38:40.224086+0000","last_undegraded":"2024-09-21T02:38:40.224086+0000","last_fullsized":"2024-09-21T02:38:40.224086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:24:13.212450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'6","reported_seq":163,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:03.212549+0000","last_change":"2024-09-21T02:36:23.732090+0000","last_active":"2024-09-21T02:39:03.212549+0000","last_peered":"2024-09-21T02:39:03.212549+0000","last_clean":"2024-09-21T02:39:03.212549+0000","last_became_active":"2024-09-21T02:36:23.731653+0000","last_became_peered":"2024-09-21T02:36:23.731653+0000","last_unstale":"2024-09-21T02:39:03.212549+0000","last_undegraded":"2024-09-21T02:39:03.212549+0000","last_fullsized":"2024-09-21T02:39:03.212549+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:02:42.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222021+0000","last_change":"2024-09-21T02:36:32.794893+0000","last_active":"2024-09-21T02:38:40.222021+0000","last_peered":"2024-09-21T02:38:40.222021+0000","last_clean":"2024-09-21T02:38:40.222021+0000","last_became_active":"2024-09-21T02:36:32.794406+0000","last_became_peered":"2024-09-21T02:36:32.794406+0000","last_unstale":"2024-09-21T02:38:40.222021+0000","last_undegraded":"2024-09-21T02:38:40.222021+0000","last_fullsized":"2024-09-21T02:38:40.222021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:40:39.621502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504064+0000","last_change":"2024-09-21T02:36:25.748220+0000","last_active":"2024-09-21T02:38:40.504064+0000","last_peered":"2024-09-21T02:38:40.504064+0000","last_clean":"2024-09-21T02:38:40.504064+0000","last_became_active":"2024-09-21T02:36:25.747553+0000","last_became_peered":"2024-09-21T02:36:25.747553+0000","last_unstale":"2024-09-21T02:38:40.504064+0000","last_undegraded":"2024-09-21T02:38:40.504064+0000","last_fullsized":"2024-09-21T02:38:40.504064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:28.757981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'6","reported_seq":161,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.219800+0000","last_change":"2024-09-21T02:36:23.801087+0000","last_active":"2024-09-21T02:39:02.219800+0000","last_peered":"2024-09-21T02:39:02.219800+0000","last_clean":"2024-09-21T02:39:02.219800+0000","last_became_active":"2024-09-21T02:36:23.800077+0000","last_became_peered":"2024-09-21T02:36:23.800077+0000","last_unstale":"2024-09-21T02:39:02.219800+0000","last_undegraded":"2024-09-21T02:39:02.219800+0000","last_fullsized":"2024-09-21T02:39:02.219800+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:50.074650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222239+0000","last_change":"2024-09-21T02:36:33.003354+0000","last_active":"2024-09-21T02:38:40.222239+0000","last_peered":"2024-09-21T02:38:40.222239+0000","last_clean":"2024-09-21T02:38:40.222239+0000","last_became_active":"2024-09-21T02:36:33.002912+0000","last_became_peered":"2024-09-21T02:36:33.002912+0000","last_unstale":"2024-09-21T02:38:40.222239+0000","last_undegraded":"2024-09-21T02:38:40.222239+0000","last_fullsized":"2024-09-21T02:38:40.222239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:21.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221848+0000","last_change":"2024-09-21T02:36:25.741701+0000","last_active":"2024-09-21T02:38:40.221848+0000","last_peered":"2024-09-21T02:38:40.221848+0000","last_clean":"2024-09-21T02:38:40.221848+0000","last_became_active":"2024-09-21T02:36:25.741222+0000","last_became_peered":"2024-09-21T02:36:25.741222+0000","last_unstale":"2024-09-21T02:38:40.221848+0000","last_undegraded":"2024-09-21T02:38:40.221848+0000","last_fullsized":"2024-09-21T02:38:40.221848+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:35:39.896852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502899+0000","last_change":"2024-09-21T02:36:23.728793+0000","last_active":"2024-09-21T02:38:40.502899+0000","last_peered":"2024-09-21T02:38:40.502899+0000","last_clean":"2024-09-21T02:38:40.502899+0000","last_became_active":"2024-09-21T02:36:23.728362+0000","last_became_peered":"2024-09-21T02:36:23.728362+0000","last_unstale":"2024-09-21T02:38:40.502899+0000","last_undegraded":"2024-09-21T02:38:40.502899+0000","last_fullsized":"2024-09-21T02:38:40.502899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:28:15.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504244+0000","last_change":"2024-09-21T02:36:33.005225+0000","last_active":"2024-09-21T02:38:40.504244+0000","last_peered":"2024-09-21T02:38:40.504244+0000","last_clean":"2024-09-21T02:38:40.504244+0000","last_became_active":"2024-09-21T02:36:33.004756+0000","last_became_peered":"2024-09-21T02:36:33.004756+0000","last_unstale":"2024-09-21T02:38:40.504244+0000","last_undegraded":"2024-09-21T02:38:40.504244+0000","last_fullsized":"2024-09-21T02:38:40.504244+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:02:16.686284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294112+0000","last_change":"2024-09-21T02:36:25.745639+0000","last_active":"2024-09-21T02:38:40.294112+0000","last_peered":"2024-09-21T02:38:40.294112+0000","last_clean":"2024-09-21T02:38:40.294112+0000","last_became_active":"2024-09-21T02:36:25.745202+0000","last_became_peered":"2024-09-21T02:36:25.745202+0000","last_unstale":"2024-09-21T02:38:40.294112+0000","last_undegraded":"2024-09-21T02:38:40.294112+0000","last_fullsized":"2024-09-21T02:38:40.294112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:08.712477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504086+0000","last_change":"2024-09-21T02:36:23.730710+0000","last_active":"2024-09-21T02:38:40.504086+0000","last_peered":"2024-09-21T02:38:40.504086+0000","last_clean":"2024-09-21T02:38:40.504086+0000","last_became_active":"2024-09-21T02:36:23.730247+0000","last_became_peered":"2024-09-21T02:36:23.730247+0000","last_unstale":"2024-09-21T02:38:40.504086+0000","last_undegraded":"2024-09-21T02:38:40.504086+0000","last_fullsized":"2024-09-21T02:38:40.504086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:43.863444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":113,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503097+0000","last_change":"2024-09-21T02:35:09.949841+0000","last_active":"2024-09-21T02:38:40.503097+0000","last_peered":"2024-09-21T02:38:40.503097+0000","last_clean":"2024-09-21T02:38:40.503097+0000","last_became_active":"2024-09-21T02:35:09.944224+0000","last_became_peered":"2024-09-21T02:35:09.944224+0000","last_unstale":"2024-09-21T02:38:40.503097+0000","last_undegraded":"2024-09-21T02:38:40.503097+0000","last_fullsized":"2024-09-21T02:38:40.503097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_clean_scrub_stamp":"2024-09-21T02:31:38.930125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:56.467576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503366+0000","last_change":"2024-09-21T02:37:39.874486+0000","last_active":"2024-09-21T02:38:40.503366+0000","last_peered":"2024-09-21T02:38:40.503366+0000","last_clean":"2024-09-21T02:38:40.503366+0000","last_became_active":"2024-09-21T02:37:39.869523+0000","last_became_peered":"2024-09-21T02:37:39.869523+0000","last_unstale":"2024-09-21T02:38:40.503366+0000","last_undegraded":"2024-09-21T02:38:40.503366+0000","last_fullsized":"2024-09-21T02:38:40.503366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:21:33.524428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503759+0000","last_change":"2024-09-21T02:36:25.741176+0000","last_active":"2024-09-21T02:38:40.503759+0000","last_peered":"2024-09-21T02:38:40.503759+0000","last_clean":"2024-09-21T02:38:40.503759+0000","last_became_active":"2024-09-21T02:36:25.740355+0000","last_became_peered":"2024-09-21T02:36:25.740355+0000","last_unstale":"2024-09-21T02:38:40.503759+0000","last_undegraded":"2024-09-21T02:38:40.503759+0000","last_fullsized":"2024-09-21T02:38:40.503759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:47:48.204768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'8","reported_seq":166,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.214476+0000","last_change":"2024-09-21T02:36:24.261500+0000","last_active":"2024-09-21T02:39:02.214476+0000","last_peered":"2024-09-21T02:39:02.214476+0000","last_clean":"2024-09-21T02:39:02.214476+0000","last_became_active":"2024-09-21T02:36:24.261089+0000","last_became_peered":"2024-09-21T02:36:24.261089+0000","last_unstale":"2024-09-21T02:39:02.214476+0000","last_undegraded":"2024-09-21T02:39:02.214476+0000","last_fullsized":"2024-09-21T02:39:02.214476+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:33:33.158177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2195,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227148+0000","last_change":"2024-09-21T02:36:32.796980+0000","last_active":"2024-09-21T02:38:40.227148+0000","last_peered":"2024-09-21T02:38:40.227148+0000","last_clean":"2024-09-21T02:38:40.227148+0000","last_became_active":"2024-09-21T02:36:32.796517+0000","last_became_peered":"2024-09-21T02:36:32.796517+0000","last_unstale":"2024-09-21T02:38:40.227148+0000","last_undegraded":"2024-09-21T02:38:40.227148+0000","last_fullsized":"2024-09-21T02:38:40.227148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:07.230177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502998+0000","last_change":"2024-09-21T02:36:25.750931+0000","last_active":"2024-09-21T02:38:40.502998+0000","last_peered":"2024-09-21T02:38:40.502998+0000","last_clean":"2024-09-21T02:38:40.502998+0000","last_became_active":"2024-09-21T02:36:25.750604+0000","last_became_peered":"2024-09-21T02:36:25.750604+0000","last_unstale":"2024-09-21T02:38:40.502998+0000","last_undegraded":"2024-09-21T02:38:40.502998+0000","last_fullsized":"2024-09-21T02:38:40.502998+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:21.325095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504562+0000","last_change":"2024-09-21T02:36:23.730700+0000","last_active":"2024-09-21T02:38:40.504562+0000","last_peered":"2024-09-21T02:38:40.504562+0000","last_clean":"2024-09-21T02:38:40.504562+0000","last_became_active":"2024-09-21T02:36:23.730367+0000","last_became_peered":"2024-09-21T02:36:23.730367+0000","last_unstale":"2024-09-21T02:38:40.504562+0000","last_undegraded":"2024-09-21T02:38:40.504562+0000","last_fullsized":"2024-09-21T02:38:40.504562+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:27:57.373077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233491+0000","last_change":"2024-09-21T02:36:32.805885+0000","last_active":"2024-09-21T02:38:40.233491+0000","last_peered":"2024-09-21T02:38:40.233491+0000","last_clean":"2024-09-21T02:38:40.233491+0000","last_became_active":"2024-09-21T02:36:32.805479+0000","last_became_peered":"2024-09-21T02:36:32.805479+0000","last_unstale":"2024-09-21T02:38:40.233491+0000","last_undegraded":"2024-09-21T02:38:40.233491+0000","last_fullsized":"2024-09-21T02:38:40.233491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:11:29.910642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503631+0000","last_change":"2024-09-21T02:36:25.742542+0000","last_active":"2024-09-21T02:38:40.503631+0000","last_peered":"2024-09-21T02:38:40.503631+0000","last_clean":"2024-09-21T02:38:40.503631+0000","last_became_active":"2024-09-21T02:36:25.742000+0000","last_became_peered":"2024-09-21T02:36:25.742000+0000","last_unstale":"2024-09-21T02:38:40.503631+0000","last_undegraded":"2024-09-21T02:38:40.503631+0000","last_fullsized":"2024-09-21T02:38:40.503631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:52.300212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222446+0000","last_change":"2024-09-21T02:36:23.723896+0000","last_active":"2024-09-21T02:38:40.222446+0000","last_peered":"2024-09-21T02:38:40.222446+0000","last_clean":"2024-09-21T02:38:40.222446+0000","last_became_active":"2024-09-21T02:36:23.723518+0000","last_became_peered":"2024-09-21T02:36:23.723518+0000","last_unstale":"2024-09-21T02:38:40.222446+0000","last_undegraded":"2024-09-21T02:38:40.222446+0000","last_fullsized":"2024-09-21T02:38:40.222446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:37.250247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503797+0000","last_change":"2024-09-21T02:36:32.799924+0000","last_active":"2024-09-21T02:38:40.503797+0000","last_peered":"2024-09-21T02:38:40.503797+0000","last_clean":"2024-09-21T02:38:40.503797+0000","last_became_active":"2024-09-21T02:36:32.799437+0000","last_became_peered":"2024-09-21T02:36:32.799437+0000","last_unstale":"2024-09-21T02:38:40.503797+0000","last_undegraded":"2024-09-21T02:38:40.503797+0000","last_fullsized":"2024-09-21T02:38:40.503797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:19.258293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331925+0000","last_change":"2024-09-21T02:36:25.747321+0000","last_active":"2024-09-21T02:38:40.331925+0000","last_peered":"2024-09-21T02:38:40.331925+0000","last_clean":"2024-09-21T02:38:40.331925+0000","last_became_active":"2024-09-21T02:36:25.746955+0000","last_became_peered":"2024-09-21T02:36:25.746955+0000","last_unstale":"2024-09-21T02:38:40.331925+0000","last_undegraded":"2024-09-21T02:38:40.331925+0000","last_fullsized":"2024-09-21T02:38:40.331925+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:25.148300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503704+0000","last_change":"2024-09-21T02:36:23.725350+0000","last_active":"2024-09-21T02:38:40.503704+0000","last_peered":"2024-09-21T02:38:40.503704+0000","last_clean":"2024-09-21T02:38:40.503704+0000","last_became_active":"2024-09-21T02:36:23.725090+0000","last_became_peered":"2024-09-21T02:36:23.725090+0000","last_unstale":"2024-09-21T02:38:40.503704+0000","last_undegraded":"2024-09-21T02:38:40.503704+0000","last_fullsized":"2024-09-21T02:38:40.503704+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:52.130345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503689+0000","last_change":"2024-09-21T02:36:32.801355+0000","last_active":"2024-09-21T02:38:40.503689+0000","last_peered":"2024-09-21T02:38:40.503689+0000","last_clean":"2024-09-21T02:38:40.503689+0000","last_became_active":"2024-09-21T02:36:32.800803+0000","last_became_peered":"2024-09-21T02:36:32.800803+0000","last_unstale":"2024-09-21T02:38:40.503689+0000","last_undegraded":"2024-09-21T02:38:40.503689+0000","last_fullsized":"2024-09-21T02:38:40.503689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:54.755733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504077+0000","last_change":"2024-09-21T02:36:25.743433+0000","last_active":"2024-09-21T02:38:40.504077+0000","last_peered":"2024-09-21T02:38:40.504077+0000","last_clean":"2024-09-21T02:38:40.504077+0000","last_became_active":"2024-09-21T02:36:25.742643+0000","last_became_peered":"2024-09-21T02:36:25.742643+0000","last_unstale":"2024-09-21T02:38:40.504077+0000","last_undegraded":"2024-09-21T02:38:40.504077+0000","last_fullsized":"2024-09-21T02:38:40.504077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:20:09.849806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331234+0000","last_change":"2024-09-21T02:37:39.625360+0000","last_active":"2024-09-21T02:38:40.331234+0000","last_peered":"2024-09-21T02:38:40.331234+0000","last_clean":"2024-09-21T02:38:40.331234+0000","last_became_active":"2024-09-21T02:37:39.611781+0000","last_became_peered":"2024-09-21T02:37:39.611781+0000","last_unstale":"2024-09-21T02:38:40.331234+0000","last_undegraded":"2024-09-21T02:38:40.331234+0000","last_fullsized":"2024-09-21T02:38:40.331234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:11.846383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222385+0000","last_change":"2024-09-21T02:36:32.799728+0000","last_active":"2024-09-21T02:38:40.222385+0000","last_peered":"2024-09-21T02:38:40.222385+0000","last_clean":"2024-09-21T02:38:40.222385+0000","last_became_active":"2024-09-21T02:36:32.798419+0000","last_became_peered":"2024-09-21T02:36:32.798419+0000","last_unstale":"2024-09-21T02:38:40.222385+0000","last_undegraded":"2024-09-21T02:38:40.222385+0000","last_fullsized":"2024-09-21T02:38:40.222385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:53.681473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502716+0000","last_change":"2024-09-21T02:36:25.747712+0000","last_active":"2024-09-21T02:38:40.502716+0000","last_peered":"2024-09-21T02:38:40.502716+0000","last_clean":"2024-09-21T02:38:40.502716+0000","last_became_active":"2024-09-21T02:36:25.747197+0000","last_became_peered":"2024-09-21T02:36:25.747197+0000","last_unstale":"2024-09-21T02:38:40.502716+0000","last_undegraded":"2024-09-21T02:38:40.502716+0000","last_fullsized":"2024-09-21T02:38:40.502716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:03.900360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":165,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:04.214988+0000","last_change":"2024-09-21T02:36:23.732156+0000","last_active":"2024-09-21T02:39:04.214988+0000","last_peered":"2024-09-21T02:39:04.214988+0000","last_clean":"2024-09-21T02:39:04.214988+0000","last_became_active":"2024-09-21T02:36:23.730238+0000","last_became_peered":"2024-09-21T02:36:23.730238+0000","last_unstale":"2024-09-21T02:39:04.214988+0000","last_undegraded":"2024-09-21T02:39:04.214988+0000","last_fullsized":"2024-09-21T02:39:04.214988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:18.053890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503510+0000","last_change":"2024-09-21T02:36:32.806674+0000","last_active":"2024-09-21T02:38:40.503510+0000","last_peered":"2024-09-21T02:38:40.503510+0000","last_clean":"2024-09-21T02:38:40.503510+0000","last_became_active":"2024-09-21T02:36:32.805373+0000","last_became_peered":"2024-09-21T02:36:32.805373+0000","last_unstale":"2024-09-21T02:38:40.503510+0000","last_undegraded":"2024-09-21T02:38:40.503510+0000","last_fullsized":"2024-09-21T02:38:40.503510+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:29.249243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503029+0000","last_change":"2024-09-21T02:36:25.748703+0000","last_active":"2024-09-21T02:38:40.503029+0000","last_peered":"2024-09-21T02:38:40.503029+0000","last_clean":"2024-09-21T02:38:40.503029+0000","last_became_active":"2024-09-21T02:36:25.748340+0000","last_became_peered":"2024-09-21T02:36:25.748340+0000","last_unstale":"2024-09-21T02:38:40.503029+0000","last_undegraded":"2024-09-21T02:38:40.503029+0000","last_fullsized":"2024-09-21T02:38:40.503029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:07.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504103+0000","last_change":"2024-09-21T02:36:23.732332+0000","last_active":"2024-09-21T02:38:40.504103+0000","last_peered":"2024-09-21T02:38:40.504103+0000","last_clean":"2024-09-21T02:38:40.504103+0000","last_became_active":"2024-09-21T02:36:23.731974+0000","last_became_peered":"2024-09-21T02:36:23.731974+0000","last_unstale":"2024-09-21T02:38:40.504103+0000","last_undegraded":"2024-09-21T02:38:40.504103+0000","last_fullsized":"2024-09-21T02:38:40.504103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:52.284192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331545+0000","last_change":"2024-09-21T02:37:39.610698+0000","last_active":"2024-09-21T02:38:40.331545+0000","last_peered":"2024-09-21T02:38:40.331545+0000","last_clean":"2024-09-21T02:38:40.331545+0000","last_became_active":"2024-09-21T02:37:39.610185+0000","last_became_peered":"2024-09-21T02:37:39.610185+0000","last_unstale":"2024-09-21T02:38:40.331545+0000","last_undegraded":"2024-09-21T02:38:40.331545+0000","last_fullsized":"2024-09-21T02:38:40.331545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:54:51.050463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221965+0000","last_change":"2024-09-21T02:36:25.751720+0000","last_active":"2024-09-21T02:38:40.221965+0000","last_peered":"2024-09-21T02:38:40.221965+0000","last_clean":"2024-09-21T02:38:40.221965+0000","last_became_active":"2024-09-21T02:36:25.751351+0000","last_became_peered":"2024-09-21T02:36:25.751351+0000","last_unstale":"2024-09-21T02:38:40.221965+0000","last_undegraded":"2024-09-21T02:38:40.221965+0000","last_fullsized":"2024-09-21T02:38:40.221965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:01.201453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224603+0000","last_change":"2024-09-21T02:36:23.729981+0000","last_active":"2024-09-21T02:38:40.224603+0000","last_peered":"2024-09-21T02:38:40.224603+0000","last_clean":"2024-09-21T02:38:40.224603+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:38:40.224603+0000","last_undegraded":"2024-09-21T02:38:40.224603+0000","last_fullsized":"2024-09-21T02:38:40.224603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:03.115712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504439+0000","last_change":"2024-09-21T02:36:32.805286+0000","last_active":"2024-09-21T02:38:40.504439+0000","last_peered":"2024-09-21T02:38:40.504439+0000","last_clean":"2024-09-21T02:38:40.504439+0000","last_became_active":"2024-09-21T02:36:32.804770+0000","last_became_peered":"2024-09-21T02:36:32.804770+0000","last_unstale":"2024-09-21T02:38:40.504439+0000","last_undegraded":"2024-09-21T02:38:40.504439+0000","last_fullsized":"2024-09-21T02:38:40.504439+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:18:34.155892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503496+0000","last_change":"2024-09-21T02:36:25.747812+0000","last_active":"2024-09-21T02:38:40.503496+0000","last_peered":"2024-09-21T02:38:40.503496+0000","last_clean":"2024-09-21T02:38:40.503496+0000","last_became_active":"2024-09-21T02:36:25.747187+0000","last_became_peered":"2024-09-21T02:36:25.747187+0000","last_unstale":"2024-09-21T02:38:40.503496+0000","last_undegraded":"2024-09-21T02:38:40.503496+0000","last_fullsized":"2024-09-21T02:38:40.503496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:51.395869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":316,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.223137+0000","last_change":"2024-09-21T02:36:23.735676+0000","last_active":"2024-09-21T02:39:02.223137+0000","last_peered":"2024-09-21T02:39:02.223137+0000","last_clean":"2024-09-21T02:39:02.223137+0000","last_became_active":"2024-09-21T02:36:23.735281+0000","last_became_peered":"2024-09-21T02:36:23.735281+0000","last_unstale":"2024-09-21T02:39:02.223137+0000","last_undegraded":"2024-09-21T02:39:02.223137+0000","last_fullsized":"2024-09-21T02:39:02.223137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:28:01.547245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68362,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":236,"num_read_kb":184,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503129+0000","last_change":"2024-09-21T02:36:32.806585+0000","last_active":"2024-09-21T02:38:40.503129+0000","last_peered":"2024-09-21T02:38:40.503129+0000","last_clean":"2024-09-21T02:38:40.503129+0000","last_became_active":"2024-09-21T02:36:32.805913+0000","last_became_peered":"2024-09-21T02:36:32.805913+0000","last_unstale":"2024-09-21T02:38:40.503129+0000","last_undegraded":"2024-09-21T02:38:40.503129+0000","last_fullsized":"2024-09-21T02:38:40.503129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:59:31.888645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228365+0000","last_change":"2024-09-21T02:36:25.742208+0000","last_active":"2024-09-21T02:38:40.228365+0000","last_peered":"2024-09-21T02:38:40.228365+0000","last_clean":"2024-09-21T02:38:40.228365+0000","last_became_active":"2024-09-21T02:36:25.741849+0000","last_became_peered":"2024-09-21T02:36:25.741849+0000","last_unstale":"2024-09-21T02:38:40.228365+0000","last_undegraded":"2024-09-21T02:38:40.228365+0000","last_fullsized":"2024-09-21T02:38:40.228365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:21:43.720481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504523+0000","last_change":"2024-09-21T02:36:23.800751+0000","last_active":"2024-09-21T02:38:40.504523+0000","last_peered":"2024-09-21T02:38:40.504523+0000","last_clean":"2024-09-21T02:38:40.504523+0000","last_became_active":"2024-09-21T02:36:23.800341+0000","last_became_peered":"2024-09-21T02:36:23.800341+0000","last_unstale":"2024-09-21T02:38:40.504523+0000","last_undegraded":"2024-09-21T02:38:40.504523+0000","last_fullsized":"2024-09-21T02:38:40.504523+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:08:02.556549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503710+0000","last_change":"2024-09-21T02:36:32.802879+0000","last_active":"2024-09-21T02:38:40.503710+0000","last_peered":"2024-09-21T02:38:40.503710+0000","last_clean":"2024-09-21T02:38:40.503710+0000","last_became_active":"2024-09-21T02:36:32.802301+0000","last_became_peered":"2024-09-21T02:36:32.802301+0000","last_unstale":"2024-09-21T02:38:40.503710+0000","last_undegraded":"2024-09-21T02:38:40.503710+0000","last_fullsized":"2024-09-21T02:38:40.503710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:08.135814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232720+0000","last_change":"2024-09-21T02:36:25.744647+0000","last_active":"2024-09-21T02:38:40.232720+0000","last_peered":"2024-09-21T02:38:40.232720+0000","last_clean":"2024-09-21T02:38:40.232720+0000","last_became_active":"2024-09-21T02:36:25.744049+0000","last_became_peered":"2024-09-21T02:36:25.744049+0000","last_unstale":"2024-09-21T02:38:40.232720+0000","last_undegraded":"2024-09-21T02:38:40.232720+0000","last_fullsized":"2024-09-21T02:38:40.232720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:52:02.210380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503652+0000","last_change":"2024-09-21T02:36:23.733590+0000","last_active":"2024-09-21T02:38:40.503652+0000","last_peered":"2024-09-21T02:38:40.503652+0000","last_clean":"2024-09-21T02:38:40.503652+0000","last_became_active":"2024-09-21T02:36:23.732922+0000","last_became_peered":"2024-09-21T02:36:23.732922+0000","last_unstale":"2024-09-21T02:38:40.503652+0000","last_undegraded":"2024-09-21T02:38:40.503652+0000","last_fullsized":"2024-09-21T02:38:40.503652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503016+0000","last_change":"2024-09-21T02:36:32.806540+0000","last_active":"2024-09-21T02:38:40.503016+0000","last_peered":"2024-09-21T02:38:40.503016+0000","last_clean":"2024-09-21T02:38:40.503016+0000","last_became_active":"2024-09-21T02:36:32.805612+0000","last_became_peered":"2024-09-21T02:36:32.805612+0000","last_unstale":"2024-09-21T02:38:40.503016+0000","last_undegraded":"2024-09-21T02:38:40.503016+0000","last_fullsized":"2024-09-21T02:38:40.503016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:23:24.686383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227650+0000","last_change":"2024-09-21T02:36:25.746229+0000","last_active":"2024-09-21T02:38:40.227650+0000","last_peered":"2024-09-21T02:38:40.227650+0000","last_clean":"2024-09-21T02:38:40.227650+0000","last_became_active":"2024-09-21T02:36:25.745892+0000","last_became_peered":"2024-09-21T02:36:25.745892+0000","last_unstale":"2024-09-21T02:38:40.227650+0000","last_undegraded":"2024-09-21T02:38:40.227650+0000","last_fullsized":"2024-09-21T02:38:40.227650+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:55.154499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"85'8","reported_seq":274,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.213998+0000","last_change":"2024-09-21T02:36:23.730987+0000","last_active":"2024-09-21T02:39:02.213998+0000","last_peered":"2024-09-21T02:39:02.213998+0000","last_clean":"2024-09-21T02:39:02.213998+0000","last_became_active":"2024-09-21T02:36:23.730566+0000","last_became_peered":"2024-09-21T02:36:23.730566+0000","last_unstale":"2024-09-21T02:39:02.213998+0000","last_undegraded":"2024-09-21T02:39:02.213998+0000","last_fullsized":"2024-09-21T02:39:02.213998+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:18:12.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293972+0000","last_change":"2024-09-21T02:36:32.804608+0000","last_active":"2024-09-21T02:38:40.293972+0000","last_peered":"2024-09-21T02:38:40.293972+0000","last_clean":"2024-09-21T02:38:40.293972+0000","last_became_active":"2024-09-21T02:36:32.803567+0000","last_became_peered":"2024-09-21T02:36:32.803567+0000","last_unstale":"2024-09-21T02:38:40.293972+0000","last_undegraded":"2024-09-21T02:38:40.293972+0000","last_fullsized":"2024-09-21T02:38:40.293972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:41.692644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232808+0000","last_change":"2024-09-21T02:36:32.804687+0000","last_active":"2024-09-21T02:38:40.232808+0000","last_peered":"2024-09-21T02:38:40.232808+0000","last_clean":"2024-09-21T02:38:40.232808+0000","last_became_active":"2024-09-21T02:36:32.804232+0000","last_became_peered":"2024-09-21T02:36:32.804232+0000","last_unstale":"2024-09-21T02:38:40.232808+0000","last_undegraded":"2024-09-21T02:38:40.232808+0000","last_fullsized":"2024-09-21T02:38:40.232808+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:43.899924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503127+0000","last_change":"2024-09-21T02:36:25.749114+0000","last_active":"2024-09-21T02:38:40.503127+0000","last_peered":"2024-09-21T02:38:40.503127+0000","last_clean":"2024-09-21T02:38:40.503127+0000","last_became_active":"2024-09-21T02:36:25.748758+0000","last_became_peered":"2024-09-21T02:36:25.748758+0000","last_unstale":"2024-09-21T02:38:40.503127+0000","last_undegraded":"2024-09-21T02:38:40.503127+0000","last_fullsized":"2024-09-21T02:38:40.503127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:57.958533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504282+0000","last_change":"2024-09-21T02:36:23.731473+0000","last_active":"2024-09-21T02:38:40.504282+0000","last_peered":"2024-09-21T02:38:40.504282+0000","last_clean":"2024-09-21T02:38:40.504282+0000","last_became_active":"2024-09-21T02:36:23.731134+0000","last_became_peered":"2024-09-21T02:36:23.731134+0000","last_unstale":"2024-09-21T02:38:40.504282+0000","last_undegraded":"2024-09-21T02:38:40.504282+0000","last_fullsized":"2024-09-21T02:38:40.504282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:05.368941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331458+0000","last_change":"2024-09-21T02:36:32.809630+0000","last_active":"2024-09-21T02:38:40.331458+0000","last_peered":"2024-09-21T02:38:40.331458+0000","last_clean":"2024-09-21T02:38:40.331458+0000","last_became_active":"2024-09-21T02:36:32.809030+0000","last_became_peered":"2024-09-21T02:36:32.809030+0000","last_unstale":"2024-09-21T02:38:40.331458+0000","last_undegraded":"2024-09-21T02:38:40.331458+0000","last_fullsized":"2024-09-21T02:38:40.331458+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:57:42.436590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293966+0000","last_change":"2024-09-21T02:36:25.742558+0000","last_active":"2024-09-21T02:38:40.293966+0000","last_peered":"2024-09-21T02:38:40.293966+0000","last_clean":"2024-09-21T02:38:40.293966+0000","last_became_active":"2024-09-21T02:36:25.742045+0000","last_became_peered":"2024-09-21T02:36:25.742045+0000","last_unstale":"2024-09-21T02:38:40.293966+0000","last_undegraded":"2024-09-21T02:38:40.293966+0000","last_fullsized":"2024-09-21T02:38:40.293966+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:57:06.705561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221778+0000","last_change":"2024-09-21T02:36:23.724345+0000","last_active":"2024-09-21T02:38:40.221778+0000","last_peered":"2024-09-21T02:38:40.221778+0000","last_clean":"2024-09-21T02:38:40.221778+0000","last_became_active":"2024-09-21T02:36:23.723635+0000","last_became_peered":"2024-09-21T02:36:23.723635+0000","last_unstale":"2024-09-21T02:38:40.221778+0000","last_undegraded":"2024-09-21T02:38:40.221778+0000","last_fullsized":"2024-09-21T02:38:40.221778+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:18.812176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223962+0000","last_change":"2024-09-21T02:36:23.732897+0000","last_active":"2024-09-21T02:38:40.223962+0000","last_peered":"2024-09-21T02:38:40.223962+0000","last_clean":"2024-09-21T02:38:40.223962+0000","last_became_active":"2024-09-21T02:36:23.732509+0000","last_became_peered":"2024-09-21T02:36:23.732509+0000","last_unstale":"2024-09-21T02:38:40.223962+0000","last_undegraded":"2024-09-21T02:38:40.223962+0000","last_fullsized":"2024-09-21T02:38:40.223962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:12:27.117962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504315+0000","last_change":"2024-09-21T02:36:25.748929+0000","last_active":"2024-09-21T02:38:40.504315+0000","last_peered":"2024-09-21T02:38:40.504315+0000","last_clean":"2024-09-21T02:38:40.504315+0000","last_became_active":"2024-09-21T02:36:25.748588+0000","last_became_peered":"2024-09-21T02:36:25.748588+0000","last_unstale":"2024-09-21T02:38:40.504315+0000","last_undegraded":"2024-09-21T02:38:40.504315+0000","last_fullsized":"2024-09-21T02:38:40.504315+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.086472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224498+0000","last_change":"2024-09-21T02:36:32.807655+0000","last_active":"2024-09-21T02:38:40.224498+0000","last_peered":"2024-09-21T02:38:40.224498+0000","last_clean":"2024-09-21T02:38:40.224498+0000","last_became_active":"2024-09-21T02:36:32.806995+0000","last_became_peered":"2024-09-21T02:36:32.806995+0000","last_unstale":"2024-09-21T02:38:40.224498+0000","last_undegraded":"2024-09-21T02:38:40.224498+0000","last_fullsized":"2024-09-21T02:38:40.224498+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:28:20.420591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'7","reported_seq":163,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:00.209025+0000","last_change":"2024-09-21T02:36:23.729427+0000","last_active":"2024-09-21T02:39:00.209025+0000","last_peered":"2024-09-21T02:39:00.209025+0000","last_clean":"2024-09-21T02:39:00.209025+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:39:00.209025+0000","last_undegraded":"2024-09-21T02:39:00.209025+0000","last_fullsized":"2024-09-21T02:39:00.209025+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:10:41.251665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502991+0000","last_change":"2024-09-21T02:36:25.743212+0000","last_active":"2024-09-21T02:38:40.502991+0000","last_peered":"2024-09-21T02:38:40.502991+0000","last_clean":"2024-09-21T02:38:40.502991+0000","last_became_active":"2024-09-21T02:36:25.742777+0000","last_became_peered":"2024-09-21T02:36:25.742777+0000","last_unstale":"2024-09-21T02:38:40.502991+0000","last_undegraded":"2024-09-21T02:38:40.502991+0000","last_fullsized":"2024-09-21T02:38:40.502991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:12:19.917214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224485+0000","last_change":"2024-09-21T02:36:32.801880+0000","last_active":"2024-09-21T02:38:40.224485+0000","last_peered":"2024-09-21T02:38:40.224485+0000","last_clean":"2024-09-21T02:38:40.224485+0000","last_became_active":"2024-09-21T02:36:32.801389+0000","last_became_peered":"2024-09-21T02:36:32.801389+0000","last_unstale":"2024-09-21T02:38:40.224485+0000","last_undegraded":"2024-09-21T02:38:40.224485+0000","last_fullsized":"2024-09-21T02:38:40.224485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:11:44.877062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331561+0000","last_change":"2024-09-21T02:36:25.754014+0000","last_active":"2024-09-21T02:38:40.331561+0000","last_peered":"2024-09-21T02:38:40.331561+0000","last_clean":"2024-09-21T02:38:40.331561+0000","last_became_active":"2024-09-21T02:36:25.753548+0000","last_became_peered":"2024-09-21T02:36:25.753548+0000","last_unstale":"2024-09-21T02:38:40.331561+0000","last_undegraded":"2024-09-21T02:38:40.331561+0000","last_fullsized":"2024-09-21T02:38:40.331561+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:03.850625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224637+0000","last_change":"2024-09-21T02:37:39.610894+0000","last_active":"2024-09-21T02:38:40.224637+0000","last_peered":"2024-09-21T02:38:40.224637+0000","last_clean":"2024-09-21T02:38:40.224637+0000","last_became_active":"2024-09-21T02:37:39.610404+0000","last_became_peered":"2024-09-21T02:37:39.610404+0000","last_unstale":"2024-09-21T02:38:40.224637+0000","last_undegraded":"2024-09-21T02:38:40.224637+0000","last_fullsized":"2024-09-21T02:38:40.224637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:54.347708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224163+0000","last_change":"2024-09-21T02:36:25.752839+0000","last_active":"2024-09-21T02:38:40.224163+0000","last_peered":"2024-09-21T02:38:40.224163+0000","last_clean":"2024-09-21T02:38:40.224163+0000","last_became_active":"2024-09-21T02:36:25.750882+0000","last_became_peered":"2024-09-21T02:36:25.750882+0000","last_unstale":"2024-09-21T02:38:40.224163+0000","last_undegraded":"2024-09-21T02:38:40.224163+0000","last_fullsized":"2024-09-21T02:38:40.224163+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:53.028611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233548+0000","last_change":"2024-09-21T02:36:25.744624+0000","last_active":"2024-09-21T02:38:40.233548+0000","last_peered":"2024-09-21T02:38:40.233548+0000","last_clean":"2024-09-21T02:38:40.233548+0000","last_became_active":"2024-09-21T02:36:25.744173+0000","last_became_peered":"2024-09-21T02:36:25.744173+0000","last_unstale":"2024-09-21T02:38:40.233548+0000","last_undegraded":"2024-09-21T02:38:40.233548+0000","last_fullsized":"2024-09-21T02:38:40.233548+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:55.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222157+0000","last_change":"2024-09-21T02:37:39.609050+0000","last_active":"2024-09-21T02:38:40.222157+0000","last_peered":"2024-09-21T02:38:40.222157+0000","last_clean":"2024-09-21T02:38:40.222157+0000","last_became_active":"2024-09-21T02:37:39.608595+0000","last_became_peered":"2024-09-21T02:37:39.608595+0000","last_unstale":"2024-09-21T02:38:40.222157+0000","last_undegraded":"2024-09-21T02:38:40.222157+0000","last_fullsized":"2024-09-21T02:38:40.222157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:26.831121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331839+0000","last_change":"2024-09-21T02:36:25.743476+0000","last_active":"2024-09-21T02:38:40.331839+0000","last_peered":"2024-09-21T02:38:40.331839+0000","last_clean":"2024-09-21T02:38:40.331839+0000","last_became_active":"2024-09-21T02:36:25.743117+0000","last_became_peered":"2024-09-21T02:36:25.743117+0000","last_unstale":"2024-09-21T02:38:40.331839+0000","last_undegraded":"2024-09-21T02:38:40.331839+0000","last_fullsized":"2024-09-21T02:38:40.331839+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:08.084109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224831+0000","last_change":"2024-09-21T02:36:25.753014+0000","last_active":"2024-09-21T02:38:40.224831+0000","last_peered":"2024-09-21T02:38:40.224831+0000","last_clean":"2024-09-21T02:38:40.224831+0000","last_became_active":"2024-09-21T02:36:25.751232+0000","last_became_peered":"2024-09-21T02:36:25.751232+0000","last_unstale":"2024-09-21T02:38:40.224831+0000","last_undegraded":"2024-09-21T02:38:40.224831+0000","last_fullsized":"2024-09-21T02:38:40.224831+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:03.052737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503339+0000","last_change":"2024-09-21T02:37:39.606525+0000","last_active":"2024-09-21T02:38:40.503339+0000","last_peered":"2024-09-21T02:38:40.503339+0000","last_clean":"2024-09-21T02:38:40.503339+0000","last_became_active":"2024-09-21T02:37:39.606078+0000","last_became_peered":"2024-09-21T02:37:39.606078+0000","last_unstale":"2024-09-21T02:38:40.503339+0000","last_undegraded":"2024-09-21T02:38:40.503339+0000","last_fullsized":"2024-09-21T02:38:40.503339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:22.423457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503992+0000","last_change":"2024-09-21T02:36:25.749605+0000","last_active":"2024-09-21T02:38:40.503992+0000","last_peered":"2024-09-21T02:38:40.503992+0000","last_clean":"2024-09-21T02:38:40.503992+0000","last_became_active":"2024-09-21T02:36:25.749165+0000","last_became_peered":"2024-09-21T02:36:25.749165+0000","last_unstale":"2024-09-21T02:38:40.503992+0000","last_undegraded":"2024-09-21T02:38:40.503992+0000","last_fullsized":"2024-09-21T02:38:40.503992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:15:51.412761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294064+0000","last_change":"2024-09-21T02:36:25.747494+0000","last_active":"2024-09-21T02:38:40.294064+0000","last_peered":"2024-09-21T02:38:40.294064+0000","last_clean":"2024-09-21T02:38:40.294064+0000","last_became_active":"2024-09-21T02:36:25.746965+0000","last_became_peered":"2024-09-21T02:36:25.746965+0000","last_unstale":"2024-09-21T02:38:40.294064+0000","last_undegraded":"2024-09-21T02:38:40.294064+0000","last_fullsized":"2024-09-21T02:38:40.294064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:51.309307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227481+0000","last_change":"2024-09-21T02:36:25.749517+0000","last_active":"2024-09-21T02:38:40.227481+0000","last_peered":"2024-09-21T02:38:40.227481+0000","last_clean":"2024-09-21T02:38:40.227481+0000","last_became_active":"2024-09-21T02:36:25.749117+0000","last_became_peered":"2024-09-21T02:36:25.749117+0000","last_unstale":"2024-09-21T02:38:40.227481+0000","last_undegraded":"2024-09-21T02:38:40.227481+0000","last_fullsized":"2024-09-21T02:38:40.227481+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:38.758019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222549+0000","last_change":"2024-09-21T02:36:25.736067+0000","last_active":"2024-09-21T02:38:40.222549+0000","last_peered":"2024-09-21T02:38:40.222549+0000","last_clean":"2024-09-21T02:38:40.222549+0000","last_became_active":"2024-09-21T02:36:25.735710+0000","last_became_peered":"2024-09-21T02:36:25.735710+0000","last_unstale":"2024-09-21T02:38:40.222549+0000","last_undegraded":"2024-09-21T02:38:40.222549+0000","last_fullsized":"2024-09-21T02:38:40.222549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.559448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331864+0000","last_change":"2024-09-21T02:36:25.746307+0000","last_active":"2024-09-21T02:38:40.331864+0000","last_peered":"2024-09-21T02:38:40.331864+0000","last_clean":"2024-09-21T02:38:40.331864+0000","last_became_active":"2024-09-21T02:36:25.745872+0000","last_became_peered":"2024-09-21T02:36:25.745872+0000","last_unstale":"2024-09-21T02:38:40.331864+0000","last_undegraded":"2024-09-21T02:38:40.331864+0000","last_fullsized":"2024-09-21T02:38:40.331864+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:58:53.074559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224609+0000","last_change":"2024-09-21T02:36:25.753393+0000","last_active":"2024-09-21T02:38:40.224609+0000","last_peered":"2024-09-21T02:38:40.224609+0000","last_clean":"2024-09-21T02:38:40.224609+0000","last_became_active":"2024-09-21T02:36:25.751871+0000","last_became_peered":"2024-09-21T02:36:25.751871+0000","last_unstale":"2024-09-21T02:38:40.224609+0000","last_undegraded":"2024-09-21T02:38:40.224609+0000","last_fullsized":"2024-09-21T02:38:40.224609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:28.199451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503585+0000","last_change":"2024-09-21T02:36:25.744845+0000","last_active":"2024-09-21T02:38:40.503585+0000","last_peered":"2024-09-21T02:38:40.503585+0000","last_clean":"2024-09-21T02:38:40.503585+0000","last_became_active":"2024-09-21T02:36:25.744492+0000","last_became_peered":"2024-09-21T02:36:25.744492+0000","last_unstale":"2024-09-21T02:38:40.503585+0000","last_undegraded":"2024-09-21T02:38:40.503585+0000","last_fullsized":"2024-09-21T02:38:40.503585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:08.524871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331829+0000","last_change":"2024-09-21T02:36:25.751187+0000","last_active":"2024-09-21T02:38:40.331829+0000","last_peered":"2024-09-21T02:38:40.331829+0000","last_clean":"2024-09-21T02:38:40.331829+0000","last_became_active":"2024-09-21T02:36:25.750813+0000","last_became_peered":"2024-09-21T02:36:25.750813+0000","last_unstale":"2024-09-21T02:38:40.331829+0000","last_undegraded":"2024-09-21T02:38:40.331829+0000","last_fullsized":"2024-09-21T02:38:40.331829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.590210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.226878+0000","last_change":"2024-09-21T02:36:25.744046+0000","last_active":"2024-09-21T02:38:40.226878+0000","last_peered":"2024-09-21T02:38:40.226878+0000","last_clean":"2024-09-21T02:38:40.226878+0000","last_became_active":"2024-09-21T02:36:25.743695+0000","last_became_peered":"2024-09-21T02:36:25.743695+0000","last_unstale":"2024-09-21T02:38:40.226878+0000","last_undegraded":"2024-09-21T02:38:40.226878+0000","last_fullsized":"2024-09-21T02:38:40.226878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:12.092940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.231911+0000","last_change":"2024-09-21T02:36:25.749113+0000","last_active":"2024-09-21T02:38:40.231911+0000","last_peered":"2024-09-21T02:38:40.231911+0000","last_clean":"2024-09-21T02:38:40.231911+0000","last_became_active":"2024-09-21T02:36:25.748759+0000","last_became_peered":"2024-09-21T02:36:25.748759+0000","last_unstale":"2024-09-21T02:38:40.231911+0000","last_undegraded":"2024-09-21T02:38:40.231911+0000","last_fullsized":"2024-09-21T02:38:40.231911+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:22.785273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232166+0000","last_change":"2024-09-21T02:36:25.745454+0000","last_active":"2024-09-21T02:38:40.232166+0000","last_peered":"2024-09-21T02:38:40.232166+0000","last_clean":"2024-09-21T02:38:40.232166+0000","last_became_active":"2024-09-21T02:36:25.745108+0000","last_became_peered":"2024-09-21T02:36:25.745108+0000","last_unstale":"2024-09-21T02:38:40.232166+0000","last_undegraded":"2024-09-21T02:38:40.232166+0000","last_fullsized":"2024-09-21T02:38:40.232166+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:36.710456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222535+0000","last_change":"2024-09-21T02:36:25.737213+0000","last_active":"2024-09-21T02:38:40.222535+0000","last_peered":"2024-09-21T02:38:40.222535+0000","last_clean":"2024-09-21T02:38:40.222535+0000","last_became_active":"2024-09-21T02:36:25.736641+0000","last_became_peered":"2024-09-21T02:36:25.736641+0000","last_unstale":"2024-09-21T02:38:40.222535+0000","last_undegraded":"2024-09-21T02:38:40.222535+0000","last_fullsized":"2024-09-21T02:38:40.222535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:47:50.258805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504287+0000","last_change":"2024-09-21T02:36:25.748230+0000","last_active":"2024-09-21T02:38:40.504287+0000","last_peered":"2024-09-21T02:38:40.504287+0000","last_clean":"2024-09-21T02:38:40.504287+0000","last_became_active":"2024-09-21T02:36:25.747859+0000","last_became_peered":"2024-09-21T02:36:25.747859+0000","last_unstale":"2024-09-21T02:38:40.504287+0000","last_undegraded":"2024-09-21T02:38:40.504287+0000","last_fullsized":"2024-09-21T02:38:40.504287+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:15:21.244825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503711+0000","last_change":"2024-09-21T02:36:25.744379+0000","last_active":"2024-09-21T02:38:40.503711+0000","last_peered":"2024-09-21T02:38:40.503711+0000","last_clean":"2024-09-21T02:38:40.503711+0000","last_became_active":"2024-09-21T02:36:25.744003+0000","last_became_peered":"2024-09-21T02:36:25.744003+0000","last_unstale":"2024-09-21T02:38:40.503711+0000","last_undegraded":"2024-09-21T02:38:40.503711+0000","last_fullsized":"2024-09-21T02:38:40.503711+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:59:59.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224543+0000","last_change":"2024-09-21T02:36:25.755616+0000","last_active":"2024-09-21T02:38:40.224543+0000","last_peered":"2024-09-21T02:38:40.224543+0000","last_clean":"2024-09-21T02:38:40.224543+0000","last_became_active":"2024-09-21T02:36:25.755304+0000","last_became_peered":"2024-09-21T02:36:25.755304+0000","last_unstale":"2024-09-21T02:38:40.224543+0000","last_undegraded":"2024-09-21T02:38:40.224543+0000","last_fullsized":"2024-09-21T02:38:40.224543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:24.813780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228147+0000","last_change":"2024-09-21T02:36:25.739747+0000","last_active":"2024-09-21T02:38:40.228147+0000","last_peered":"2024-09-21T02:38:40.228147+0000","last_clean":"2024-09-21T02:38:40.228147+0000","last_became_active":"2024-09-21T02:36:25.739404+0000","last_became_peered":"2024-09-21T02:36:25.739404+0000","last_unstale":"2024-09-21T02:38:40.228147+0000","last_undegraded":"2024-09-21T02:38:40.228147+0000","last_fullsized":"2024-09-21T02:38:40.228147+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:17.975419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293973+0000","last_change":"2024-09-21T02:36:25.747723+0000","last_active":"2024-09-21T02:38:40.293973+0000","last_peered":"2024-09-21T02:38:40.293973+0000","last_clean":"2024-09-21T02:38:40.293973+0000","last_became_active":"2024-09-21T02:36:25.747368+0000","last_became_peered":"2024-09-21T02:36:25.747368+0000","last_unstale":"2024-09-21T02:38:40.293973+0000","last_undegraded":"2024-09-21T02:38:40.293973+0000","last_fullsized":"2024-09-21T02:38:40.293973+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.319756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503361+0000","last_change":"2024-09-21T02:36:25.743772+0000","last_active":"2024-09-21T02:38:40.503361+0000","last_peered":"2024-09-21T02:38:40.503361+0000","last_clean":"2024-09-21T02:38:40.503361+0000","last_became_active":"2024-09-21T02:36:25.743403+0000","last_became_peered":"2024-09-21T02:36:25.743403+0000","last_unstale":"2024-09-21T02:38:40.503361+0000","last_undegraded":"2024-09-21T02:38:40.503361+0000","last_fullsized":"2024-09-21T02:38:40.503361+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:11:29.930248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223893+0000","last_change":"2024-09-21T02:36:25.746911+0000","last_active":"2024-09-21T02:38:40.223893+0000","last_peered":"2024-09-21T02:38:40.223893+0000","last_clean":"2024-09-21T02:38:40.223893+0000","last_became_active":"2024-09-21T02:36:25.746426+0000","last_became_peered":"2024-09-21T02:36:25.746426+0000","last_unstale":"2024-09-21T02:38:40.223893+0000","last_undegraded":"2024-09-21T02:38:40.223893+0000","last_fullsized":"2024-09-21T02:38:40.223893+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:12.835851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503097+0000","last_change":"2024-09-21T02:36:25.751278+0000","last_active":"2024-09-21T02:38:40.503097+0000","last_peered":"2024-09-21T02:38:40.503097+0000","last_clean":"2024-09-21T02:38:40.503097+0000","last_became_active":"2024-09-21T02:36:25.750414+0000","last_became_peered":"2024-09-21T02:36:25.750414+0000","last_unstale":"2024-09-21T02:38:40.503097+0000","last_undegraded":"2024-09-21T02:38:40.503097+0000","last_fullsized":"2024-09-21T02:38:40.503097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:13:56.080122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504014+0000","last_change":"2024-09-21T02:36:25.752852+0000","last_active":"2024-09-21T02:38:40.504014+0000","last_peered":"2024-09-21T02:38:40.504014+0000","last_clean":"2024-09-21T02:38:40.504014+0000","last_became_active":"2024-09-21T02:36:25.752514+0000","last_became_peered":"2024-09-21T02:36:25.752514+0000","last_unstale":"2024-09-21T02:38:40.504014+0000","last_undegraded":"2024-09-21T02:38:40.504014+0000","last_fullsized":"2024-09-21T02:38:40.504014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:55.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.231464+0000","last_change":"2024-09-21T02:36:25.750081+0000","last_active":"2024-09-21T02:38:40.231464+0000","last_peered":"2024-09-21T02:38:40.231464+0000","last_clean":"2024-09-21T02:38:40.231464+0000","last_became_active":"2024-09-21T02:36:25.749602+0000","last_became_peered":"2024-09-21T02:36:25.749602+0000","last_unstale":"2024-09-21T02:38:40.231464+0000","last_undegraded":"2024-09-21T02:38:40.231464+0000","last_fullsized":"2024-09-21T02:38:40.231464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:06.785998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232934+0000","last_change":"2024-09-21T02:36:25.746897+0000","last_active":"2024-09-21T02:38:40.232934+0000","last_peered":"2024-09-21T02:38:40.232934+0000","last_clean":"2024-09-21T02:38:40.232934+0000","last_became_active":"2024-09-21T02:36:25.746477+0000","last_became_peered":"2024-09-21T02:36:25.746477+0000","last_unstale":"2024-09-21T02:38:40.232934+0000","last_undegraded":"2024-09-21T02:38:40.232934+0000","last_fullsized":"2024-09-21T02:38:40.232934+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:34:00.739410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331749+0000","last_change":"2024-09-21T02:36:25.745233+0000","last_active":"2024-09-21T02:38:40.331749+0000","last_peered":"2024-09-21T02:38:40.331749+0000","last_clean":"2024-09-21T02:38:40.331749+0000","last_became_active":"2024-09-21T02:36:25.744637+0000","last_became_peered":"2024-09-21T02:36:25.744637+0000","last_unstale":"2024-09-21T02:38:40.331749+0000","last_undegraded":"2024-09-21T02:38:40.331749+0000","last_fullsized":"2024-09-21T02:38:40.331749+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:34:59.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77327,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1181,"num_read_kb":673,"num_write":272,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":232865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249064,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":69,"seq":296352743469,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":950272,"data_stored":486338,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906994,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1144968,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070520,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":485131,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266749,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":954368,"data_stored":485397,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462979,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":1083626,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659208,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":481128,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855437,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1024000,"data_stored":1077729,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018962,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":485202,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215190,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":480343,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":552297,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1015808,"data_stored":549534,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":966656,"data_stored":482623,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":70591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:39:08.459 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T02:39:08.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:39:08.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:08 smithi190 ceph-mon[33871]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:08.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:08 smithi029 ceph-mon[31006]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:08.719 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:08.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:08 smithi066 ceph-mon[33436]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 -- 172.21.15.29:0/1116229382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc801083a0 msgr2=0x7fbc801087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 --2- 172.21.15.29:0/1116229382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc801083a0 0x7fbc801087b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009900 tx=0x7fbc7402a410 comp rx=0 tx=0).stop 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 -- 172.21.15.29:0/1116229382 shutdown_connections 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 --2- 172.21.15.29:0/1116229382 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc80108cf0 0x7fbc8010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 --2- 172.21.15.29:0/1116229382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc801083a0 0x7fbc801087b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 --2- 172.21.15.29:0/1116229382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc80102a60 0x7fbc80102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 -- 172.21.15.29:0/1116229382 >> 172.21.15.29:0/1116229382 conn(0x7fbc80079730 msgr2=0x7fbc80079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 -- 172.21.15.29:0/1116229382 shutdown_connections 2024-09-21T02:39:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.225+0000 7fbc8652f640 1 -- 172.21.15.29:0/1116229382 wait complete. 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc8652f640 1 Processor -- start 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc8652f640 1 -- start start 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc8652f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc80102a60 0x7fbc8019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc8652f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc801083a0 0x7fbc8019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc8652f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc80108cf0 0x7fbc8019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc8652f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbc80113930 con 0x7fbc80102a60 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc8652f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbc801137b0 con 0x7fbc801083a0 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc8652f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbc80113ab0 con 0x7fbc80108cf0 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc7ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc80102a60 0x7fbc8019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc7ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc80102a60 0x7fbc8019a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48014/0 (socket says 172.21.15.29:48014) 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc7ffff640 1 -- 172.21.15.29:0/683892330 learned_addr learned my addr 172.21.15.29:0/683892330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc84acc640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc80108cf0 0x7fbc8019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.226+0000 7fbc7f7fe640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc801083a0 0x7fbc8019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7ffff640 1 -- 172.21.15.29:0/683892330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc80108cf0 msgr2=0x7fbc8019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7ffff640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc80108cf0 0x7fbc8019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7ffff640 1 -- 172.21.15.29:0/683892330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc801083a0 msgr2=0x7fbc8019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7ffff640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc801083a0 0x7fbc8019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7ffff640 1 -- 172.21.15.29:0/683892330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc801a03b0 con 0x7fbc80102a60 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7ffff640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc80102a60 0x7fbc8019a440 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000d950 tx=0x7fbc7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7d7fa640 1 -- 172.21.15.29:0/683892330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc70017070 con 0x7fbc80102a60 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7d7fa640 1 -- 172.21.15.29:0/683892330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc70009c30 con 0x7fbc80102a60 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc7d7fa640 1 -- 172.21.15.29:0/683892330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7000f620 con 0x7fbc80102a60 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc801a06a0 con 0x7fbc80102a60 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.227+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc801a6e00 con 0x7fbc80102a60 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.228+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc80102e70 con 0x7fbc80102a60 2024-09-21T02:39:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.230+0000 7fbc7d7fa640 1 -- 172.21.15.29:0/683892330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7fbc70010460 con 0x7fbc80102a60 2024-09-21T02:39:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.230+0000 7fbc7d7fa640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbc680b6620 0x7fbc680b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.231+0000 7fbc7f7fe640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbc680b6620 0x7fbc680b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.231+0000 7fbc7d7fa640 1 -- 172.21.15.29:0/683892330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbc700e7310 con 0x7fbc80102a60 2024-09-21T02:39:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.231+0000 7fbc7f7fe640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbc680b6620 0x7fbc680b8b30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbc74004770 tx=0x7fbc740058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.235+0000 7fbc7d7fa640 1 -- 172.21.15.29:0/683892330 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbc70009de0 con 0x7fbc80102a60 2024-09-21T02:39:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.420+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 --> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc80063800 con 0x7fbc680b6620 2024-09-21T02:39:09.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:09 smithi029 ceph-mon[31006]: pgmap v390: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-09-21T02:39:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.426+0000 7fbc7d7fa640 1 -- 172.21.15.29:0/683892330 <== mgr.14152 v2:172.21.15.29:6800/257624116 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340785 (secure 0 0 0) 0x7fbc80063800 con 0x7fbc680b6620 2024-09-21T02:39:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T02:39:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbc680b6620 msgr2=0x7fbc680b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbc680b6620 0x7fbc680b8b30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbc74004770 tx=0x7fbc740058d0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc80102a60 msgr2=0x7fbc8019a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc80102a60 0x7fbc8019a440 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000d950 tx=0x7fbc7000de10 comp rx=0 tx=0).stop 2024-09-21T02:39:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 shutdown_connections 2024-09-21T02:39:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbc680b6620 0x7fbc680b8b30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc80108cf0 0x7fbc8019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc801083a0 0x7fbc8019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 --2- 172.21.15.29:0/683892330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc80102a60 0x7fbc8019a440 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 >> 172.21.15.29:0/683892330 conn(0x7fbc80079730 msgr2=0x7fbc80104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 shutdown_connections 2024-09-21T02:39:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:09.429+0000 7fbc8652f640 1 -- 172.21.15.29:0/683892330 wait complete. 2024-09-21T02:39:09.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:09 smithi190 ceph-mon[33871]: pgmap v390: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-09-21T02:39:09.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:09 smithi066 ceph-mon[33436]: pgmap v390: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-09-21T02:39:10.212 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-21T02:39:07.976068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668813,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1264,"num_read_kb":739,"num_write":401,"num_write_kb":2320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2952069,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":830,"ondisk_log_size":830,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369180,"kb_used_data":11608,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572644,"statfs":{"total":1151940427776,"available":1151562387456,"internally_reserved":0,"allocated":11886592,"data_stored":7803354,"data_compressed":463314,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003951"},"pg_stats":[{"pgid":"2.1f","version":"85'14","reported_seq":168,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.209564+0000","last_change":"2024-09-21T02:36:23.729337+0000","last_active":"2024-09-21T02:39:02.209564+0000","last_peered":"2024-09-21T02:39:02.209564+0000","last_clean":"2024-09-21T02:39:02.209564+0000","last_became_active":"2024-09-21T02:36:23.728576+0000","last_became_peered":"2024-09-21T02:36:23.728576+0000","last_unstale":"2024-09-21T02:39:02.209564+0000","last_undegraded":"2024-09-21T02:39:02.209564+0000","last_fullsized":"2024-09-21T02:39:02.209564+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:31.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294323+0000","last_change":"2024-09-21T02:36:25.748925+0000","last_active":"2024-09-21T02:38:40.294323+0000","last_peered":"2024-09-21T02:38:40.294323+0000","last_clean":"2024-09-21T02:38:40.294323+0000","last_became_active":"2024-09-21T02:36:25.748333+0000","last_became_peered":"2024-09-21T02:36:25.748333+0000","last_unstale":"2024-09-21T02:38:40.294323+0000","last_undegraded":"2024-09-21T02:38:40.294323+0000","last_fullsized":"2024-09-21T02:38:40.294323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:46.889497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224117+0000","last_change":"2024-09-21T02:36:32.807771+0000","last_active":"2024-09-21T02:38:40.224117+0000","last_peered":"2024-09-21T02:38:40.224117+0000","last_clean":"2024-09-21T02:38:40.224117+0000","last_became_active":"2024-09-21T02:36:32.807291+0000","last_became_peered":"2024-09-21T02:36:32.807291+0000","last_unstale":"2024-09-21T02:38:40.224117+0000","last_undegraded":"2024-09-21T02:38:40.224117+0000","last_fullsized":"2024-09-21T02:38:40.224117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:41.468318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294537+0000","last_change":"2024-09-21T02:36:23.800779+0000","last_active":"2024-09-21T02:38:40.294537+0000","last_peered":"2024-09-21T02:38:40.294537+0000","last_clean":"2024-09-21T02:38:40.294537+0000","last_became_active":"2024-09-21T02:36:23.800323+0000","last_became_peered":"2024-09-21T02:36:23.800323+0000","last_unstale":"2024-09-21T02:38:40.294537+0000","last_undegraded":"2024-09-21T02:38:40.294537+0000","last_fullsized":"2024-09-21T02:38:40.294537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:24.843233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232565+0000","last_change":"2024-09-21T02:36:25.745277+0000","last_active":"2024-09-21T02:38:40.232565+0000","last_peered":"2024-09-21T02:38:40.232565+0000","last_clean":"2024-09-21T02:38:40.232565+0000","last_became_active":"2024-09-21T02:36:25.744501+0000","last_became_peered":"2024-09-21T02:36:25.744501+0000","last_unstale":"2024-09-21T02:38:40.232565+0000","last_undegraded":"2024-09-21T02:38:40.232565+0000","last_fullsized":"2024-09-21T02:38:40.232565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:14.776250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502659+0000","last_change":"2024-09-21T02:36:32.801978+0000","last_active":"2024-09-21T02:38:40.502659+0000","last_peered":"2024-09-21T02:38:40.502659+0000","last_clean":"2024-09-21T02:38:40.502659+0000","last_became_active":"2024-09-21T02:36:32.801436+0000","last_became_peered":"2024-09-21T02:36:32.801436+0000","last_unstale":"2024-09-21T02:38:40.502659+0000","last_undegraded":"2024-09-21T02:38:40.502659+0000","last_fullsized":"2024-09-21T02:38:40.502659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:37.380417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503616+0000","last_change":"2024-09-21T02:36:23.729336+0000","last_active":"2024-09-21T02:38:40.503616+0000","last_peered":"2024-09-21T02:38:40.503616+0000","last_clean":"2024-09-21T02:38:40.503616+0000","last_became_active":"2024-09-21T02:36:23.728724+0000","last_became_peered":"2024-09-21T02:36:23.728724+0000","last_unstale":"2024-09-21T02:38:40.503616+0000","last_undegraded":"2024-09-21T02:38:40.503616+0000","last_fullsized":"2024-09-21T02:38:40.503616+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:31:52.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228248+0000","last_change":"2024-09-21T02:36:25.737577+0000","last_active":"2024-09-21T02:38:40.228248+0000","last_peered":"2024-09-21T02:38:40.228248+0000","last_clean":"2024-09-21T02:38:40.228248+0000","last_became_active":"2024-09-21T02:36:25.737221+0000","last_became_peered":"2024-09-21T02:36:25.737221+0000","last_unstale":"2024-09-21T02:38:40.228248+0000","last_undegraded":"2024-09-21T02:38:40.228248+0000","last_fullsized":"2024-09-21T02:38:40.228248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:42:54.786464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504222+0000","last_change":"2024-09-21T02:36:32.802507+0000","last_active":"2024-09-21T02:38:40.504222+0000","last_peered":"2024-09-21T02:38:40.504222+0000","last_clean":"2024-09-21T02:38:40.504222+0000","last_became_active":"2024-09-21T02:36:32.801879+0000","last_became_peered":"2024-09-21T02:36:32.801879+0000","last_unstale":"2024-09-21T02:38:40.504222+0000","last_undegraded":"2024-09-21T02:38:40.504222+0000","last_fullsized":"2024-09-21T02:38:40.504222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:09.873825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503447+0000","last_change":"2024-09-21T02:36:23.730889+0000","last_active":"2024-09-21T02:38:40.503447+0000","last_peered":"2024-09-21T02:38:40.503447+0000","last_clean":"2024-09-21T02:38:40.503447+0000","last_became_active":"2024-09-21T02:36:23.730551+0000","last_became_peered":"2024-09-21T02:36:23.730551+0000","last_unstale":"2024-09-21T02:38:40.503447+0000","last_undegraded":"2024-09-21T02:38:40.503447+0000","last_fullsized":"2024-09-21T02:38:40.503447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:17.178848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504430+0000","last_change":"2024-09-21T02:37:39.869272+0000","last_active":"2024-09-21T02:38:40.504430+0000","last_peered":"2024-09-21T02:38:40.504430+0000","last_clean":"2024-09-21T02:38:40.504430+0000","last_became_active":"2024-09-21T02:37:39.868824+0000","last_became_peered":"2024-09-21T02:37:39.868824+0000","last_unstale":"2024-09-21T02:38:40.504430+0000","last_undegraded":"2024-09-21T02:38:40.504430+0000","last_fullsized":"2024-09-21T02:38:40.504430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:05:31.503359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503414+0000","last_change":"2024-09-21T02:36:32.800489+0000","last_active":"2024-09-21T02:38:40.503414+0000","last_peered":"2024-09-21T02:38:40.503414+0000","last_clean":"2024-09-21T02:38:40.503414+0000","last_became_active":"2024-09-21T02:36:32.799876+0000","last_became_peered":"2024-09-21T02:36:32.799876+0000","last_unstale":"2024-09-21T02:38:40.503414+0000","last_undegraded":"2024-09-21T02:38:40.503414+0000","last_fullsized":"2024-09-21T02:38:40.503414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:02:56.711257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502829+0000","last_change":"2024-09-21T02:36:25.744561+0000","last_active":"2024-09-21T02:38:40.502829+0000","last_peered":"2024-09-21T02:38:40.502829+0000","last_clean":"2024-09-21T02:38:40.502829+0000","last_became_active":"2024-09-21T02:36:25.743757+0000","last_became_peered":"2024-09-21T02:36:25.743757+0000","last_unstale":"2024-09-21T02:38:40.502829+0000","last_undegraded":"2024-09-21T02:38:40.502829+0000","last_fullsized":"2024-09-21T02:38:40.502829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:30.235517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503205+0000","last_change":"2024-09-21T02:37:39.616628+0000","last_active":"2024-09-21T02:38:40.503205+0000","last_peered":"2024-09-21T02:38:40.503205+0000","last_clean":"2024-09-21T02:38:40.503205+0000","last_became_active":"2024-09-21T02:37:39.609108+0000","last_became_peered":"2024-09-21T02:37:39.609108+0000","last_unstale":"2024-09-21T02:38:40.503205+0000","last_undegraded":"2024-09-21T02:38:40.503205+0000","last_fullsized":"2024-09-21T02:38:40.503205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:11:28.944244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504031+0000","last_change":"2024-09-21T02:36:33.003670+0000","last_active":"2024-09-21T02:38:40.504031+0000","last_peered":"2024-09-21T02:38:40.504031+0000","last_clean":"2024-09-21T02:38:40.504031+0000","last_became_active":"2024-09-21T02:36:33.003022+0000","last_became_peered":"2024-09-21T02:36:33.003022+0000","last_unstale":"2024-09-21T02:38:40.504031+0000","last_undegraded":"2024-09-21T02:38:40.504031+0000","last_fullsized":"2024-09-21T02:38:40.504031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:12.753375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294239+0000","last_change":"2024-09-21T02:36:25.749404+0000","last_active":"2024-09-21T02:38:40.294239+0000","last_peered":"2024-09-21T02:38:40.294239+0000","last_clean":"2024-09-21T02:38:40.294239+0000","last_became_active":"2024-09-21T02:36:25.747854+0000","last_became_peered":"2024-09-21T02:36:25.747854+0000","last_unstale":"2024-09-21T02:38:40.294239+0000","last_undegraded":"2024-09-21T02:38:40.294239+0000","last_fullsized":"2024-09-21T02:38:40.294239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:41.224740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224160+0000","last_change":"2024-09-21T02:36:23.726689+0000","last_active":"2024-09-21T02:38:40.224160+0000","last_peered":"2024-09-21T02:38:40.224160+0000","last_clean":"2024-09-21T02:38:40.224160+0000","last_became_active":"2024-09-21T02:36:23.726165+0000","last_became_peered":"2024-09-21T02:36:23.726165+0000","last_unstale":"2024-09-21T02:38:40.224160+0000","last_undegraded":"2024-09-21T02:38:40.224160+0000","last_fullsized":"2024-09-21T02:38:40.224160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:58.166457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503916+0000","last_change":"2024-09-21T02:36:32.809594+0000","last_active":"2024-09-21T02:38:40.503916+0000","last_peered":"2024-09-21T02:38:40.503916+0000","last_clean":"2024-09-21T02:38:40.503916+0000","last_became_active":"2024-09-21T02:36:32.808893+0000","last_became_peered":"2024-09-21T02:36:32.808893+0000","last_unstale":"2024-09-21T02:38:40.503916+0000","last_undegraded":"2024-09-21T02:38:40.503916+0000","last_fullsized":"2024-09-21T02:38:40.503916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:55.187241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331726+0000","last_change":"2024-09-21T02:36:25.755955+0000","last_active":"2024-09-21T02:38:40.331726+0000","last_peered":"2024-09-21T02:38:40.331726+0000","last_clean":"2024-09-21T02:38:40.331726+0000","last_became_active":"2024-09-21T02:36:25.755577+0000","last_became_peered":"2024-09-21T02:36:25.755577+0000","last_unstale":"2024-09-21T02:38:40.331726+0000","last_undegraded":"2024-09-21T02:38:40.331726+0000","last_fullsized":"2024-09-21T02:38:40.331726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:48.447758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294101+0000","last_change":"2024-09-21T02:36:23.730043+0000","last_active":"2024-09-21T02:38:40.294101+0000","last_peered":"2024-09-21T02:38:40.294101+0000","last_clean":"2024-09-21T02:38:40.294101+0000","last_became_active":"2024-09-21T02:36:23.729576+0000","last_became_peered":"2024-09-21T02:36:23.729576+0000","last_unstale":"2024-09-21T02:38:40.294101+0000","last_undegraded":"2024-09-21T02:38:40.294101+0000","last_fullsized":"2024-09-21T02:38:40.294101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:48:09.530913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503449+0000","last_change":"2024-09-21T02:36:32.806923+0000","last_active":"2024-09-21T02:38:40.503449+0000","last_peered":"2024-09-21T02:38:40.503449+0000","last_clean":"2024-09-21T02:38:40.503449+0000","last_became_active":"2024-09-21T02:36:32.806176+0000","last_became_peered":"2024-09-21T02:36:32.806176+0000","last_unstale":"2024-09-21T02:38:40.503449+0000","last_undegraded":"2024-09-21T02:38:40.503449+0000","last_fullsized":"2024-09-21T02:38:40.503449+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.165638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502828+0000","last_change":"2024-09-21T02:36:25.747501+0000","last_active":"2024-09-21T02:38:40.502828+0000","last_peered":"2024-09-21T02:38:40.502828+0000","last_clean":"2024-09-21T02:38:40.502828+0000","last_became_active":"2024-09-21T02:36:25.747111+0000","last_became_peered":"2024-09-21T02:36:25.747111+0000","last_unstale":"2024-09-21T02:38:40.502828+0000","last_undegraded":"2024-09-21T02:38:40.502828+0000","last_fullsized":"2024-09-21T02:38:40.502828+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:03:04.061537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.335704+0000","last_change":"2024-09-21T02:38:40.335626+0000","last_active":"2024-09-21T02:38:40.335704+0000","last_peered":"2024-09-21T02:38:40.335704+0000","last_clean":"2024-09-21T02:38:40.335704+0000","last_became_active":"2024-09-21T02:38:40.332735+0000","last_became_peered":"2024-09-21T02:38:40.332735+0000","last_unstale":"2024-09-21T02:38:40.335704+0000","last_undegraded":"2024-09-21T02:38:40.335704+0000","last_fullsized":"2024-09-21T02:38:40.335704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:42:37.763918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504172+0000","last_change":"2024-09-21T02:37:39.870041+0000","last_active":"2024-09-21T02:38:40.504172+0000","last_peered":"2024-09-21T02:38:40.504172+0000","last_clean":"2024-09-21T02:38:40.504172+0000","last_became_active":"2024-09-21T02:37:39.869518+0000","last_became_peered":"2024-09-21T02:37:39.869518+0000","last_unstale":"2024-09-21T02:38:40.504172+0000","last_undegraded":"2024-09-21T02:38:40.504172+0000","last_fullsized":"2024-09-21T02:38:40.504172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:20.276471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233047+0000","last_change":"2024-09-21T02:36:25.746139+0000","last_active":"2024-09-21T02:38:40.233047+0000","last_peered":"2024-09-21T02:38:40.233047+0000","last_clean":"2024-09-21T02:38:40.233047+0000","last_became_active":"2024-09-21T02:36:25.744418+0000","last_became_peered":"2024-09-21T02:36:25.744418+0000","last_unstale":"2024-09-21T02:38:40.233047+0000","last_undegraded":"2024-09-21T02:38:40.233047+0000","last_fullsized":"2024-09-21T02:38:40.233047+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:21.706782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503137+0000","last_change":"2024-09-21T02:36:24.261945+0000","last_active":"2024-09-21T02:38:40.503137+0000","last_peered":"2024-09-21T02:38:40.503137+0000","last_clean":"2024-09-21T02:38:40.503137+0000","last_became_active":"2024-09-21T02:36:24.261486+0000","last_became_peered":"2024-09-21T02:36:24.261486+0000","last_unstale":"2024-09-21T02:38:40.503137+0000","last_undegraded":"2024-09-21T02:38:40.503137+0000","last_fullsized":"2024-09-21T02:38:40.503137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:45:18.331035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224142+0000","last_change":"2024-09-21T02:36:32.799464+0000","last_active":"2024-09-21T02:38:40.224142+0000","last_peered":"2024-09-21T02:38:40.224142+0000","last_clean":"2024-09-21T02:38:40.224142+0000","last_became_active":"2024-09-21T02:36:32.799000+0000","last_became_peered":"2024-09-21T02:36:32.799000+0000","last_unstale":"2024-09-21T02:38:40.224142+0000","last_undegraded":"2024-09-21T02:38:40.224142+0000","last_fullsized":"2024-09-21T02:38:40.224142+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:56:36.271172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294464+0000","last_change":"2024-09-21T02:36:25.741151+0000","last_active":"2024-09-21T02:38:40.294464+0000","last_peered":"2024-09-21T02:38:40.294464+0000","last_clean":"2024-09-21T02:38:40.294464+0000","last_became_active":"2024-09-21T02:36:25.740803+0000","last_became_peered":"2024-09-21T02:36:25.740803+0000","last_unstale":"2024-09-21T02:38:40.294464+0000","last_undegraded":"2024-09-21T02:38:40.294464+0000","last_fullsized":"2024-09-21T02:38:40.294464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:44.585136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'18","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503589+0000","last_change":"2024-09-21T02:36:23.725155+0000","last_active":"2024-09-21T02:38:40.503589+0000","last_peered":"2024-09-21T02:38:40.503589+0000","last_clean":"2024-09-21T02:38:40.503589+0000","last_became_active":"2024-09-21T02:36:23.724791+0000","last_became_peered":"2024-09-21T02:36:23.724791+0000","last_unstale":"2024-09-21T02:38:40.503589+0000","last_undegraded":"2024-09-21T02:38:40.503589+0000","last_fullsized":"2024-09-21T02:38:40.503589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:06.520465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293747+0000","last_change":"2024-09-21T02:36:33.003401+0000","last_active":"2024-09-21T02:38:40.293747+0000","last_peered":"2024-09-21T02:38:40.293747+0000","last_clean":"2024-09-21T02:38:40.293747+0000","last_became_active":"2024-09-21T02:36:33.002930+0000","last_became_peered":"2024-09-21T02:36:33.002930+0000","last_unstale":"2024-09-21T02:38:40.293747+0000","last_undegraded":"2024-09-21T02:38:40.293747+0000","last_fullsized":"2024-09-21T02:38:40.293747+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:40:23.629763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222161+0000","last_change":"2024-09-21T02:36:25.735501+0000","last_active":"2024-09-21T02:38:40.222161+0000","last_peered":"2024-09-21T02:38:40.222161+0000","last_clean":"2024-09-21T02:38:40.222161+0000","last_became_active":"2024-09-21T02:36:25.734453+0000","last_became_peered":"2024-09-21T02:36:25.734453+0000","last_unstale":"2024-09-21T02:38:40.222161+0000","last_undegraded":"2024-09-21T02:38:40.222161+0000","last_fullsized":"2024-09-21T02:38:40.222161+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:50.238529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503639+0000","last_change":"2024-09-21T02:36:23.800338+0000","last_active":"2024-09-21T02:38:40.503639+0000","last_peered":"2024-09-21T02:38:40.503639+0000","last_clean":"2024-09-21T02:38:40.503639+0000","last_became_active":"2024-09-21T02:36:23.799796+0000","last_became_peered":"2024-09-21T02:36:23.799796+0000","last_unstale":"2024-09-21T02:38:40.503639+0000","last_undegraded":"2024-09-21T02:38:40.503639+0000","last_fullsized":"2024-09-21T02:38:40.503639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:00:18.459515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331347+0000","last_change":"2024-09-21T02:37:39.611513+0000","last_active":"2024-09-21T02:38:40.331347+0000","last_peered":"2024-09-21T02:38:40.331347+0000","last_clean":"2024-09-21T02:38:40.331347+0000","last_became_active":"2024-09-21T02:37:39.610843+0000","last_became_peered":"2024-09-21T02:37:39.610843+0000","last_unstale":"2024-09-21T02:38:40.331347+0000","last_undegraded":"2024-09-21T02:38:40.331347+0000","last_fullsized":"2024-09-21T02:38:40.331347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:51.911830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223967+0000","last_change":"2024-09-21T02:36:25.745497+0000","last_active":"2024-09-21T02:38:40.223967+0000","last_peered":"2024-09-21T02:38:40.223967+0000","last_clean":"2024-09-21T02:38:40.223967+0000","last_became_active":"2024-09-21T02:36:25.745126+0000","last_became_peered":"2024-09-21T02:36:25.745126+0000","last_unstale":"2024-09-21T02:38:40.223967+0000","last_undegraded":"2024-09-21T02:38:40.223967+0000","last_fullsized":"2024-09-21T02:38:40.223967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:42:17.835784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503417+0000","last_change":"2024-09-21T02:36:23.802440+0000","last_active":"2024-09-21T02:38:40.503417+0000","last_peered":"2024-09-21T02:38:40.503417+0000","last_clean":"2024-09-21T02:38:40.503417+0000","last_became_active":"2024-09-21T02:36:23.802065+0000","last_became_peered":"2024-09-21T02:36:23.802065+0000","last_unstale":"2024-09-21T02:38:40.503417+0000","last_undegraded":"2024-09-21T02:38:40.503417+0000","last_fullsized":"2024-09-21T02:38:40.503417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:22:20.647476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502936+0000","last_change":"2024-09-21T02:36:33.003465+0000","last_active":"2024-09-21T02:38:40.502936+0000","last_peered":"2024-09-21T02:38:40.502936+0000","last_clean":"2024-09-21T02:38:40.502936+0000","last_became_active":"2024-09-21T02:36:33.002874+0000","last_became_peered":"2024-09-21T02:36:33.002874+0000","last_unstale":"2024-09-21T02:38:40.502936+0000","last_undegraded":"2024-09-21T02:38:40.502936+0000","last_fullsized":"2024-09-21T02:38:40.502936+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:02.388770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224086+0000","last_change":"2024-09-21T02:36:25.746092+0000","last_active":"2024-09-21T02:38:40.224086+0000","last_peered":"2024-09-21T02:38:40.224086+0000","last_clean":"2024-09-21T02:38:40.224086+0000","last_became_active":"2024-09-21T02:36:25.745730+0000","last_became_peered":"2024-09-21T02:36:25.745730+0000","last_unstale":"2024-09-21T02:38:40.224086+0000","last_undegraded":"2024-09-21T02:38:40.224086+0000","last_fullsized":"2024-09-21T02:38:40.224086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:24:13.212450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'6","reported_seq":163,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:03.212549+0000","last_change":"2024-09-21T02:36:23.732090+0000","last_active":"2024-09-21T02:39:03.212549+0000","last_peered":"2024-09-21T02:39:03.212549+0000","last_clean":"2024-09-21T02:39:03.212549+0000","last_became_active":"2024-09-21T02:36:23.731653+0000","last_became_peered":"2024-09-21T02:36:23.731653+0000","last_unstale":"2024-09-21T02:39:03.212549+0000","last_undegraded":"2024-09-21T02:39:03.212549+0000","last_fullsized":"2024-09-21T02:39:03.212549+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:02:42.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222021+0000","last_change":"2024-09-21T02:36:32.794893+0000","last_active":"2024-09-21T02:38:40.222021+0000","last_peered":"2024-09-21T02:38:40.222021+0000","last_clean":"2024-09-21T02:38:40.222021+0000","last_became_active":"2024-09-21T02:36:32.794406+0000","last_became_peered":"2024-09-21T02:36:32.794406+0000","last_unstale":"2024-09-21T02:38:40.222021+0000","last_undegraded":"2024-09-21T02:38:40.222021+0000","last_fullsized":"2024-09-21T02:38:40.222021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:40:39.621502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504064+0000","last_change":"2024-09-21T02:36:25.748220+0000","last_active":"2024-09-21T02:38:40.504064+0000","last_peered":"2024-09-21T02:38:40.504064+0000","last_clean":"2024-09-21T02:38:40.504064+0000","last_became_active":"2024-09-21T02:36:25.747553+0000","last_became_peered":"2024-09-21T02:36:25.747553+0000","last_unstale":"2024-09-21T02:38:40.504064+0000","last_undegraded":"2024-09-21T02:38:40.504064+0000","last_fullsized":"2024-09-21T02:38:40.504064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:28.757981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'6","reported_seq":161,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.219800+0000","last_change":"2024-09-21T02:36:23.801087+0000","last_active":"2024-09-21T02:39:02.219800+0000","last_peered":"2024-09-21T02:39:02.219800+0000","last_clean":"2024-09-21T02:39:02.219800+0000","last_became_active":"2024-09-21T02:36:23.800077+0000","last_became_peered":"2024-09-21T02:36:23.800077+0000","last_unstale":"2024-09-21T02:39:02.219800+0000","last_undegraded":"2024-09-21T02:39:02.219800+0000","last_fullsized":"2024-09-21T02:39:02.219800+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:50.074650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222239+0000","last_change":"2024-09-21T02:36:33.003354+0000","last_active":"2024-09-21T02:38:40.222239+0000","last_peered":"2024-09-21T02:38:40.222239+0000","last_clean":"2024-09-21T02:38:40.222239+0000","last_became_active":"2024-09-21T02:36:33.002912+0000","last_became_peered":"2024-09-21T02:36:33.002912+0000","last_unstale":"2024-09-21T02:38:40.222239+0000","last_undegraded":"2024-09-21T02:38:40.222239+0000","last_fullsized":"2024-09-21T02:38:40.222239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:21.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221848+0000","last_change":"2024-09-21T02:36:25.741701+0000","last_active":"2024-09-21T02:38:40.221848+0000","last_peered":"2024-09-21T02:38:40.221848+0000","last_clean":"2024-09-21T02:38:40.221848+0000","last_became_active":"2024-09-21T02:36:25.741222+0000","last_became_peered":"2024-09-21T02:36:25.741222+0000","last_unstale":"2024-09-21T02:38:40.221848+0000","last_undegraded":"2024-09-21T02:38:40.221848+0000","last_fullsized":"2024-09-21T02:38:40.221848+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:35:39.896852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502899+0000","last_change":"2024-09-21T02:36:23.728793+0000","last_active":"2024-09-21T02:38:40.502899+0000","last_peered":"2024-09-21T02:38:40.502899+0000","last_clean":"2024-09-21T02:38:40.502899+0000","last_became_active":"2024-09-21T02:36:23.728362+0000","last_became_peered":"2024-09-21T02:36:23.728362+0000","last_unstale":"2024-09-21T02:38:40.502899+0000","last_undegraded":"2024-09-21T02:38:40.502899+0000","last_fullsized":"2024-09-21T02:38:40.502899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:28:15.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504244+0000","last_change":"2024-09-21T02:36:33.005225+0000","last_active":"2024-09-21T02:38:40.504244+0000","last_peered":"2024-09-21T02:38:40.504244+0000","last_clean":"2024-09-21T02:38:40.504244+0000","last_became_active":"2024-09-21T02:36:33.004756+0000","last_became_peered":"2024-09-21T02:36:33.004756+0000","last_unstale":"2024-09-21T02:38:40.504244+0000","last_undegraded":"2024-09-21T02:38:40.504244+0000","last_fullsized":"2024-09-21T02:38:40.504244+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:02:16.686284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294112+0000","last_change":"2024-09-21T02:36:25.745639+0000","last_active":"2024-09-21T02:38:40.294112+0000","last_peered":"2024-09-21T02:38:40.294112+0000","last_clean":"2024-09-21T02:38:40.294112+0000","last_became_active":"2024-09-21T02:36:25.745202+0000","last_became_peered":"2024-09-21T02:36:25.745202+0000","last_unstale":"2024-09-21T02:38:40.294112+0000","last_undegraded":"2024-09-21T02:38:40.294112+0000","last_fullsized":"2024-09-21T02:38:40.294112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:08.712477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504086+0000","last_change":"2024-09-21T02:36:23.730710+0000","last_active":"2024-09-21T02:38:40.504086+0000","last_peered":"2024-09-21T02:38:40.504086+0000","last_clean":"2024-09-21T02:38:40.504086+0000","last_became_active":"2024-09-21T02:36:23.730247+0000","last_became_peered":"2024-09-21T02:36:23.730247+0000","last_unstale":"2024-09-21T02:38:40.504086+0000","last_undegraded":"2024-09-21T02:38:40.504086+0000","last_fullsized":"2024-09-21T02:38:40.504086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:43.863444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":113,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503097+0000","last_change":"2024-09-21T02:35:09.949841+0000","last_active":"2024-09-21T02:38:40.503097+0000","last_peered":"2024-09-21T02:38:40.503097+0000","last_clean":"2024-09-21T02:38:40.503097+0000","last_became_active":"2024-09-21T02:35:09.944224+0000","last_became_peered":"2024-09-21T02:35:09.944224+0000","last_unstale":"2024-09-21T02:38:40.503097+0000","last_undegraded":"2024-09-21T02:38:40.503097+0000","last_fullsized":"2024-09-21T02:38:40.503097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:31:38.930125+0000","last_clean_scrub_stamp":"2024-09-21T02:31:38.930125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:56.467576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503366+0000","last_change":"2024-09-21T02:37:39.874486+0000","last_active":"2024-09-21T02:38:40.503366+0000","last_peered":"2024-09-21T02:38:40.503366+0000","last_clean":"2024-09-21T02:38:40.503366+0000","last_became_active":"2024-09-21T02:37:39.869523+0000","last_became_peered":"2024-09-21T02:37:39.869523+0000","last_unstale":"2024-09-21T02:38:40.503366+0000","last_undegraded":"2024-09-21T02:38:40.503366+0000","last_fullsized":"2024-09-21T02:38:40.503366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:21:33.524428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503759+0000","last_change":"2024-09-21T02:36:25.741176+0000","last_active":"2024-09-21T02:38:40.503759+0000","last_peered":"2024-09-21T02:38:40.503759+0000","last_clean":"2024-09-21T02:38:40.503759+0000","last_became_active":"2024-09-21T02:36:25.740355+0000","last_became_peered":"2024-09-21T02:36:25.740355+0000","last_unstale":"2024-09-21T02:38:40.503759+0000","last_undegraded":"2024-09-21T02:38:40.503759+0000","last_fullsized":"2024-09-21T02:38:40.503759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:47:48.204768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'8","reported_seq":166,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.214476+0000","last_change":"2024-09-21T02:36:24.261500+0000","last_active":"2024-09-21T02:39:02.214476+0000","last_peered":"2024-09-21T02:39:02.214476+0000","last_clean":"2024-09-21T02:39:02.214476+0000","last_became_active":"2024-09-21T02:36:24.261089+0000","last_became_peered":"2024-09-21T02:36:24.261089+0000","last_unstale":"2024-09-21T02:39:02.214476+0000","last_undegraded":"2024-09-21T02:39:02.214476+0000","last_fullsized":"2024-09-21T02:39:02.214476+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:33:33.158177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2195,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227148+0000","last_change":"2024-09-21T02:36:32.796980+0000","last_active":"2024-09-21T02:38:40.227148+0000","last_peered":"2024-09-21T02:38:40.227148+0000","last_clean":"2024-09-21T02:38:40.227148+0000","last_became_active":"2024-09-21T02:36:32.796517+0000","last_became_peered":"2024-09-21T02:36:32.796517+0000","last_unstale":"2024-09-21T02:38:40.227148+0000","last_undegraded":"2024-09-21T02:38:40.227148+0000","last_fullsized":"2024-09-21T02:38:40.227148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:07.230177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502998+0000","last_change":"2024-09-21T02:36:25.750931+0000","last_active":"2024-09-21T02:38:40.502998+0000","last_peered":"2024-09-21T02:38:40.502998+0000","last_clean":"2024-09-21T02:38:40.502998+0000","last_became_active":"2024-09-21T02:36:25.750604+0000","last_became_peered":"2024-09-21T02:36:25.750604+0000","last_unstale":"2024-09-21T02:38:40.502998+0000","last_undegraded":"2024-09-21T02:38:40.502998+0000","last_fullsized":"2024-09-21T02:38:40.502998+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:21.325095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504562+0000","last_change":"2024-09-21T02:36:23.730700+0000","last_active":"2024-09-21T02:38:40.504562+0000","last_peered":"2024-09-21T02:38:40.504562+0000","last_clean":"2024-09-21T02:38:40.504562+0000","last_became_active":"2024-09-21T02:36:23.730367+0000","last_became_peered":"2024-09-21T02:36:23.730367+0000","last_unstale":"2024-09-21T02:38:40.504562+0000","last_undegraded":"2024-09-21T02:38:40.504562+0000","last_fullsized":"2024-09-21T02:38:40.504562+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:27:57.373077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233491+0000","last_change":"2024-09-21T02:36:32.805885+0000","last_active":"2024-09-21T02:38:40.233491+0000","last_peered":"2024-09-21T02:38:40.233491+0000","last_clean":"2024-09-21T02:38:40.233491+0000","last_became_active":"2024-09-21T02:36:32.805479+0000","last_became_peered":"2024-09-21T02:36:32.805479+0000","last_unstale":"2024-09-21T02:38:40.233491+0000","last_undegraded":"2024-09-21T02:38:40.233491+0000","last_fullsized":"2024-09-21T02:38:40.233491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:11:29.910642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503631+0000","last_change":"2024-09-21T02:36:25.742542+0000","last_active":"2024-09-21T02:38:40.503631+0000","last_peered":"2024-09-21T02:38:40.503631+0000","last_clean":"2024-09-21T02:38:40.503631+0000","last_became_active":"2024-09-21T02:36:25.742000+0000","last_became_peered":"2024-09-21T02:36:25.742000+0000","last_unstale":"2024-09-21T02:38:40.503631+0000","last_undegraded":"2024-09-21T02:38:40.503631+0000","last_fullsized":"2024-09-21T02:38:40.503631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:52.300212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222446+0000","last_change":"2024-09-21T02:36:23.723896+0000","last_active":"2024-09-21T02:38:40.222446+0000","last_peered":"2024-09-21T02:38:40.222446+0000","last_clean":"2024-09-21T02:38:40.222446+0000","last_became_active":"2024-09-21T02:36:23.723518+0000","last_became_peered":"2024-09-21T02:36:23.723518+0000","last_unstale":"2024-09-21T02:38:40.222446+0000","last_undegraded":"2024-09-21T02:38:40.222446+0000","last_fullsized":"2024-09-21T02:38:40.222446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:37.250247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503797+0000","last_change":"2024-09-21T02:36:32.799924+0000","last_active":"2024-09-21T02:38:40.503797+0000","last_peered":"2024-09-21T02:38:40.503797+0000","last_clean":"2024-09-21T02:38:40.503797+0000","last_became_active":"2024-09-21T02:36:32.799437+0000","last_became_peered":"2024-09-21T02:36:32.799437+0000","last_unstale":"2024-09-21T02:38:40.503797+0000","last_undegraded":"2024-09-21T02:38:40.503797+0000","last_fullsized":"2024-09-21T02:38:40.503797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:19.258293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331925+0000","last_change":"2024-09-21T02:36:25.747321+0000","last_active":"2024-09-21T02:38:40.331925+0000","last_peered":"2024-09-21T02:38:40.331925+0000","last_clean":"2024-09-21T02:38:40.331925+0000","last_became_active":"2024-09-21T02:36:25.746955+0000","last_became_peered":"2024-09-21T02:36:25.746955+0000","last_unstale":"2024-09-21T02:38:40.331925+0000","last_undegraded":"2024-09-21T02:38:40.331925+0000","last_fullsized":"2024-09-21T02:38:40.331925+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:25.148300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"85'3","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:41.078921+0000","last_change":"2024-09-21T02:36:23.725350+0000","last_active":"2024-09-21T02:38:41.078921+0000","last_peered":"2024-09-21T02:38:41.078921+0000","last_clean":"2024-09-21T02:38:41.078921+0000","last_became_active":"2024-09-21T02:36:23.725090+0000","last_became_peered":"2024-09-21T02:36:23.725090+0000","last_unstale":"2024-09-21T02:38:41.078921+0000","last_undegraded":"2024-09-21T02:38:41.078921+0000","last_fullsized":"2024-09-21T02:38:41.078921+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:52.130345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503689+0000","last_change":"2024-09-21T02:36:32.801355+0000","last_active":"2024-09-21T02:38:40.503689+0000","last_peered":"2024-09-21T02:38:40.503689+0000","last_clean":"2024-09-21T02:38:40.503689+0000","last_became_active":"2024-09-21T02:36:32.800803+0000","last_became_peered":"2024-09-21T02:36:32.800803+0000","last_unstale":"2024-09-21T02:38:40.503689+0000","last_undegraded":"2024-09-21T02:38:40.503689+0000","last_fullsized":"2024-09-21T02:38:40.503689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:54.755733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504077+0000","last_change":"2024-09-21T02:36:25.743433+0000","last_active":"2024-09-21T02:38:40.504077+0000","last_peered":"2024-09-21T02:38:40.504077+0000","last_clean":"2024-09-21T02:38:40.504077+0000","last_became_active":"2024-09-21T02:36:25.742643+0000","last_became_peered":"2024-09-21T02:36:25.742643+0000","last_unstale":"2024-09-21T02:38:40.504077+0000","last_undegraded":"2024-09-21T02:38:40.504077+0000","last_fullsized":"2024-09-21T02:38:40.504077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:20:09.849806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331234+0000","last_change":"2024-09-21T02:37:39.625360+0000","last_active":"2024-09-21T02:38:40.331234+0000","last_peered":"2024-09-21T02:38:40.331234+0000","last_clean":"2024-09-21T02:38:40.331234+0000","last_became_active":"2024-09-21T02:37:39.611781+0000","last_became_peered":"2024-09-21T02:37:39.611781+0000","last_unstale":"2024-09-21T02:38:40.331234+0000","last_undegraded":"2024-09-21T02:38:40.331234+0000","last_fullsized":"2024-09-21T02:38:40.331234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:11.846383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222385+0000","last_change":"2024-09-21T02:36:32.799728+0000","last_active":"2024-09-21T02:38:40.222385+0000","last_peered":"2024-09-21T02:38:40.222385+0000","last_clean":"2024-09-21T02:38:40.222385+0000","last_became_active":"2024-09-21T02:36:32.798419+0000","last_became_peered":"2024-09-21T02:36:32.798419+0000","last_unstale":"2024-09-21T02:38:40.222385+0000","last_undegraded":"2024-09-21T02:38:40.222385+0000","last_fullsized":"2024-09-21T02:38:40.222385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:53.681473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502716+0000","last_change":"2024-09-21T02:36:25.747712+0000","last_active":"2024-09-21T02:38:40.502716+0000","last_peered":"2024-09-21T02:38:40.502716+0000","last_clean":"2024-09-21T02:38:40.502716+0000","last_became_active":"2024-09-21T02:36:25.747197+0000","last_became_peered":"2024-09-21T02:36:25.747197+0000","last_unstale":"2024-09-21T02:38:40.502716+0000","last_undegraded":"2024-09-21T02:38:40.502716+0000","last_fullsized":"2024-09-21T02:38:40.502716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:41:03.900360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":165,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:04.214988+0000","last_change":"2024-09-21T02:36:23.732156+0000","last_active":"2024-09-21T02:39:04.214988+0000","last_peered":"2024-09-21T02:39:04.214988+0000","last_clean":"2024-09-21T02:39:04.214988+0000","last_became_active":"2024-09-21T02:36:23.730238+0000","last_became_peered":"2024-09-21T02:36:23.730238+0000","last_unstale":"2024-09-21T02:39:04.214988+0000","last_undegraded":"2024-09-21T02:39:04.214988+0000","last_fullsized":"2024-09-21T02:39:04.214988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:18.053890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503510+0000","last_change":"2024-09-21T02:36:32.806674+0000","last_active":"2024-09-21T02:38:40.503510+0000","last_peered":"2024-09-21T02:38:40.503510+0000","last_clean":"2024-09-21T02:38:40.503510+0000","last_became_active":"2024-09-21T02:36:32.805373+0000","last_became_peered":"2024-09-21T02:36:32.805373+0000","last_unstale":"2024-09-21T02:38:40.503510+0000","last_undegraded":"2024-09-21T02:38:40.503510+0000","last_fullsized":"2024-09-21T02:38:40.503510+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:29.249243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503029+0000","last_change":"2024-09-21T02:36:25.748703+0000","last_active":"2024-09-21T02:38:40.503029+0000","last_peered":"2024-09-21T02:38:40.503029+0000","last_clean":"2024-09-21T02:38:40.503029+0000","last_became_active":"2024-09-21T02:36:25.748340+0000","last_became_peered":"2024-09-21T02:36:25.748340+0000","last_unstale":"2024-09-21T02:38:40.503029+0000","last_undegraded":"2024-09-21T02:38:40.503029+0000","last_fullsized":"2024-09-21T02:38:40.503029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:07.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504103+0000","last_change":"2024-09-21T02:36:23.732332+0000","last_active":"2024-09-21T02:38:40.504103+0000","last_peered":"2024-09-21T02:38:40.504103+0000","last_clean":"2024-09-21T02:38:40.504103+0000","last_became_active":"2024-09-21T02:36:23.731974+0000","last_became_peered":"2024-09-21T02:36:23.731974+0000","last_unstale":"2024-09-21T02:38:40.504103+0000","last_undegraded":"2024-09-21T02:38:40.504103+0000","last_fullsized":"2024-09-21T02:38:40.504103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:52.284192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331545+0000","last_change":"2024-09-21T02:37:39.610698+0000","last_active":"2024-09-21T02:38:40.331545+0000","last_peered":"2024-09-21T02:38:40.331545+0000","last_clean":"2024-09-21T02:38:40.331545+0000","last_became_active":"2024-09-21T02:37:39.610185+0000","last_became_peered":"2024-09-21T02:37:39.610185+0000","last_unstale":"2024-09-21T02:38:40.331545+0000","last_undegraded":"2024-09-21T02:38:40.331545+0000","last_fullsized":"2024-09-21T02:38:40.331545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:54:51.050463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221965+0000","last_change":"2024-09-21T02:36:25.751720+0000","last_active":"2024-09-21T02:38:40.221965+0000","last_peered":"2024-09-21T02:38:40.221965+0000","last_clean":"2024-09-21T02:38:40.221965+0000","last_became_active":"2024-09-21T02:36:25.751351+0000","last_became_peered":"2024-09-21T02:36:25.751351+0000","last_unstale":"2024-09-21T02:38:40.221965+0000","last_undegraded":"2024-09-21T02:38:40.221965+0000","last_fullsized":"2024-09-21T02:38:40.221965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:01.201453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224603+0000","last_change":"2024-09-21T02:36:23.729981+0000","last_active":"2024-09-21T02:38:40.224603+0000","last_peered":"2024-09-21T02:38:40.224603+0000","last_clean":"2024-09-21T02:38:40.224603+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:38:40.224603+0000","last_undegraded":"2024-09-21T02:38:40.224603+0000","last_fullsized":"2024-09-21T02:38:40.224603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:03.115712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504439+0000","last_change":"2024-09-21T02:36:32.805286+0000","last_active":"2024-09-21T02:38:40.504439+0000","last_peered":"2024-09-21T02:38:40.504439+0000","last_clean":"2024-09-21T02:38:40.504439+0000","last_became_active":"2024-09-21T02:36:32.804770+0000","last_became_peered":"2024-09-21T02:36:32.804770+0000","last_unstale":"2024-09-21T02:38:40.504439+0000","last_undegraded":"2024-09-21T02:38:40.504439+0000","last_fullsized":"2024-09-21T02:38:40.504439+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:18:34.155892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503496+0000","last_change":"2024-09-21T02:36:25.747812+0000","last_active":"2024-09-21T02:38:40.503496+0000","last_peered":"2024-09-21T02:38:40.503496+0000","last_clean":"2024-09-21T02:38:40.503496+0000","last_became_active":"2024-09-21T02:36:25.747187+0000","last_became_peered":"2024-09-21T02:36:25.747187+0000","last_unstale":"2024-09-21T02:38:40.503496+0000","last_undegraded":"2024-09-21T02:38:40.503496+0000","last_fullsized":"2024-09-21T02:38:40.503496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:51.395869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":316,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.223137+0000","last_change":"2024-09-21T02:36:23.735676+0000","last_active":"2024-09-21T02:39:02.223137+0000","last_peered":"2024-09-21T02:39:02.223137+0000","last_clean":"2024-09-21T02:39:02.223137+0000","last_became_active":"2024-09-21T02:36:23.735281+0000","last_became_peered":"2024-09-21T02:36:23.735281+0000","last_unstale":"2024-09-21T02:39:02.223137+0000","last_undegraded":"2024-09-21T02:39:02.223137+0000","last_fullsized":"2024-09-21T02:39:02.223137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:28:01.547245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68362,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":236,"num_read_kb":184,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503129+0000","last_change":"2024-09-21T02:36:32.806585+0000","last_active":"2024-09-21T02:38:40.503129+0000","last_peered":"2024-09-21T02:38:40.503129+0000","last_clean":"2024-09-21T02:38:40.503129+0000","last_became_active":"2024-09-21T02:36:32.805913+0000","last_became_peered":"2024-09-21T02:36:32.805913+0000","last_unstale":"2024-09-21T02:38:40.503129+0000","last_undegraded":"2024-09-21T02:38:40.503129+0000","last_fullsized":"2024-09-21T02:38:40.503129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:59:31.888645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228365+0000","last_change":"2024-09-21T02:36:25.742208+0000","last_active":"2024-09-21T02:38:40.228365+0000","last_peered":"2024-09-21T02:38:40.228365+0000","last_clean":"2024-09-21T02:38:40.228365+0000","last_became_active":"2024-09-21T02:36:25.741849+0000","last_became_peered":"2024-09-21T02:36:25.741849+0000","last_unstale":"2024-09-21T02:38:40.228365+0000","last_undegraded":"2024-09-21T02:38:40.228365+0000","last_fullsized":"2024-09-21T02:38:40.228365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:21:43.720481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504523+0000","last_change":"2024-09-21T02:36:23.800751+0000","last_active":"2024-09-21T02:38:40.504523+0000","last_peered":"2024-09-21T02:38:40.504523+0000","last_clean":"2024-09-21T02:38:40.504523+0000","last_became_active":"2024-09-21T02:36:23.800341+0000","last_became_peered":"2024-09-21T02:36:23.800341+0000","last_unstale":"2024-09-21T02:38:40.504523+0000","last_undegraded":"2024-09-21T02:38:40.504523+0000","last_fullsized":"2024-09-21T02:38:40.504523+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:08:02.556549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503710+0000","last_change":"2024-09-21T02:36:32.802879+0000","last_active":"2024-09-21T02:38:40.503710+0000","last_peered":"2024-09-21T02:38:40.503710+0000","last_clean":"2024-09-21T02:38:40.503710+0000","last_became_active":"2024-09-21T02:36:32.802301+0000","last_became_peered":"2024-09-21T02:36:32.802301+0000","last_unstale":"2024-09-21T02:38:40.503710+0000","last_undegraded":"2024-09-21T02:38:40.503710+0000","last_fullsized":"2024-09-21T02:38:40.503710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:22:08.135814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232720+0000","last_change":"2024-09-21T02:36:25.744647+0000","last_active":"2024-09-21T02:38:40.232720+0000","last_peered":"2024-09-21T02:38:40.232720+0000","last_clean":"2024-09-21T02:38:40.232720+0000","last_became_active":"2024-09-21T02:36:25.744049+0000","last_became_peered":"2024-09-21T02:36:25.744049+0000","last_unstale":"2024-09-21T02:38:40.232720+0000","last_undegraded":"2024-09-21T02:38:40.232720+0000","last_fullsized":"2024-09-21T02:38:40.232720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:52:02.210380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503652+0000","last_change":"2024-09-21T02:36:23.733590+0000","last_active":"2024-09-21T02:38:40.503652+0000","last_peered":"2024-09-21T02:38:40.503652+0000","last_clean":"2024-09-21T02:38:40.503652+0000","last_became_active":"2024-09-21T02:36:23.732922+0000","last_became_peered":"2024-09-21T02:36:23.732922+0000","last_unstale":"2024-09-21T02:38:40.503652+0000","last_undegraded":"2024-09-21T02:38:40.503652+0000","last_fullsized":"2024-09-21T02:38:40.503652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503016+0000","last_change":"2024-09-21T02:36:32.806540+0000","last_active":"2024-09-21T02:38:40.503016+0000","last_peered":"2024-09-21T02:38:40.503016+0000","last_clean":"2024-09-21T02:38:40.503016+0000","last_became_active":"2024-09-21T02:36:32.805612+0000","last_became_peered":"2024-09-21T02:36:32.805612+0000","last_unstale":"2024-09-21T02:38:40.503016+0000","last_undegraded":"2024-09-21T02:38:40.503016+0000","last_fullsized":"2024-09-21T02:38:40.503016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:23:24.686383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227650+0000","last_change":"2024-09-21T02:36:25.746229+0000","last_active":"2024-09-21T02:38:40.227650+0000","last_peered":"2024-09-21T02:38:40.227650+0000","last_clean":"2024-09-21T02:38:40.227650+0000","last_became_active":"2024-09-21T02:36:25.745892+0000","last_became_peered":"2024-09-21T02:36:25.745892+0000","last_unstale":"2024-09-21T02:38:40.227650+0000","last_undegraded":"2024-09-21T02:38:40.227650+0000","last_fullsized":"2024-09-21T02:38:40.227650+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:55.154499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"85'8","reported_seq":274,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:02.213998+0000","last_change":"2024-09-21T02:36:23.730987+0000","last_active":"2024-09-21T02:39:02.213998+0000","last_peered":"2024-09-21T02:39:02.213998+0000","last_clean":"2024-09-21T02:39:02.213998+0000","last_became_active":"2024-09-21T02:36:23.730566+0000","last_became_peered":"2024-09-21T02:36:23.730566+0000","last_unstale":"2024-09-21T02:39:02.213998+0000","last_undegraded":"2024-09-21T02:39:02.213998+0000","last_fullsized":"2024-09-21T02:39:02.213998+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:18:12.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293972+0000","last_change":"2024-09-21T02:36:32.804608+0000","last_active":"2024-09-21T02:38:40.293972+0000","last_peered":"2024-09-21T02:38:40.293972+0000","last_clean":"2024-09-21T02:38:40.293972+0000","last_became_active":"2024-09-21T02:36:32.803567+0000","last_became_peered":"2024-09-21T02:36:32.803567+0000","last_unstale":"2024-09-21T02:38:40.293972+0000","last_undegraded":"2024-09-21T02:38:40.293972+0000","last_fullsized":"2024-09-21T02:38:40.293972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:41.692644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232808+0000","last_change":"2024-09-21T02:36:32.804687+0000","last_active":"2024-09-21T02:38:40.232808+0000","last_peered":"2024-09-21T02:38:40.232808+0000","last_clean":"2024-09-21T02:38:40.232808+0000","last_became_active":"2024-09-21T02:36:32.804232+0000","last_became_peered":"2024-09-21T02:36:32.804232+0000","last_unstale":"2024-09-21T02:38:40.232808+0000","last_undegraded":"2024-09-21T02:38:40.232808+0000","last_fullsized":"2024-09-21T02:38:40.232808+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:43.899924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503127+0000","last_change":"2024-09-21T02:36:25.749114+0000","last_active":"2024-09-21T02:38:40.503127+0000","last_peered":"2024-09-21T02:38:40.503127+0000","last_clean":"2024-09-21T02:38:40.503127+0000","last_became_active":"2024-09-21T02:36:25.748758+0000","last_became_peered":"2024-09-21T02:36:25.748758+0000","last_unstale":"2024-09-21T02:38:40.503127+0000","last_undegraded":"2024-09-21T02:38:40.503127+0000","last_fullsized":"2024-09-21T02:38:40.503127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:57.958533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504282+0000","last_change":"2024-09-21T02:36:23.731473+0000","last_active":"2024-09-21T02:38:40.504282+0000","last_peered":"2024-09-21T02:38:40.504282+0000","last_clean":"2024-09-21T02:38:40.504282+0000","last_became_active":"2024-09-21T02:36:23.731134+0000","last_became_peered":"2024-09-21T02:36:23.731134+0000","last_unstale":"2024-09-21T02:38:40.504282+0000","last_undegraded":"2024-09-21T02:38:40.504282+0000","last_fullsized":"2024-09-21T02:38:40.504282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:05.368941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331458+0000","last_change":"2024-09-21T02:36:32.809630+0000","last_active":"2024-09-21T02:38:40.331458+0000","last_peered":"2024-09-21T02:38:40.331458+0000","last_clean":"2024-09-21T02:38:40.331458+0000","last_became_active":"2024-09-21T02:36:32.809030+0000","last_became_peered":"2024-09-21T02:36:32.809030+0000","last_unstale":"2024-09-21T02:38:40.331458+0000","last_undegraded":"2024-09-21T02:38:40.331458+0000","last_fullsized":"2024-09-21T02:38:40.331458+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:57:42.436590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293966+0000","last_change":"2024-09-21T02:36:25.742558+0000","last_active":"2024-09-21T02:38:40.293966+0000","last_peered":"2024-09-21T02:38:40.293966+0000","last_clean":"2024-09-21T02:38:40.293966+0000","last_became_active":"2024-09-21T02:36:25.742045+0000","last_became_peered":"2024-09-21T02:36:25.742045+0000","last_unstale":"2024-09-21T02:38:40.293966+0000","last_undegraded":"2024-09-21T02:38:40.293966+0000","last_fullsized":"2024-09-21T02:38:40.293966+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:57:06.705561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.221778+0000","last_change":"2024-09-21T02:36:23.724345+0000","last_active":"2024-09-21T02:38:40.221778+0000","last_peered":"2024-09-21T02:38:40.221778+0000","last_clean":"2024-09-21T02:38:40.221778+0000","last_became_active":"2024-09-21T02:36:23.723635+0000","last_became_peered":"2024-09-21T02:36:23.723635+0000","last_unstale":"2024-09-21T02:38:40.221778+0000","last_undegraded":"2024-09-21T02:38:40.221778+0000","last_fullsized":"2024-09-21T02:38:40.221778+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:18.812176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223962+0000","last_change":"2024-09-21T02:36:23.732897+0000","last_active":"2024-09-21T02:38:40.223962+0000","last_peered":"2024-09-21T02:38:40.223962+0000","last_clean":"2024-09-21T02:38:40.223962+0000","last_became_active":"2024-09-21T02:36:23.732509+0000","last_became_peered":"2024-09-21T02:36:23.732509+0000","last_unstale":"2024-09-21T02:38:40.223962+0000","last_undegraded":"2024-09-21T02:38:40.223962+0000","last_fullsized":"2024-09-21T02:38:40.223962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:12:27.117962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504315+0000","last_change":"2024-09-21T02:36:25.748929+0000","last_active":"2024-09-21T02:38:40.504315+0000","last_peered":"2024-09-21T02:38:40.504315+0000","last_clean":"2024-09-21T02:38:40.504315+0000","last_became_active":"2024-09-21T02:36:25.748588+0000","last_became_peered":"2024-09-21T02:36:25.748588+0000","last_unstale":"2024-09-21T02:38:40.504315+0000","last_undegraded":"2024-09-21T02:38:40.504315+0000","last_fullsized":"2024-09-21T02:38:40.504315+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.086472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224498+0000","last_change":"2024-09-21T02:36:32.807655+0000","last_active":"2024-09-21T02:38:40.224498+0000","last_peered":"2024-09-21T02:38:40.224498+0000","last_clean":"2024-09-21T02:38:40.224498+0000","last_became_active":"2024-09-21T02:36:32.806995+0000","last_became_peered":"2024-09-21T02:36:32.806995+0000","last_unstale":"2024-09-21T02:38:40.224498+0000","last_undegraded":"2024-09-21T02:38:40.224498+0000","last_fullsized":"2024-09-21T02:38:40.224498+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:28:20.420591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'9","reported_seq":172,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:39:05.217495+0000","last_change":"2024-09-21T02:36:23.729427+0000","last_active":"2024-09-21T02:39:05.217495+0000","last_peered":"2024-09-21T02:39:05.217495+0000","last_clean":"2024-09-21T02:39:05.217495+0000","last_became_active":"2024-09-21T02:36:23.729077+0000","last_became_peered":"2024-09-21T02:36:23.729077+0000","last_unstale":"2024-09-21T02:39:05.217495+0000","last_undegraded":"2024-09-21T02:39:05.217495+0000","last_fullsized":"2024-09-21T02:39:05.217495+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:22.711389+0000","last_clean_scrub_stamp":"2024-09-21T02:36:22.711389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:10:41.251665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.502991+0000","last_change":"2024-09-21T02:36:25.743212+0000","last_active":"2024-09-21T02:38:40.502991+0000","last_peered":"2024-09-21T02:38:40.502991+0000","last_clean":"2024-09-21T02:38:40.502991+0000","last_became_active":"2024-09-21T02:36:25.742777+0000","last_became_peered":"2024-09-21T02:36:25.742777+0000","last_unstale":"2024-09-21T02:38:40.502991+0000","last_undegraded":"2024-09-21T02:38:40.502991+0000","last_fullsized":"2024-09-21T02:38:40.502991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:12:19.917214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224485+0000","last_change":"2024-09-21T02:36:32.801880+0000","last_active":"2024-09-21T02:38:40.224485+0000","last_peered":"2024-09-21T02:38:40.224485+0000","last_clean":"2024-09-21T02:38:40.224485+0000","last_became_active":"2024-09-21T02:36:32.801389+0000","last_became_peered":"2024-09-21T02:36:32.801389+0000","last_unstale":"2024-09-21T02:38:40.224485+0000","last_undegraded":"2024-09-21T02:38:40.224485+0000","last_fullsized":"2024-09-21T02:38:40.224485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:31.769890+0000","last_clean_scrub_stamp":"2024-09-21T02:36:31.769890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:11:44.877062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331561+0000","last_change":"2024-09-21T02:36:25.754014+0000","last_active":"2024-09-21T02:38:40.331561+0000","last_peered":"2024-09-21T02:38:40.331561+0000","last_clean":"2024-09-21T02:38:40.331561+0000","last_became_active":"2024-09-21T02:36:25.753548+0000","last_became_peered":"2024-09-21T02:36:25.753548+0000","last_unstale":"2024-09-21T02:38:40.331561+0000","last_undegraded":"2024-09-21T02:38:40.331561+0000","last_fullsized":"2024-09-21T02:38:40.331561+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:03.850625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224637+0000","last_change":"2024-09-21T02:37:39.610894+0000","last_active":"2024-09-21T02:38:40.224637+0000","last_peered":"2024-09-21T02:38:40.224637+0000","last_clean":"2024-09-21T02:38:40.224637+0000","last_became_active":"2024-09-21T02:37:39.610404+0000","last_became_peered":"2024-09-21T02:37:39.610404+0000","last_unstale":"2024-09-21T02:38:40.224637+0000","last_undegraded":"2024-09-21T02:38:40.224637+0000","last_fullsized":"2024-09-21T02:38:40.224637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:54.347708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224163+0000","last_change":"2024-09-21T02:36:25.752839+0000","last_active":"2024-09-21T02:38:40.224163+0000","last_peered":"2024-09-21T02:38:40.224163+0000","last_clean":"2024-09-21T02:38:40.224163+0000","last_became_active":"2024-09-21T02:36:25.750882+0000","last_became_peered":"2024-09-21T02:36:25.750882+0000","last_unstale":"2024-09-21T02:38:40.224163+0000","last_undegraded":"2024-09-21T02:38:40.224163+0000","last_fullsized":"2024-09-21T02:38:40.224163+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:53.028611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.233548+0000","last_change":"2024-09-21T02:36:25.744624+0000","last_active":"2024-09-21T02:38:40.233548+0000","last_peered":"2024-09-21T02:38:40.233548+0000","last_clean":"2024-09-21T02:38:40.233548+0000","last_became_active":"2024-09-21T02:36:25.744173+0000","last_became_peered":"2024-09-21T02:36:25.744173+0000","last_unstale":"2024-09-21T02:38:40.233548+0000","last_undegraded":"2024-09-21T02:38:40.233548+0000","last_fullsized":"2024-09-21T02:38:40.233548+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:55.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222157+0000","last_change":"2024-09-21T02:37:39.609050+0000","last_active":"2024-09-21T02:38:40.222157+0000","last_peered":"2024-09-21T02:38:40.222157+0000","last_clean":"2024-09-21T02:38:40.222157+0000","last_became_active":"2024-09-21T02:37:39.608595+0000","last_became_peered":"2024-09-21T02:37:39.608595+0000","last_unstale":"2024-09-21T02:38:40.222157+0000","last_undegraded":"2024-09-21T02:38:40.222157+0000","last_fullsized":"2024-09-21T02:38:40.222157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:26.831121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331839+0000","last_change":"2024-09-21T02:36:25.743476+0000","last_active":"2024-09-21T02:38:40.331839+0000","last_peered":"2024-09-21T02:38:40.331839+0000","last_clean":"2024-09-21T02:38:40.331839+0000","last_became_active":"2024-09-21T02:36:25.743117+0000","last_became_peered":"2024-09-21T02:36:25.743117+0000","last_unstale":"2024-09-21T02:38:40.331839+0000","last_undegraded":"2024-09-21T02:38:40.331839+0000","last_fullsized":"2024-09-21T02:38:40.331839+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:08.084109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224831+0000","last_change":"2024-09-21T02:36:25.753014+0000","last_active":"2024-09-21T02:38:40.224831+0000","last_peered":"2024-09-21T02:38:40.224831+0000","last_clean":"2024-09-21T02:38:40.224831+0000","last_became_active":"2024-09-21T02:36:25.751232+0000","last_became_peered":"2024-09-21T02:36:25.751232+0000","last_unstale":"2024-09-21T02:38:40.224831+0000","last_undegraded":"2024-09-21T02:38:40.224831+0000","last_fullsized":"2024-09-21T02:38:40.224831+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:03.052737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503339+0000","last_change":"2024-09-21T02:37:39.606525+0000","last_active":"2024-09-21T02:38:40.503339+0000","last_peered":"2024-09-21T02:38:40.503339+0000","last_clean":"2024-09-21T02:38:40.503339+0000","last_became_active":"2024-09-21T02:37:39.606078+0000","last_became_peered":"2024-09-21T02:37:39.606078+0000","last_unstale":"2024-09-21T02:38:40.503339+0000","last_undegraded":"2024-09-21T02:38:40.503339+0000","last_fullsized":"2024-09-21T02:38:40.503339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:22.423457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503992+0000","last_change":"2024-09-21T02:36:25.749605+0000","last_active":"2024-09-21T02:38:40.503992+0000","last_peered":"2024-09-21T02:38:40.503992+0000","last_clean":"2024-09-21T02:38:40.503992+0000","last_became_active":"2024-09-21T02:36:25.749165+0000","last_became_peered":"2024-09-21T02:36:25.749165+0000","last_unstale":"2024-09-21T02:38:40.503992+0000","last_undegraded":"2024-09-21T02:38:40.503992+0000","last_fullsized":"2024-09-21T02:38:40.503992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:15:51.412761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.294064+0000","last_change":"2024-09-21T02:36:25.747494+0000","last_active":"2024-09-21T02:38:40.294064+0000","last_peered":"2024-09-21T02:38:40.294064+0000","last_clean":"2024-09-21T02:38:40.294064+0000","last_became_active":"2024-09-21T02:36:25.746965+0000","last_became_peered":"2024-09-21T02:36:25.746965+0000","last_unstale":"2024-09-21T02:38:40.294064+0000","last_undegraded":"2024-09-21T02:38:40.294064+0000","last_fullsized":"2024-09-21T02:38:40.294064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:51.309307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.227481+0000","last_change":"2024-09-21T02:36:25.749517+0000","last_active":"2024-09-21T02:38:40.227481+0000","last_peered":"2024-09-21T02:38:40.227481+0000","last_clean":"2024-09-21T02:38:40.227481+0000","last_became_active":"2024-09-21T02:36:25.749117+0000","last_became_peered":"2024-09-21T02:36:25.749117+0000","last_unstale":"2024-09-21T02:38:40.227481+0000","last_undegraded":"2024-09-21T02:38:40.227481+0000","last_fullsized":"2024-09-21T02:38:40.227481+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:38.758019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222549+0000","last_change":"2024-09-21T02:36:25.736067+0000","last_active":"2024-09-21T02:38:40.222549+0000","last_peered":"2024-09-21T02:38:40.222549+0000","last_clean":"2024-09-21T02:38:40.222549+0000","last_became_active":"2024-09-21T02:36:25.735710+0000","last_became_peered":"2024-09-21T02:36:25.735710+0000","last_unstale":"2024-09-21T02:38:40.222549+0000","last_undegraded":"2024-09-21T02:38:40.222549+0000","last_fullsized":"2024-09-21T02:38:40.222549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.559448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331864+0000","last_change":"2024-09-21T02:36:25.746307+0000","last_active":"2024-09-21T02:38:40.331864+0000","last_peered":"2024-09-21T02:38:40.331864+0000","last_clean":"2024-09-21T02:38:40.331864+0000","last_became_active":"2024-09-21T02:36:25.745872+0000","last_became_peered":"2024-09-21T02:36:25.745872+0000","last_unstale":"2024-09-21T02:38:40.331864+0000","last_undegraded":"2024-09-21T02:38:40.331864+0000","last_fullsized":"2024-09-21T02:38:40.331864+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:58:53.074559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224609+0000","last_change":"2024-09-21T02:36:25.753393+0000","last_active":"2024-09-21T02:38:40.224609+0000","last_peered":"2024-09-21T02:38:40.224609+0000","last_clean":"2024-09-21T02:38:40.224609+0000","last_became_active":"2024-09-21T02:36:25.751871+0000","last_became_peered":"2024-09-21T02:36:25.751871+0000","last_unstale":"2024-09-21T02:38:40.224609+0000","last_undegraded":"2024-09-21T02:38:40.224609+0000","last_fullsized":"2024-09-21T02:38:40.224609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:28.199451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503585+0000","last_change":"2024-09-21T02:36:25.744845+0000","last_active":"2024-09-21T02:38:40.503585+0000","last_peered":"2024-09-21T02:38:40.503585+0000","last_clean":"2024-09-21T02:38:40.503585+0000","last_became_active":"2024-09-21T02:36:25.744492+0000","last_became_peered":"2024-09-21T02:36:25.744492+0000","last_unstale":"2024-09-21T02:38:40.503585+0000","last_undegraded":"2024-09-21T02:38:40.503585+0000","last_fullsized":"2024-09-21T02:38:40.503585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:08.524871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331829+0000","last_change":"2024-09-21T02:36:25.751187+0000","last_active":"2024-09-21T02:38:40.331829+0000","last_peered":"2024-09-21T02:38:40.331829+0000","last_clean":"2024-09-21T02:38:40.331829+0000","last_became_active":"2024-09-21T02:36:25.750813+0000","last_became_peered":"2024-09-21T02:36:25.750813+0000","last_unstale":"2024-09-21T02:38:40.331829+0000","last_undegraded":"2024-09-21T02:38:40.331829+0000","last_fullsized":"2024-09-21T02:38:40.331829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.590210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.226878+0000","last_change":"2024-09-21T02:36:25.744046+0000","last_active":"2024-09-21T02:38:40.226878+0000","last_peered":"2024-09-21T02:38:40.226878+0000","last_clean":"2024-09-21T02:38:40.226878+0000","last_became_active":"2024-09-21T02:36:25.743695+0000","last_became_peered":"2024-09-21T02:36:25.743695+0000","last_unstale":"2024-09-21T02:38:40.226878+0000","last_undegraded":"2024-09-21T02:38:40.226878+0000","last_fullsized":"2024-09-21T02:38:40.226878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:12.092940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.231911+0000","last_change":"2024-09-21T02:36:25.749113+0000","last_active":"2024-09-21T02:38:40.231911+0000","last_peered":"2024-09-21T02:38:40.231911+0000","last_clean":"2024-09-21T02:38:40.231911+0000","last_became_active":"2024-09-21T02:36:25.748759+0000","last_became_peered":"2024-09-21T02:36:25.748759+0000","last_unstale":"2024-09-21T02:38:40.231911+0000","last_undegraded":"2024-09-21T02:38:40.231911+0000","last_fullsized":"2024-09-21T02:38:40.231911+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:22.785273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232166+0000","last_change":"2024-09-21T02:36:25.745454+0000","last_active":"2024-09-21T02:38:40.232166+0000","last_peered":"2024-09-21T02:38:40.232166+0000","last_clean":"2024-09-21T02:38:40.232166+0000","last_became_active":"2024-09-21T02:36:25.745108+0000","last_became_peered":"2024-09-21T02:36:25.745108+0000","last_unstale":"2024-09-21T02:38:40.232166+0000","last_undegraded":"2024-09-21T02:38:40.232166+0000","last_fullsized":"2024-09-21T02:38:40.232166+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:36.710456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.222535+0000","last_change":"2024-09-21T02:36:25.737213+0000","last_active":"2024-09-21T02:38:40.222535+0000","last_peered":"2024-09-21T02:38:40.222535+0000","last_clean":"2024-09-21T02:38:40.222535+0000","last_became_active":"2024-09-21T02:36:25.736641+0000","last_became_peered":"2024-09-21T02:36:25.736641+0000","last_unstale":"2024-09-21T02:38:40.222535+0000","last_undegraded":"2024-09-21T02:38:40.222535+0000","last_fullsized":"2024-09-21T02:38:40.222535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:47:50.258805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504287+0000","last_change":"2024-09-21T02:36:25.748230+0000","last_active":"2024-09-21T02:38:40.504287+0000","last_peered":"2024-09-21T02:38:40.504287+0000","last_clean":"2024-09-21T02:38:40.504287+0000","last_became_active":"2024-09-21T02:36:25.747859+0000","last_became_peered":"2024-09-21T02:36:25.747859+0000","last_unstale":"2024-09-21T02:38:40.504287+0000","last_undegraded":"2024-09-21T02:38:40.504287+0000","last_fullsized":"2024-09-21T02:38:40.504287+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:15:21.244825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503711+0000","last_change":"2024-09-21T02:36:25.744379+0000","last_active":"2024-09-21T02:38:40.503711+0000","last_peered":"2024-09-21T02:38:40.503711+0000","last_clean":"2024-09-21T02:38:40.503711+0000","last_became_active":"2024-09-21T02:36:25.744003+0000","last_became_peered":"2024-09-21T02:36:25.744003+0000","last_unstale":"2024-09-21T02:38:40.503711+0000","last_undegraded":"2024-09-21T02:38:40.503711+0000","last_fullsized":"2024-09-21T02:38:40.503711+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:59:59.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.224543+0000","last_change":"2024-09-21T02:36:25.755616+0000","last_active":"2024-09-21T02:38:40.224543+0000","last_peered":"2024-09-21T02:38:40.224543+0000","last_clean":"2024-09-21T02:38:40.224543+0000","last_became_active":"2024-09-21T02:36:25.755304+0000","last_became_peered":"2024-09-21T02:36:25.755304+0000","last_unstale":"2024-09-21T02:38:40.224543+0000","last_undegraded":"2024-09-21T02:38:40.224543+0000","last_fullsized":"2024-09-21T02:38:40.224543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:24.813780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.228147+0000","last_change":"2024-09-21T02:36:25.739747+0000","last_active":"2024-09-21T02:38:40.228147+0000","last_peered":"2024-09-21T02:38:40.228147+0000","last_clean":"2024-09-21T02:38:40.228147+0000","last_became_active":"2024-09-21T02:36:25.739404+0000","last_became_peered":"2024-09-21T02:36:25.739404+0000","last_unstale":"2024-09-21T02:38:40.228147+0000","last_undegraded":"2024-09-21T02:38:40.228147+0000","last_fullsized":"2024-09-21T02:38:40.228147+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:17.975419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.293973+0000","last_change":"2024-09-21T02:36:25.747723+0000","last_active":"2024-09-21T02:38:40.293973+0000","last_peered":"2024-09-21T02:38:40.293973+0000","last_clean":"2024-09-21T02:38:40.293973+0000","last_became_active":"2024-09-21T02:36:25.747368+0000","last_became_peered":"2024-09-21T02:36:25.747368+0000","last_unstale":"2024-09-21T02:38:40.293973+0000","last_undegraded":"2024-09-21T02:38:40.293973+0000","last_fullsized":"2024-09-21T02:38:40.293973+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.319756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503361+0000","last_change":"2024-09-21T02:36:25.743772+0000","last_active":"2024-09-21T02:38:40.503361+0000","last_peered":"2024-09-21T02:38:40.503361+0000","last_clean":"2024-09-21T02:38:40.503361+0000","last_became_active":"2024-09-21T02:36:25.743403+0000","last_became_peered":"2024-09-21T02:36:25.743403+0000","last_unstale":"2024-09-21T02:38:40.503361+0000","last_undegraded":"2024-09-21T02:38:40.503361+0000","last_fullsized":"2024-09-21T02:38:40.503361+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:11:29.930248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.223893+0000","last_change":"2024-09-21T02:36:25.746911+0000","last_active":"2024-09-21T02:38:40.223893+0000","last_peered":"2024-09-21T02:38:40.223893+0000","last_clean":"2024-09-21T02:38:40.223893+0000","last_became_active":"2024-09-21T02:36:25.746426+0000","last_became_peered":"2024-09-21T02:36:25.746426+0000","last_unstale":"2024-09-21T02:38:40.223893+0000","last_undegraded":"2024-09-21T02:38:40.223893+0000","last_fullsized":"2024-09-21T02:38:40.223893+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:12.835851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.503097+0000","last_change":"2024-09-21T02:36:25.751278+0000","last_active":"2024-09-21T02:38:40.503097+0000","last_peered":"2024-09-21T02:38:40.503097+0000","last_clean":"2024-09-21T02:38:40.503097+0000","last_became_active":"2024-09-21T02:36:25.750414+0000","last_became_peered":"2024-09-21T02:36:25.750414+0000","last_unstale":"2024-09-21T02:38:40.503097+0000","last_undegraded":"2024-09-21T02:38:40.503097+0000","last_fullsized":"2024-09-21T02:38:40.503097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:13:56.080122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.504014+0000","last_change":"2024-09-21T02:36:25.752852+0000","last_active":"2024-09-21T02:38:40.504014+0000","last_peered":"2024-09-21T02:38:40.504014+0000","last_clean":"2024-09-21T02:38:40.504014+0000","last_became_active":"2024-09-21T02:36:25.752514+0000","last_became_peered":"2024-09-21T02:36:25.752514+0000","last_unstale":"2024-09-21T02:38:40.504014+0000","last_undegraded":"2024-09-21T02:38:40.504014+0000","last_fullsized":"2024-09-21T02:38:40.504014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:55.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.231464+0000","last_change":"2024-09-21T02:36:25.750081+0000","last_active":"2024-09-21T02:38:40.231464+0000","last_peered":"2024-09-21T02:38:40.231464+0000","last_clean":"2024-09-21T02:38:40.231464+0000","last_became_active":"2024-09-21T02:36:25.749602+0000","last_became_peered":"2024-09-21T02:36:25.749602+0000","last_unstale":"2024-09-21T02:38:40.231464+0000","last_undegraded":"2024-09-21T02:38:40.231464+0000","last_fullsized":"2024-09-21T02:38:40.231464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:06.785998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.232934+0000","last_change":"2024-09-21T02:36:25.746897+0000","last_active":"2024-09-21T02:38:40.232934+0000","last_peered":"2024-09-21T02:38:40.232934+0000","last_clean":"2024-09-21T02:38:40.232934+0000","last_became_active":"2024-09-21T02:36:25.746477+0000","last_became_peered":"2024-09-21T02:36:25.746477+0000","last_unstale":"2024-09-21T02:38:40.232934+0000","last_undegraded":"2024-09-21T02:38:40.232934+0000","last_fullsized":"2024-09-21T02:38:40.232934+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:34:00.739410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-21T02:38:40.331749+0000","last_change":"2024-09-21T02:36:25.745233+0000","last_active":"2024-09-21T02:38:40.331749+0000","last_peered":"2024-09-21T02:38:40.331749+0000","last_clean":"2024-09-21T02:38:40.331749+0000","last_became_active":"2024-09-21T02:36:25.744637+0000","last_became_peered":"2024-09-21T02:36:25.744637+0000","last_unstale":"2024-09-21T02:38:40.331749+0000","last_undegraded":"2024-09-21T02:38:40.331749+0000","last_fullsized":"2024-09-21T02:38:40.331749+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:36:24.715344+0000","last_clean_scrub_stamp":"2024-09-21T02:36:24.715344+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:34:59.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78211,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1188,"num_read_kb":675,"num_write":276,"num_write_kb":168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":240917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249064,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":69,"seq":296352743469,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":950272,"data_stored":486338,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906994,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1144968,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070520,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":485131,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266750,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":485397,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462979,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":1083626,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659208,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":481128,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855438,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1078613,"data_compressed":65815,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018962,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":485202,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215190,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":480343,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":552297,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":557195,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":966656,"data_stored":483116,"data_compressed":29541,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34656,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36274,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":77282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":70591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:39:10.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:39:10.482 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:10.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:10 smithi029 ceph-mon[31006]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:10.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:10 smithi190 ceph-mon[33871]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:10.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:10 smithi066 ceph-mon[33436]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:39:11.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 -- 172.21.15.29:0/845758973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c102a80 msgr2=0x7f8a4c102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:11.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 --2- 172.21.15.29:0/845758973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c102a80 0x7f8a4c102e90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009900 tx=0x7f8a3c02a410 comp rx=0 tx=0).stop 2024-09-21T02:39:11.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 -- 172.21.15.29:0/845758973 shutdown_connections 2024-09-21T02:39:11.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 --2- 172.21.15.29:0/845758973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a4c108d10 0x7f8a4c10f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 --2- 172.21.15.29:0/845758973 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a4c1083c0 0x7f8a4c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 --2- 172.21.15.29:0/845758973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c102a80 0x7f8a4c102e90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 -- 172.21.15.29:0/845758973 >> 172.21.15.29:0/845758973 conn(0x7f8a4c079740 msgr2=0x7f8a4c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 -- 172.21.15.29:0/845758973 shutdown_connections 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 -- 172.21.15.29:0/845758973 wait complete. 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.014+0000 7f8a51d25640 1 Processor -- start 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a51d25640 1 -- start start 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a51d25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a4c102a80 0x7f8a4c199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a51d25640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c1083c0 0x7f8a4c199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a51d25640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a4c108d10 0x7f8a4c199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a51d25640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c112830 con 0x7f8a4c1083c0 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a51d25640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c1126b0 con 0x7f8a4c108d10 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a51d25640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c1129b0 con 0x7f8a4c102a80 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c1083c0 0x7f8a4c199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c1083c0 0x7f8a4c199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48046/0 (socket says 172.21.15.29:48046) 2024-09-21T02:39:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a4affd640 1 -- 172.21.15.29:0/56241403 learned_addr learned my addr 172.21.15.29:0/56241403 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a4c102a80 0x7f8a4c199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.015+0000 7f8a4bfff640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a4c108d10 0x7f8a4c199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a4affd640 1 -- 172.21.15.29:0/56241403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a4c102a80 msgr2=0x7f8a4c199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a4affd640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a4c102a80 0x7f8a4c199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a4affd640 1 -- 172.21.15.29:0/56241403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a4c108d10 msgr2=0x7f8a4c199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a4affd640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a4c108d10 0x7f8a4c199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a4affd640 1 -- 172.21.15.29:0/56241403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c1a86e0 con 0x7f8a4c1083c0 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a4affd640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c1083c0 0x7f8a4c199570 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c970 tx=0x7f8a4000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a48ff9640 1 -- 172.21.15.29:0/56241403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40007bb0 con 0x7f8a4c1083c0 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c1a89d0 con 0x7f8a4c1083c0 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a48ff9640 1 -- 172.21.15.29:0/56241403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a40007d50 con 0x7f8a4c1083c0 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a48ff9640 1 -- 172.21.15.29:0/56241403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40004db0 con 0x7f8a4c1083c0 2024-09-21T02:39:11.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.016+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c1a8f60 con 0x7f8a4c1083c0 2024-09-21T02:39:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.017+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c103d00 con 0x7f8a4c1083c0 2024-09-21T02:39:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.019+0000 7f8a48ff9640 1 -- 172.21.15.29:0/56241403 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f8a40004f70 con 0x7f8a4c1083c0 2024-09-21T02:39:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.019+0000 7f8a48ff9640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8a340b65a0 0x7f8a340b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.019+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8a340b65a0 0x7f8a340b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.019+0000 7f8a48ff9640 1 -- 172.21.15.29:0/56241403 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8a400e77f0 con 0x7f8a4c1083c0 2024-09-21T02:39:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.020+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8a340b65a0 0x7f8a340b8ab0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c079c60 tx=0x7f8a3c0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.022+0000 7f8a48ff9640 1 -- 172.21.15.29:0/56241403 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8a40014030 con 0x7f8a4c1083c0 2024-09-21T02:39:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.253+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a4c102e90 con 0x7f8a4c1083c0 2024-09-21T02:39:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.254+0000 7f8a48ff9640 1 -- 172.21.15.29:0/56241403 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11971 (secure 0 0 0) 0x7f8a4c102e90 con 0x7f8a4c1083c0 2024-09-21T02:39:11.256 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:11.256 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-09-21T02:39:04:269980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:39:04.253002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:39:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-09-21T02:39:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8a340b65a0 msgr2=0x7f8a340b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8a340b65a0 0x7f8a340b8ab0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c079c60 tx=0x7f8a3c0058d0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c1083c0 msgr2=0x7f8a4c199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c1083c0 0x7f8a4c199570 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c970 tx=0x7f8a4000ce30 comp rx=0 tx=0).stop 2024-09-21T02:39:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 shutdown_connections 2024-09-21T02:39:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8a340b65a0 0x7f8a340b8ab0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a4c108d10 0x7f8a4c199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c1083c0 0x7f8a4c199570 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 --2- 172.21.15.29:0/56241403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a4c102a80 0x7f8a4c199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 >> 172.21.15.29:0/56241403 conn(0x7f8a4c079740 msgr2=0x7f8a4c1016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 shutdown_connections 2024-09-21T02:39:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:11.257+0000 7f8a51d25640 1 -- 172.21.15.29:0/56241403 wait complete. 2024-09-21T02:39:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:11 smithi190 ceph-mon[33871]: pgmap v391: 129 pgs: 129 active+clean; 660 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-21T02:39:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:11 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/56241403' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:11.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:11 smithi029 ceph-mon[31006]: pgmap v391: 129 pgs: 129 active+clean; 660 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-21T02:39:11.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:11 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/56241403' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:11.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:11 smithi066 ceph-mon[33436]: pgmap v391: 129 pgs: 129 active+clean; 660 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-21T02:39:11.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:11 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/56241403' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:12.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:39:12.511 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 -- 172.21.15.29:0/4009124400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108103b60 msgr2=0x7fe108103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 --2- 172.21.15.29:0/4009124400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108103b60 0x7fe108103f70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009900 tx=0x7fe0f802a410 comp rx=0 tx=0).stop 2024-09-21T02:39:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 -- 172.21.15.29:0/4009124400 shutdown_connections 2024-09-21T02:39:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 --2- 172.21.15.29:0/4009124400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe108109df0 0x7fe108110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 --2- 172.21.15.29:0/4009124400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe1081094a0 0x7fe1081098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 --2- 172.21.15.29:0/4009124400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108103b60 0x7fe108103f70 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 -- 172.21.15.29:0/4009124400 >> 172.21.15.29:0/4009124400 conn(0x7fe10807b850 msgr2=0x7fe10807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 -- 172.21.15.29:0/4009124400 shutdown_connections 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.047+0000 7fe10e30c640 1 -- 172.21.15.29:0/4009124400 wait complete. 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10e30c640 1 Processor -- start 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10e30c640 1 -- start start 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10e30c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe108103b60 0x7fe10819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10e30c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1081094a0 0x7fe10819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10e30c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108109df0 0x7fe10819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10e30c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe108113960 con 0x7fe108109df0 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10e30c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe1081137e0 con 0x7fe108103b60 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10e30c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe108113ae0 con 0x7fe1081094a0 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10c8a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108109df0 0x7fe10819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10c8a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108109df0 0x7fe10819fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48062/0 (socket says 172.21.15.29:48062) 2024-09-21T02:39:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe10c8a9640 1 -- 172.21.15.29:0/1487630682 learned_addr learned my addr 172.21.15.29:0/1487630682 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe107fff640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe108103b60 0x7fe10819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.048+0000 7fe1077fe640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1081094a0 0x7fe10819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe10c8a9640 1 -- 172.21.15.29:0/1487630682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1081094a0 msgr2=0x7fe10819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe10c8a9640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1081094a0 0x7fe10819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe10c8a9640 1 -- 172.21.15.29:0/1487630682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe108103b60 msgr2=0x7fe10819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe10c8a9640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe108103b60 0x7fe10819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe10c8a9640 1 -- 172.21.15.29:0/1487630682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1081a0320 con 0x7fe108109df0 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe1077fe640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1081094a0 0x7fe10819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe10c8a9640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108109df0 0x7fe10819fbf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe10000eb10 tx=0x7fe10000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe1057fa640 1 -- 172.21.15.29:0/1487630682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100004b20 con 0x7fe108109df0 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe1057fa640 1 -- 172.21.15.29:0/1487630682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe100004cc0 con 0x7fe108109df0 2024-09-21T02:39:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1081a0610 con 0x7fe108109df0 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe1057fa640 1 -- 172.21.15.29:0/1487630682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100010690 con 0x7fe108109df0 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.049+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1081a6d70 con 0x7fe108109df0 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.050+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe108103f70 con 0x7fe108109df0 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.052+0000 7fe1057fa640 1 -- 172.21.15.29:0/1487630682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7fe100005000 con 0x7fe108109df0 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.052+0000 7fe1057fa640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0f00b65a0 0x7fe0f00b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.053+0000 7fe107fff640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0f00b65a0 0x7fe0f00b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.053+0000 7fe1057fa640 1 -- 172.21.15.29:0/1487630682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe1000e80b0 con 0x7fe108109df0 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.053+0000 7fe107fff640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0f00b65a0 0x7fe0f00b8ab0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8000c00 tx=0x7fe0f8009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:13.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.056+0000 7fe1057fa640 1 -- 172.21.15.29:0/1487630682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe100056710 con 0x7fe108109df0 2024-09-21T02:39:13.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.283+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe10819b4e0 con 0x7fe108109df0 2024-09-21T02:39:13.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.283+0000 7fe1057fa640 1 -- 172.21.15.29:0/1487630682 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11971 (secure 0 0 0) 0x7fe10819b4e0 con 0x7fe108109df0 2024-09-21T02:39:13.285 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:13.285 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-09-21T02:39:04:269980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:39:04.253002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:39:13.285 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-09-21T02:39:13.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0f00b65a0 msgr2=0x7fe0f00b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:13.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0f00b65a0 0x7fe0f00b8ab0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8000c00 tx=0x7fe0f8009680 comp rx=0 tx=0).stop 2024-09-21T02:39:13.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108109df0 msgr2=0x7fe10819fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:13.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108109df0 0x7fe10819fbf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe10000eb10 tx=0x7fe10000c2d0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 shutdown_connections 2024-09-21T02:39:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0f00b65a0 0x7fe0f00b8ab0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe108109df0 0x7fe10819fbf0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1081094a0 0x7fe10819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 --2- 172.21.15.29:0/1487630682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe108103b60 0x7fe10819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 >> 172.21.15.29:0/1487630682 conn(0x7fe10807b850 msgr2=0x7fe1081075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 shutdown_connections 2024-09-21T02:39:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:13.286+0000 7fe10e30c640 1 -- 172.21.15.29:0/1487630682 wait complete. 2024-09-21T02:39:13.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:13 smithi190 ceph-mon[33871]: pgmap v392: 129 pgs: 129 active+clean; 660 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-21T02:39:13.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:13 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1487630682' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:13.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:13 smithi029 ceph-mon[31006]: pgmap v392: 129 pgs: 129 active+clean; 660 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-21T02:39:13.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:13 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1487630682' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:13.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:13 smithi066 ceph-mon[33436]: pgmap v392: 129 pgs: 129 active+clean; 660 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-21T02:39:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:13 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1487630682' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:14.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:39:14.401 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:14.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 -- 172.21.15.29:0/3783676447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50601083a0 msgr2=0x7f50601087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:14.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3783676447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50601083a0 0x7f50601087b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5054009900 tx=0x7f505402a410 comp rx=0 tx=0).stop 2024-09-21T02:39:14.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 -- 172.21.15.29:0/3783676447 shutdown_connections 2024-09-21T02:39:14.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3783676447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5060108cf0 0x7f506010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:14.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3783676447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50601083a0 0x7f50601087b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:14.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3783676447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5060102a60 0x7f5060102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 -- 172.21.15.29:0/3783676447 >> 172.21.15.29:0/3783676447 conn(0x7f5060079730 msgr2=0x7f5060079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 -- 172.21.15.29:0/3783676447 shutdown_connections 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.928+0000 7f5065ef0640 1 -- 172.21.15.29:0/3783676447 wait complete. 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.929+0000 7f5065ef0640 1 Processor -- start 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.929+0000 7f5065ef0640 1 -- start start 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.929+0000 7f5065ef0640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5060102a60 0x7f506019b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.929+0000 7f5065ef0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50601083a0 0x7f506019b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.929+0000 7f5065ef0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5060108cf0 0x7f506019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.929+0000 7f5065ef0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5060112810 con 0x7f5060108cf0 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.929+0000 7f5065ef0640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5060112690 con 0x7f5060102a60 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.929+0000 7f5065ef0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5060112990 con 0x7f50601083a0 2024-09-21T02:39:14.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5060108cf0 0x7f506019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5060108cf0 0x7f506019bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52868/0 (socket says 172.21.15.29:52868) 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505f7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5060102a60 0x7f506019b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 -- 172.21.15.29:0/3709796069 learned_addr learned my addr 172.21.15.29:0/3709796069 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505effd640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50601083a0 0x7f506019b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 -- 172.21.15.29:0/3709796069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50601083a0 msgr2=0x7f506019b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50601083a0 0x7f506019b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 -- 172.21.15.29:0/3709796069 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5060102a60 msgr2=0x7f506019b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5060102a60 0x7f506019b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 -- 172.21.15.29:0/3709796069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50601a6430 con 0x7f5060108cf0 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505effd640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50601083a0 0x7f506019b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505ffff640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5060108cf0 0x7f506019bc20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f505800cad0 tx=0x7f505800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505cff9640 1 -- 172.21.15.29:0/3709796069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505800ae20 con 0x7f5060108cf0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50601a6720 con 0x7f5060108cf0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505cff9640 1 -- 172.21.15.29:0/3709796069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5058005020 con 0x7f5060108cf0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f505cff9640 1 -- 172.21.15.29:0/3709796069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505800f6b0 con 0x7f5060108cf0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.930+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50601a6c30 con 0x7f5060108cf0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.932+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5060102e70 con 0x7f5060108cf0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.933+0000 7f505cff9640 1 -- 172.21.15.29:0/3709796069 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f50580051d0 con 0x7f5060108cf0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.934+0000 7f505cff9640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f50480b65d0 0x7f50480b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.934+0000 7f505f7fe640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f50480b65d0 0x7f50480b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.934+0000 7f505cff9640 1 -- 172.21.15.29:0/3709796069 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f50580e7bb0 con 0x7f5060108cf0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.934+0000 7f505f7fe640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f50480b65d0 0x7f50480b8ae0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f506006b9c0 tx=0x7f505000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:14.938+0000 7f505cff9640 1 -- 172.21.15.29:0/3709796069 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5058014030 con 0x7f5060108cf0 2024-09-21T02:39:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.129+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f506010a550 con 0x7f5060108cf0 2024-09-21T02:39:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.131+0000 7f505cff9640 1 -- 172.21.15.29:0/3709796069 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24773 (secure 0 0 0) 0x7f50580adf60 con 0x7f5060108cf0 2024-09-21T02:39:15.132 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:15.133 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T02:38:40.213155+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:39:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.133+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f50480b65d0 msgr2=0x7f50480b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.133+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f50480b65d0 0x7f50480b8ae0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f506006b9c0 tx=0x7f505000b390 comp rx=0 tx=0).stop 2024-09-21T02:39:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.133+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5060108cf0 msgr2=0x7f506019bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.133+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5060108cf0 0x7f506019bc20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f505800cad0 tx=0x7f505800cf90 comp rx=0 tx=0).stop 2024-09-21T02:39:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.134+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 shutdown_connections 2024-09-21T02:39:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.134+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f50480b65d0 0x7f50480b8ae0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.134+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5060108cf0 0x7f506019bc20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.134+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50601083a0 0x7f506019b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.134+0000 7f5065ef0640 1 --2- 172.21.15.29:0/3709796069 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5060102a60 0x7f506019b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.134+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 >> 172.21.15.29:0/3709796069 conn(0x7f5060079730 msgr2=0x7f50601044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.134+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 shutdown_connections 2024-09-21T02:39:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:15.134+0000 7f5065ef0640 1 -- 172.21.15.29:0/3709796069 wait complete. 2024-09-21T02:39:15.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:15 smithi029 ceph-mon[31006]: pgmap v393: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-21T02:39:15.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:15 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3709796069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:39:15.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:15 smithi190 ceph-mon[33871]: pgmap v393: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-21T02:39:15.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:15 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3709796069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:39:15.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:15 smithi066 ceph-mon[33436]: pgmap v393: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-21T02:39:15.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:15 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3709796069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:39:16.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mon.a --format=json config get run_dir 2024-09-21T02:39:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:17.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:17 smithi190 ceph-mon[33871]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:17.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:17 smithi029 ceph-mon[31006]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:17.694 INFO:teuthology.orchestra.run.smithi029.stdout:{"run_dir":"/var/run/ceph"} 2024-09-21T02:39:17.695 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-21T02:39:17.695 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 7864 2024-09-21T02:39:17.695 DEBUG:tasks.quiescer:Yielding 2024-09-21T02:39:17.695 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T02:39:17.709 INFO:tasks.workunit:Pulling workunits from ref 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:39:17.710 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T02:39:17.710 INFO:tasks.workunit:timeout=3h 2024-09-21T02:39:17.710 INFO:tasks.workunit:cleanup=True 2024-09-21T02:39:17.710 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:17.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:39:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-21T02:39:17.731 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 78h/120d Inode: 1 Links: 2 2024-09-21T02:39:17.731 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:39:17.731 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:fusefs_t:s0 2024-09-21T02:39:17.731 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 02:37:13.744444346 +0000 2024-09-21T02:39:17.731 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 02:37:13.744444346 +0000 2024-09-21T02:39:17.731 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 02:39:03.850492023 +0000 2024-09-21T02:39:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-09-21T02:39:17.731 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-21T02:39:17.731 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-21T02:39:17.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:17 smithi066 ceph-mon[33436]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:17.805 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:39:17.863 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.890+0000 7f3f52932640 1 -- 172.21.15.29:0/1442606449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1036d0 msgr2=0x7f3f4c103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.890+0000 7f3f52932640 1 --2- 172.21.15.29:0/1442606449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1036d0 0x7f3f4c103b40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009900 tx=0x7f3f3c02a410 comp rx=0 tx=0).stop 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.890+0000 7f3f52932640 1 -- 172.21.15.29:0/1442606449 shutdown_connections 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.890+0000 7f3f52932640 1 --2- 172.21.15.29:0/1442606449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4c1050f0 0x7f3f4c10ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.890+0000 7f3f52932640 1 --2- 172.21.15.29:0/1442606449 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c104750 0x7f3f4c104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.890+0000 7f3f52932640 1 --2- 172.21.15.29:0/1442606449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1036d0 0x7f3f4c103b40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.890+0000 7f3f52932640 1 -- 172.21.15.29:0/1442606449 >> 172.21.15.29:0/1442606449 conn(0x7f3f4c0ff030 msgr2=0x7f3f4c1014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.890+0000 7f3f52932640 1 -- 172.21.15.29:0/1442606449 shutdown_connections 2024-09-21T02:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.891+0000 7f3f52932640 1 -- 172.21.15.29:0/1442606449 wait complete. 2024-09-21T02:39:17.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.891+0000 7f3f52932640 1 Processor -- start 2024-09-21T02:39:17.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.891+0000 7f3f52932640 1 -- start start 2024-09-21T02:39:17.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.891+0000 7f3f52932640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c1036d0 0x7f3f4c199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f52932640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4c104750 0x7f3f4c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f52932640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1050f0 0x7f3f4c19b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f52932640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c10f1a0 con 0x7f3f4c1050f0 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f52932640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c10f020 con 0x7f3f4c1036d0 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f52932640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c10f320 con 0x7f3f4c104750 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1050f0 0x7f3f4c19b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1050f0 0x7f3f4c19b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52890/0 (socket says 172.21.15.29:52890) 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 -- 172.21.15.29:0/779961639 learned_addr learned my addr 172.21.15.29:0/779961639 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c1036d0 0x7f3f4c199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4c104750 0x7f3f4c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 -- 172.21.15.29:0/779961639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4c104750 msgr2=0x7f3f4c19a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4c104750 0x7f3f4c19a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 -- 172.21.15.29:0/779961639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c1036d0 msgr2=0x7f3f4c199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c1036d0 0x7f3f4c199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 -- 172.21.15.29:0/779961639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4c19b890 con 0x7f3f4c1050f0 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4c104750 0x7f3f4c19a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f50ecf640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1050f0 0x7f3f4c19b070 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400eaa0 tx=0x7f3f4400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f497fa640 1 -- 172.21.15.29:0/779961639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4400cdb0 con 0x7f3f4c1050f0 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f52932640 1 -- 172.21.15.29:0/779961639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4c19bb80 con 0x7f3f4c1050f0 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f497fa640 1 -- 172.21.15.29:0/779961639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f44005020 con 0x7f3f4c1050f0 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.892+0000 7f3f497fa640 1 -- 172.21.15.29:0/779961639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f440116d0 con 0x7f3f4c1050f0 2024-09-21T02:39:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.893+0000 7f3f52932640 1 -- 172.21.15.29:0/779961639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4c1a2270 con 0x7f3f4c1050f0 2024-09-21T02:39:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.895+0000 7f3f598a0740 1 Processor -- start 2024-09-21T02:39:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.895+0000 7f3f497fa640 1 -- 172.21.15.29:0/779961639 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f3f440051d0 con 0x7f3f4c1050f0 2024-09-21T02:39:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.895+0000 7f3f598a0740 1 -- start start 2024-09-21T02:39:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 0x5645f91648d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d974d0 0x5645f9165420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f598a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5645f90d26f0 con 0x5645f8db81b0 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f598a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5645f90c1b20 con 0x5645f8d974d0 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f598a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5645f90c5930 con 0x5645f8d8f2f0 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9164260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52906/0 (socket says 172.21.15.29:52906) 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/761190257 learned_addr learned my addr 172.21.15.29:0/761190257 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 0x5645f91648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d974d0 0x5645f9165420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f497fa640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3f340b65a0 0x7f3f340b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/761190257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 msgr2=0x5645f91648d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 0x5645f91648d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/761190257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d974d0 msgr2=0x5645f9165420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d974d0 0x5645f9165420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/761190257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645f9087530 con 0x5645f8db81b0 2024-09-21T02:39:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.896+0000 7f3f50ecf640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3f340b65a0 0x7f3f340b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.897+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9164260 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000beb0 tx=0x7f3f4000b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd69c5cbe0ce670b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.897+0000 7f3f497fa640 1 -- 172.21.15.29:0/779961639 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3f440e7d80 con 0x7f3f4c1050f0 2024-09-21T02:39:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.897+0000 7f3f327fc640 1 -- 172.21.15.29:0/761190257 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40013d10 con 0x5645f8db81b0 2024-09-21T02:39:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.897+0000 7f3f327fc640 1 -- 172.21.15.29:0/761190257 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f40011880 con 0x5645f8db81b0 2024-09-21T02:39:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.897+0000 7f3f327fc640 1 -- 172.21.15.29:0/761190257 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f400044c0 con 0x5645f8db81b0 2024-09-21T02:39:17.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f50ecf640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3f340b65a0 0x7f3f340b8ab0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f44021000 tx=0x7f3f44079040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:17.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 -- 172.21.15.29:0/761190257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 msgr2=0x5645f9164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9164260 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000beb0 tx=0x7f3f4000b750 comp rx=0 tx=0).stop 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 -- 172.21.15.29:0/761190257 shutdown_connections 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d974d0 0x5645f9165420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 0x5645f91648d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 --2- 172.21.15.29:0/761190257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9164260 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 -- 172.21.15.29:0/761190257 >> 172.21.15.29:0/761190257 conn(0x5645f8dac300 msgr2=0x5645f915b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 -- 172.21.15.29:0/761190257 shutdown_connections 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.898+0000 7f3f598a0740 1 -- 172.21.15.29:0/761190257 wait complete. 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 1 -- start start 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 1 Processor -- start 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 0x5645f916d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d974d0 0x5645f916e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 0x5645f916ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5645f8ff9ab0 con 0x5645f8d974d0 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5645f9067730 con 0x5645f8db81b0 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5645f9165f40 con 0x5645f8d8f2f0 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f598a0740 10 client.0 _ll_register_callbacks cb 0x5645f90bb6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d974d0 0x5645f916e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d974d0 0x5645f916e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52916/0 (socket says 172.21.15.29:52916) 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f4bfff640 1 -- 172.21.15.29:0/3894956888 learned_addr learned my addr 172.21.15.29:0/3894956888 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.899+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 0x5645f916d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 0x5645f916ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f4bfff640 1 -- 172.21.15.29:0/3894956888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 msgr2=0x5645f916d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 0x5645f916d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f4bfff640 1 -- 172.21.15.29:0/3894956888 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 msgr2=0x5645f916ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 0x5645f916ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f4bfff640 1 -- 172.21.15.29:0/3894956888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645f8e23790 con 0x5645f8d974d0 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 0x5645f916ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d974d0 0x5645f916e0a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009900 tx=0x7f3f3c031890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4fd73514376558 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/3894956888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c035040 con 0x5645f8d974d0 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/3894956888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3f3c004240 con 0x5645f8d974d0 2024-09-21T02:39:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f598a0740 1 -- 172.21.15.29:0/3894956888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f40011880 con 0x5645f8d974d0 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/3894956888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f3c04e050 con 0x5645f8d974d0 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f598a0740 10 client.15552 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.900+0000 7f3f598a0740 1 -- 172.21.15.29:0/3894956888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5645f8e9c620 con 0x5645f8d974d0 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/3894956888 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f3c004440 con 0x5645f8d974d0 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/3894956888 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3f3c03c770 con 0x5645f8d974d0 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f598a0740 10 client.15552 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f598a0740 10 client.15552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f598a0740 20 client.15552 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f598a0740 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5645f8d76d80 0x5645f917b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f598a0740 4 client.15552 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f598a0740 1 -- 172.21.15.29:0/3894956888 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3f40011880 con 0x5645f8d76d80 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.901+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5645f8d76d80 0x5645f917b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.902+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5645f8d76d80 0x5645f917b150 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=517ca808d80687c9 server_cookie=94c6e82dca510b73 in_seq=0 out_seq=0 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.902+0000 7f3f2b7fe640 10 client.15552 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/3894956888 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f3f40011880 con 0x5645f8d76d80 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f2b7fe640 10 client.15552 handle_command_reply: tid=0 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 client.15552 shutdown 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 -- 172.21.15.29:0/3894956888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d974d0 msgr2=0x5645f916e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d974d0 0x5645f916e0a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009900 tx=0x7f3f3c031890 comp rx=0 tx=0).stop 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 -- 172.21.15.29:0/3894956888 shutdown_connections 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 0x5645f916ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5645f8d76d80 0x5645f917b150 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d974d0 0x5645f916e0a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 --2- 172.21.15.29:0/3894956888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d8f2f0 0x5645f916d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f2b7fe640 0 client.15552 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 -- 172.21.15.29:0/3894956888 >> 172.21.15.29:0/3894956888 conn(0x5645f8dac300 msgr2=0x5645f9159ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 -- 172.21.15.29:0/3894956888 shutdown_connections 2024-09-21T02:39:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 1 -- 172.21.15.29:0/3894956888 wait complete. 2024-09-21T02:39:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.903+0000 7f3f598a0740 20 client.15552 trim_cache size 0 max 16384 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f598a0740 1 Processor -- start 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f598a0740 1 -- start start 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d8f2f0 0x5645f915b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 0x5645f916d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 0x5645f916df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f598a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1aa380 con 0x5645f8d8f2f0 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f598a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1abec0 con 0x5645f8db81b0 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f598a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1aaa50 con 0x5645f8d974d0 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d8f2f0 0x5645f915b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d8f2f0 0x5645f915b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52924/0 (socket says 172.21.15.29:52924) 2024-09-21T02:39:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.919+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/2680406138 learned_addr learned my addr 172.21.15.29:0/2680406138 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:17.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.920+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/2680406138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 0x5645f916d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.920+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/2680406138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 msgr2=0x5645f916d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:17.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.920+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/2680406138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 0x5645f916d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.920+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/2680406138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 msgr2=0x5645f916df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:17.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.920+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/2680406138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 0x5645f916df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.920+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/2680406138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645f91570a0 con 0x5645f8d8f2f0 2024-09-21T02:39:17.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.920+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/2680406138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d8f2f0 0x5645f915b290 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3f40005fd0 tx=0x7f3f4000b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855505f98ca4baee server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:17.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.921+0000 7f3f31ffb640 1 -- 172.21.15.29:0/2680406138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40013d10 con 0x5645f8d8f2f0 2024-09-21T02:39:17.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.921+0000 7f3f31ffb640 1 -- 172.21.15.29:0/2680406138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f40011800 con 0x5645f8d8f2f0 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.921+0000 7f3f31ffb640 1 -- 172.21.15.29:0/2680406138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40005020 con 0x5645f8d8f2f0 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 -- 172.21.15.29:0/2680406138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d8f2f0 msgr2=0x5645f915b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2680406138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d8f2f0 0x5645f915b290 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3f40005fd0 tx=0x7f3f4000b750 comp rx=0 tx=0).stop 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 -- 172.21.15.29:0/2680406138 shutdown_connections 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2680406138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8db81b0 0x5645f916df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2680406138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 0x5645f916d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2680406138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8d8f2f0 0x5645f915b290 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 -- 172.21.15.29:0/2680406138 >> 172.21.15.29:0/2680406138 conn(0x5645f8dac300 msgr2=0x5645f91650f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 -- 172.21.15.29:0/2680406138 shutdown_connections 2024-09-21T02:39:17.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 -- 172.21.15.29:0/2680406138 wait complete. 2024-09-21T02:39:17.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 -- start start 2024-09-21T02:39:17.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.922+0000 7f3f598a0740 1 Processor -- start 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d8f2f0 0x5645f91754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 0x5645f9174310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f598a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9174a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f598a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1ab7f0 con 0x5645f8db81b0 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f598a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1ab120 con 0x5645f8d8f2f0 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f598a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1a8360 con 0x5645f8d974d0 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f598a0740 10 client.0 _ll_register_callbacks cb 0x7f3f3c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9174a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9174a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52928/0 (socket says 172.21.15.29:52928) 2024-09-21T02:39:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/2636674928 learned_addr learned my addr 172.21.15.29:0/2636674928 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 0x5645f9174310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.923+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d8f2f0 0x5645f91754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/2636674928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 msgr2=0x5645f9174310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 0x5645f9174310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/2636674928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 msgr2=0x5645f9174a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9174a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/2636674928 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645f9172e50 con 0x5645f8d8f2f0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d8f2f0 0x5645f91754f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f40013ed0 tx=0x7f3f4000bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89e9dc0f0937a258 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/2636674928 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40004020 con 0x5645f8d8f2f0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f598a0740 1 -- 172.21.15.29:0/2636674928 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f40011800 con 0x5645f8d8f2f0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/2636674928 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3f4001a040 con 0x5645f8d8f2f0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/2636674928 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f400041c0 con 0x5645f8d8f2f0 2024-09-21T02:39:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f598a0740 10 client.25178 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.924+0000 7f3f598a0740 1 -- 172.21.15.29:0/2636674928 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5645f91e5dd0 con 0x5645f8d8f2f0 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/2636674928 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f40011800 con 0x5645f8d8f2f0 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/2636674928 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3f400035f0 con 0x5645f8d8f2f0 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f598a0740 10 client.25178 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f598a0740 10 client.25178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f598a0740 20 client.25178 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5645f8d4c1c0 0x5645f91e8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f598a0740 4 client.25178 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f598a0740 1 -- 172.21.15.29:0/2636674928 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5645f90d4640 con 0x5645f8d4c1c0 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.925+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5645f8d4c1c0 0x5645f91e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.926+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5645f8d4c1c0 0x5645f91e8610 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff041f48d350a2 server_cookie=bc693f5d51dbf7fb in_seq=0 out_seq=0 2024-09-21T02:39:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.926+0000 7f3f2b7fe640 10 client.25178 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.938+0000 7f3f2b7fe640 1 -- 172.21.15.29:0/2636674928 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5645f90d4640 con 0x5645f8d4c1c0 2024-09-21T02:39:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f2b7fe640 10 client.25178 handle_command_reply: tid=0 2024-09-21T02:39:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 client.25178 shutdown 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 -- 172.21.15.29:0/2636674928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d8f2f0 msgr2=0x5645f91754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d8f2f0 0x5645f91754f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f40013ed0 tx=0x7f3f4000bfa0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 -- 172.21.15.29:0/2636674928 shutdown_connections 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5645f8d4c1c0 0x5645f91e8610 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645f8db81b0 0x5645f9174a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5645f8d974d0 0x5645f9174310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f2b7fe640 0 client.25178 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 --2- 172.21.15.29:0/2636674928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5645f8d8f2f0 0x5645f91754f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 -- 172.21.15.29:0/2636674928 >> 172.21.15.29:0/2636674928 conn(0x5645f8dac300 msgr2=0x5645f915b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 -- 172.21.15.29:0/2636674928 shutdown_connections 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 1 -- 172.21.15.29:0/2636674928 wait complete. 2024-09-21T02:39:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.939+0000 7f3f598a0740 20 client.25178 trim_cache size 0 max 16384 2024-09-21T02:39:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:39:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:39:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:39:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 -- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3f340b65a0 msgr2=0x7f3f340b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3f340b65a0 0x7f3f340b8ab0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f44021000 tx=0x7f3f44079040 comp rx=0 tx=0).stop 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 -- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1050f0 msgr2=0x7f3f4c19b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1050f0 0x7f3f4c19b070 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400eaa0 tx=0x7f3f4400ef60 comp rx=0 tx=0).stop 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 -- 172.21.15.29:0/779961639 shutdown_connections 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3f340b65a0 0x7f3f340b8ab0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c1050f0 0x7f3f4c19b070 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4c104750 0x7f3f4c19a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 --2- 172.21.15.29:0/779961639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c1036d0 0x7f3f4c199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 -- 172.21.15.29:0/779961639 >> 172.21.15.29:0/779961639 conn(0x7f3f4c0ff030 msgr2=0x7f3f4c107a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 -- 172.21.15.29:0/779961639 shutdown_connections 2024-09-21T02:39:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:17.940+0000 7f3f52932640 1 -- 172.21.15.29:0/779961639 wait complete. 2024-09-21T02:39:18.000 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:18.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 -- 172.21.15.29:0/1302613176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38102a80 msgr2=0x7f9a38102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:18.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 --2- 172.21.15.29:0/1302613176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38102a80 0x7f9a38102e90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009900 tx=0x7f9a3402a410 comp rx=0 tx=0).stop 2024-09-21T02:39:18.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 -- 172.21.15.29:0/1302613176 shutdown_connections 2024-09-21T02:39:18.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 --2- 172.21.15.29:0/1302613176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a38108d10 0x7f9a3810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 --2- 172.21.15.29:0/1302613176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a381083c0 0x7f9a381087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 --2- 172.21.15.29:0/1302613176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38102a80 0x7f9a38102e90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 -- 172.21.15.29:0/1302613176 >> 172.21.15.29:0/1302613176 conn(0x7f9a38079740 msgr2=0x7f9a38079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 -- 172.21.15.29:0/1302613176 shutdown_connections 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.543+0000 7f9a401f1640 1 -- 172.21.15.29:0/1302613176 wait complete. 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a401f1640 1 Processor -- start 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a401f1640 1 -- start start 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a401f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a38102a80 0x7f9a3819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a401f1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a381083c0 0x7f9a3819b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a401f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38108d10 0x7f9a3819bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a401f1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9a38112830 con 0x7f9a38108d10 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a401f1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381126b0 con 0x7f9a381083c0 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a401f1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381129b0 con 0x7f9a38102a80 2024-09-21T02:39:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3e78e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38108d10 0x7f9a3819bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3e78e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38108d10 0x7f9a3819bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52946/0 (socket says 172.21.15.29:52946) 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3e78e640 1 -- 172.21.15.29:0/2652823028 learned_addr learned my addr 172.21.15.29:0/2652823028 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3d78c640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a381083c0 0x7f9a3819b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3df8d640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a38102a80 0x7f9a3819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3e78e640 1 -- 172.21.15.29:0/2652823028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a38102a80 msgr2=0x7f9a3819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3e78e640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a38102a80 0x7f9a3819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3e78e640 1 -- 172.21.15.29:0/2652823028 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a381083c0 msgr2=0x7f9a3819b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.544+0000 7f9a3e78e640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a381083c0 0x7f9a3819b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.545+0000 7f9a3e78e640 1 -- 172.21.15.29:0/2652823028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a381a6410 con 0x7f9a38108d10 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.545+0000 7f9a3df8d640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a38102a80 0x7f9a3819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.545+0000 7f9a3e78e640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38108d10 0x7f9a3819bb80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000eb10 tx=0x7f9a3000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.545+0000 7f9a2effd640 1 -- 172.21.15.29:0/2652823028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a30004b20 con 0x7f9a38108d10 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.545+0000 7f9a2effd640 1 -- 172.21.15.29:0/2652823028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a30004cc0 con 0x7f9a38108d10 2024-09-21T02:39:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.545+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a381a6700 con 0x7f9a38108d10 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.545+0000 7f9a2effd640 1 -- 172.21.15.29:0/2652823028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a30010690 con 0x7f9a38108d10 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.545+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a381a6c40 con 0x7f9a38108d10 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.546+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a38103d00 con 0x7f9a38108d10 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.548+0000 7f9a2effd640 1 -- 172.21.15.29:0/2652823028 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f9a30005020 con 0x7f9a38108d10 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.548+0000 7f9a2effd640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9a240b65a0 0x7f9a240b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.548+0000 7f9a3df8d640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9a240b65a0 0x7f9a240b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.549+0000 7f9a2effd640 1 -- 172.21.15.29:0/2652823028 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9a300e7e00 con 0x7f9a38108d10 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.549+0000 7f9a3df8d640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9a240b65a0 0x7f9a240b8ab0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9a34000c00 tx=0x7f9a34032c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.553+0000 7f9a2effd640 1 -- 172.21.15.29:0/2652823028 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9a300ae1b0 con 0x7f9a38108d10 2024-09-21T02:39:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.777+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a3806bfb0 con 0x7f9a38108d10 2024-09-21T02:39:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.778+0000 7f9a2effd640 1 -- 172.21.15.29:0/2652823028 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11971 (secure 0 0 0) 0x7f9a300a9030 con 0x7f9a38108d10 2024-09-21T02:39:18.779 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:18.779 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-09-21T02:39:04:269980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:39:04.253002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:39:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-09-21T02:39:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9a240b65a0 msgr2=0x7f9a240b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9a240b65a0 0x7f9a240b8ab0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9a34000c00 tx=0x7f9a34032c80 comp rx=0 tx=0).stop 2024-09-21T02:39:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38108d10 msgr2=0x7f9a3819bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38108d10 0x7f9a3819bb80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000eb10 tx=0x7f9a3000c2d0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 shutdown_connections 2024-09-21T02:39:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9a240b65a0 0x7f9a240b8ab0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a38108d10 0x7f9a3819bb80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a381083c0 0x7f9a3819b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 --2- 172.21.15.29:0/2652823028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a38102a80 0x7f9a3819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 >> 172.21.15.29:0/2652823028 conn(0x7f9a38079740 msgr2=0x7f9a381014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 shutdown_connections 2024-09-21T02:39:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:18.781+0000 7f9a401f1640 1 -- 172.21.15.29:0/2652823028 wait complete. 2024-09-21T02:39:19.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:19 smithi029 ceph-mon[31006]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-21T02:39:19.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:19 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2652823028' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:19.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T02:36:26.562695+0000', 'modified': '2024-09-21T02:39:04.253002+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24637, 'mds_1': 24632, 'mds_2': 24625, 'mds_3': 24613, 'mds_4': 24599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14781': {'gid': 14781, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6839/886485890', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 886485890}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 886485890}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14787': {'gid': 14787, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6841/1488187270', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6840', 'nonce': 1488187270}, {'type': 'v1', 'addr': '172.21.15.29:6841', 'nonce': 1488187270}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.66:6833/2291646688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6832', 'nonce': 2291646688}, {'type': 'v1', 'addr': '172.21.15.66:6833', 'nonce': 2291646688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24574': {'gid': 24574, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6833/3478087331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6832', 'nonce': 3478087331}, {'type': 'v1', 'addr': '172.21.15.190:6833', 'nonce': 3478087331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.66:6835/4067532455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6834', 'nonce': 4067532455}, {'type': 'v1', 'addr': '172.21.15.66:6835', 'nonce': 4067532455}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24599': {'gid': 24599, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.66:6837/3184195773', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6836', 'nonce': 3184195773}, {'type': 'v1', 'addr': '172.21.15.66:6837', 'nonce': 3184195773}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24613': {'gid': 24613, 'name': 'f', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.190:6835/131555194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 131555194}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 131555194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24625': {'gid': 24625, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.190:6837/3758896499', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 3758896499}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 3758896499}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24632': {'gid': 24632, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.66:6839/3618856137', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.66:6838', 'nonce': 3618856137}, {'type': 'v1', 'addr': '172.21.15.66:6839', 'nonce': 3618856137}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24637': {'gid': 24637, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.190:6839/3083654868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 3083654868}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 3083654868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24637, 'qdb_cluster': [24599, 24625, 24613, 24632, 24637]} 2024-09-21T02:39:19.547 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-21T02:39:19.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.h status 2024-09-21T02:39:19.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:19 smithi190 ceph-mon[33871]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-21T02:39:19.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:19 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2652823028' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:19.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:19 smithi066 ceph-mon[33436]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-21T02:39:19.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:19 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2652823028' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.349+0000 7faf4d4d2640 1 -- 172.21.15.29:0/1961737870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 msgr2=0x7faf48109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.349+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/1961737870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 0x7faf48109900 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7faf30009900 tx=0x7faf3002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.349+0000 7faf4d4d2640 1 -- 172.21.15.29:0/1961737870 shutdown_connections 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.349+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/1961737870 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf48109e40 0x7faf48110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.349+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/1961737870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 0x7faf48109900 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.349+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/1961737870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf48103bb0 0x7faf48103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.349+0000 7faf4d4d2640 1 -- 172.21.15.29:0/1961737870 >> 172.21.15.29:0/1961737870 conn(0x7faf480ff600 msgr2=0x7faf48101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.349+0000 7faf4d4d2640 1 -- 172.21.15.29:0/1961737870 shutdown_connections 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.350+0000 7faf4d4d2640 1 -- 172.21.15.29:0/1961737870 wait complete. 2024-09-21T02:39:20.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.350+0000 7faf4d4d2640 1 Processor -- start 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.350+0000 7faf4d4d2640 1 -- start start 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf4d4d2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf48103bb0 0x7faf4819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf4d4d2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 0x7faf4819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf4d4d2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf48109e40 0x7faf4819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf4d4d2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faf481139b0 con 0x7faf481094f0 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf4d4d2640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faf48113830 con 0x7faf48109e40 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf4d4d2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faf48113b30 con 0x7faf48103bb0 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf3e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 0x7faf4819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf3e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 0x7faf4819aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52962/0 (socket says 172.21.15.29:52962) 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf3e7fc640 1 -- 172.21.15.29:0/3747517234 learned_addr learned my addr 172.21.15.29:0/3747517234 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf46ffd640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf48103bb0 0x7faf4819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf477fe640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf48109e40 0x7faf4819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf3e7fc640 1 -- 172.21.15.29:0/3747517234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf48103bb0 msgr2=0x7faf4819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf48103bb0 0x7faf4819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf3e7fc640 1 -- 172.21.15.29:0/3747517234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf48109e40 msgr2=0x7faf4819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf48109e40 0x7faf4819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf3e7fc640 1 -- 172.21.15.29:0/3747517234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf481a0310 con 0x7faf481094f0 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.351+0000 7faf477fe640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf48109e40 0x7faf4819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.352+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 0x7faf4819aa00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7faf30004770 tx=0x7faf30004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.352+0000 7faf44ff9640 1 -- 172.21.15.29:0/3747517234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3003f030 con 0x7faf481094f0 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.352+0000 7faf4d4d2640 1 -- 172.21.15.29:0/3747517234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf481a05a0 con 0x7faf481094f0 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.352+0000 7faf44ff9640 1 -- 172.21.15.29:0/3747517234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf30004440 con 0x7faf481094f0 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.352+0000 7faf44ff9640 1 -- 172.21.15.29:0/3747517234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf30035070 con 0x7faf481094f0 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.352+0000 7faf4d4d2640 1 -- 172.21.15.29:0/3747517234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf481a0ad0 con 0x7faf481094f0 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.354+0000 7faf53a46740 1 Processor -- start 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf53a46740 1 -- start start 2024-09-21T02:39:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf44ff9640 1 -- 172.21.15.29:0/3747517234 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7faf30042030 con 0x7faf481094f0 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf53a46740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 0x561390fcff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf53a46740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fd0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf53a46740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf53a46740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561390f47c10 con 0x561390c09280 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf53a46740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561390f37040 con 0x561390c010a0 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf53a46740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561390f3ae50 con 0x561390c29f60 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf3e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf477fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 0x561390fcff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf3e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52968/0 (socket says 172.21.15.29:52968) 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf3e7fc640 1 -- 172.21.15.29:0/2077659832 learned_addr learned my addr 172.21.15.29:0/2077659832 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.355+0000 7faf46ffd640 1 --2- 172.21.15.29:0/2077659832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fd0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3e7fc640 1 -- 172.21.15.29:0/2077659832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 msgr2=0x561390fcff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/2077659832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 0x561390fcff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3e7fc640 1 -- 172.21.15.29:0/2077659832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 msgr2=0x561390fd0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/2077659832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fd0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3e7fc640 1 -- 172.21.15.29:0/2077659832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561390c95540 con 0x561390c09280 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/2077659832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd0980 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7faf30105120 tx=0x7faf3010be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20adda5fe900a2ec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf44ff9640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf340b65d0 0x7faf340b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3d7fa640 1 -- 172.21.15.29:0/2077659832 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf300c9eb0 con 0x561390c09280 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3d7fa640 1 -- 172.21.15.29:0/2077659832 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf300d90f0 con 0x561390c09280 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf477fe640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf340b65d0 0x7faf340b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf3d7fa640 1 -- 172.21.15.29:0/2077659832 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf300cd1b0 con 0x561390c09280 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.356+0000 7faf44ff9640 1 -- 172.21.15.29:0/3747517234 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faf30107bb0 con 0x7faf481094f0 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 -- 172.21.15.29:0/2077659832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 msgr2=0x561390fd0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 --2- 172.21.15.29:0/2077659832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd0980 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7faf30105120 tx=0x7faf3010be80 comp rx=0 tx=0).stop 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf477fe640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf340b65d0 0x7faf340b8ae0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faf40005fd0 tx=0x7faf400086a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 -- 172.21.15.29:0/2077659832 shutdown_connections 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 --2- 172.21.15.29:0/2077659832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd0980 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 --2- 172.21.15.29:0/2077659832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fd0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 --2- 172.21.15.29:0/2077659832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 0x561390fcff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 -- 172.21.15.29:0/2077659832 >> 172.21.15.29:0/2077659832 conn(0x561390c1e350 msgr2=0x561390fcddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 -- 172.21.15.29:0/2077659832 shutdown_connections 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.357+0000 7faf53a46740 1 -- 172.21.15.29:0/2077659832 wait complete. 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 1 -- start start 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 1 Processor -- start 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c010a0 0x561390fdba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fdbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c29f60 0x561390fe7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561390d985f0 con 0x561390c09280 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561390ed2010 con 0x561390c29f60 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561390fde4a0 con 0x561390c010a0 2024-09-21T02:39:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf53a46740 10 client.0 _ll_register_callbacks cb 0x561390f31340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf3e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fdbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf3e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fdbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52974/0 (socket says 172.21.15.29:52974) 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf3e7fc640 1 -- 172.21.15.29:0/555828817 learned_addr learned my addr 172.21.15.29:0/555828817 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf477fe640 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c29f60 0x561390fe7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.358+0000 7faf46ffd640 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c010a0 0x561390fdba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf3e7fc640 1 -- 172.21.15.29:0/555828817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c010a0 msgr2=0x561390fdba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c010a0 0x561390fdba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf3e7fc640 1 -- 172.21.15.29:0/555828817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c29f60 msgr2=0x561390fe7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c29f60 0x561390fe7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf3e7fc640 1 -- 172.21.15.29:0/555828817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561390effa00 con 0x561390c09280 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fdbf90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7faf30105760 tx=0x7faf30105660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87339025f7ad7336 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf2e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf2e7fc640 1 -- 172.21.15.29:0/555828817 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf30049070 con 0x561390c09280 2024-09-21T02:39:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf2e7fc640 1 -- 172.21.15.29:0/555828817 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faf30003c80 con 0x561390c09280 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf53a46740 1 -- 172.21.15.29:0/555828817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf300d90f0 con 0x561390c09280 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf2e7fc640 1 -- 172.21.15.29:0/555828817 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faf300cd3a0 con 0x561390c09280 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf53a46740 10 client.15576 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf53a46740 1 -- 172.21.15.29:0/555828817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561390fea420 con 0x561390c09280 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.359+0000 7faf2e7fc640 1 -- 172.21.15.29:0/555828817 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf300d90f0 con 0x561390c09280 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.360+0000 7faf2e7fc640 1 -- 172.21.15.29:0/555828817 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7faf300d0c60 con 0x561390c09280 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.360+0000 7faf53a46740 10 client.15576 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.360+0000 7faf53a46740 10 client.15576 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.360+0000 7faf53a46740 20 client.15576 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.360+0000 7faf53a46740 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x561390be88e0 0x561390fedc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.360+0000 7faf53a46740 4 client.15576 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.360+0000 7faf53a46740 1 -- 172.21.15.29:0/555828817 --> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561390f49b60 con 0x561390be88e0 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.360+0000 7faf46ffd640 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x561390be88e0 0x561390fedc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.361+0000 7faf46ffd640 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x561390be88e0 0x561390fedc00 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5b5143297add0ac3 server_cookie=f09ec9dc4f35f831 in_seq=0 out_seq=0 2024-09-21T02:39:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.361+0000 7faf2e7fc640 10 client.15576 ms_handle_connect on v2:172.21.15.66:6836/3184195773 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.361+0000 7faf2e7fc640 1 -- 172.21.15.29:0/555828817 <== mds.4 v2:172.21.15.66:6836/3184195773 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561390f49b60 con 0x561390be88e0 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.361+0000 7faf2e7fc640 10 client.15576 handle_command_reply: tid=0 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.361+0000 7faf53a46740 1 client.15576 shutdown 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 -- 172.21.15.29:0/555828817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 msgr2=0x561390fdbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fdbf90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7faf30105760 tx=0x7faf30105660 comp rx=0 tx=0).stop 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 -- 172.21.15.29:0/555828817 shutdown_connections 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x561390be88e0 0x561390fedc00 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c29f60 0x561390fe7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fdbf90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 --2- 172.21.15.29:0/555828817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c010a0 0x561390fdba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf2e7fc640 0 client.15576 ms_handle_reset on v2:172.21.15.66:6836/3184195773 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 -- 172.21.15.29:0/555828817 >> 172.21.15.29:0/555828817 conn(0x561390c1e350 msgr2=0x561390fd3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 -- 172.21.15.29:0/555828817 shutdown_connections 2024-09-21T02:39:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 1 -- 172.21.15.29:0/555828817 wait complete. 2024-09-21T02:39:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.362+0000 7faf53a46740 20 client.15576 trim_cache size 0 max 16384 2024-09-21T02:39:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.380+0000 7faf53a46740 1 Processor -- start 2024-09-21T02:39:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.380+0000 7faf53a46740 1 -- start start 2024-09-21T02:39:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf53a46740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fdba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf53a46740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf53a46740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 0x561390fd1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf53a46740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561390febf40 con 0x561390c09280 2024-09-21T02:39:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf53a46740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561390fec610 con 0x561390c010a0 2024-09-21T02:39:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf53a46740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561390fecce0 con 0x561390c29f60 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52986/0 (socket says 172.21.15.29:52986) 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 -- 172.21.15.29:0/3958582284 learned_addr learned my addr 172.21.15.29:0/3958582284 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf477fe640 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 0x561390fd1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf46ffd640 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fdba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 -- 172.21.15.29:0/3958582284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 msgr2=0x561390fd1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 0x561390fd1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 -- 172.21.15.29:0/3958582284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 msgr2=0x561390fdba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fdba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 -- 172.21.15.29:0/3958582284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561390fc8080 con 0x561390c09280 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf46ffd640 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fdba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.381+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd5fe0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faf300cd370 tx=0x7faf300f1050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19f7c70b4caf187 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.382+0000 7faf3cff9640 1 -- 172.21.15.29:0/3958582284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf30049070 con 0x561390c09280 2024-09-21T02:39:20.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.382+0000 7faf3cff9640 1 -- 172.21.15.29:0/3958582284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf300c9350 con 0x561390c09280 2024-09-21T02:39:20.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.382+0000 7faf3cff9640 1 -- 172.21.15.29:0/3958582284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf300cd630 con 0x561390c09280 2024-09-21T02:39:20.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 -- 172.21.15.29:0/3958582284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 msgr2=0x561390fd5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd5fe0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faf300cd370 tx=0x7faf300f1050 comp rx=0 tx=0).stop 2024-09-21T02:39:20.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 -- 172.21.15.29:0/3958582284 shutdown_connections 2024-09-21T02:39:20.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c29f60 0x561390fd1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c09280 0x561390fd5fe0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 --2- 172.21.15.29:0/3958582284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390fdba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 -- 172.21.15.29:0/3958582284 >> 172.21.15.29:0/3958582284 conn(0x561390c1e350 msgr2=0x561390fd0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:20.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 -- 172.21.15.29:0/3958582284 shutdown_connections 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 -- 172.21.15.29:0/3958582284 wait complete. 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 -- start start 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.383+0000 7faf53a46740 1 Processor -- start 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf53a46740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390feefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf53a46740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c09280 0x561390fef530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf53a46740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c29f60 0x56139105c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf53a46740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561390feda80 con 0x561390c29f60 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf53a46740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561390fed3b0 con 0x561390c010a0 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf53a46740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561390fee150 con 0x561390c09280 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf53a46740 10 client.0 _ll_register_callbacks cb 0x7faf380097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c29f60 0x56139105c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c29f60 0x56139105c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53002/0 (socket says 172.21.15.29:53002) 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 -- 172.21.15.29:0/2298036567 learned_addr learned my addr 172.21.15.29:0/2298036567 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf3e7fc640 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c09280 0x561390fef530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf46ffd640 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390feefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 -- 172.21.15.29:0/2298036567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c09280 msgr2=0x561390fef530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c09280 0x561390fef530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 -- 172.21.15.29:0/2298036567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 msgr2=0x561390feefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390feefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 -- 172.21.15.29:0/2298036567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56139105c710 con 0x561390c29f60 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf477fe640 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c29f60 0x56139105c100 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7faf40010c90 tx=0x7faf40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a008a4f2b0aa77 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.384+0000 7faf2e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf2e7fc640 1 -- 172.21.15.29:0/2298036567 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4001ce20 con 0x561390c29f60 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf2e7fc640 1 -- 172.21.15.29:0/2298036567 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faf40004b20 con 0x561390c29f60 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 1 -- 172.21.15.29:0/2298036567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf300c9350 con 0x561390c29f60 2024-09-21T02:39:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf2e7fc640 1 -- 172.21.15.29:0/2298036567 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faf40032050 con 0x561390c29f60 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 10 client.15582 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 1 -- 172.21.15.29:0/2298036567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56139105d9f0 con 0x561390c29f60 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf2e7fc640 1 -- 172.21.15.29:0/2298036567 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf300c9350 con 0x561390c29f60 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf2e7fc640 1 -- 172.21.15.29:0/2298036567 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7faf40026070 con 0x561390c29f60 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 10 client.15582 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 10 client.15582 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 20 client.15582 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x561390bbdc90 0x5613910613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 4 client.15582 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.385+0000 7faf53a46740 1 -- 172.21.15.29:0/2298036567 --> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] -- command(tid 0: {"prefix": "status"}) -- 0x561390f45610 con 0x561390bbdc90 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.386+0000 7faf46ffd640 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x561390bbdc90 0x5613910613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.386+0000 7faf2e7fc640 10 client.15582 ms_handle_connect on v2:172.21.15.66:6836/3184195773 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.386+0000 7faf46ffd640 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x561390bbdc90 0x5613910613a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1a1ba5bdb1ce7f5b server_cookie=2c9b57ed62e6a435 in_seq=0 out_seq=0 2024-09-21T02:39:20.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.386+0000 7faf2e7fc640 1 -- 172.21.15.29:0/2298036567 <== mds.4 v2:172.21.15.66:6836/3184195773 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561390f45610 con 0x561390bbdc90 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.386+0000 7faf2e7fc640 10 client.15582 handle_command_reply: tid=0 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.386+0000 7faf53a46740 1 client.15582 shutdown 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.386+0000 7faf53a46740 1 -- 172.21.15.29:0/2298036567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c29f60 msgr2=0x56139105c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c29f60 0x56139105c100 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7faf40010c90 tx=0x7faf40004a40 comp rx=0 tx=0).stop 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 -- 172.21.15.29:0/2298036567 shutdown_connections 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.66:6836/3184195773,v1:172.21.15.66:6837/3184195773] conn(0x561390bbdc90 0x5613910613a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561390c29f60 0x56139105c100 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561390c09280 0x561390fef530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 --2- 172.21.15.29:0/2298036567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561390c010a0 0x561390feefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 -- 172.21.15.29:0/2298036567 >> 172.21.15.29:0/2298036567 conn(0x561390c1e350 msgr2=0x561390fdcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf2e7fc640 0 client.15582 ms_handle_reset on v2:172.21.15.66:6836/3184195773 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 -- 172.21.15.29:0/2298036567 shutdown_connections 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 1 -- 172.21.15.29:0/2298036567 wait complete. 2024-09-21T02:39:20.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf53a46740 20 client.15582 trim_cache size 0 max 16384 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24599, 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 131.35492169299999, 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-09-21T02:39:20.393 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:39:20.394 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-09-21T02:39:20.394 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 196.606223613 2024-09-21T02:39:20.394 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:39:20.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 -- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf340b65d0 msgr2=0x7faf340b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf340b65d0 0x7faf340b8ae0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faf40005fd0 tx=0x7faf400086a0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 -- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 msgr2=0x7faf4819aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:20.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 0x7faf4819aa00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7faf30004770 tx=0x7faf30004040 comp rx=0 tx=0).stop 2024-09-21T02:39:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 -- 172.21.15.29:0/3747517234 shutdown_connections 2024-09-21T02:39:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faf340b65d0 0x7faf340b8ae0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf48109e40 0x7faf4819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf481094f0 0x7faf4819aa00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 --2- 172.21.15.29:0/3747517234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf48103bb0 0x7faf4819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.387+0000 7faf4d4d2640 1 -- 172.21.15.29:0/3747517234 >> 172.21.15.29:0/3747517234 conn(0x7faf480ff600 msgr2=0x7faf48101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.388+0000 7faf4d4d2640 1 -- 172.21.15.29:0/3747517234 shutdown_connections 2024-09-21T02:39:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:20.388+0000 7faf4d4d2640 1 -- 172.21.15.29:0/3747517234 wait complete. 2024-09-21T02:39:21.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.f status 2024-09-21T02:39:21.374 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:21.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:21 smithi190 ceph-mon[33871]: pgmap v396: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-09-21T02:39:21.798 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:21 smithi066 ceph-mon[33436]: pgmap v396: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-09-21T02:39:21.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:21 smithi029 ceph-mon[31006]: pgmap v396: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-09-21T02:39:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.928+0000 7f3414d9f640 1 -- 172.21.15.29:0/1156092066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410103990 msgr2=0x7f3410103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.928+0000 7f3414d9f640 1 --2- 172.21.15.29:0/1156092066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410103990 0x7f3410103da0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3400009900 tx=0x7f340002a410 comp rx=0 tx=0).stop 2024-09-21T02:39:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.929+0000 7f3414d9f640 1 -- 172.21.15.29:0/1156092066 shutdown_connections 2024-09-21T02:39:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.929+0000 7f3414d9f640 1 --2- 172.21.15.29:0/1156092066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3410109c20 0x7f3410110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.929+0000 7f3414d9f640 1 --2- 172.21.15.29:0/1156092066 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34101092d0 0x7f34101096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.929+0000 7f3414d9f640 1 --2- 172.21.15.29:0/1156092066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410103990 0x7f3410103da0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.929+0000 7f3414d9f640 1 -- 172.21.15.29:0/1156092066 >> 172.21.15.29:0/1156092066 conn(0x7f34100ff3e0 msgr2=0x7f3410101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.929+0000 7f3414d9f640 1 -- 172.21.15.29:0/1156092066 shutdown_connections 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.929+0000 7f3414d9f640 1 -- 172.21.15.29:0/1156092066 wait complete. 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.929+0000 7f3414d9f640 1 Processor -- start 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f3414d9f640 1 -- start start 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f3414d9f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3410103990 0x7f341019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f3414d9f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34101092d0 0x7f341019a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f3414d9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410109c20 0x7f341019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f3414d9f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3410113850 con 0x7f3410109c20 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f3414d9f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f34101136d0 con 0x7f3410103990 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f3414d9f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34101139d0 con 0x7f34101092d0 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f340ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410109c20 0x7f341019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f340ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410109c20 0x7f341019fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53016/0 (socket says 172.21.15.29:53016) 2024-09-21T02:39:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f340ffff640 1 -- 172.21.15.29:0/417003598 learned_addr learned my addr 172.21.15.29:0/417003598 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f340f7fe640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3410103990 0x7f341019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.930+0000 7f340effd640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34101092d0 0x7f341019a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340ffff640 1 -- 172.21.15.29:0/417003598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34101092d0 msgr2=0x7f341019a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340ffff640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34101092d0 0x7f341019a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340ffff640 1 -- 172.21.15.29:0/417003598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3410103990 msgr2=0x7f341019a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340ffff640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3410103990 0x7f341019a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340ffff640 1 -- 172.21.15.29:0/417003598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34101a02b0 con 0x7f3410109c20 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340ffff640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410109c20 0x7f341019fb80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f340800b970 tx=0x7f340800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340cff9640 1 -- 172.21.15.29:0/417003598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3408019070 con 0x7f3410109c20 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f3414d9f640 1 -- 172.21.15.29:0/417003598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34101a05a0 con 0x7f3410109c20 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340cff9640 1 -- 172.21.15.29:0/417003598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3408014440 con 0x7f3410109c20 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f340cff9640 1 -- 172.21.15.29:0/417003598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34080075d0 con 0x7f3410109c20 2024-09-21T02:39:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.931+0000 7f3414d9f640 1 -- 172.21.15.29:0/417003598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34101a6d00 con 0x7f3410109c20 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f341c575740 1 Processor -- start 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f340cff9640 1 -- 172.21.15.29:0/417003598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f340801c070 con 0x7f3410109c20 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f341c575740 1 -- start start 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f341c575740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d5651940f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f341c575740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dc50a0 0x55d565194630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f341c575740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d565194b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f341c575740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d56510bc10 con 0x55d564dcd280 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f341c575740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d5650fb040 con 0x55d564dc50a0 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.934+0000 7f341c575740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d5650fee50 con 0x55d564dedf60 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d565194b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d565194b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53026/0 (socket says 172.21.15.29:53026) 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 -- 172.21.15.29:0/3563174724 learned_addr learned my addr 172.21.15.29:0/3563174724 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340f7fe640 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dc50a0 0x55d565194630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d5651940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 -- 172.21.15.29:0/3563174724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 msgr2=0x55d5651940f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d5651940f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 -- 172.21.15.29:0/3563174724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dc50a0 msgr2=0x55d565194630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dc50a0 0x55d565194630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 -- 172.21.15.29:0/3563174724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d564e59540 con 0x55d564dcd280 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340f7fe640 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dc50a0 0x55d565194630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340cff9640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f33f80b6620 0x7f33f80b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340f7fe640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f33f80b6620 0x7f33f80b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340effd640 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d565194b70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3404009330 tx=0x7f34040094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8391f965bba20b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f340cff9640 1 -- 172.21.15.29:0/417003598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f34080e6c00 con 0x7f3410109c20 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.935+0000 7f33f5ffb640 1 -- 172.21.15.29:0/3563174724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404017e60 con 0x55d564dcd280 2024-09-21T02:39:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f33f5ffb640 1 -- 172.21.15.29:0/3563174724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340401d070 con 0x55d564dcd280 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f33f5ffb640 1 -- 172.21.15.29:0/3563174724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404018840 con 0x55d564dcd280 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f341c575740 1 -- 172.21.15.29:0/3563174724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 msgr2=0x55d565194b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f341c575740 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d565194b70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3404009330 tx=0x7f34040094d0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f340f7fe640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f33f80b6620 0x7f33f80b8b30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f34000057c0 tx=0x7f34000058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f341c575740 1 -- 172.21.15.29:0/3563174724 shutdown_connections 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f341c575740 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d565194b70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f341c575740 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dc50a0 0x55d565194630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f341c575740 1 --2- 172.21.15.29:0/3563174724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d5651940f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.936+0000 7f341c575740 1 -- 172.21.15.29:0/3563174724 >> 172.21.15.29:0/3563174724 conn(0x55d564de2350 msgr2=0x55d565191fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 -- 172.21.15.29:0/3563174724 shutdown_connections 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 -- 172.21.15.29:0/3563174724 wait complete. 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 -- start start 2024-09-21T02:39:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 Processor -- start 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d56519fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 0x55d5651a0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d5651abb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d564f5c5f0 con 0x55d564dc50a0 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.937+0000 7f341c575740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d565096010 con 0x55d564dcd280 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f341c575740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d5651a2870 con 0x55d564dedf60 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d56519fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f341c575740 10 client.0 _ll_register_callbacks cb 0x55d5650f5340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d56519fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53038/0 (socket says 172.21.15.29:53038) 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 -- 172.21.15.29:0/2787336566 learned_addr learned my addr 172.21.15.29:0/2787336566 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340ffff640 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d5651abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340effd640 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 0x55d5651a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 -- 172.21.15.29:0/2787336566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 msgr2=0x55d5651abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d5651abb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 -- 172.21.15.29:0/2787336566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 msgr2=0x55d5651a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 0x55d5651a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 -- 172.21.15.29:0/2787336566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5650c3a00 con 0x55d564dc50a0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f340f7fe640 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d56519fc10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3400009900 tx=0x7f3400009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9f306391c7974e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f33eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f33eeffd640 1 -- 172.21.15.29:0/2787336566 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400035040 con 0x55d564dc50a0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f33eeffd640 1 -- 172.21.15.29:0/2787336566 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f34000035f0 con 0x55d564dc50a0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.938+0000 7f341c575740 1 -- 172.21.15.29:0/2787336566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f340401d070 con 0x55d564dc50a0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f33eeffd640 1 -- 172.21.15.29:0/2787336566 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f34000044c0 con 0x55d564dc50a0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f341c575740 10 client.15594 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f341c575740 1 -- 172.21.15.29:0/2787336566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5651ae810 con 0x55d564dc50a0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f33eeffd640 1 -- 172.21.15.29:0/2787336566 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340401d070 con 0x55d564dc50a0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f33eeffd640 1 -- 172.21.15.29:0/2787336566 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3400044070 con 0x55d564dc50a0 2024-09-21T02:39:21.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f341c575740 10 client.15594 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f341c575740 10 client.15594 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f341c575740 20 client.15594 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f341c575740 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x55d564dac8e0 0x55d5651b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f341c575740 4 client.15594 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.939+0000 7f341c575740 1 -- 172.21.15.29:0/2787336566 --> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d56510db60 con 0x55d564dac8e0 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.940+0000 7f340effd640 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x55d564dac8e0 0x55d5651b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.940+0000 7f340effd640 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x55d564dac8e0 0x55d5651b22a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c0a87da1690109a5 server_cookie=180034cb0b137df3 in_seq=0 out_seq=0 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.940+0000 7f33eeffd640 10 client.15594 ms_handle_connect on v2:172.21.15.190:6834/131555194 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f33eeffd640 1 -- 172.21.15.29:0/2787336566 <== mds.3 v2:172.21.15.190:6834/131555194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d56510db60 con 0x55d564dac8e0 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f33eeffd640 10 client.15594 handle_command_reply: tid=0 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 client.15594 shutdown 2024-09-21T02:39:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 -- 172.21.15.29:0/2787336566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 msgr2=0x55d56519fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d56519fc10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3400009900 tx=0x7f3400009260 comp rx=0 tx=0).stop 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 -- 172.21.15.29:0/2787336566 shutdown_connections 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x55d564dac8e0 0x55d5651b22a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d5651abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 0x55d5651a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f33eeffd640 0 client.15594 ms_handle_reset on v2:172.21.15.190:6834/131555194 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 --2- 172.21.15.29:0/2787336566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d56519fc10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 -- 172.21.15.29:0/2787336566 >> 172.21.15.29:0/2787336566 conn(0x55d564de2350 msgr2=0x55d565195ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 -- 172.21.15.29:0/2787336566 shutdown_connections 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 1 -- 172.21.15.29:0/2787336566 wait complete. 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.941+0000 7f341c575740 20 client.15594 trim_cache size 0 max 16384 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f341c575740 1 Processor -- start 2024-09-21T02:39:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f341c575740 1 -- start start 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f341c575740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d5651abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f341c575740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 0x55d56519e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f341c575740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d56519e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f341c575740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d5651b05e0 con 0x55d564dc50a0 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f341c575740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d5651b0cb0 con 0x55d564dcd280 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f341c575740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d5651b1380 con 0x55d564dedf60 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d5651abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d5651abd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53048/0 (socket says 172.21.15.29:53048) 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340f7fe640 1 -- 172.21.15.29:0/1469625422 learned_addr learned my addr 172.21.15.29:0/1469625422 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340ffff640 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d56519e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340f7fe640 1 -- 172.21.15.29:0/1469625422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 msgr2=0x55d56519e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340f7fe640 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d56519e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340f7fe640 1 -- 172.21.15.29:0/1469625422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 msgr2=0x55d56519e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340effd640 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 0x55d56519e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340f7fe640 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 0x55d56519e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340f7fe640 1 -- 172.21.15.29:0/1469625422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5651a0cd0 con 0x55d564dc50a0 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.960+0000 7f340ffff640 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d56519e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.961+0000 7f340f7fe640 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d5651abd10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3400009230 tx=0x7f3400004450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb290ee8bc82a996 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.961+0000 7f33f57fa640 1 -- 172.21.15.29:0/1469625422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400035040 con 0x55d564dc50a0 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.961+0000 7f33f57fa640 1 -- 172.21.15.29:0/1469625422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340003dbd0 con 0x55d564dc50a0 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.961+0000 7f33f57fa640 1 -- 172.21.15.29:0/1469625422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400003d10 con 0x55d564dc50a0 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.961+0000 7f341c575740 1 -- 172.21.15.29:0/1469625422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 msgr2=0x55d5651abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.961+0000 7f341c575740 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d5651abd10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3400009230 tx=0x7f3400004450 comp rx=0 tx=0).stop 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 -- 172.21.15.29:0/1469625422 shutdown_connections 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dedf60 0x55d56519e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dcd280 0x55d56519e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 --2- 172.21.15.29:0/1469625422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dc50a0 0x55d5651abd10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 -- 172.21.15.29:0/1469625422 >> 172.21.15.29:0/1469625422 conn(0x55d564de2350 msgr2=0x55d5651afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 -- 172.21.15.29:0/1469625422 shutdown_connections 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 -- 172.21.15.29:0/1469625422 wait complete. 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 -- start start 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.962+0000 7f341c575740 1 Processor -- start 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f341c575740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dc50a0 0x55d5651baf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f341c575740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d5651b4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f341c575740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dedf60 0x55d5651b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f341c575740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d5651b2120 con 0x55d564dcd280 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f341c575740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d5651b1a50 con 0x55d564dedf60 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f341c575740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d5651b27f0 con 0x55d564dc50a0 2024-09-21T02:39:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f341c575740 10 client.0 _ll_register_callbacks cb 0x7f3400009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d5651b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dc50a0 0x55d5651baf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d5651b4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53054/0 (socket says 172.21.15.29:53054) 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 -- 172.21.15.29:0/492859757 learned_addr learned my addr 172.21.15.29:0/492859757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340ffff640 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dedf60 0x55d5651b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 -- 172.21.15.29:0/492859757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dc50a0 msgr2=0x55d5651baf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dc50a0 0x55d5651baf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 -- 172.21.15.29:0/492859757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dedf60 msgr2=0x55d5651b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dedf60 0x55d5651b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 -- 172.21.15.29:0/492859757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5651b9b90 con 0x55d564dcd280 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340f7fe640 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dc50a0 0x55d5651baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340ffff640 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dedf60 0x55d5651b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:21.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f340effd640 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d5651b4460 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3404009300 tx=0x7f34040094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b4b25c3f81ed9d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.963+0000 7f33eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.964+0000 7f33eeffd640 1 -- 172.21.15.29:0/492859757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404015b40 con 0x55d564dcd280 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.964+0000 7f33eeffd640 1 -- 172.21.15.29:0/492859757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3404015ce0 con 0x55d564dcd280 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.964+0000 7f341c575740 1 -- 172.21.15.29:0/492859757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f340003dbd0 con 0x55d564dcd280 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.964+0000 7f33eeffd640 1 -- 172.21.15.29:0/492859757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3404037050 con 0x55d564dcd280 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.964+0000 7f341c575740 10 client.15600 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.964+0000 7f341c575740 1 -- 172.21.15.29:0/492859757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d56521d860 con 0x55d564dcd280 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.964+0000 7f33eeffd640 1 -- 172.21.15.29:0/492859757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340003dbd0 con 0x55d564dcd280 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.964+0000 7f33eeffd640 1 -- 172.21.15.29:0/492859757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3404029400 con 0x55d564dcd280 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f341c575740 10 client.15600 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f341c575740 10 client.15600 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f341c575740 20 client.15600 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f341c575740 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x55d564d81c90 0x55d565220d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f341c575740 4 client.15600 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:39:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f341c575740 1 -- 172.21.15.29:0/492859757 --> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] -- command(tid 0: {"prefix": "status"}) -- 0x55d565109610 con 0x55d564d81c90 2024-09-21T02:39:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f340f7fe640 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x55d564d81c90 0x55d565220d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f33eeffd640 10 client.15600 ms_handle_connect on v2:172.21.15.190:6834/131555194 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.965+0000 7f340f7fe640 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x55d564d81c90 0x55d565220d00 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=73b1fa14046c44b3 server_cookie=1b4092cbebc93093 in_seq=0 out_seq=0 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f33eeffd640 1 -- 172.21.15.29:0/492859757 <== mds.3 v2:172.21.15.190:6834/131555194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d565109610 con 0x55d564d81c90 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f33eeffd640 10 client.15600 handle_command_reply: tid=0 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 client.15600 shutdown 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 -- 172.21.15.29:0/492859757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 msgr2=0x55d5651b4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d5651b4460 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3404009300 tx=0x7f34040094c0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 -- 172.21.15.29:0/492859757 shutdown_connections 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.190:6834/131555194,v1:172.21.15.190:6835/131555194] conn(0x55d564d81c90 0x55d565220d00 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d564dedf60 0x55d5651b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d564dcd280 0x55d5651b4460 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f33eeffd640 0 client.15600 ms_handle_reset on v2:172.21.15.190:6834/131555194 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 --2- 172.21.15.29:0/492859757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d564dc50a0 0x55d5651baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 -- 172.21.15.29:0/492859757 >> 172.21.15.29:0/492859757 conn(0x55d564de2350 msgr2=0x55d56519abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 -- 172.21.15.29:0/492859757 shutdown_connections 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 1 -- 172.21.15.29:0/492859757 wait complete. 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.966+0000 7f341c575740 20 client.15600 trim_cache size 0 max 16384 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-09-21T02:39:21.983 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24613, 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 133.94307067700001, 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 203.32327060099999 2024-09-21T02:39:21.984 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 -- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f33f80b6620 msgr2=0x7f33f80b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f33f80b6620 0x7f33f80b8b30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f34000057c0 tx=0x7f34000058d0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 -- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410109c20 msgr2=0x7f341019fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410109c20 0x7f341019fb80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f340800b970 tx=0x7f340800bd70 comp rx=0 tx=0).stop 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 -- 172.21.15.29:0/417003598 shutdown_connections 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f33f80b6620 0x7f33f80b8b30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3410109c20 0x7f341019fb80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34101092d0 0x7f341019a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 --2- 172.21.15.29:0/417003598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3410103990 0x7f341019a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 -- 172.21.15.29:0/417003598 >> 172.21.15.29:0/417003598 conn(0x7f34100ff3e0 msgr2=0x7f3410101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 -- 172.21.15.29:0/417003598 shutdown_connections 2024-09-21T02:39:21.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:21.967+0000 7f3414d9f640 1 -- 172.21.15.29:0/417003598 wait complete. 2024-09-21T02:39:22.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.c status 2024-09-21T02:39:23.051 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:23.477 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:23 smithi029 ceph-mon[31006]: pgmap v397: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-21T02:39:23.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 -- 172.21.15.29:0/2163620407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664108cf0 msgr2=0x7fb66410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 --2- 172.21.15.29:0/2163620407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664108cf0 0x7fb66410f520 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009ca0 tx=0x7fb65c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 -- 172.21.15.29:0/2163620407 shutdown_connections 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 --2- 172.21.15.29:0/2163620407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664108cf0 0x7fb66410f520 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 --2- 172.21.15.29:0/2163620407 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb6641083a0 0x7fb6641087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 --2- 172.21.15.29:0/2163620407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb664102a60 0x7fb664102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 -- 172.21.15.29:0/2163620407 >> 172.21.15.29:0/2163620407 conn(0x7fb664079730 msgr2=0x7fb664079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 -- 172.21.15.29:0/2163620407 shutdown_connections 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.569+0000 7fb66ab16640 1 -- 172.21.15.29:0/2163620407 wait complete. 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb66ab16640 1 Processor -- start 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb66ab16640 1 -- start start 2024-09-21T02:39:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb66ab16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664102a60 0x7fb66419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb66ab16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6641083a0 0x7fb66419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb66ab16640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb664108cf0 0x7fb66419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb66ab16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb664112860 con 0x7fb664102a60 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb66ab16640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb6641126e0 con 0x7fb664108cf0 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb66ab16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb6641129e0 con 0x7fb6641083a0 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb6688b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664102a60 0x7fb66419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb6688b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664102a60 0x7fb66419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53060/0 (socket says 172.21.15.29:53060) 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.570+0000 7fb6688b2640 1 -- 172.21.15.29:0/489139404 learned_addr learned my addr 172.21.15.29:0/489139404 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb663fff640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6641083a0 0x7fb66419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb6690b3640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb664108cf0 0x7fb66419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb6688b2640 1 -- 172.21.15.29:0/489139404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6641083a0 msgr2=0x7fb66419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb6688b2640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6641083a0 0x7fb66419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb6688b2640 1 -- 172.21.15.29:0/489139404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb664108cf0 msgr2=0x7fb66419fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb6688b2640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb664108cf0 0x7fb66419fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb6688b2640 1 -- 172.21.15.29:0/489139404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6641a0390 con 0x7fb664102a60 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb6688b2640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664102a60 0x7fb66419a420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb65800d770 tx=0x7fb65800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb661ffb640 1 -- 172.21.15.29:0/489139404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658017070 con 0x7fb664102a60 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb661ffb640 1 -- 172.21.15.29:0/489139404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6580054b0 con 0x7fb664102a60 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb66ab16640 1 -- 172.21.15.29:0/489139404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6641a0680 con 0x7fb664102a60 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb661ffb640 1 -- 172.21.15.29:0/489139404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658004c60 con 0x7fb664102a60 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.571+0000 7fb66ab16640 1 -- 172.21.15.29:0/489139404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6641a6de0 con 0x7fb664102a60 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb661ffb640 1 -- 172.21.15.29:0/489139404 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7fb658004e00 con 0x7fb664102a60 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb67108a740 1 Processor -- start 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb67108a740 1 -- start start 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb67108a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a75f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb67108a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 0x564893a76440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb67108a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a76980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb67108a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5648939edc10 con 0x5648936cff60 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb67108a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5648939dd040 con 0x5648936af280 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb67108a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5648939e0e50 con 0x5648936a70a0 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb6690b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a75f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb6690b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a75f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53074/0 (socket says 172.21.15.29:53074) 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.574+0000 7fb6690b3640 1 -- 172.21.15.29:0/883572801 learned_addr learned my addr 172.21.15.29:0/883572801 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb663fff640 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a76980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb6688b2640 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 0x564893a76440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb661ffb640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6540b65a0 0x7fb6540b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb6690b3640 1 -- 172.21.15.29:0/883572801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 msgr2=0x564893a76440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb6690b3640 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 0x564893a76440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb6690b3640 1 -- 172.21.15.29:0/883572801 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 msgr2=0x564893a76980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb6690b3640 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a76980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb6690b3640 1 -- 172.21.15.29:0/883572801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489373b540 con 0x5648936cff60 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb663fff640 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a76980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb661ffb640 1 -- 172.21.15.29:0/489139404 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb6580e7340 con 0x7fb664102a60 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb6690b3640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6540b65a0 0x7fb6540b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb6690b3640 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a75f00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb65c005cc0 tx=0x7fb65c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3927991dc90a62f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.575+0000 7fb64effd640 1 -- 172.21.15.29:0/883572801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c04d070 con 0x5648936cff60 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb64effd640 1 -- 172.21.15.29:0/883572801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb65c036040 con 0x5648936cff60 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb64effd640 1 -- 172.21.15.29:0/883572801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0307c0 con 0x5648936cff60 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb67108a740 1 -- 172.21.15.29:0/883572801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 msgr2=0x564893a75f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb6690b3640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6540b65a0 0x7fb6540b8ab0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb65c04a000 tx=0x7fb65c04b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb67108a740 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a75f00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb65c005cc0 tx=0x7fb65c009a20 comp rx=0 tx=0).stop 2024-09-21T02:39:23.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb67108a740 1 -- 172.21.15.29:0/883572801 shutdown_connections 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb67108a740 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a76980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb67108a740 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 0x564893a76440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb67108a740 1 --2- 172.21.15.29:0/883572801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a75f00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb67108a740 1 -- 172.21.15.29:0/883572801 >> 172.21.15.29:0/883572801 conn(0x5648936c4350 msgr2=0x564893a73df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.576+0000 7fb67108a740 1 -- 172.21.15.29:0/883572801 shutdown_connections 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.577+0000 7fb67108a740 1 -- 172.21.15.29:0/883572801 wait complete. 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.577+0000 7fb67108a740 1 -- start start 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.577+0000 7fb67108a740 1 Processor -- start 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb67108a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936a70a0 0x564893a81a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb67108a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 0x564893a81fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb67108a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936cff60 0x564893a8da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb67108a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56489383e5f0 con 0x5648936a70a0 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb67108a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564893978010 con 0x5648936cff60 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb67108a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564893a846e0 con 0x5648936af280 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936a70a0 0x564893a81a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb67108a740 10 client.0 _ll_register_callbacks cb 0x5648939d7340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936a70a0 0x564893a81a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53088/0 (socket says 172.21.15.29:53088) 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 -- 172.21.15.29:0/963664059 learned_addr learned my addr 172.21.15.29:0/963664059 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb663fff640 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 0x564893a81fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6690b3640 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936cff60 0x564893a8da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 -- 172.21.15.29:0/963664059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 msgr2=0x564893a81fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 0x564893a81fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 -- 172.21.15.29:0/963664059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936cff60 msgr2=0x564893a8da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936cff60 0x564893a8da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 -- 172.21.15.29:0/963664059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648939a5a00 con 0x5648936a70a0 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6690b3640 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936cff60 0x564893a8da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb6688b2640 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936a70a0 0x564893a81a80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb6580aa260 tx=0x7fb6580a8e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5287718b2bf2aa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb643fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb643fff640 1 -- 172.21.15.29:0/963664059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658020070 con 0x5648936a70a0 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb643fff640 1 -- 172.21.15.29:0/963664059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb6580050f0 con 0x5648936a70a0 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.578+0000 7fb67108a740 1 -- 172.21.15.29:0/963664059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb65c036040 con 0x5648936a70a0 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb643fff640 1 -- 172.21.15.29:0/963664059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb6580ac760 con 0x5648936a70a0 2024-09-21T02:39:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb67108a740 10 client.15612 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb67108a740 1 -- 172.21.15.29:0/963664059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564893a90600 con 0x5648936a70a0 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb643fff640 1 -- 172.21.15.29:0/963664059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb65c036040 con 0x5648936a70a0 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb643fff640 1 -- 172.21.15.29:0/963664059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb6580981a0 con 0x5648936a70a0 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb67108a740 10 client.15612 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb67108a740 10 client.15612 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb67108a740 20 client.15612 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb67108a740 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56489368e8e0 0x564893a94040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb67108a740 4 client.15612 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb67108a740 1 -- 172.21.15.29:0/963664059 --> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648939efb60 con 0x56489368e8e0 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.579+0000 7fb663fff640 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56489368e8e0 0x564893a94040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.580+0000 7fb643fff640 10 client.15612 ms_handle_connect on v2:172.21.15.190:6836/3758896499 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.580+0000 7fb663fff640 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56489368e8e0 0x564893a94040 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bc4c270e66028407 server_cookie=45e6322600353f48 in_seq=0 out_seq=0 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb643fff640 1 -- 172.21.15.29:0/963664059 <== mds.2 v2:172.21.15.190:6836/3758896499 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5648939efb60 con 0x56489368e8e0 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb643fff640 10 client.15612 handle_command_reply: tid=0 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 client.15612 shutdown 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 -- 172.21.15.29:0/963664059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936a70a0 msgr2=0x564893a81a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936a70a0 0x564893a81a80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb6580aa260 tx=0x7fb6580a8e90 comp rx=0 tx=0).stop 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 -- 172.21.15.29:0/963664059 shutdown_connections 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x56489368e8e0 0x564893a94040 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936cff60 0x564893a8da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 0x564893a81fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 --2- 172.21.15.29:0/963664059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936a70a0 0x564893a81a80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb643fff640 0 client.15612 ms_handle_reset on v2:172.21.15.190:6836/3758896499 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 -- 172.21.15.29:0/963664059 >> 172.21.15.29:0/963664059 conn(0x5648936c4350 msgr2=0x564893a790e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 -- 172.21.15.29:0/963664059 shutdown_connections 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 1 -- 172.21.15.29:0/963664059 wait complete. 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.581+0000 7fb67108a740 20 client.15612 trim_cache size 0 max 16384 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.599+0000 7fb67108a740 1 Processor -- start 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.599+0000 7fb67108a740 1 -- start start 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb67108a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936a70a0 0x564893a7caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb67108a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 0x564893a7cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb67108a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a80100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb67108a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564893a92380 con 0x5648936cff60 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb67108a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564893a92a50 con 0x5648936a70a0 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb67108a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564893a93120 con 0x5648936af280 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a80100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a80100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53100/0 (socket says 172.21.15.29:53100) 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 -- 172.21.15.29:0/2694741041 learned_addr learned my addr 172.21.15.29:0/2694741041 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6688b2640 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936a70a0 0x564893a7caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb663fff640 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 0x564893a7cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 -- 172.21.15.29:0/2694741041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 msgr2=0x564893a7cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 0x564893a7cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 -- 172.21.15.29:0/2694741041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936a70a0 msgr2=0x564893a7caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936a70a0 0x564893a7caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 -- 172.21.15.29:0/2694741041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564893a781f0 con 0x5648936cff60 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6688b2640 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936a70a0 0x564893a7caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.600+0000 7fb6690b3640 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a80100 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb65c031ee0 tx=0x7fb65c050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422d3976c57b8a3b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb64e7fc640 1 -- 172.21.15.29:0/2694741041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c060070 con 0x5648936cff60 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb64e7fc640 1 -- 172.21.15.29:0/2694741041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb65c036050 con 0x5648936cff60 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb64e7fc640 1 -- 172.21.15.29:0/2694741041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c04d040 con 0x5648936cff60 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb67108a740 1 -- 172.21.15.29:0/2694741041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 msgr2=0x564893a80100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb67108a740 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a80100 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb65c031ee0 tx=0x7fb65c050000 comp rx=0 tx=0).stop 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb67108a740 1 -- 172.21.15.29:0/2694741041 shutdown_connections 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb67108a740 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a80100 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb67108a740 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936af280 0x564893a7cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb67108a740 1 --2- 172.21.15.29:0/2694741041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936a70a0 0x564893a7caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb67108a740 1 -- 172.21.15.29:0/2694741041 >> 172.21.15.29:0/2694741041 conn(0x5648936c4350 msgr2=0x564893a81a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.601+0000 7fb67108a740 1 -- 172.21.15.29:0/2694741041 shutdown_connections 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.602+0000 7fb67108a740 1 -- 172.21.15.29:0/2694741041 wait complete. 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.602+0000 7fb67108a740 1 -- start start 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.602+0000 7fb67108a740 1 Processor -- start 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.602+0000 7fb67108a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 0x564893a9c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.602+0000 7fb67108a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a9cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.602+0000 7fb67108a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a95a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb67108a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564893a93ec0 con 0x5648936cff60 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb67108a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564893a937f0 con 0x5648936af280 2024-09-21T02:39:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb67108a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564893a94590 con 0x5648936a70a0 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a95a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a95a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53110/0 (socket says 172.21.15.29:53110) 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 -- 172.21.15.29:0/2015434318 learned_addr learned my addr 172.21.15.29:0/2015434318 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6688b2640 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 0x564893a9c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb67108a740 10 client.0 _ll_register_callbacks cb 0x7fb658009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb663fff640 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a9cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 -- 172.21.15.29:0/2015434318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 msgr2=0x564893a9c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 0x564893a9c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 -- 172.21.15.29:0/2015434318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 msgr2=0x564893a9cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a9cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 -- 172.21.15.29:0/2015434318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564893a96220 con 0x5648936cff60 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb6690b3640 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a95a90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb65c02ade0 tx=0x7fb65c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b95f2ef14a9da37 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb643fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb643fff640 1 -- 172.21.15.29:0/2015434318 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c060070 con 0x5648936cff60 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb643fff640 1 -- 172.21.15.29:0/2015434318 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb65c003f00 con 0x5648936cff60 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.603+0000 7fb67108a740 1 -- 172.21.15.29:0/2015434318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb65c036050 con 0x5648936cff60 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb643fff640 1 -- 172.21.15.29:0/2015434318 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb65c06c050 con 0x5648936cff60 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb67108a740 10 client.15618 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb67108a740 1 -- 172.21.15.29:0/2015434318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564893a9bb20 con 0x5648936cff60 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb643fff640 1 -- 172.21.15.29:0/2015434318 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb65c036050 con 0x5648936cff60 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb643fff640 1 -- 172.21.15.29:0/2015434318 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb65c043050 con 0x5648936cff60 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb67108a740 10 client.15618 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb67108a740 10 client.15618 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:39:23.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb67108a740 20 client.15618 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:23.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb67108a740 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x564893663c90 0x564893b021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:23.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb67108a740 4 client.15618 mds_command: new command op to 24625 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:39:23.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb67108a740 1 -- 172.21.15.29:0/2015434318 --> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] -- command(tid 0: {"prefix": "status"}) -- 0x5648939eb610 con 0x564893663c90 2024-09-21T02:39:23.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.604+0000 7fb6688b2640 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x564893663c90 0x564893b021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:23.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb6688b2640 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x564893663c90 0x564893b021c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4cf2c032f686fd45 server_cookie=f93c56fe6bf932db in_seq=0 out_seq=0 2024-09-21T02:39:23.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb643fff640 10 client.15618 ms_handle_connect on v2:172.21.15.190:6836/3758896499 2024-09-21T02:39:23.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb643fff640 1 -- 172.21.15.29:0/2015434318 <== mds.2 v2:172.21.15.190:6836/3758896499 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5648939eb610 con 0x564893663c90 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb643fff640 10 client.15618 handle_command_reply: tid=0 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 client.15618 shutdown 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 -- 172.21.15.29:0/2015434318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 msgr2=0x564893a95a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a95a90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb65c02ade0 tx=0x7fb65c000f40 comp rx=0 tx=0).stop 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 -- 172.21.15.29:0/2015434318 shutdown_connections 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.190:6836/3758896499,v1:172.21.15.190:6837/3758896499] conn(0x564893663c90 0x564893b021c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb643fff640 0 client.15618 ms_handle_reset on v2:172.21.15.190:6836/3758896499 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648936cff60 0x564893a95a90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5648936af280 0x564893a9cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 --2- 172.21.15.29:0/2015434318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648936a70a0 0x564893a9c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.605+0000 7fb67108a740 1 -- 172.21.15.29:0/2015434318 >> 172.21.15.29:0/2015434318 conn(0x5648936c4350 msgr2=0x564893a81a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb67108a740 1 -- 172.21.15.29:0/2015434318 shutdown_connections 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb67108a740 1 -- 172.21.15.29:0/2015434318 wait complete. 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb67108a740 20 client.15618 trim_cache size 0 max 16384 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24625, 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:39:23.625 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:39:23.626 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 136.59133911800001, 2024-09-21T02:39:23.626 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-09-21T02:39:23.626 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:39:23.626 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-09-21T02:39:23.626 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 202.458500829 2024-09-21T02:39:23.626 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 -- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6540b65a0 msgr2=0x7fb6540b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6540b65a0 0x7fb6540b8ab0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb65c04a000 tx=0x7fb65c04b000 comp rx=0 tx=0).stop 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 -- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664102a60 msgr2=0x7fb66419a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664102a60 0x7fb66419a420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb65800d770 tx=0x7fb65800dc30 comp rx=0 tx=0).stop 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 -- 172.21.15.29:0/489139404 shutdown_connections 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6540b65a0 0x7fb6540b8ab0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb664108cf0 0x7fb66419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6641083a0 0x7fb66419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 --2- 172.21.15.29:0/489139404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb664102a60 0x7fb66419a420 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 -- 172.21.15.29:0/489139404 >> 172.21.15.29:0/489139404 conn(0x7fb664079730 msgr2=0x7fb664101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 -- 172.21.15.29:0/489139404 shutdown_connections 2024-09-21T02:39:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:23.606+0000 7fb66ab16640 1 -- 172.21.15.29:0/489139404 wait complete. 2024-09-21T02:39:23.628 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:23 smithi190 ceph-mon[33871]: pgmap v397: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-21T02:39:23.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:23 smithi066 ceph-mon[33436]: pgmap v397: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-21T02:39:24.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T02:39:24.878 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 -- 172.21.15.29:0/3575752993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48109e10 msgr2=0x7ffb48110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/3575752993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48109e10 0x7ffb48110640 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009ca0 tx=0x7ffb4002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 -- 172.21.15.29:0/3575752993 shutdown_connections 2024-09-21T02:39:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/3575752993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48109e10 0x7ffb48110640 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/3575752993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb481094c0 0x7ffb481098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/3575752993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb48103b80 0x7ffb48103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 -- 172.21.15.29:0/3575752993 >> 172.21.15.29:0/3575752993 conn(0x7ffb480ff610 msgr2=0x7ffb48101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 -- 172.21.15.29:0/3575752993 shutdown_connections 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 -- 172.21.15.29:0/3575752993 wait complete. 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.405+0000 7ffb4ed75640 1 Processor -- start 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4ed75640 1 -- start start 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4ed75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48103b80 0x7ffb4819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4ed75640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb481094c0 0x7ffb4819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4ed75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb48109e10 0x7ffb4819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4ed75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffb481139d0 con 0x7ffb48103b80 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4ed75640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ffb48113850 con 0x7ffb481094c0 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4ed75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffb48113b50 con 0x7ffb48109e10 2024-09-21T02:39:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4cb11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48103b80 0x7ffb4819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4cb11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48103b80 0x7ffb4819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33618/0 (socket says 172.21.15.29:33618) 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4cb11640 1 -- 172.21.15.29:0/67730700 learned_addr learned my addr 172.21.15.29:0/67730700 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4d312640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb48109e10 0x7ffb4819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb47fff640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb481094c0 0x7ffb4819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4cb11640 1 -- 172.21.15.29:0/67730700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb48109e10 msgr2=0x7ffb4819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb48109e10 0x7ffb4819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4cb11640 1 -- 172.21.15.29:0/67730700 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb481094c0 msgr2=0x7ffb4819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb481094c0 0x7ffb4819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb4cb11640 1 -- 172.21.15.29:0/67730700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb481a0210 con 0x7ffb48103b80 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.406+0000 7ffb47fff640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb481094c0 0x7ffb4819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.407+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48103b80 0x7ffb4819a360 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00c920 tx=0x7ffb3c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.407+0000 7ffb45ffb640 1 -- 172.21.15.29:0/67730700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c007c70 con 0x7ffb48103b80 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.407+0000 7ffb45ffb640 1 -- 172.21.15.29:0/67730700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb3c007e10 con 0x7ffb48103b80 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.407+0000 7ffb4ed75640 1 -- 172.21.15.29:0/67730700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb481a0500 con 0x7ffb48103b80 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.407+0000 7ffb45ffb640 1 -- 172.21.15.29:0/67730700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c018710 con 0x7ffb48103b80 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.407+0000 7ffb4ed75640 1 -- 172.21.15.29:0/67730700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb481a6d20 con 0x7ffb48103b80 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb45ffb640 1 -- 172.21.15.29:0/67730700 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7ffb3c00f450 con 0x7ffb48103b80 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb552e9740 1 Processor -- start 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb552e9740 1 -- start start 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c35df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c35e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb552e9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55900c2d5c10 con 0x55900bfb7f60 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb552e9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55900c2c5040 con 0x55900bf8f0a0 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb552e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55900c2c8e50 con 0x55900bf97280 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb4d312640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c35df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb4d312640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c35df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33632/0 (socket says 172.21.15.29:33632) 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb4d312640 1 -- 172.21.15.29:0/1245805006 learned_addr learned my addr 172.21.15.29:0/1245805006 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb47fff640 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c35e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.410+0000 7ffb45ffb640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb380b65d0 0x7ffb380b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4d312640 1 -- 172.21.15.29:0/1245805006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 msgr2=0x55900c35e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4d312640 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c35e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4d312640 1 -- 172.21.15.29:0/1245805006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 msgr2=0x55900c35e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4d312640 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4d312640 1 -- 172.21.15.29:0/1245805006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55900c023540 con 0x55900bfb7f60 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb47fff640 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c35e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4d312640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb380b65d0 0x7ffb380b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb45ffb640 1 -- 172.21.15.29:0/67730700 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffb3c0e7a80 con 0x7ffb48103b80 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4d312640 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c35df00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffb40005cc0 tx=0x7ffb40031670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9a429e68af106b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb32ffd640 1 -- 172.21.15.29:0/1245805006 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40031dc0 con 0x55900bfb7f60 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb32ffd640 1 -- 172.21.15.29:0/1245805006 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb40036040 con 0x55900bfb7f60 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb4d312640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb380b65d0 0x7ffb380b8ae0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffb4004c000 tx=0x7ffb40049000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.411+0000 7ffb32ffd640 1 -- 172.21.15.29:0/1245805006 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40030800 con 0x55900bfb7f60 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 -- 172.21.15.29:0/1245805006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 msgr2=0x55900c35df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c35df00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffb40005cc0 tx=0x7ffb40031670 comp rx=0 tx=0).stop 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 -- 172.21.15.29:0/1245805006 shutdown_connections 2024-09-21T02:39:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c35e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1245805006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c35df00 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 -- 172.21.15.29:0/1245805006 >> 172.21.15.29:0/1245805006 conn(0x55900bfac350 msgr2=0x55900c35bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 -- 172.21.15.29:0/1245805006 shutdown_connections 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.412+0000 7ffb552e9740 1 -- 172.21.15.29:0/1245805006 wait complete. 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 1 -- start start 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 1 Processor -- start 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf8f0a0 0x55900c369a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c369f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55900c1265f0 con 0x55900bfb7f60 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55900c260010 con 0x55900bf97280 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55900c36c560 con 0x55900bf8f0a0 2024-09-21T02:39:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb552e9740 10 client.0 _ll_register_callbacks cb 0x55900c2bf340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb4d312640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb4d312640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33648/0 (socket says 172.21.15.29:33648) 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.413+0000 7ffb4d312640 1 -- 172.21.15.29:0/3520062342 learned_addr learned my addr 172.21.15.29:0/3520062342 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf8f0a0 0x55900c369a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb47fff640 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c369f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb4d312640 1 -- 172.21.15.29:0/3520062342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf8f0a0 msgr2=0x55900c369a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb4d312640 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf8f0a0 0x55900c369a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb4d312640 1 -- 172.21.15.29:0/3520062342 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 msgr2=0x55900c369f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb4d312640 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c369f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb4d312640 1 -- 172.21.15.29:0/3520062342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55900c28da00 con 0x55900bfb7f60 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb4d312640 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375850 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffb4002a900 tx=0x7ffb40009410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1311d0b1a126d3f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb27fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb27fff640 1 -- 172.21.15.29:0/3520062342 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40009800 con 0x55900bfb7f60 2024-09-21T02:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb27fff640 1 -- 172.21.15.29:0/3520062342 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffb4004e050 con 0x55900bfb7f60 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb552e9740 1 -- 172.21.15.29:0/3520062342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb40036040 con 0x55900bfb7f60 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb27fff640 1 -- 172.21.15.29:0/3520062342 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffb40030cf0 con 0x55900bfb7f60 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb552e9740 10 client.15630 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.414+0000 7ffb552e9740 1 -- 172.21.15.29:0/3520062342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55900c3784d0 con 0x55900bfb7f60 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb27fff640 1 -- 172.21.15.29:0/3520062342 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb40036040 con 0x55900bfb7f60 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb27fff640 1 -- 172.21.15.29:0/3520062342 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ffb40052710 con 0x55900bfb7f60 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb552e9740 10 client.15630 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb552e9740 10 client.15630 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb552e9740 20 client.15630 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb552e9740 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x55900bf768e0 0x55900c37bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb552e9740 4 client.15630 mds_command: new command op to 24632 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb552e9740 1 -- 172.21.15.29:0/3520062342 --> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55900c2d7b60 con 0x55900bf768e0 2024-09-21T02:39:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x55900bf768e0 0x55900c37bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb27fff640 10 client.15630 ms_handle_connect on v2:172.21.15.66:6838/3618856137 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.415+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x55900bf768e0 0x55900c37bbc0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa46b12d851d5282 server_cookie=9f3d22e5d08a375e in_seq=0 out_seq=0 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.416+0000 7ffb27fff640 1 -- 172.21.15.29:0/3520062342 <== mds.1 v2:172.21.15.66:6838/3618856137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55900c2d7b60 con 0x55900bf768e0 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.416+0000 7ffb27fff640 10 client.15630 handle_command_reply: tid=0 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.416+0000 7ffb552e9740 1 client.15630 shutdown 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 -- 172.21.15.29:0/3520062342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 msgr2=0x55900c375850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375850 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffb4002a900 tx=0x7ffb40009410 comp rx=0 tx=0).stop 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 -- 172.21.15.29:0/3520062342 shutdown_connections 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x55900bf768e0 0x55900c37bbc0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375850 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c369f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb27fff640 0 client.15630 ms_handle_reset on v2:172.21.15.66:6838/3618856137 2024-09-21T02:39:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 --2- 172.21.15.29:0/3520062342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf8f0a0 0x55900c369a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 -- 172.21.15.29:0/3520062342 >> 172.21.15.29:0/3520062342 conn(0x55900bfac350 msgr2=0x55900c3610e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 -- 172.21.15.29:0/3520062342 shutdown_connections 2024-09-21T02:39:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 1 -- 172.21.15.29:0/3520062342 wait complete. 2024-09-21T02:39:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.417+0000 7ffb552e9740 20 client.15630 trim_cache size 0 max 16384 2024-09-21T02:39:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb552e9740 1 Processor -- start 2024-09-21T02:39:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb552e9740 1 -- start start 2024-09-21T02:39:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c3757c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb552e9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55900c379f00 con 0x55900bfb7f60 2024-09-21T02:39:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb552e9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55900c37a5d0 con 0x55900bf8f0a0 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb552e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55900c37aca0 con 0x55900bf97280 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb4d312640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb4d312640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33650/0 (socket says 172.21.15.29:33650) 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb4d312640 1 -- 172.21.15.29:0/2209053437 learned_addr learned my addr 172.21.15.29:0/2209053437 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.436+0000 7ffb47fff640 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c3757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb4d312640 1 -- 172.21.15.29:0/2209053437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 msgr2=0x55900c3757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb4d312640 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c3757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb4d312640 1 -- 172.21.15.29:0/2209053437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 msgr2=0x55900c35f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb4d312640 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb4d312640 1 -- 172.21.15.29:0/2209053437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55900c3601f0 con 0x55900bfb7f60 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb4d312640 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375d00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ffb400309f0 tx=0x7ffb40009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201b4caf40225be server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb327fc640 1 -- 172.21.15.29:0/2209053437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40009800 con 0x55900bfb7f60 2024-09-21T02:39:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb327fc640 1 -- 172.21.15.29:0/2209053437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb40036030 con 0x55900bfb7f60 2024-09-21T02:39:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.437+0000 7ffb327fc640 1 -- 172.21.15.29:0/2209053437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40004480 con 0x55900bfb7f60 2024-09-21T02:39:25.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 -- 172.21.15.29:0/2209053437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 msgr2=0x55900c375d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375d00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ffb400309f0 tx=0x7ffb40009380 comp rx=0 tx=0).stop 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 -- 172.21.15.29:0/2209053437 shutdown_connections 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bfb7f60 0x55900c375d00 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bf97280 0x55900c3757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 --2- 172.21.15.29:0/2209053437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf8f0a0 0x55900c35f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 -- 172.21.15.29:0/2209053437 >> 172.21.15.29:0/2209053437 conn(0x55900bfac350 msgr2=0x55900c369a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 -- 172.21.15.29:0/2209053437 shutdown_connections 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 -- 172.21.15.29:0/2209053437 wait complete. 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.438+0000 7ffb552e9740 1 -- start start 2024-09-21T02:39:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb552e9740 1 Processor -- start 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bf8f0a0 0x55900c3842f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c37d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb552e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bfb7f60 0x55900c37dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb552e9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55900c37ba40 con 0x55900bf8f0a0 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb552e9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55900c37b370 con 0x55900bf97280 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb552e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55900c37c110 con 0x55900bfb7f60 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb552e9740 10 client.0 _ll_register_callbacks cb 0x7ffb4004b020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb4cb11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bf8f0a0 0x55900c3842f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb4cb11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bf8f0a0 0x55900c3842f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33656/0 (socket says 172.21.15.29:33656) 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb4cb11640 1 -- 172.21.15.29:0/1567113269 learned_addr learned my addr 172.21.15.29:0/1567113269 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb4d312640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bfb7f60 0x55900c37dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb47fff640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c37d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb4cb11640 1 -- 172.21.15.29:0/1567113269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bfb7f60 msgr2=0x55900c37dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bfb7f60 0x55900c37dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb4cb11640 1 -- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 msgr2=0x55900c37d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.439+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c37d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb4cb11640 1 -- 172.21.15.29:0/1567113269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55900c37e270 con 0x55900bf8f0a0 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb47fff640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c37d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb4d312640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bfb7f60 0x55900c37dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb4cb11640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bf8f0a0 0x55900c3842f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c01f000 tx=0x7ffb3c00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba74bb0021721c18 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb27fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb27fff640 1 -- 172.21.15.29:0/1567113269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c0ed070 con 0x55900bf8f0a0 2024-09-21T02:39:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb27fff640 1 -- 172.21.15.29:0/1567113269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffb3c0a99a0 con 0x55900bf8f0a0 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb552e9740 1 -- 172.21.15.29:0/1567113269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb40036030 con 0x55900bf8f0a0 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb27fff640 1 -- 172.21.15.29:0/1567113269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffb3c0b2210 con 0x55900bf8f0a0 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb552e9740 10 client.15636 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb552e9740 1 -- 172.21.15.29:0/1567113269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55900c383b70 con 0x55900bf8f0a0 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb27fff640 1 -- 172.21.15.29:0/1567113269 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb40036030 con 0x55900bf8f0a0 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb27fff640 1 -- 172.21.15.29:0/1567113269 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ffb3c0a9080 con 0x55900bf8f0a0 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.440+0000 7ffb552e9740 10 client.15636 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.441+0000 7ffb552e9740 10 client.15636 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.441+0000 7ffb552e9740 20 client.15636 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.441+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x55900bf4bc90 0x55900c3ea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.441+0000 7ffb552e9740 4 client.15636 mds_command: new command op to 24632 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:39:25.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.441+0000 7ffb552e9740 1 -- 172.21.15.29:0/1567113269 --> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] -- command(tid 0: {"prefix": "status"}) -- 0x55900c2d3610 con 0x55900bf4bc90 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.441+0000 7ffb47fff640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x55900bf4bc90 0x55900c3ea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.441+0000 7ffb47fff640 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x55900bf4bc90 0x55900c3ea600 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=303d07dd2cac185 server_cookie=8b8454890b84e541 in_seq=0 out_seq=0 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.441+0000 7ffb27fff640 10 client.15636 ms_handle_connect on v2:172.21.15.66:6838/3618856137 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb27fff640 1 -- 172.21.15.29:0/1567113269 <== mds.1 v2:172.21.15.66:6838/3618856137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55900c2d3610 con 0x55900bf4bc90 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb27fff640 10 client.15636 handle_command_reply: tid=0 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 client.15636 shutdown 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 -- 172.21.15.29:0/1567113269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bf8f0a0 msgr2=0x55900c3842f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bf8f0a0 0x55900c3842f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c01f000 tx=0x7ffb3c00fd40 comp rx=0 tx=0).stop 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 -- 172.21.15.29:0/1567113269 shutdown_connections 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:6838/3618856137,v1:172.21.15.66:6839/3618856137] conn(0x55900bf4bc90 0x55900c3ea600 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55900bfb7f60 0x55900c37dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55900bf97280 0x55900c37d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 --2- 172.21.15.29:0/1567113269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55900bf8f0a0 0x55900c3842f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 -- 172.21.15.29:0/1567113269 >> 172.21.15.29:0/1567113269 conn(0x55900bfac350 msgr2=0x55900c35b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb27fff640 0 client.15636 ms_handle_reset on v2:172.21.15.66:6838/3618856137 2024-09-21T02:39:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 -- 172.21.15.29:0/1567113269 shutdown_connections 2024-09-21T02:39:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 1 -- 172.21.15.29:0/1567113269 wait complete. 2024-09-21T02:39:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.442+0000 7ffb552e9740 20 client.15636 trim_cache size 0 max 16384 2024-09-21T02:39:25.449 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:39:25.449 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24632, 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 139.43470861500001, 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 189.80622421999999 2024-09-21T02:39:25.450 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:39:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 -- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb380b65d0 msgr2=0x7ffb380b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb380b65d0 0x7ffb380b8ae0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffb4004c000 tx=0x7ffb40049000 comp rx=0 tx=0).stop 2024-09-21T02:39:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 -- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48103b80 msgr2=0x7ffb4819a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48103b80 0x7ffb4819a360 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00c920 tx=0x7ffb3c00cde0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 -- 172.21.15.29:0/67730700 shutdown_connections 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb380b65d0 0x7ffb380b8ae0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb48109e10 0x7ffb4819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb481094c0 0x7ffb4819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 --2- 172.21.15.29:0/67730700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb48103b80 0x7ffb4819a360 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 -- 172.21.15.29:0/67730700 >> 172.21.15.29:0/67730700 conn(0x7ffb480ff610 msgr2=0x7ffb48108b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 -- 172.21.15.29:0/67730700 shutdown_connections 2024-09-21T02:39:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:25.443+0000 7ffb4ed75640 1 -- 172.21.15.29:0/67730700 wait complete. 2024-09-21T02:39:25.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:25 smithi190 ceph-mon[33871]: pgmap v398: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-09-21T02:39:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:25 smithi029 ceph-mon[31006]: pgmap v398: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-09-21T02:39:25.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:25 smithi066 ceph-mon[33436]: pgmap v398: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-09-21T02:39:26.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T02:39:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:27.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.160+0000 7f7873ef7640 1 -- 172.21.15.29:0/3726414104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c103b60 msgr2=0x7f786c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.160+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3726414104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c103b60 0x7f786c103f70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7868009900 tx=0x7f786802a410 comp rx=0 tx=0).stop 2024-09-21T02:39:27.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.160+0000 7f7873ef7640 1 -- 172.21.15.29:0/3726414104 shutdown_connections 2024-09-21T02:39:27.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.160+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3726414104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f786c109df0 0x7f786c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.160+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3726414104 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f786c1094a0 0x7f786c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.160+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3726414104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c103b60 0x7f786c103f70 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.160+0000 7f7873ef7640 1 -- 172.21.15.29:0/3726414104 >> 172.21.15.29:0/3726414104 conn(0x7f786c07b850 msgr2=0x7f786c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.161+0000 7f7873ef7640 1 -- 172.21.15.29:0/3726414104 shutdown_connections 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.161+0000 7f7873ef7640 1 -- 172.21.15.29:0/3726414104 wait complete. 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.161+0000 7f7873ef7640 1 Processor -- start 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.161+0000 7f7873ef7640 1 -- start start 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.161+0000 7f7873ef7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f786c103b60 0x7f786c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.161+0000 7f7873ef7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c1094a0 0x7f786c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7873ef7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f786c109df0 0x7f786c1a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7873ef7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f786c113960 con 0x7f786c1094a0 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7873ef7640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f786c1137e0 con 0x7f786c109df0 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7873ef7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f786c113ae0 con 0x7f786c103b60 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c1094a0 0x7f786c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c1094a0 0x7f786c19cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33686/0 (socket says 172.21.15.29:33686) 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 -- 172.21.15.29:0/3045604454 learned_addr learned my addr 172.21.15.29:0/3045604454 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871c93640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f786c103b60 0x7f786c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7872494640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f786c109df0 0x7f786c1a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 -- 172.21.15.29:0/3045604454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f786c103b60 msgr2=0x7f786c19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f786c103b60 0x7f786c19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 -- 172.21.15.29:0/3045604454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f786c109df0 msgr2=0x7f786c1a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f786c109df0 0x7f786c1a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 -- 172.21.15.29:0/3045604454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c1a2600 con 0x7f786c1094a0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7871492640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c1094a0 0x7f786c19cc30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f785c00c970 tx=0x7f785c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7862ffd640 1 -- 172.21.15.29:0/3045604454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c007bb0 con 0x7f786c1094a0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7862ffd640 1 -- 172.21.15.29:0/3045604454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f785c007d50 con 0x7f786c1094a0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7873ef7640 1 -- 172.21.15.29:0/3045604454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786c1a28f0 con 0x7f786c1094a0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.162+0000 7f7862ffd640 1 -- 172.21.15.29:0/3045604454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c004da0 con 0x7f786c1094a0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.163+0000 7f7873ef7640 1 -- 172.21.15.29:0/3045604454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786c1a9050 con 0x7f786c1094a0 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.165+0000 7f787a46b740 1 Processor -- start 2024-09-21T02:39:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.165+0000 7f787a46b740 1 -- start start 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.165+0000 7f7862ffd640 1 -- 172.21.15.29:0/3045604454 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f785c004f80 con 0x7f786c1094a0 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f787a46b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 0x55a39fdd0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f787a46b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa020a0 0x55a39fdd1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f787a46b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdd1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f787a46b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a39fd48c10 con 0x55a39fa0a280 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f787a46b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a39fd38040 con 0x55a39fa020a0 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f787a46b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a39fd3be50 con 0x55a39fa2af60 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871492640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdd1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871492640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdd1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33696/0 (socket says 172.21.15.29:33696) 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871492640 1 -- 172.21.15.29:0/3739671476 learned_addr learned my addr 172.21.15.29:0/3739671476 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7872494640 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 0x55a39fdd0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871c93640 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa020a0 0x55a39fdd1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871492640 1 -- 172.21.15.29:0/3739671476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 msgr2=0x55a39fdd0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871492640 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 0x55a39fdd0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7862ffd640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f78580b65a0 0x7f78580b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871492640 1 -- 172.21.15.29:0/3739671476 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa020a0 msgr2=0x55a39fdd1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871492640 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa020a0 0x55a39fdd1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871492640 1 -- 172.21.15.29:0/3739671476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a39fa96540 con 0x55a39fa0a280 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7871c93640 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa020a0 0x55a39fdd1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.166+0000 7f7872494640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f78580b65a0 0x7f78580b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.167+0000 7f7862ffd640 1 -- 172.21.15.29:0/3045604454 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f785c0e74d0 con 0x7f786c1094a0 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.167+0000 7f7871492640 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdd1980 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f785c0e4a40 tx=0x7f785c0ad420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8d16db717ca89e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.167+0000 7f7857fff640 1 -- 172.21.15.29:0/3739671476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c01d070 con 0x55a39fa0a280 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.167+0000 7f7857fff640 1 -- 172.21.15.29:0/3739671476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f785c0046c0 con 0x55a39fa0a280 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.167+0000 7f7857fff640 1 -- 172.21.15.29:0/3739671476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c0ac9a0 con 0x55a39fa0a280 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 -- 172.21.15.29:0/3739671476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 msgr2=0x55a39fdd1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f7872494640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f78580b65a0 0x7f78580b8ab0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f786400aeb0 tx=0x7f786400a4d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdd1980 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f785c0e4a40 tx=0x7f785c0ad420 comp rx=0 tx=0).stop 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 -- 172.21.15.29:0/3739671476 shutdown_connections 2024-09-21T02:39:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdd1980 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa020a0 0x55a39fdd1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 --2- 172.21.15.29:0/3739671476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 0x55a39fdd0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 -- 172.21.15.29:0/3739671476 >> 172.21.15.29:0/3739671476 conn(0x55a39fa1f350 msgr2=0x55a39fdcedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 -- 172.21.15.29:0/3739671476 shutdown_connections 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 -- 172.21.15.29:0/3739671476 wait complete. 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 -- start start 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.168+0000 7f787a46b740 1 Processor -- start 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f787a46b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fddc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f787a46b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa0a280 0x55a39fddcf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f787a46b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 0x55a39fde89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f787a46b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a39fb995f0 con 0x55a39fa020a0 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f787a46b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a39fcd3010 con 0x55a39fa2af60 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f787a46b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a39fddf6e0 con 0x55a39fa0a280 2024-09-21T02:39:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871c93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fddc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871c93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fddc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33712/0 (socket says 172.21.15.29:33712) 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f787a46b740 10 client.0 _ll_register_callbacks cb 0x55a39fd32340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871c93640 1 -- 172.21.15.29:0/509636991 learned_addr learned my addr 172.21.15.29:0/509636991 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871492640 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa0a280 0x55a39fddcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7872494640 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 0x55a39fde89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871c93640 1 -- 172.21.15.29:0/509636991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa0a280 msgr2=0x55a39fddcf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871c93640 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa0a280 0x55a39fddcf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871c93640 1 -- 172.21.15.29:0/509636991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 msgr2=0x55a39fde89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871c93640 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 0x55a39fde89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.169+0000 7f7871c93640 1 -- 172.21.15.29:0/509636991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a39fd00a00 con 0x55a39fa020a0 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f7871c93640 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fddc9f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f786802a930 tx=0x7f7868031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c273b9156d35d26 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f78557fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f78557fa640 1 -- 172.21.15.29:0/509636991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868035070 con 0x55a39fa020a0 2024-09-21T02:39:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f78557fa640 1 -- 172.21.15.29:0/509636991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7868004240 con 0x55a39fa020a0 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 1 -- 172.21.15.29:0/509636991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a39fd4ab60 con 0x55a39fa020a0 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f78557fa640 1 -- 172.21.15.29:0/509636991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f786804d050 con 0x55a39fa020a0 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 10 client.15648 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 1 -- 172.21.15.29:0/509636991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a39fdeb620 con 0x55a39fa020a0 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f78557fa640 1 -- 172.21.15.29:0/509636991 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55a39fd4ab60 con 0x55a39fa020a0 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f78557fa640 1 -- 172.21.15.29:0/509636991 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f786803b430 con 0x55a39fa020a0 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 10 client.15648 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 10 client.15648 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 20 client.15648 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a39f9e98e0 0x55a39fdef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 4 client.15648 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.170+0000 7f787a46b740 1 -- 172.21.15.29:0/509636991 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a39fd46610 con 0x55a39f9e98e0 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.171+0000 7f7871492640 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a39f9e98e0 0x55a39fdef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.171+0000 7f7871492640 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a39f9e98e0 0x55a39fdef0f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7553bfb4c550c9ea server_cookie=98dc081047bd2e9c in_seq=0 out_seq=0 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.171+0000 7f78557fa640 10 client.15648 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f78557fa640 1 -- 172.21.15.29:0/509636991 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a39fd46610 con 0x55a39f9e98e0 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f78557fa640 10 client.15648 handle_command_reply: tid=0 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 client.15648 shutdown 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 -- 172.21.15.29:0/509636991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 msgr2=0x55a39fddc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fddc9f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f786802a930 tx=0x7f7868031400 comp rx=0 tx=0).stop 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 -- 172.21.15.29:0/509636991 shutdown_connections 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 0x55a39fde89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa0a280 0x55a39fddcf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a39f9e98e0 0x55a39fdef0f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 --2- 172.21.15.29:0/509636991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fddc9f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 -- 172.21.15.29:0/509636991 >> 172.21.15.29:0/509636991 conn(0x55a39fa1f350 msgr2=0x55a39fdd4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f78557fa640 0 client.15648 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 -- 172.21.15.29:0/509636991 shutdown_connections 2024-09-21T02:39:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 1 -- 172.21.15.29:0/509636991 wait complete. 2024-09-21T02:39:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.172+0000 7f787a46b740 20 client.15648 trim_cache size 0 max 16384 2024-09-21T02:39:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f787a46b740 1 Processor -- start 2024-09-21T02:39:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f787a46b740 1 -- start start 2024-09-21T02:39:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f787a46b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fde8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f787a46b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa0a280 0x55a39fde8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f787a46b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 0x55a39fdd79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f787a46b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a39fded430 con 0x55a39fa020a0 2024-09-21T02:39:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f787a46b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a39fdedb00 con 0x55a39fa0a280 2024-09-21T02:39:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f787a46b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a39fdee1d0 con 0x55a39fa2af60 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.191+0000 7f7871c93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fde8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871c93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fde8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33716/0 (socket says 172.21.15.29:33716) 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871c93640 1 -- 172.21.15.29:0/2118738191 learned_addr learned my addr 172.21.15.29:0/2118738191 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871492640 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa0a280 0x55a39fde8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7872494640 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 0x55a39fdd79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871c93640 1 -- 172.21.15.29:0/2118738191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 msgr2=0x55a39fdd79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871c93640 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 0x55a39fdd79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871c93640 1 -- 172.21.15.29:0/2118738191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa0a280 msgr2=0x55a39fde8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871c93640 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa0a280 0x55a39fde8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871c93640 1 -- 172.21.15.29:0/2118738191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a39fdddab0 con 0x55a39fa020a0 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f7871c93640 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fde8940 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f78680057c0 tx=0x7f786802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7b3dc5cc3e994a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f78577fe640 1 -- 172.21.15.29:0/2118738191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868056070 con 0x55a39fa020a0 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f78577fe640 1 -- 172.21.15.29:0/2118738191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f786803ba40 con 0x55a39fa020a0 2024-09-21T02:39:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.192+0000 7f78577fe640 1 -- 172.21.15.29:0/2118738191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868035040 con 0x55a39fa020a0 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 -- 172.21.15.29:0/2118738191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 msgr2=0x55a39fde8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fde8940 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f78680057c0 tx=0x7f786802a410 comp rx=0 tx=0).stop 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 -- 172.21.15.29:0/2118738191 shutdown_connections 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa2af60 0x55a39fdd79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa0a280 0x55a39fde8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 --2- 172.21.15.29:0/2118738191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa020a0 0x55a39fde8940 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 -- 172.21.15.29:0/2118738191 >> 172.21.15.29:0/2118738191 conn(0x55a39fa1f350 msgr2=0x55a39fdc9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 -- 172.21.15.29:0/2118738191 shutdown_connections 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.193+0000 7f787a46b740 1 -- 172.21.15.29:0/2118738191 wait complete. 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 1 -- start start 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 1 Processor -- start 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa020a0 0x55a39fdf7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdf0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 0x55a39fdf1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a39fdeef70 con 0x55a39fa0a280 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a39fdee8a0 con 0x55a39fa2af60 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a39fdef640 con 0x55a39fa020a0 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f787a46b740 10 client.0 _ll_register_callbacks cb 0x7f7868005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f7871492640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdf0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f7871492640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdf0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33726/0 (socket says 172.21.15.29:33726) 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f7871492640 1 -- 172.21.15.29:0/1935792512 learned_addr learned my addr 172.21.15.29:0/1935792512 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.194+0000 7f7872494640 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 0x55a39fdf1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f7871c93640 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa020a0 0x55a39fdf7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f7871492640 1 -- 172.21.15.29:0/1935792512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa020a0 msgr2=0x55a39fdf7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f7871492640 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa020a0 0x55a39fdf7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f7871492640 1 -- 172.21.15.29:0/1935792512 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 msgr2=0x55a39fdf1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:27.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f7871492640 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 0x55a39fdf1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f7871492640 1 -- 172.21.15.29:0/1935792512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a39fdf1b80 con 0x55a39fa0a280 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f7871492640 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdf0f50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f785c0aa4a0 tx=0x7f785c0acb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1615dcf798fd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f78557fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f78557fa640 1 -- 172.21.15.29:0/1935792512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c01d040 con 0x55a39fa0a280 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f78557fa640 1 -- 172.21.15.29:0/1935792512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f785c005270 con 0x55a39fa0a280 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f787a46b740 1 -- 172.21.15.29:0/1935792512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f786803ba40 con 0x55a39fa0a280 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f78557fa640 1 -- 172.21.15.29:0/1935792512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f785c0ac9a0 con 0x55a39fa0a280 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f787a46b740 10 client.15654 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f787a46b740 1 -- 172.21.15.29:0/1935792512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a39fe5a5c0 con 0x55a39fa0a280 2024-09-21T02:39:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.195+0000 7f78557fa640 1 -- 172.21.15.29:0/1935792512 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f786803ba40 con 0x55a39fa0a280 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f78557fa640 1 -- 172.21.15.29:0/1935792512 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f785c082ce0 con 0x55a39fa0a280 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f787a46b740 10 client.15654 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f787a46b740 10 client.15654 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f787a46b740 20 client.15654 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f787a46b740 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a39f9bec90 0x55a39fe5d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f787a46b740 4 client.15654 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f787a46b740 1 -- 172.21.15.29:0/1935792512 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "status"}) -- 0x55a39fd43a40 con 0x55a39f9bec90 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f7871c93640 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a39f9bec90 0x55a39fe5d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f78557fa640 10 client.15654 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.196+0000 7f7871c93640 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a39f9bec90 0x55a39fe5d880 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63423ba395ab95d2 server_cookie=61bca7ff4a81b935 in_seq=0 out_seq=0 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f78557fa640 1 -- 172.21.15.29:0/1935792512 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a39fd43a40 con 0x55a39f9bec90 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f78557fa640 10 client.15654 handle_command_reply: tid=0 2024-09-21T02:39:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 client.15654 shutdown 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 -- 172.21.15.29:0/1935792512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 msgr2=0x55a39fdf0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdf0f50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f785c0aa4a0 tx=0x7f785c0acb80 comp rx=0 tx=0).stop 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 -- 172.21.15.29:0/1935792512 shutdown_connections 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a39fa2af60 0x55a39fdf1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a39f9bec90 0x55a39fe5d880 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a39fa0a280 0x55a39fdf0f50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 --2- 172.21.15.29:0/1935792512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a39fa020a0 0x55a39fdf7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f78557fa640 0 client.15654 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 -- 172.21.15.29:0/1935792512 >> 172.21.15.29:0/1935792512 conn(0x55a39fa1f350 msgr2=0x55a39fdd79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 -- 172.21.15.29:0/1935792512 shutdown_connections 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 1 -- 172.21.15.29:0/1935792512 wait complete. 2024-09-21T02:39:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.197+0000 7f787a46b740 20 client.15654 trim_cache size 0 max 16384 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "e1049baa-77c0-11ef-baf5-efdc52797490", 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24637, 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 180.462504908, 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 200.90277323999999 2024-09-21T02:39:27.207 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:39:27.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 -- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f78580b65a0 msgr2=0x7f78580b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f78580b65a0 0x7f78580b8ab0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f786400aeb0 tx=0x7f786400a4d0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 -- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c1094a0 msgr2=0x7f786c19cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:27.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c1094a0 0x7f786c19cc30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f785c00c970 tx=0x7f785c00ce30 comp rx=0 tx=0).stop 2024-09-21T02:39:27.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 -- 172.21.15.29:0/3045604454 shutdown_connections 2024-09-21T02:39:27.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f78580b65a0 0x7f78580b8ab0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f786c109df0 0x7f786c1a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786c1094a0 0x7f786c19cc30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 --2- 172.21.15.29:0/3045604454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f786c103b60 0x7f786c19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:27.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 -- 172.21.15.29:0/3045604454 >> 172.21.15.29:0/3045604454 conn(0x7f786c07b850 msgr2=0x7f786c1073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:27.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 -- 172.21.15.29:0/3045604454 shutdown_connections 2024-09-21T02:39:27.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:27.198+0000 7f7873ef7640 1 -- 172.21.15.29:0/3045604454 wait complete. 2024-09-21T02:39:27.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:27 smithi190 ceph-mon[33871]: pgmap v399: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-21T02:39:27.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:27 smithi029 ceph-mon[31006]: pgmap v399: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-21T02:39:27.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:27 smithi066 ceph-mon[33436]: pgmap v399: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-21T02:39:28.146 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-21T02:39:29.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:29 smithi190 ceph-mon[33871]: pgmap v400: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-09-21T02:39:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:39:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:29 smithi029 ceph-mon[31006]: pgmap v400: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-09-21T02:39:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:39:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:29 smithi066 ceph-mon[33436]: pgmap v400: 129 pgs: 129 active+clean; 667 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-09-21T02:39:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:39:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:30.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:39:30.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:39:30.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:39:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:39:30.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:30.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:39:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:39:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:39:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.057+0000 7fb75183f640 1 -- 172.21.15.29:0/765997049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c1025d0 msgr2=0x7fb74c102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.057+0000 7fb75183f640 1 --2- 172.21.15.29:0/765997049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c1025d0 0x7fb74c102a40 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009900 tx=0x7fb73c02a410 comp rx=0 tx=0).stop 2024-09-21T02:39:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.058+0000 7fb75183f640 1 -- 172.21.15.29:0/765997049 shutdown_connections 2024-09-21T02:39:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.058+0000 7fb75183f640 1 --2- 172.21.15.29:0/765997049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c103ff0 0x7fb74c10a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.058+0000 7fb75183f640 1 --2- 172.21.15.29:0/765997049 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb74c103650 0x7fb74c103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.058+0000 7fb75183f640 1 --2- 172.21.15.29:0/765997049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c1025d0 0x7fb74c102a40 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.058+0000 7fb75183f640 1 -- 172.21.15.29:0/765997049 >> 172.21.15.29:0/765997049 conn(0x7fb74c0fdf30 msgr2=0x7fb74c1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.058+0000 7fb75183f640 1 -- 172.21.15.29:0/765997049 shutdown_connections 2024-09-21T02:39:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.058+0000 7fb75183f640 1 -- 172.21.15.29:0/765997049 wait complete. 2024-09-21T02:39:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.058+0000 7fb75183f640 1 Processor -- start 2024-09-21T02:39:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb75183f640 1 -- start start 2024-09-21T02:39:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb75183f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c1025d0 0x7fb74c06c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb75183f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c103650 0x7fb74c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb75183f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb74c103ff0 0x7fb74c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb75183f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c10e0a0 con 0x7fb74c103650 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb75183f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c10df20 con 0x7fb74c103ff0 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb75183f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c10e220 con 0x7fb74c1025d0 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c103650 0x7fb74c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c103650 0x7fb74c19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33746/0 (socket says 172.21.15.29:33746) 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 -- 172.21.15.29:0/209475631 learned_addr learned my addr 172.21.15.29:0/209475631 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb74c103ff0 0x7fb74c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74affd640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c1025d0 0x7fb74c06c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 -- 172.21.15.29:0/209475631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c1025d0 msgr2=0x7fb74c06c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c1025d0 0x7fb74c06c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 -- 172.21.15.29:0/209475631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb74c103ff0 msgr2=0x7fb74c1a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb74c103ff0 0x7fb74c1a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 -- 172.21.15.29:0/209475631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74c1a1920 con 0x7fb74c103650 2024-09-21T02:39:31.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.059+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c103650 0x7fb74c19ebd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d950 tx=0x7fb74000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:31.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.060+0000 7fb75083d640 1 -- 172.21.15.29:0/209475631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb740017070 con 0x7fb74c103650 2024-09-21T02:39:31.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.060+0000 7fb75083d640 1 -- 172.21.15.29:0/209475631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb740009c30 con 0x7fb74c103650 2024-09-21T02:39:31.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.060+0000 7fb75183f640 1 -- 172.21.15.29:0/209475631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74c1a1c10 con 0x7fb74c103650 2024-09-21T02:39:31.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.060+0000 7fb75083d640 1 -- 172.21.15.29:0/209475631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74000f620 con 0x7fb74c103650 2024-09-21T02:39:31.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.060+0000 7fb75183f640 1 -- 172.21.15.29:0/209475631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74c1a21b0 con 0x7fb74c103650 2024-09-21T02:39:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.062+0000 7fb75888c740 1 Processor -- start 2024-09-21T02:39:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.062+0000 7fb75888c740 1 -- start start 2024-09-21T02:39:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.062+0000 7fb75083d640 1 -- 172.21.15.29:0/209475631 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7fb740010460 con 0x7fb74c103650 2024-09-21T02:39:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb75888c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed2de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb75888c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e95d2f0 0x56480ed2e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb75888c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed2f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb75888c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56480eca06f0 con 0x56480e9861b0 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb75888c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56480ec8fb20 con 0x56480e95d2f0 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb75888c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56480ec93930 con 0x56480e9654d0 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed2de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed2de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33750/0 (socket says 172.21.15.29:33750) 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74b7fe640 1 -- 172.21.15.29:0/3617292956 learned_addr learned my addr 172.21.15.29:0/3617292956 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed2f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74affd640 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e95d2f0 0x56480ed2e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74b7fe640 1 -- 172.21.15.29:0/3617292956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 msgr2=0x56480ed2f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed2f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74b7fe640 1 -- 172.21.15.29:0/3617292956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e95d2f0 msgr2=0x56480ed2e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e95d2f0 0x56480ed2e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74b7fe640 1 -- 172.21.15.29:0/3617292956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56480ec55530 con 0x56480e9861b0 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed2f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.063+0000 7fb75083d640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7340b6620 0x7fb7340b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.064+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed2de50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb7440039f0 tx=0x7fb744003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b728b95b48a23004 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.064+0000 7fb74affd640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7340b6620 0x7fb7340b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.064+0000 7fb75083d640 1 -- 172.21.15.29:0/209475631 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb7400e7420 con 0x7fb74c103650 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.064+0000 7fb731ffb640 1 -- 172.21.15.29:0/3617292956 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74401a470 con 0x56480e9861b0 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.064+0000 7fb731ffb640 1 -- 172.21.15.29:0/3617292956 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb744015070 con 0x56480e9861b0 2024-09-21T02:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.064+0000 7fb731ffb640 1 -- 172.21.15.29:0/3617292956 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb744010770 con 0x56480e9861b0 2024-09-21T02:39:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 -- 172.21.15.29:0/3617292956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 msgr2=0x56480ed2de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb74affd640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7340b6620 0x7fb7340b8b30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb73c000c00 tx=0x7fb73c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed2de50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb7440039f0 tx=0x7fb744003eb0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 -- 172.21.15.29:0/3617292956 shutdown_connections 2024-09-21T02:39:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed2f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e95d2f0 0x56480ed2e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 --2- 172.21.15.29:0/3617292956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed2de50 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 -- 172.21.15.29:0/3617292956 >> 172.21.15.29:0/3617292956 conn(0x56480e97a300 msgr2=0x56480ed298e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 -- 172.21.15.29:0/3617292956 shutdown_connections 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 -- 172.21.15.29:0/3617292956 wait complete. 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.065+0000 7fb75888c740 1 -- start start 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb75888c740 1 Processor -- start 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb75888c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed37630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb75888c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed37cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb75888c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed38860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb75888c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56480ebc7ab0 con 0x56480e95d2f0 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb75888c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56480ec35730 con 0x56480e9861b0 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb75888c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56480ed2fb30 con 0x56480e9654d0 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb75888c740 10 client.0 _ll_register_callbacks cb 0x56480ec896b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed37630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed37630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33766/0 (socket says 172.21.15.29:33766) 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74affd640 1 -- 172.21.15.29:0/960701673 learned_addr learned my addr 172.21.15.29:0/960701673 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed37cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74affd640 1 -- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 msgr2=0x56480ed37cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed38860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74affd640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed37cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74affd640 1 -- 172.21.15.29:0/960701673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 msgr2=0x56480ed38860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74affd640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed38860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74affd640 1 -- 172.21.15.29:0/960701673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56480e9f1790 con 0x56480e95d2f0 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed37cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.066+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed38860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb74affd640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed37630 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb73c033000 tx=0x7fb73c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1163f1ff2be1007 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb72affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb72affd640 1 -- 172.21.15.29:0/960701673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c04c030 con 0x56480e95d2f0 2024-09-21T02:39:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb72affd640 1 -- 172.21.15.29:0/960701673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb73c004310 con 0x56480e95d2f0 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 1 -- 172.21.15.29:0/960701673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb744015070 con 0x56480e95d2f0 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb72affd640 1 -- 172.21.15.29:0/960701673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb73c059050 con 0x56480e95d2f0 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 10 client.15666 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 1 -- 172.21.15.29:0/960701673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56480ea6a620 con 0x56480e95d2f0 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb72affd640 1 -- 172.21.15.29:0/960701673 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb744015070 con 0x56480e95d2f0 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb72affd640 1 -- 172.21.15.29:0/960701673 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb73c048580 con 0x56480e95d2f0 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 10 client.15666 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 10 client.15666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 20 client.15666 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56480e944d80 0x56480ed44ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 4 client.15666 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.067+0000 7fb75888c740 1 -- 172.21.15.29:0/960701673 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56480eca2640 con 0x56480e944d80 2024-09-21T02:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.068+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56480e944d80 0x56480ed44ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.068+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56480e944d80 0x56480ed44ea0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3b6e324a91caf4b server_cookie=4cb7a8f8288f14fa in_seq=0 out_seq=0 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.068+0000 7fb72affd640 10 client.15666 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb72affd640 1 -- 172.21.15.29:0/960701673 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56480eca2640 con 0x56480e944d80 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb72affd640 10 client.15666 handle_command_reply: tid=0 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 client.15666 shutdown 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 -- 172.21.15.29:0/960701673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 msgr2=0x56480ed37630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed37630 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb73c033000 tx=0x7fb73c009260 comp rx=0 tx=0).stop 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 -- 172.21.15.29:0/960701673 shutdown_connections 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed38860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed37cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56480e944d80 0x56480ed44ea0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 --2- 172.21.15.29:0/960701673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed37630 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb72affd640 0 client.15666 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 -- 172.21.15.29:0/960701673 >> 172.21.15.29:0/960701673 conn(0x56480e97a300 msgr2=0x56480ed27ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 -- 172.21.15.29:0/960701673 shutdown_connections 2024-09-21T02:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 1 -- 172.21.15.29:0/960701673 wait complete. 2024-09-21T02:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.069+0000 7fb75888c740 20 client.15666 trim_cache size 0 max 16384 2024-09-21T02:39:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.084+0000 7fb75888c740 1 Processor -- start 2024-09-21T02:39:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.084+0000 7fb75888c740 1 -- start start 2024-09-21T02:39:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.084+0000 7fb75888c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed29220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.084+0000 7fb75888c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed36830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.084+0000 7fb75888c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed28610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.084+0000 7fb75888c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c1ac430 con 0x56480e95d2f0 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.084+0000 7fb75888c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c1adf70 con 0x56480e9861b0 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.084+0000 7fb75888c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c1acb00 con 0x56480e9654d0 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed29220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed29220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33780/0 (socket says 172.21.15.29:33780) 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 -- 172.21.15.29:0/2091191215 learned_addr learned my addr 172.21.15.29:0/2091191215 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed28610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed36830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 -- 172.21.15.29:0/2091191215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 msgr2=0x56480ed36830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed36830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 -- 172.21.15.29:0/2091191215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 msgr2=0x56480ed28610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed28610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 -- 172.21.15.29:0/2091191215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56480ed357c0 con 0x56480e95d2f0 2024-09-21T02:39:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed36830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb74affd640 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed29220 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb73c04b040 tx=0x7fb73c0045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b400cea0720cd6d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb7317fa640 1 -- 172.21.15.29:0/2091191215 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c04c040 con 0x56480e95d2f0 2024-09-21T02:39:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb7317fa640 1 -- 172.21.15.29:0/2091191215 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb73c048de0 con 0x56480e95d2f0 2024-09-21T02:39:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.085+0000 7fb7317fa640 1 -- 172.21.15.29:0/2091191215 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c035050 con 0x56480e95d2f0 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 -- 172.21.15.29:0/2091191215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 msgr2=0x56480ed29220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed29220 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb73c04b040 tx=0x7fb73c0045d0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 -- 172.21.15.29:0/2091191215 shutdown_connections 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9861b0 0x56480ed28610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e9654d0 0x56480ed36830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 --2- 172.21.15.29:0/2091191215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e95d2f0 0x56480ed29220 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 -- 172.21.15.29:0/2091191215 >> 172.21.15.29:0/2091191215 conn(0x56480e97a300 msgr2=0x56480ed38e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 -- 172.21.15.29:0/2091191215 shutdown_connections 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 -- 172.21.15.29:0/2091191215 wait complete. 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.086+0000 7fb75888c740 1 -- start start 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb75888c740 1 Processor -- start 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb75888c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e95d2f0 0x56480ed3eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb75888c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9654d0 0x56480ed3f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb75888c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed3d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb75888c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c1ad8a0 con 0x56480e9861b0 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb75888c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c1ad1d0 con 0x56480e9654d0 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb75888c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c1aa410 con 0x56480e95d2f0 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb75888c740 10 client.0 _ll_register_callbacks cb 0x7fb73c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed3d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed3d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33792/0 (socket says 172.21.15.29:33792) 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74b7fe640 1 -- 172.21.15.29:0/960823214 learned_addr learned my addr 172.21.15.29:0/960823214 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74affd640 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e95d2f0 0x56480ed3eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74a7fc640 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9654d0 0x56480ed3f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74b7fe640 1 -- 172.21.15.29:0/960823214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e95d2f0 msgr2=0x56480ed3eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e95d2f0 0x56480ed3eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74b7fe640 1 -- 172.21.15.29:0/960823214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9654d0 msgr2=0x56480ed3f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9654d0 0x56480ed3f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.087+0000 7fb74b7fe640 1 -- 172.21.15.29:0/960823214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56480ed3e170 con 0x56480e9861b0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb74b7fe640 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed3d9e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb744006010 tx=0x7fb744009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b803dd1bab46080a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb72affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb72affd640 1 -- 172.21.15.29:0/960823214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb744004e30 con 0x56480e9861b0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb72affd640 1 -- 172.21.15.29:0/960823214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb744005040 con 0x56480e9861b0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb75888c740 1 -- 172.21.15.29:0/960823214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb73c048de0 con 0x56480e9861b0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb72affd640 1 -- 172.21.15.29:0/960823214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb744032050 con 0x56480e9861b0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb75888c740 10 client.15672 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb75888c740 1 -- 172.21.15.29:0/960823214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56480ed3cfd0 con 0x56480e9861b0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb72affd640 1 -- 172.21.15.29:0/960823214 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb73c048de0 con 0x56480e9861b0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb72affd640 1 -- 172.21.15.29:0/960823214 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb744024720 con 0x56480e9861b0 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb75888c740 10 client.15672 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.088+0000 7fb75888c740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:39:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.089+0000 7fb75888c740 20 client.15672 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.089+0000 7fb75888c740 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56480e91a1c0 0x56480edb1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.089+0000 7fb75888c740 4 client.15672 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:39:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.089+0000 7fb75888c740 1 -- 172.21.15.29:0/960823214 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56480ec9e0f0 con 0x56480e91a1c0 2024-09-21T02:39:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.089+0000 7fb74affd640 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56480e91a1c0 0x56480edb1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.089+0000 7fb74affd640 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56480e91a1c0 0x56480edb1b80 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=430d1ad09683bd25 server_cookie=31e408d11c6e35e3 in_seq=0 out_seq=0 2024-09-21T02:39:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.089+0000 7fb72affd640 10 client.15672 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:31.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb72affd640 1 -- 172.21.15.29:0/960823214 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56480ec9e0f0 con 0x56480e91a1c0 2024-09-21T02:39:31.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb72affd640 10 client.15672 handle_command_reply: tid=0 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 client.15672 shutdown 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 -- 172.21.15.29:0/960823214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 msgr2=0x56480ed3d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed3d9e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb744006010 tx=0x7fb744009b20 comp rx=0 tx=0).stop 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 -- 172.21.15.29:0/960823214 shutdown_connections 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56480e91a1c0 0x56480edb1b80 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56480e9861b0 0x56480ed3d9e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56480e9654d0 0x56480ed3f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb72affd640 0 client.15672 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 --2- 172.21.15.29:0/960823214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56480e95d2f0 0x56480ed3eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 -- 172.21.15.29:0/960823214 >> 172.21.15.29:0/960823214 conn(0x56480e97a300 msgr2=0x56480ed38e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 -- 172.21.15.29:0/960823214 shutdown_connections 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 1 -- 172.21.15.29:0/960823214 wait complete. 2024-09-21T02:39:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.097+0000 7fb75888c740 20 client.15672 trim_cache size 0 max 16384 2024-09-21T02:39:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:39:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:39:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:39:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:39:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 -- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7340b6620 msgr2=0x7fb7340b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7340b6620 0x7fb7340b8b30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb73c000c00 tx=0x7fb73c005d50 comp rx=0 tx=0).stop 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 -- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c103650 msgr2=0x7fb74c19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c103650 0x7fb74c19ebd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d950 tx=0x7fb74000de10 comp rx=0 tx=0).stop 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 -- 172.21.15.29:0/209475631 shutdown_connections 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7340b6620 0x7fb7340b8b30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb74c103ff0 0x7fb74c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74c103650 0x7fb74c19ebd0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 --2- 172.21.15.29:0/209475631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c1025d0 0x7fb74c06c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 -- 172.21.15.29:0/209475631 >> 172.21.15.29:0/209475631 conn(0x7fb74c0fdf30 msgr2=0x7fb74c106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 -- 172.21.15.29:0/209475631 shutdown_connections 2024-09-21T02:39:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:31.098+0000 7fb75183f640 1 -- 172.21.15.29:0/209475631 wait complete. 2024-09-21T02:39:31.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:31 smithi066 ceph-mon[33436]: pgmap v401: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.9 KiB/s wr, 16 op/s 2024-09-21T02:39:31.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:31 smithi190 ceph-mon[33871]: pgmap v401: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.9 KiB/s wr, 16 op/s 2024-09-21T02:39:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:31 smithi029 ceph-mon[31006]: pgmap v401: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.9 KiB/s wr, 16 op/s 2024-09-21T02:39:33.684 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:33 smithi029 ceph-mon[31006]: pgmap v402: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:33.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:33 smithi066 ceph-mon[33436]: pgmap v402: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:33 smithi190 ceph-mon[33871]: pgmap v402: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:35 smithi066 ceph-mon[33436]: pgmap v403: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-21T02:39:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:35 smithi190 ceph-mon[33871]: pgmap v403: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-21T02:39:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:35 smithi029 ceph-mon[31006]: pgmap v403: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-21T02:39:37.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:37 smithi066 ceph-mon[33436]: pgmap v404: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:37 smithi190 ceph-mon[33871]: pgmap v404: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:37.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:37 smithi029 ceph-mon[31006]: pgmap v404: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:39 smithi066 ceph-mon[33436]: pgmap v405: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:39 smithi190 ceph-mon[33871]: pgmap v405: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:39 smithi029 ceph-mon[31006]: pgmap v405: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T02:39:41.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:41 smithi066 ceph-mon[33436]: pgmap v406: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-21T02:39:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:41 smithi190 ceph-mon[33871]: pgmap v406: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-21T02:39:41.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:41 smithi029 ceph-mon[31006]: pgmap v406: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-21T02:39:43.147 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.5 2024-09-21T02:39:43.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:39:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:43.554 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:43 smithi029 ceph-mon[31006]: pgmap v407: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-21T02:39:43.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:43 smithi066 ceph-mon[33436]: pgmap v407: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-21T02:39:43.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:43 smithi190 ceph-mon[33871]: pgmap v407: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-21T02:39:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 -- 172.21.15.29:0/2414138883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 msgr2=0x7f29901087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 --2- 172.21.15.29:0/2414138883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 0x7f29901087b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2980009980 tx=0x7f298002e820 comp rx=0 tx=0).stop 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 -- 172.21.15.29:0/2414138883 shutdown_connections 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 --2- 172.21.15.29:0/2414138883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2990108cf0 0x7f299010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 --2- 172.21.15.29:0/2414138883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 0x7f29901087b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 --2- 172.21.15.29:0/2414138883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2990102a60 0x7f2990102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 -- 172.21.15.29:0/2414138883 >> 172.21.15.29:0/2414138883 conn(0x7f2990079730 msgr2=0x7f2990079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 -- 172.21.15.29:0/2414138883 shutdown_connections 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.979+0000 7f299824b640 1 -- 172.21.15.29:0/2414138883 wait complete. 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.980+0000 7f299824b640 1 Processor -- start 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.980+0000 7f299824b640 1 -- start start 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f299824b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2990102a60 0x7f29901981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f299824b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 0x7f2990198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f299824b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2990108cf0 0x7f299019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f299824b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2990111710 con 0x7f29901083a0 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f299824b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2990111590 con 0x7f2990108cf0 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f299824b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2990111890 con 0x7f2990102a60 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f29957e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 0x7f2990198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f29957e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 0x7f2990198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58048/0 (socket says 172.21.15.29:58048) 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f29957e6640 1 -- 172.21.15.29:0/3354337 learned_addr learned my addr 172.21.15.29:0/3354337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f2995fe7640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2990102a60 0x7f29901981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f29967e8640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2990108cf0 0x7f299019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f29957e6640 1 -- 172.21.15.29:0/3354337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2990102a60 msgr2=0x7f29901981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f29957e6640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2990102a60 0x7f29901981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f29957e6640 1 -- 172.21.15.29:0/3354337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2990108cf0 msgr2=0x7f299019da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.981+0000 7f29957e6640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2990108cf0 0x7f299019da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.982+0000 7f29957e6640 1 -- 172.21.15.29:0/3354337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299019e130 con 0x7f29901083a0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.982+0000 7f29957e6640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 0x7f2990198760 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f298002ed40 tx=0x7f29800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.982+0000 7f2986ffd640 1 -- 172.21.15.29:0/3354337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980045020 con 0x7f29901083a0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.982+0000 7f2986ffd640 1 -- 172.21.15.29:0/3354337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29800367f0 con 0x7f29901083a0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.982+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299019e3c0 con 0x7f29901083a0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.982+0000 7f2986ffd640 1 -- 172.21.15.29:0/3354337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980040820 con 0x7f29901083a0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.982+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29901a4b80 con 0x7f29901083a0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.983+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2990102e70 con 0x7f29901083a0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.985+0000 7f2986ffd640 1 -- 172.21.15.29:0/3354337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f2980048020 con 0x7f29901083a0 2024-09-21T02:39:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.986+0000 7f2986ffd640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f297c0b6620 0x7f297c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.986+0000 7f2995fe7640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f297c0b6620 0x7f297c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.986+0000 7f2986ffd640 1 -- 172.21.15.29:0/3354337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f298010b350 con 0x7f29901083a0 2024-09-21T02:39:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.987+0000 7f2995fe7640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f297c0b6620 0x7f297c0b8b30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f298c00bc40 tx=0x7f298c00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:43.991+0000 7f2986ffd640 1 -- 172.21.15.29:0/3354337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f29800d5b80 con 0x7f29901083a0 2024-09-21T02:39:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 -- 172.21.15.29:0/702499520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 msgr2=0x7f494410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 --2- 172.21.15.29:0/702499520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 0x7f494410a960 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f493c009c90 tx=0x7f493c02a860 comp rx=0 tx=0).stop 2024-09-21T02:39:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 -- 172.21.15.29:0/702499520 shutdown_connections 2024-09-21T02:39:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 --2- 172.21.15.29:0/702499520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 0x7f494410a960 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 --2- 172.21.15.29:0/702499520 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4944103620 0x7f4944103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 --2- 172.21.15.29:0/702499520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49441025a0 0x7f4944102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 -- 172.21.15.29:0/702499520 >> 172.21.15.29:0/702499520 conn(0x7f49440fdee0 msgr2=0x7f4944100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 -- 172.21.15.29:0/702499520 shutdown_connections 2024-09-21T02:39:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.201+0000 7f4949cbd640 1 -- 172.21.15.29:0/702499520 wait complete. 2024-09-21T02:39:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.202+0000 7f4949cbd640 1 Processor -- start 2024-09-21T02:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.202+0000 7f4949cbd640 1 -- start start 2024-09-21T02:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.202+0000 7f4949cbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49441025a0 0x7f494406b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4949cbd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4944103620 0x7f494419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4949cbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 0x7f49441a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4949cbd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f494410e0b0 con 0x7f4944103fc0 2024-09-21T02:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4949cbd640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f494410df30 con 0x7f4944103620 2024-09-21T02:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4949cbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f494410e230 con 0x7f49441025a0 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 0x7f49441a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 0x7f49441a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51072/0 (socket says 172.21.15.29:51072) 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 -- 172.21.15.29:0/1374916850 learned_addr learned my addr 172.21.15.29:0/1374916850 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f49437fe640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49441025a0 0x7f494406b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4942ffd640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4944103620 0x7f494419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 -- 172.21.15.29:0/1374916850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49441025a0 msgr2=0x7f494406b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49441025a0 0x7f494406b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 -- 172.21.15.29:0/1374916850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4944103620 msgr2=0x7f494419ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4944103620 0x7f494419ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 -- 172.21.15.29:0/1374916850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49441a1980 con 0x7f4944103fc0 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4943fff640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 0x7f49441a1180 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f493c0011e0 tx=0x7f493c032a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.203+0000 7f4940ff9640 1 -- 172.21.15.29:0/1374916850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c003bc0 con 0x7f4944103fc0 2024-09-21T02:39:44.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.204+0000 7f4949cbd640 1 -- 172.21.15.29:0/1374916850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49441a1c10 con 0x7f4944103fc0 2024-09-21T02:39:44.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.204+0000 7f4940ff9640 1 -- 172.21.15.29:0/1374916850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f493c040070 con 0x7f4944103fc0 2024-09-21T02:39:44.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.204+0000 7f4940ff9640 1 -- 172.21.15.29:0/1374916850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c0307c0 con 0x7f4944103fc0 2024-09-21T02:39:44.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.204+0000 7f4949cbd640 1 -- 172.21.15.29:0/1374916850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49441a2160 con 0x7f4944103fc0 2024-09-21T02:39:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.207+0000 7f4950c29740 1 Processor -- start 2024-09-21T02:39:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.207+0000 7f4940ff9640 1 -- 172.21.15.29:0/1374916850 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f493c004050 con 0x7f4944103fc0 2024-09-21T02:39:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4950c29740 1 -- start start 2024-09-21T02:39:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4950c29740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613ae1b0 0x56106175a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4950c29740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613852f0 0x56106175a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4950c29740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56106138d4d0 0x56106175b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4950c29740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5610616c86f0 con 0x5610613ae1b0 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4950c29740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5610616b7b20 con 0x5610613852f0 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4950c29740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5610616bb930 con 0x56106138d4d0 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613ae1b0 0x56106175a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613ae1b0 0x56106175a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51082/0 (socket says 172.21.15.29:51082) 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 -- 172.21.15.29:0/2047471827 learned_addr learned my addr 172.21.15.29:0/2047471827 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f49437fe640 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613852f0 0x56106175a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56106138d4d0 0x56106175b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 -- 172.21.15.29:0/2047471827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56106138d4d0 msgr2=0x56106175b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4940ff9640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f492c0b65a0 0x7f492c0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56106138d4d0 0x56106175b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 -- 172.21.15.29:0/2047471827 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613852f0 msgr2=0x56106175a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613852f0 0x56106175a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f4942ffd640 1 -- 172.21.15.29:0/2047471827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56106167d530 con 0x5610613ae1b0 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.208+0000 7f49437fe640 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613852f0 0x56106175a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.209+0000 7f4943fff640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f492c0b65a0 0x7f492c0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.209+0000 7f4940ff9640 1 -- 172.21.15.29:0/1374916850 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f493c1086c0 con 0x7f4944103fc0 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.209+0000 7f4942ffd640 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613ae1b0 0x56106175a260 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4938013610 tx=0x7f4938009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215ed7e12de32f91 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.209+0000 7f4929ffb640 1 -- 172.21.15.29:0/2047471827 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493800f600 con 0x5610613ae1b0 2024-09-21T02:39:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.209+0000 7f4929ffb640 1 -- 172.21.15.29:0/2047471827 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f493800fc30 con 0x5610613ae1b0 2024-09-21T02:39:44.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.209+0000 7f4929ffb640 1 -- 172.21.15.29:0/2047471827 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938010a60 con 0x5610613ae1b0 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 -- 172.21.15.29:0/2047471827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613ae1b0 msgr2=0x56106175a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613ae1b0 0x56106175a260 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4938013610 tx=0x7f4938009890 comp rx=0 tx=0).stop 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4943fff640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f492c0b65a0 0x7f492c0b8ab0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f493c0ca940 tx=0x7f493c041e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 -- 172.21.15.29:0/2047471827 shutdown_connections 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56106138d4d0 0x56106175b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613852f0 0x56106175a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 --2- 172.21.15.29:0/2047471827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613ae1b0 0x56106175a260 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 -- 172.21.15.29:0/2047471827 >> 172.21.15.29:0/2047471827 conn(0x5610613a2300 msgr2=0x5610617518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 -- 172.21.15.29:0/2047471827 shutdown_connections 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.210+0000 7f4950c29740 1 -- 172.21.15.29:0/2047471827 wait complete. 2024-09-21T02:39:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 1 -- start start 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 1 Processor -- start 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613852f0 0x561061763a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56106138d4d0 0x561061764100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613ae1b0 0x561061764c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5610615efab0 con 0x5610613852f0 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56106165d730 con 0x56106138d4d0 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56106175bf40 con 0x5610613ae1b0 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4942ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613852f0 0x561061763a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4950c29740 10 client.0 _ll_register_callbacks cb 0x5610616b16b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f49437fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56106138d4d0 0x561061764100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4942ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613852f0 0x561061763a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51084/0 (socket says 172.21.15.29:51084) 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.211+0000 7f4942ffd640 1 -- 172.21.15.29:0/2655043730 learned_addr learned my addr 172.21.15.29:0/2655043730 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4942ffd640 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613ae1b0 0x561061764c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4942ffd640 1 -- 172.21.15.29:0/2655043730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613ae1b0 msgr2=0x561061764c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4942ffd640 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613ae1b0 0x561061764c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4942ffd640 1 -- 172.21.15.29:0/2655043730 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56106138d4d0 msgr2=0x561061764100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4942ffd640 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56106138d4d0 0x561061764100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4942ffd640 1 -- 172.21.15.29:0/2655043730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561061419790 con 0x5610613852f0 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f49437fe640 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56106138d4d0 0x561061764100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4942ffd640 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613852f0 0x561061763a40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f49380057c0 tx=0x7f49380056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e489a53dff5c3a7b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4922ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4922ffd640 1 -- 172.21.15.29:0/2655043730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938029020 con 0x5610613852f0 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4922ffd640 1 -- 172.21.15.29:0/2655043730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4938010bb0 con 0x5610613852f0 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4950c29740 1 -- 172.21.15.29:0/2655043730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f493800fc30 con 0x5610613852f0 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4922ffd640 1 -- 172.21.15.29:0/2655043730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4938034050 con 0x5610613852f0 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4950c29740 10 client.15690 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.212+0000 7f4950c29740 1 -- 172.21.15.29:0/2655043730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561061492620 con 0x5610613852f0 2024-09-21T02:39:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.213+0000 7f4922ffd640 1 -- 172.21.15.29:0/2655043730 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f493800fc30 con 0x5610613852f0 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.213+0000 7f4922ffd640 1 -- 172.21.15.29:0/2655043730 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f493801faf0 con 0x5610613852f0 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.213+0000 7f4950c29740 10 client.15690 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.213+0000 7f4950c29740 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.213+0000 7f4950c29740 20 client.15690 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.213+0000 7f4950c29740 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56106136cd80 0x561061771010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.213+0000 7f4950c29740 4 client.15690 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.213+0000 7f4950c29740 1 -- 172.21.15.29:0/2655043730 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610616ca640 con 0x56106136cd80 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.214+0000 7f49437fe640 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56106136cd80 0x561061771010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.214+0000 7f49437fe640 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56106136cd80 0x561061771010 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e09a8033fc4cc63 server_cookie=f4092ffbbe399f4d in_seq=0 out_seq=0 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.214+0000 7f4922ffd640 10 client.15690 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4922ffd640 1 -- 172.21.15.29:0/2655043730 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5610616ca640 con 0x56106136cd80 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4922ffd640 10 client.15690 handle_command_reply: tid=0 2024-09-21T02:39:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 client.15690 shutdown 2024-09-21T02:39:44.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 -- 172.21.15.29:0/2655043730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613852f0 msgr2=0x561061763a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613852f0 0x561061763a40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f49380057c0 tx=0x7f49380056d0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 -- 172.21.15.29:0/2655043730 shutdown_connections 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613ae1b0 0x561061764c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56106138d4d0 0x561061764100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56106136cd80 0x561061771010 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 --2- 172.21.15.29:0/2655043730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610613852f0 0x561061763a40 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 -- 172.21.15.29:0/2655043730 >> 172.21.15.29:0/2655043730 conn(0x5610613a2300 msgr2=0x56106174fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4922ffd640 0 client.15690 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.215+0000 7f4950c29740 1 -- 172.21.15.29:0/2655043730 shutdown_connections 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.216+0000 7f4950c29740 1 -- 172.21.15.29:0/2655043730 wait complete. 2024-09-21T02:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.216+0000 7f4950c29740 20 client.15690 trim_cache size 0 max 16384 2024-09-21T02:39:44.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.235+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f299010ab20 con 0x7f29901083a0 2024-09-21T02:39:44.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.235+0000 7f2986ffd640 1 -- 172.21.15.29:0/3354337 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11971 (secure 0 0 0) 0x7f29800daac0 con 0x7f29901083a0 2024-09-21T02:39:44.237 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:44.237 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-09-21T02:39:04:269980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:39:04.253002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:39:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-09-21T02:39:44.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f297c0b6620 msgr2=0x7f297c0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f297c0b6620 0x7f297c0b8b30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f298c00bc40 tx=0x7f298c00d040 comp rx=0 tx=0).stop 2024-09-21T02:39:44.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 msgr2=0x7f2990198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 0x7f2990198760 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f298002ed40 tx=0x7f29800042f0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 shutdown_connections 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f297c0b6620 0x7f297c0b8b30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2990108cf0 0x7f299019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29901083a0 0x7f2990198760 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 --2- 172.21.15.29:0/3354337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2990102a60 0x7f29901981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 >> 172.21.15.29:0/3354337 conn(0x7f2990079730 msgr2=0x7f2990104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 shutdown_connections 2024-09-21T02:39:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:44.238+0000 7f299824b640 1 -- 172.21.15.29:0/3354337 wait complete. 2024-09-21T02:39:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.243+0000 7f4950c29740 1 Processor -- start 2024-09-21T02:39:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.243+0000 7f4950c29740 1 -- start start 2024-09-21T02:39:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.243+0000 7f4950c29740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061762c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.243+0000 7f4950c29740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061750610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.243+0000 7f4950c29740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 0x561061750b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.243+0000 7f4950c29740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f49441aa200 con 0x56106138d4d0 2024-09-21T02:39:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.243+0000 7f4950c29740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f49441abd40 con 0x5610613ae1b0 2024-09-21T02:39:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.243+0000 7f4950c29740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f49441aa8d0 con 0x5610613852f0 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061750610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061750610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51094/0 (socket says 172.21.15.29:51094) 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 -- 172.21.15.29:0/1582468386 learned_addr learned my addr 172.21.15.29:0/1582468386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f4942ffd640 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061762c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f4942ffd640 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 0x561061750b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 -- 172.21.15.29:0/1582468386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 msgr2=0x561061762c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061762c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 -- 172.21.15.29:0/1582468386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 msgr2=0x561061750b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 0x561061750b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 -- 172.21.15.29:0/1582468386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56106176bf10 con 0x56106138d4d0 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f4942ffd640 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061762c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49437fe640 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061750610 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4934012470 tx=0x7f4934005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5acb3f023fffe1f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49297fa640 1 -- 172.21.15.29:0/1582468386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493402b070 con 0x56106138d4d0 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49297fa640 1 -- 172.21.15.29:0/1582468386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f493400fd30 con 0x56106138d4d0 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.244+0000 7f49297fa640 1 -- 172.21.15.29:0/1582468386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4934012e60 con 0x56106138d4d0 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 -- 172.21.15.29:0/1582468386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 msgr2=0x561061750610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061750610 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4934012470 tx=0x7f4934005f90 comp rx=0 tx=0).stop 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 -- 172.21.15.29:0/1582468386 shutdown_connections 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 0x561061750b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061750610 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 --2- 172.21.15.29:0/1582468386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061762c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 -- 172.21.15.29:0/1582468386 >> 172.21.15.29:0/1582468386 conn(0x5610613a2300 msgr2=0x561061751290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 -- 172.21.15.29:0/1582468386 shutdown_connections 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.245+0000 7f4950c29740 1 -- 172.21.15.29:0/1582468386 wait complete. 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 1 -- start start 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 1 Processor -- start 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061768720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061768de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 0x561061769890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f49441ab670 con 0x56106138d4d0 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f49441aafa0 con 0x5610613ae1b0 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f49441a81e0 con 0x5610613852f0 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.246+0000 7f4950c29740 10 client.0 _ll_register_callbacks cb 0x7f49380134f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061768de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061768de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51106/0 (socket says 172.21.15.29:51106) 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 -- 172.21.15.29:0/439782776 learned_addr learned my addr 172.21.15.29:0/439782776 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f4942ffd640 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061768720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f4942ffd640 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 0x561061769890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 -- 172.21.15.29:0/439782776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 msgr2=0x561061768720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061768720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 -- 172.21.15.29:0/439782776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 msgr2=0x561061769890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 0x561061769890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 -- 172.21.15.29:0/439782776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561061769ff0 con 0x56106138d4d0 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f49437fe640 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061768de0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4934000c00 tx=0x7f4934009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6f6301abddb9fc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f4922ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.247+0000 7f4922ffd640 1 -- 172.21.15.29:0/439782776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493402b040 con 0x56106138d4d0 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4922ffd640 1 -- 172.21.15.29:0/439782776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4934010430 con 0x56106138d4d0 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 1 -- 172.21.15.29:0/439782776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f493400fd30 con 0x56106138d4d0 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4922ffd640 1 -- 172.21.15.29:0/439782776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f493403c050 con 0x56106138d4d0 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 10 client.15696 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 1 -- 172.21.15.29:0/439782776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56106176b310 con 0x56106138d4d0 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4922ffd640 1 -- 172.21.15.29:0/439782776 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f493400fd30 con 0x56106138d4d0 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4922ffd640 1 -- 172.21.15.29:0/439782776 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f493402e070 con 0x56106138d4d0 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 10 client.15696 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 20 client.15696 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5610613421c0 0x5610617dd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 4 client.15696 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:39:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.248+0000 7f4950c29740 1 -- 172.21.15.29:0/439782776 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5610616c60f0 con 0x5610613421c0 2024-09-21T02:39:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.249+0000 7f4942ffd640 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5610613421c0 0x5610617dd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.249+0000 7f4922ffd640 10 client.15696 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.249+0000 7f4942ffd640 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5610613421c0 0x5610617dd9a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6352c9e3933f7463 server_cookie=e18245edd974789a in_seq=0 out_seq=0 2024-09-21T02:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.255+0000 7f4922ffd640 1 -- 172.21.15.29:0/439782776 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5610616c60f0 con 0x5610613421c0 2024-09-21T02:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.255+0000 7f4922ffd640 10 client.15696 handle_command_reply: tid=0 2024-09-21T02:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.255+0000 7f4950c29740 1 client.15696 shutdown 2024-09-21T02:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 -- 172.21.15.29:0/439782776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 msgr2=0x561061768de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061768de0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4934000c00 tx=0x7f4934009e00 comp rx=0 tx=0).stop 2024-09-21T02:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 -- 172.21.15.29:0/439782776 shutdown_connections 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610613ae1b0 0x561061769890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5610613421c0 0x5610617dd9a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56106138d4d0 0x561061768de0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 --2- 172.21.15.29:0/439782776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610613852f0 0x561061768720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4922ffd640 0 client.15696 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 -- 172.21.15.29:0/439782776 >> 172.21.15.29:0/439782776 conn(0x5610613a2300 msgr2=0x561061751220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 -- 172.21.15.29:0/439782776 shutdown_connections 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 1 -- 172.21.15.29:0/439782776 wait complete. 2024-09-21T02:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4950c29740 20 client.15696 trim_cache size 0 max 16384 2024-09-21T02:39:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:39:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:39:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:39:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.256+0000 7f4949cbd640 1 -- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f492c0b65a0 msgr2=0x7f492c0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f492c0b65a0 0x7f492c0b8ab0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f493c0ca940 tx=0x7f493c041e70 comp rx=0 tx=0).stop 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 -- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 msgr2=0x7f49441a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 0x7f49441a1180 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f493c0011e0 tx=0x7f493c032a90 comp rx=0 tx=0).stop 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 -- 172.21.15.29:0/1374916850 shutdown_connections 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f492c0b65a0 0x7f492c0b8ab0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944103fc0 0x7f49441a1180 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4944103620 0x7f494419ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 --2- 172.21.15.29:0/1374916850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49441025a0 0x7f494406b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 -- 172.21.15.29:0/1374916850 >> 172.21.15.29:0/1374916850 conn(0x7f49440fdee0 msgr2=0x7f4944106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 -- 172.21.15.29:0/1374916850 shutdown_connections 2024-09-21T02:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:44.257+0000 7f4949cbd640 1 -- 172.21.15.29:0/1374916850 wait complete. 2024-09-21T02:39:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:44 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3354337' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:44.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:44 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3354337' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:44.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:44 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3354337' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:45.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:45 smithi066 ceph-mon[33436]: pgmap v408: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-21T02:39:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:45 smithi190 ceph-mon[33871]: pgmap v408: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-21T02:39:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:45 smithi029 ceph-mon[31006]: pgmap v408: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-21T02:39:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:47 smithi066 ceph-mon[33436]: pgmap v409: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T02:39:47.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:47 smithi190 ceph-mon[33871]: pgmap v409: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T02:39:47.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:47 smithi029 ceph-mon[31006]: pgmap v409: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T02:39:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:49 smithi066 ceph-mon[33436]: pgmap v410: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:39:49.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:49 smithi190 ceph-mon[33871]: pgmap v410: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:39:49.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:49 smithi029 ceph-mon[31006]: pgmap v410: 129 pgs: 129 active+clean; 677 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T02:39:51.676 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:51 smithi190 ceph-mon[33871]: pgmap v411: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:39:51.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:51 smithi066 ceph-mon[33436]: pgmap v411: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:39:51.794 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:51 smithi029 ceph-mon[31006]: pgmap v411: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:39:51.823 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 134.5 --await-for 45.0' 2024-09-21T02:39:51.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 134.5 --await-for 45.0 2024-09-21T02:39:52.112 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:52.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 -- 172.21.15.29:0/25512717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea41094c0 msgr2=0x7fcea41098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 --2- 172.21.15.29:0/25512717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea41094c0 0x7fcea41098d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fce90009900 tx=0x7fce9002a410 comp rx=0 tx=0).stop 2024-09-21T02:39:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 -- 172.21.15.29:0/25512717 shutdown_connections 2024-09-21T02:39:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 --2- 172.21.15.29:0/25512717 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea4109e10 0x7fcea4110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 --2- 172.21.15.29:0/25512717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea41094c0 0x7fcea41098d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 --2- 172.21.15.29:0/25512717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcea4101f80 0x7fcea4102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 -- 172.21.15.29:0/25512717 >> 172.21.15.29:0/25512717 conn(0x7fcea4079f50 msgr2=0x7fcea407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 -- 172.21.15.29:0/25512717 shutdown_connections 2024-09-21T02:39:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.677+0000 7fceaae1c640 1 -- 172.21.15.29:0/25512717 wait complete. 2024-09-21T02:39:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.678+0000 7fceaae1c640 1 Processor -- start 2024-09-21T02:39:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.678+0000 7fceaae1c640 1 -- start start 2024-09-21T02:39:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.678+0000 7fceaae1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea4101f80 0x7fcea419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.678+0000 7fceaae1c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea41094c0 0x7fcea419a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.678+0000 7fceaae1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcea4109e10 0x7fcea419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.678+0000 7fceaae1c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcea4113980 con 0x7fcea4101f80 2024-09-21T02:39:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.678+0000 7fceaae1c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcea4113800 con 0x7fcea41094c0 2024-09-21T02:39:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.678+0000 7fceaae1c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcea4113b00 con 0x7fcea4109e10 2024-09-21T02:39:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea41094c0 0x7fcea419a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea41094c0 0x7fcea419a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:55150/0 (socket says 172.21.15.29:55150) 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 -- 172.21.15.29:0/4280472605 learned_addr learned my addr 172.21.15.29:0/4280472605 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea93b9640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcea4109e10 0x7fcea419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 -- 172.21.15.29:0/4280472605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcea4109e10 msgr2=0x7fcea419fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcea4109e10 0x7fcea419fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 -- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea4101f80 msgr2=0x7fcea419a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea4101f80 0x7fcea419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 -- 172.21.15.29:0/4280472605 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea41a0380 con 0x7fcea41094c0 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.679+0000 7fcea3fff640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea41094c0 0x7fcea419a9b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fce90004770 tx=0x7fce90004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.680+0000 7fcea1ffb640 1 -- 172.21.15.29:0/4280472605 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9003f030 con 0x7fcea41094c0 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.680+0000 7fcea1ffb640 1 -- 172.21.15.29:0/4280472605 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce90004440 con 0x7fcea41094c0 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.680+0000 7fceaae1c640 1 -- 172.21.15.29:0/4280472605 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea41a0610 con 0x7fcea41094c0 2024-09-21T02:39:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.680+0000 7fcea1ffb640 1 -- 172.21.15.29:0/4280472605 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce90035070 con 0x7fcea41094c0 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.680+0000 7fceaae1c640 1 -- 172.21.15.29:0/4280472605 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea41a6dd0 con 0x7fcea41094c0 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.683+0000 7fcea1ffb640 1 -- 172.21.15.29:0/4280472605 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7fce90042030 con 0x7fcea41094c0 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.683+0000 7fceb1390740 1 Processor -- start 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fceb1390740 1 -- start start 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fceb1390740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa8b840 0x563cfae32600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fcea1ffb640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce940b65a0 0x7fce940b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fceb1390740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae32b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fceb1390740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae33080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fceb1390740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563cfada8b90 con 0x563cfaa6ab60 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fceb1390740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x563cfad97fc0 con 0x563cfaa8b840 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fceb1390740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563cfad9bdd0 con 0x563cfaa62980 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fcea1ffb640 1 -- 172.21.15.29:0/4280472605 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fce90107c50 con 0x7fcea41094c0 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce940b65a0 0x7fce940b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae32b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae32b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:59980/0 (socket says 172.21.15.29:59980) 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.684+0000 7fcea3fff640 1 -- 172.21.15.29:0/3334118865 learned_addr learned my addr 172.21.15.29:0/3334118865 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.685+0000 7fcea93b9640 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae33080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.685+0000 7fcea3fff640 1 -- 172.21.15.29:0/3334118865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa8b840 msgr2=0x563cfae32600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.685+0000 7fcea3fff640 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa8b840 0x563cfae32600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.685+0000 7fcea3fff640 1 -- 172.21.15.29:0/3334118865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 msgr2=0x563cfae33080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.685+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce940b65a0 0x7fce940b8ab0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fce98005fd0 tx=0x7fce98005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.685+0000 7fcea3fff640 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae33080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.685+0000 7fcea3fff640 1 -- 172.21.15.29:0/3334118865 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cfad1f060 con 0x563cfaa62980 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.685+0000 7fcea93b9640 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae33080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.686+0000 7fcea3fff640 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae32b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce90006010 tx=0x7fce900cd1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b39fb29f2493c466 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.686+0000 7fce8effd640 1 -- 172.21.15.29:0/3334118865 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce90049040 con 0x563cfaa62980 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.686+0000 7fce8effd640 1 -- 172.21.15.29:0/3334118865 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce900cd3b0 con 0x563cfaa62980 2024-09-21T02:39:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.686+0000 7fce8effd640 1 -- 172.21.15.29:0/3334118865 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce900f15a0 con 0x563cfaa62980 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 -- 172.21.15.29:0/3334118865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 msgr2=0x563cfae32b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae32b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce90006010 tx=0x7fce900cd1f0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 -- 172.21.15.29:0/3334118865 shutdown_connections 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae33080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae32b40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 --2- 172.21.15.29:0/3334118865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa8b840 0x563cfae32600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 -- 172.21.15.29:0/3334118865 >> 172.21.15.29:0/3334118865 conn(0x563cfaa7f990 msgr2=0x563cfae304f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 -- 172.21.15.29:0/3334118865 shutdown_connections 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.687+0000 7fceb1390740 1 -- 172.21.15.29:0/3334118865 wait complete. 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.688+0000 7fceb1390740 1 -- start start 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.688+0000 7fceb1390740 1 Processor -- start 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.688+0000 7fceb1390740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae42680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.688+0000 7fceb1390740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae42c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.688+0000 7fceb1390740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.688+0000 7fceb1390740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563cfabf97a0 con 0x563cfaa6ab60 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fceb1390740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x563cfad3d0a0 con 0x563cfaa62980 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fceb1390740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563cfad9bdd0 con 0x563cfaa8b840 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fceb1390740 10 client.0 _ll_register_callbacks cb 0x563cfad1a3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae42c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae42c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51126/0 (socket says 172.21.15.29:51126) 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea93b9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea3fff640 1 -- 172.21.15.29:0/2582224975 learned_addr learned my addr 172.21.15.29:0/2582224975 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae42680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea93b9640 1 -- 172.21.15.29:0/2582224975 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 msgr2=0x563cfae42680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea93b9640 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae42680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea93b9640 1 -- 172.21.15.29:0/2582224975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 msgr2=0x563cfae42c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea93b9640 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae42c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea93b9640 1 -- 172.21.15.29:0/2582224975 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cfad2b880 con 0x563cfaa8b840 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.689+0000 7fcea3fff640 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae42c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fcea93b9640 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4e080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0108e0 tx=0x7fce9c010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de1d04d68de0df79 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fce83fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fce83fff640 1 -- 172.21.15.29:0/2582224975 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c003c40 con 0x563cfaa8b840 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fceb1390740 1 -- 172.21.15.29:0/2582224975 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce900cd3b0 con 0x563cfaa8b840 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fce83fff640 1 -- 172.21.15.29:0/2582224975 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fce9c003de0 con 0x563cfaa8b840 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fce83fff640 1 -- 172.21.15.29:0/2582224975 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce9c00f8a0 con 0x563cfaa8b840 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fceb1390740 10 client.25360 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fceb1390740 1 -- 172.21.15.29:0/2582224975 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563cfab05b90 con 0x563cfaa8b840 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fce83fff640 1 -- 172.21.15.29:0/2582224975 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce900cd3b0 con 0x563cfaa8b840 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.690+0000 7fce83fff640 1 -- 172.21.15.29:0/2582224975 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fce9c01a420 con 0x563cfaa8b840 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.691+0000 7fceb1390740 10 client.25360 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.691+0000 7fceb1390740 10 client.25360 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:39:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.691+0000 7fceb1390740 20 client.25360 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.691+0000 7fceb1390740 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563cfaa4a1c0 0x563cfae541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.691+0000 7fceb1390740 4 client.25360 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.691+0000 7fceb1390740 1 -- 172.21.15.29:0/2582224975 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cfadaaae0 con 0x563cfaa4a1c0 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.691+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563cfaa4a1c0 0x563cfae541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.692+0000 7fce83fff640 10 client.25360 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.692+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563cfaa4a1c0 0x563cfae541e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e09b7d8a7e39031 server_cookie=7986dc3423ac4994 in_seq=0 out_seq=0 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.692+0000 7fce83fff640 1 -- 172.21.15.29:0/2582224975 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563cfadaaae0 con 0x563cfaa4a1c0 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.692+0000 7fce83fff640 10 client.25360 handle_command_reply: tid=0 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 client.25360 shutdown 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 -- 172.21.15.29:0/2582224975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 msgr2=0x563cfae4e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4e080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0108e0 tx=0x7fce9c010ce0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 -- 172.21.15.29:0/2582224975 shutdown_connections 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4e080 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563cfaa4a1c0 0x563cfae541e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae42c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fce83fff640 0 client.25360 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 --2- 172.21.15.29:0/2582224975 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae42680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 -- 172.21.15.29:0/2582224975 >> 172.21.15.29:0/2582224975 conn(0x563cfaa7f990 msgr2=0x563cfae2ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 -- 172.21.15.29:0/2582224975 shutdown_connections 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 1 -- 172.21.15.29:0/2582224975 wait complete. 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.693+0000 7fceb1390740 20 client.25360 trim_cache size 0 max 16384 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.726+0000 7fceb1390740 1 Processor -- start 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.726+0000 7fceb1390740 1 -- start start 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.726+0000 7fceb1390740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae2ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.726+0000 7fceb1390740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae2efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fceb1390740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fcea8bb8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae2ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fcea8bb8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae2ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:55172/0 (socket says 172.21.15.29:55172) 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae2efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae2efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51128/0 (socket says 172.21.15.29:51128) 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fcea8bb8640 1 -- 172.21.15.29:0/4135407902 learned_addr learned my addr 172.21.15.29:0/4135407902 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fceb1390740 1 -- 172.21.15.29:0/4135407902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcea41ae7b0 con 0x563cfaa6ab60 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fcea93b9640 1 --2- 172.21.15.29:0/4135407902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fceb1390740 1 -- 172.21.15.29:0/4135407902 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcea41b02f0 con 0x563cfaa62980 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.727+0000 7fceb1390740 1 -- 172.21.15.29:0/4135407902 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcea41aee80 con 0x563cfaa8b840 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fcea3fff640 1 -- 172.21.15.29:0/4135407902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 msgr2=0x563cfae4df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fcea3fff640 1 --2- 172.21.15.29:0/4135407902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fcea3fff640 1 -- 172.21.15.29:0/4135407902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 msgr2=0x563cfae2ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fcea3fff640 1 --2- 172.21.15.29:0/4135407902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae2ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fcea3fff640 1 -- 172.21.15.29:0/4135407902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cfae30a30 con 0x563cfaa6ab60 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fcea3fff640 1 --2- 172.21.15.29:0/4135407902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae2efd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fce9003d040 tx=0x7fce900cd320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb18410cfe4f41dc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fce8e7fc640 1 -- 172.21.15.29:0/4135407902 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce90049070 con 0x563cfaa6ab60 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fce8e7fc640 1 -- 172.21.15.29:0/4135407902 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce900cd3b0 con 0x563cfaa6ab60 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.728+0000 7fce8e7fc640 1 -- 172.21.15.29:0/4135407902 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce90105470 con 0x563cfaa6ab60 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 -- 172.21.15.29:0/4135407902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 msgr2=0x563cfae2efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 --2- 172.21.15.29:0/4135407902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae2efd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fce9003d040 tx=0x7fce900cd320 comp rx=0 tx=0).stop 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 -- 172.21.15.29:0/4135407902 shutdown_connections 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 --2- 172.21.15.29:0/4135407902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa8b840 0x563cfae4df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 --2- 172.21.15.29:0/4135407902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa6ab60 0x563cfae2efd0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 --2- 172.21.15.29:0/4135407902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa62980 0x563cfae2ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 -- 172.21.15.29:0/4135407902 >> 172.21.15.29:0/4135407902 conn(0x563cfaa7f990 msgr2=0x563cfae2fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 -- 172.21.15.29:0/4135407902 shutdown_connections 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 -- 172.21.15.29:0/4135407902 wait complete. 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 -- start start 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.730+0000 7fceb1390740 1 Processor -- start 2024-09-21T02:39:52.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fceb1390740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae4a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fceb1390740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa6ab60 0x563cfae46bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fceb1390740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa8b840 0x563cfae470f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fceb1390740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcea41afc20 con 0x563cfaa8b840 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fceb1390740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcea41af550 con 0x563cfaa6ab60 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fcea93b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa8b840 0x563cfae470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fcea93b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa8b840 0x563cfae470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51134/0 (socket says 172.21.15.29:51134) 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fcea93b9640 1 -- 172.21.15.29:0/4100317528 learned_addr learned my addr 172.21.15.29:0/4100317528 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcea41ac790 con 0x563cfaa62980 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fcea3fff640 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa6ab60 0x563cfae46bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.731+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae4a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fceb1390740 10 client.0 _ll_register_callbacks cb 0x7fce9c005ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fcea93b9640 1 -- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 msgr2=0x563cfae4a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fcea93b9640 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae4a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fcea93b9640 1 -- 172.21.15.29:0/4100317528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa6ab60 msgr2=0x563cfae46bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fcea93b9640 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa6ab60 0x563cfae46bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fcea93b9640 1 -- 172.21.15.29:0/4100317528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cfae3dcd0 con 0x563cfaa8b840 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae4a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fcea93b9640 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa8b840 0x563cfae470f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fce9c01a860 tx=0x7fce9c01a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b3a9997fb9f267 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fce83fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fce83fff640 1 -- 172.21.15.29:0/4100317528 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c02d070 con 0x563cfaa8b840 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.732+0000 7fce83fff640 1 -- 172.21.15.29:0/4100317528 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fce9c00fc50 con 0x563cfaa8b840 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.733+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce900cd3b0 con 0x563cfaa8b840 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.733+0000 7fce83fff640 1 -- 172.21.15.29:0/4100317528 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce9c00fe90 con 0x563cfaa8b840 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.733+0000 7fceb1390740 10 client.15708 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.733+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563cfaebf780 con 0x563cfaa8b840 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.733+0000 7fce83fff640 1 -- 172.21.15.29:0/4100317528 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce9c005560 con 0x563cfaa8b840 2024-09-21T02:39:52.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.733+0000 7fce83fff640 1 -- 172.21.15.29:0/4100317528 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fce9c029760 con 0x563cfaa8b840 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.734+0000 7fceb1390740 10 client.15708 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.734+0000 7fceb1390740 10 client.15708 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.734+0000 7fceb1390740 20 client.15708 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.734+0000 7fceb1390740 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563cfaa1f570 0x563cfaec1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.734+0000 7fceb1390740 4 client.15708 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 134.5, "await_for": 45.0}] 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.734+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 134.5, "await_for": 45.0}) -- 0x7fce900cd3b0 con 0x563cfaa1f570 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.735+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563cfaa1f570 0x563cfaec1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.735+0000 7fce83fff640 10 client.15708 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.735+0000 7fcea8bb8640 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563cfaa1f570 0x563cfaec1f80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4a452f98df9133a server_cookie=66a26b2ac7ccf54 in_seq=0 out_seq=0 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fce83fff640 1 -- 172.21.15.29:0/4100317528 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7fce900cd3b0 con 0x563cfaa1f570 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fce83fff640 10 client.15708 handle_command_reply: tid=0 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 client.15708 shutdown 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa8b840 msgr2=0x563cfae470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa8b840 0x563cfae470f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fce9c01a860 tx=0x7fce9c01a650 comp rx=0 tx=0).stop 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 shutdown_connections 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563cfaa1f570 0x563cfaec1f80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fce83fff640 0 client.15708 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfaa8b840 0x563cfae470f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563cfaa6ab60 0x563cfae46bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 --2- 172.21.15.29:0/4100317528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfaa62980 0x563cfae4a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.747+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 >> 172.21.15.29:0/4100317528 conn(0x563cfaa7f990 msgr2=0x563cfae2ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 shutdown_connections 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceb1390740 1 -- 172.21.15.29:0/4100317528 wait complete. 2024-09-21T02:39:52.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceb1390740 20 client.15708 trim_cache size 0 max 16384 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 29, 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 2, 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "b213b215": { 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 2, 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:39:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 134.5, 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:39:52.966 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 -- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce940b65a0 msgr2=0x7fce940b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce940b65a0 0x7fce940b8ab0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fce98005fd0 tx=0x7fce98005850 comp rx=0 tx=0).stop 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 -- 172.21.15.29:0/4280472605 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea41094c0 msgr2=0x7fcea419a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea41094c0 0x7fcea419a9b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fce90004770 tx=0x7fce90004040 comp rx=0 tx=0).stop 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 -- 172.21.15.29:0/4280472605 shutdown_connections 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce940b65a0 0x7fce940b8ab0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcea4109e10 0x7fcea419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea41094c0 0x7fcea419a9b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 --2- 172.21.15.29:0/4280472605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea4101f80 0x7fcea419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.748+0000 7fceaae1c640 1 -- 172.21.15.29:0/4280472605 >> 172.21.15.29:0/4280472605 conn(0x7fcea4079f50 msgr2=0x7fcea4107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:52.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.749+0000 7fceaae1c640 1 -- 172.21.15.29:0/4280472605 shutdown_connections 2024-09-21T02:39:52.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:52.749+0000 7fceaae1c640 1 -- 172.21.15.29:0/4280472605 wait complete. 2024-09-21T02:39:53.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:53 smithi029 ceph-mon[31006]: pgmap v412: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:39:53.616 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b213b215', quiesce took 10.5 seconds. Will release after: 4.0 2024-09-21T02:39:53.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:53 smithi066 ceph-mon[33436]: pgmap v412: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:39:53.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:53 smithi190 ceph-mon[33871]: pgmap v412: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:39:55.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:55 smithi066 ceph-mon[33436]: pgmap v413: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:39:55.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:55 smithi190 ceph-mon[33871]: pgmap v413: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:39:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:55 smithi029 ceph-mon[31006]: pgmap v413: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 -- 172.21.15.29:0/3540047895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 msgr2=0x7f3a10103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 --2- 172.21.15.29:0/3540047895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 0x7f3a10103a30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009900 tx=0x7f3a0002a410 comp rx=0 tx=0).stop 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 -- 172.21.15.29:0/3540047895 shutdown_connections 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 --2- 172.21.15.29:0/3540047895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a10103fc0 0x7f3a1010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 --2- 172.21.15.29:0/3540047895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 0x7f3a10103a30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 --2- 172.21.15.29:0/3540047895 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a101025a0 0x7f3a10102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 -- 172.21.15.29:0/3540047895 >> 172.21.15.29:0/3540047895 conn(0x7f3a100fdee0 msgr2=0x7f3a10100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 -- 172.21.15.29:0/3540047895 shutdown_connections 2024-09-21T02:39:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.410+0000 7f3a17a77640 1 -- 172.21.15.29:0/3540047895 wait complete. 2024-09-21T02:39:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.412+0000 7f3a17a77640 1 Processor -- start 2024-09-21T02:39:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.412+0000 7f3a17a77640 1 -- start start 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.412+0000 7f3a17a77640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a101025a0 0x7f3a10198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.412+0000 7f3a17a77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 0x7f3a101993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.412+0000 7f3a17a77640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a10103fc0 0x7f3a10199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a17a77640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1010e0b0 con 0x7f3a10103620 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a15012640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 0x7f3a101993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a15012640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 0x7f3a101993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48808/0 (socket says 172.21.15.29:48808) 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a15012640 1 -- 172.21.15.29:0/2716358236 learned_addr learned my addr 172.21.15.29:0/2716358236 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a17a77640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1010df30 con 0x7f3a10103fc0 2024-09-21T02:39:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a15813640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a101025a0 0x7f3a10198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a16014640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a10103fc0 0x7f3a10199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1010e230 con 0x7f3a101025a0 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a16014640 1 -- 172.21.15.29:0/2716358236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a101025a0 msgr2=0x7f3a10198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a16014640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a101025a0 0x7f3a10198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a16014640 1 -- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 msgr2=0x7f3a101993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a16014640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 0x7f3a101993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a16014640 1 -- 172.21.15.29:0/2716358236 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1019a7c0 con 0x7f3a10103fc0 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.413+0000 7f3a15012640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 0x7f3a101993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.414+0000 7f3a16014640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a10103fc0 0x7f3a10199fa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800d980 tx=0x7f3a0800de40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.414+0000 7f3a06ffd640 1 -- 172.21.15.29:0/2716358236 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08017070 con 0x7f3a10103fc0 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.414+0000 7f3a06ffd640 1 -- 172.21.15.29:0/2716358236 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a08003e50 con 0x7f3a10103fc0 2024-09-21T02:39:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.414+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1019aab0 con 0x7f3a10103fc0 2024-09-21T02:39:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.414+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a101a11a0 con 0x7f3a10103fc0 2024-09-21T02:39:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.415+0000 7f3a06ffd640 1 -- 172.21.15.29:0/2716358236 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a080047e0 con 0x7f3a10103fc0 2024-09-21T02:39:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.418+0000 7f3a1e9df740 1 Processor -- start 2024-09-21T02:39:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.419+0000 7f3a06ffd640 1 -- 172.21.15.29:0/2716358236 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f3a080039c0 con 0x7f3a10103fc0 2024-09-21T02:39:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.419+0000 7f3a1e9df740 1 -- start start 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.419+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e124e1b0 0x5589e15f8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.419+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.419+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.419+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5589e15686f0 con 0x5589e122d4d0 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.419+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5589e1557b20 con 0x5589e12252f0 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.419+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5589e155b930 con 0x5589e124e1b0 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15813640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15012640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15012640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48820/0 (socket says 172.21.15.29:48820) 2024-09-21T02:39:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15012640 1 -- 172.21.15.29:0/3233018596 learned_addr learned my addr 172.21.15.29:0/3233018596 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15813640 1 -- 172.21.15.29:0/3233018596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e124e1b0 msgr2=0x5589e15f8060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:39:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15813640 1 --2- 172.21.15.29:0/3233018596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e124e1b0 0x5589e15f8060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15813640 1 -- 172.21.15.29:0/3233018596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 msgr2=0x5589e15f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15813640 1 --2- 172.21.15.29:0/3233018596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15813640 1 -- 172.21.15.29:0/3233018596 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589e151d530 con 0x5589e12252f0 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.420+0000 7f3a15012640 1 --2- 172.21.15.29:0/3233018596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.421+0000 7f3a06ffd640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f39fc0b6620 0x7f39fc0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.421+0000 7f3a15813640 1 --2- 172.21.15.29:0/3233018596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f86d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009c20 tx=0x7f3a0c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89962300195176ac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.421+0000 7f3a06ffd640 1 -- 172.21.15.29:0/2716358236 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3a080e78a0 con 0x7f3a10103fc0 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.421+0000 7f39fbfff640 1 -- 172.21.15.29:0/3233018596 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004050 con 0x5589e12252f0 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.421+0000 7f3a15012640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f39fc0b6620 0x7f39fc0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.421+0000 7f39fbfff640 1 -- 172.21.15.29:0/3233018596 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a0c005020 con 0x5589e12252f0 2024-09-21T02:39:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.421+0000 7f39fbfff640 1 -- 172.21.15.29:0/3233018596 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004ba0 con 0x5589e12252f0 2024-09-21T02:39:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.422+0000 7f3a1e9df740 1 -- 172.21.15.29:0/3233018596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 msgr2=0x5589e15f86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.422+0000 7f3a15012640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f39fc0b6620 0x7f39fc0b8b30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009230 tx=0x7f3a00031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.422+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/3233018596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f86d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009c20 tx=0x7f3a0c011810 comp rx=0 tx=0).stop 2024-09-21T02:39:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.422+0000 7f3a1e9df740 1 -- 172.21.15.29:0/3233018596 shutdown_connections 2024-09-21T02:39:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.422+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/3233018596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.422+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/3233018596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f86d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.422+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/3233018596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e124e1b0 0x5589e15f8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.422+0000 7f3a1e9df740 1 -- 172.21.15.29:0/3233018596 >> 172.21.15.29:0/3233018596 conn(0x5589e1242300 msgr2=0x5589e15f18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.423+0000 7f3a1e9df740 1 -- 172.21.15.29:0/3233018596 shutdown_connections 2024-09-21T02:39:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.423+0000 7f3a1e9df740 1 -- 172.21.15.29:0/3233018596 wait complete. 2024-09-21T02:39:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.423+0000 7f3a1e9df740 1 -- start start 2024-09-21T02:39:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.423+0000 7f3a1e9df740 1 Processor -- start 2024-09-21T02:39:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e15ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e1608600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e124e1b0 0x5589e160ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5589e148fab0 con 0x5589e124e1b0 2024-09-21T02:39:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5589e14fd730 con 0x5589e122d4d0 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5589e15f9d40 con 0x5589e12252f0 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a1e9df740 10 client.0 _ll_register_callbacks cb 0x5589e15516b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a15813640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e15ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a15813640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e15ffae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34804/0 (socket says 172.21.15.29:34804) 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a15813640 1 -- 172.21.15.29:0/1200516660 learned_addr learned my addr 172.21.15.29:0/1200516660 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a15012640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e1608600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.424+0000 7f3a16014640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e124e1b0 0x5589e160ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a15813640 1 -- 172.21.15.29:0/1200516660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 msgr2=0x5589e1608600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a15813640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e1608600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a15813640 1 -- 172.21.15.29:0/1200516660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e124e1b0 msgr2=0x5589e160ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a15813640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e124e1b0 0x5589e160ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a15813640 1 -- 172.21.15.29:0/1200516660 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589e12b9790 con 0x5589e12252f0 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a15012640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e1608600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a16014640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e124e1b0 0x5589e160ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:57.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a15813640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e15ffae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c01b040 tx=0x7f3a0c004280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9518b942c01cfa6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:57.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f39f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:39:57.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f39f97fa640 1 -- 172.21.15.29:0/1200516660 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c029070 con 0x5589e12252f0 2024-09-21T02:39:57.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f39f97fa640 1 -- 172.21.15.29:0/1200516660 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3a0c004920 con 0x5589e12252f0 2024-09-21T02:39:57.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.425+0000 7f3a1e9df740 1 -- 172.21.15.29:0/1200516660 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a0c005020 con 0x5589e12252f0 2024-09-21T02:39:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.432+0000 7f39f97fa640 1 -- 172.21.15.29:0/1200516660 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3a0c035050 con 0x5589e12252f0 2024-09-21T02:39:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.432+0000 7f3a1e9df740 10 client.25375 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.432+0000 7f3a1e9df740 1 -- 172.21.15.29:0/1200516660 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5589e1332620 con 0x5589e12252f0 2024-09-21T02:39:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.433+0000 7f39f97fa640 1 -- 172.21.15.29:0/1200516660 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a0c005020 con 0x5589e12252f0 2024-09-21T02:39:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.434+0000 7f39f97fa640 1 -- 172.21.15.29:0/1200516660 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3a0c023430 con 0x5589e12252f0 2024-09-21T02:39:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.434+0000 7f3a1e9df740 10 client.25375 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.434+0000 7f3a1e9df740 10 client.25375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:39:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.434+0000 7f3a1e9df740 20 client.25375 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.434+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5589e120cd80 0x5589e1610f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.434+0000 7f3a1e9df740 4 client.25375 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:39:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.434+0000 7f3a1e9df740 1 -- 172.21.15.29:0/1200516660 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589e156a640 con 0x5589e120cd80 2024-09-21T02:39:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.434+0000 7f3a15012640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5589e120cd80 0x5589e1610f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.435+0000 7f39f97fa640 10 client.25375 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.435+0000 7f3a15012640 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5589e120cd80 0x5589e1610f90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b394796ae30f9f53 server_cookie=90c8e329a652468d in_seq=0 out_seq=0 2024-09-21T02:39:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.435+0000 7f39f97fa640 1 -- 172.21.15.29:0/1200516660 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5589e156a640 con 0x5589e120cd80 2024-09-21T02:39:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.435+0000 7f39f97fa640 10 client.25375 handle_command_reply: tid=0 2024-09-21T02:39:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.435+0000 7f3a1e9df740 1 client.25375 shutdown 2024-09-21T02:39:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 -- 172.21.15.29:0/1200516660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 msgr2=0x5589e15ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e15ffae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c01b040 tx=0x7f3a0c004280 comp rx=0 tx=0).stop 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 -- 172.21.15.29:0/1200516660 shutdown_connections 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5589e120cd80 0x5589e1610f90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e124e1b0 0x5589e160ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f39f97fa640 0 client.25375 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e1608600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/1200516660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e15ffae0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 -- 172.21.15.29:0/1200516660 >> 172.21.15.29:0/1200516660 conn(0x5589e1242300 msgr2=0x5589e15efea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 -- 172.21.15.29:0/1200516660 shutdown_connections 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 1 -- 172.21.15.29:0/1200516660 wait complete. 2024-09-21T02:39:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.436+0000 7f3a1e9df740 20 client.25375 trim_cache size 0 max 16384 2024-09-21T02:39:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a1e9df740 1 Processor -- start 2024-09-21T02:39:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a1e9df740 1 -- start start 2024-09-21T02:39:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e120cd80 0x5589e15f3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101aa310 con 0x5589e122d4d0 2024-09-21T02:39:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101abe50 con 0x5589e12252f0 2024-09-21T02:39:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a1e9df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101aa9e0 con 0x5589e120cd80 2024-09-21T02:39:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a15813640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e120cd80 0x5589e15f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a15012640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a16014640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a15012640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:40082/0 (socket says 172.21.15.29:40082) 2024-09-21T02:39:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.466+0000 7f3a15012640 1 -- 172.21.15.29:0/4035888992 learned_addr learned my addr 172.21.15.29:0/4035888992 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f3a15012640 1 -- 172.21.15.29:0/4035888992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e120cd80 msgr2=0x5589e15f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f3a15012640 1 --2- 172.21.15.29:0/4035888992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e120cd80 0x5589e15f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f3a15012640 1 -- 172.21.15.29:0/4035888992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 msgr2=0x5589e15f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f3a15012640 1 --2- 172.21.15.29:0/4035888992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f3a15012640 1 -- 172.21.15.29:0/4035888992 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589e160b070 con 0x5589e12252f0 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f3a15012640 1 --2- 172.21.15.29:0/4035888992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f0750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a000319b0 tx=0x7f3a00044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2184678cf14379ca server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f39fb7fe640 1 -- 172.21.15.29:0/4035888992 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00035050 con 0x5589e12252f0 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f39fb7fe640 1 -- 172.21.15.29:0/4035888992 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a00003d00 con 0x5589e12252f0 2024-09-21T02:39:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.467+0000 7f39fb7fe640 1 -- 172.21.15.29:0/4035888992 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0004abf0 con 0x5589e12252f0 2024-09-21T02:39:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.468+0000 7f3a1e9df740 1 -- 172.21.15.29:0/4035888992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 msgr2=0x5589e15f0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.468+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/4035888992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f0750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a000319b0 tx=0x7f3a00044000 comp rx=0 tx=0).stop 2024-09-21T02:39:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 -- 172.21.15.29:0/4035888992 shutdown_connections 2024-09-21T02:39:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/4035888992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e122d4d0 0x5589e15f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/4035888992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e12252f0 0x5589e15f0750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/4035888992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e120cd80 0x5589e15f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 -- 172.21.15.29:0/4035888992 >> 172.21.15.29:0/4035888992 conn(0x5589e1242300 msgr2=0x5589e15ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 -- 172.21.15.29:0/4035888992 shutdown_connections 2024-09-21T02:39:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 -- 172.21.15.29:0/4035888992 wait complete. 2024-09-21T02:39:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 -- start start 2024-09-21T02:39:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.469+0000 7f3a1e9df740 1 Processor -- start 2024-09-21T02:39:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.470+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e120cd80 0x5589e16201e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.470+0000 7f3a1e9df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e16208a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.470+0000 7f3a15813640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e120cd80 0x5589e16201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.470+0000 7f3a15813640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e120cd80 0x5589e16201e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48848/0 (socket says 172.21.15.29:48848) 2024-09-21T02:39:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.470+0000 7f3a15813640 1 -- 172.21.15.29:0/2311302319 learned_addr learned my addr 172.21.15.29:0/2311302319 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.470+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e16213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101ab780 con 0x5589e120cd80 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101ab0b0 con 0x5589e122d4d0 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101a82f0 con 0x5589e12252f0 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a1e9df740 10 client.0 _ll_register_callbacks cb 0x7f3a00009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a16014640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e16213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a15813640 1 -- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 msgr2=0x5589e16208a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a15012640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e16208a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a15813640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e16208a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a15813640 1 -- 172.21.15.29:0/2311302319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 msgr2=0x5589e16213a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a15813640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e16213a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a15813640 1 -- 172.21.15.29:0/2311302319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589e1621b00 con 0x5589e120cd80 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a15012640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e16208a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:39:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a16014640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e16213a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f3a15813640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e120cd80 0x5589e16201e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x5589e15ec890 tx=0x7f3a0c005550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4c3d884789c255 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.471+0000 7f39f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f39f97fa640 1 -- 172.21.15.29:0/2311302319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c029040 con 0x5589e120cd80 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f39f97fa640 1 -- 172.21.15.29:0/2311302319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3a0c023430 con 0x5589e120cd80 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a00003d00 con 0x5589e120cd80 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f39f97fa640 1 -- 172.21.15.29:0/2311302319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3a0c023630 con 0x5589e120cd80 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f3a1e9df740 10 client.15726 fetch_fsmap learned FSMap version 29 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5589e1622e20 con 0x5589e120cd80 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f39f97fa640 1 -- 172.21.15.29:0/2311302319 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a0c005b90 con 0x5589e120cd80 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f39f97fa640 1 -- 172.21.15.29:0/2311302319 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3a0c0035f0 con 0x5589e120cd80 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f3a1e9df740 10 client.15726 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f3a1e9df740 10 client.15726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f3a1e9df740 20 client.15726 populate_metadata read hostname 'smithi029' 2024-09-21T02:39:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.472+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5589e124e1b0 0x5589e167da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:57.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.473+0000 7f3a1e9df740 4 client.15726 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:39:57.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.473+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f3a00003d00 con 0x5589e124e1b0 2024-09-21T02:39:57.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.473+0000 7f3a15012640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5589e124e1b0 0x5589e167da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:57.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.473+0000 7f39f97fa640 10 client.15726 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:57.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.473+0000 7f3a15012640 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5589e124e1b0 0x5589e167da50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2533981a27bfeef9 server_cookie=a57c3a3ae7b77b58 in_seq=0 out_seq=0 2024-09-21T02:39:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f39f97fa640 1 -- 172.21.15.29:0/2311302319 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f3a00003d00 con 0x5589e124e1b0 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f39f97fa640 10 client.15726 handle_command_reply: tid=0 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 client.15726 shutdown 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e120cd80 msgr2=0x5589e16201e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e120cd80 0x5589e16201e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x5589e15ec890 tx=0x7f3a0c005550 comp rx=0 tx=0).stop 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 shutdown_connections 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5589e122d4d0 0x5589e16213a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589e12252f0 0x5589e16208a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5589e124e1b0 0x5589e167da50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 --2- 172.21.15.29:0/2311302319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5589e120cd80 0x5589e16201e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f39f97fa640 0 client.15726 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 >> 172.21.15.29:0/2311302319 conn(0x5589e1242300 msgr2=0x5589e15ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 shutdown_connections 2024-09-21T02:39:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.480+0000 7f3a1e9df740 1 -- 172.21.15.29:0/2311302319 wait complete. 2024-09-21T02:39:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a1e9df740 20 client.15726 trim_cache size 0 max 16384 2024-09-21T02:39:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:39:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:39:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:39:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f39fc0b6620 msgr2=0x7f39fc0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f39fc0b6620 0x7f39fc0b8b30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009230 tx=0x7f3a00031400 comp rx=0 tx=0).stop 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a10103fc0 msgr2=0x7f3a10199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a10103fc0 0x7f3a10199fa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800d980 tx=0x7f3a0800de40 comp rx=0 tx=0).stop 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 shutdown_connections 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f39fc0b6620 0x7f39fc0b8b30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a10103fc0 0x7f3a10199fa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a10103620 0x7f3a101993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 --2- 172.21.15.29:0/2716358236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a101025a0 0x7f3a10198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 >> 172.21.15.29:0/2716358236 conn(0x7f3a100fdee0 msgr2=0x7f3a10106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 shutdown_connections 2024-09-21T02:39:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:39:57.481+0000 7f3a17a77640 1 -- 172.21.15.29:0/2716358236 wait complete. 2024-09-21T02:39:57.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:39:57.762 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:57 smithi066 ceph-mon[33436]: pgmap v414: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T02:39:57.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:57 smithi190 ceph-mon[33871]: pgmap v414: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T02:39:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:57 smithi029 ceph-mon[31006]: pgmap v414: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T02:39:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:39:58.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 -- 172.21.15.29:0/245184562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f90105310 msgr2=0x7f5f9010d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 --2- 172.21.15.29:0/245184562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f90105310 0x7f5f9010d860 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009900 tx=0x7f5f8402a410 comp rx=0 tx=0).stop 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 -- 172.21.15.29:0/245184562 shutdown_connections 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 --2- 172.21.15.29:0/245184562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f9010ddf0 0x7f5f90110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 --2- 172.21.15.29:0/245184562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f90105310 0x7f5f9010d860 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 --2- 172.21.15.29:0/245184562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f901049c0 0x7f5f90104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 -- 172.21.15.29:0/245184562 >> 172.21.15.29:0/245184562 conn(0x7f5f90079680 msgr2=0x7f5f90079a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 -- 172.21.15.29:0/245184562 shutdown_connections 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.550+0000 7f5f957ea640 1 -- 172.21.15.29:0/245184562 wait complete. 2024-09-21T02:39:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.551+0000 7f5f957ea640 1 Processor -- start 2024-09-21T02:39:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.551+0000 7f5f957ea640 1 -- start start 2024-09-21T02:39:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.551+0000 7f5f957ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f901049c0 0x7f5f901a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.551+0000 7f5f957ea640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f90105310 0x7f5f901a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.551+0000 7f5f957ea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f9010ddf0 0x7f5f901ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f957ea640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5f90113920 con 0x7f5f901049c0 2024-09-21T02:39:58.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f901049c0 0x7f5f901a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:58.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f957ea640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5f901137a0 con 0x7f5f90105310 2024-09-21T02:39:58.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f901049c0 0x7f5f901a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48864/0 (socket says 172.21.15.29:48864) 2024-09-21T02:39:58.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8effd640 1 -- 172.21.15.29:0/2505705832 learned_addr learned my addr 172.21.15.29:0/2505705832 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:39:58.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5f90113aa0 con 0x7f5f9010ddf0 2024-09-21T02:39:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8f7fe640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f9010ddf0 0x7f5f901ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8e7fc640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f90105310 0x7f5f901a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8effd640 1 -- 172.21.15.29:0/2505705832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f9010ddf0 msgr2=0x7f5f901ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8effd640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f9010ddf0 0x7f5f901ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8effd640 1 -- 172.21.15.29:0/2505705832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f90105310 msgr2=0x7f5f901a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8effd640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f90105310 0x7f5f901a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8effd640 1 -- 172.21.15.29:0/2505705832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f901acb80 con 0x7f5f901049c0 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.552+0000 7f5f8e7fc640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f90105310 0x7f5f901a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.553+0000 7f5f8effd640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f901049c0 0x7f5f901a15c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5f8000da70 tx=0x7f5f8000df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.553+0000 7f5f77fff640 1 -- 172.21.15.29:0/2505705832 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f80017030 con 0x7f5f901049c0 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.553+0000 7f5f77fff640 1 -- 172.21.15.29:0/2505705832 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f800099a0 con 0x7f5f901049c0 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.553+0000 7f5f77fff640 1 -- 172.21.15.29:0/2505705832 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f80005620 con 0x7f5f901049c0 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.553+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f901acdb0 con 0x7f5f901049c0 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.553+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f901ad210 con 0x7f5f901049c0 2024-09-21T02:39:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.554+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f9006c230 con 0x7f5f901049c0 2024-09-21T02:39:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.557+0000 7f5f77fff640 1 -- 172.21.15.29:0/2505705832 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f5f80004020 con 0x7f5f901049c0 2024-09-21T02:39:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.558+0000 7f5f77fff640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5f780b6620 0x7f5f780b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:39:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.558+0000 7f5f77fff640 1 -- 172.21.15.29:0/2505705832 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5f800e7810 con 0x7f5f901049c0 2024-09-21T02:39:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.558+0000 7f5f8e7fc640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5f780b6620 0x7f5f780b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:39:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.559+0000 7f5f8e7fc640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5f780b6620 0x7f5f780b8b30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5f840057c0 tx=0x7f5f84009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:39:58.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.564+0000 7f5f77fff640 1 -- 172.21.15.29:0/2505705832 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5f800adc40 con 0x7f5f901049c0 2024-09-21T02:39:58.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.864+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f9010c1a0 con 0x7f5f901049c0 2024-09-21T02:39:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.865+0000 7f5f77fff640 1 -- 172.21.15.29:0/2505705832 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11971 (secure 0 0 0) 0x7f5f9010c1a0 con 0x7f5f901049c0 2024-09-21T02:39:58.867 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:39:58.867 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-09-21T02:39:04:269980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:39:04.253002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:39:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-09-21T02:39:58.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5f780b6620 msgr2=0x7f5f780b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:58.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5f780b6620 0x7f5f780b8b30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5f840057c0 tx=0x7f5f84009680 comp rx=0 tx=0).stop 2024-09-21T02:39:58.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f901049c0 msgr2=0x7f5f901a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:39:58.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f901049c0 0x7f5f901a15c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5f8000da70 tx=0x7f5f8000df30 comp rx=0 tx=0).stop 2024-09-21T02:39:58.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 shutdown_connections 2024-09-21T02:39:58.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5f780b6620 0x7f5f780b8b30 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5f840057c0 tx=0x7f5f84009680 comp rx=0 tx=0).stop 2024-09-21T02:39:58.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f9010ddf0 0x7f5f901ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:58.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f90105310 0x7f5f901a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:58.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 --2- 172.21.15.29:0/2505705832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f901049c0 0x7f5f901a15c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:39:58.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.868+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 >> 172.21.15.29:0/2505705832 conn(0x7f5f90079680 msgr2=0x7f5f9010cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:39:58.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.869+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 shutdown_connections 2024-09-21T02:39:58.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:39:58.869+0000 7f5f957ea640 1 -- 172.21.15.29:0/2505705832 wait complete. 2024-09-21T02:39:59.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:59 smithi029 ceph-mon[31006]: pgmap v415: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:39:59.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:39:59 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2505705832' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:59.650 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id b213b215 --release --await' 2024-09-21T02:39:59.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id b213b215 --release --await 2024-09-21T02:39:59.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:59 smithi066 ceph-mon[33436]: pgmap v415: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:39:59.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:39:59 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2505705832' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:59.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:59 smithi190 ceph-mon[33871]: pgmap v415: 129 pgs: 129 active+clean; 678 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:39:59.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:39:59 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2505705832' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:39:59.934 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:40:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.582+0000 7f69e9212640 1 -- 172.21.15.29:0/1034552918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e4102a80 msgr2=0x7f69e4102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.582+0000 7f69e9212640 1 --2- 172.21.15.29:0/1034552918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e4102a80 0x7f69e4102e90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009900 tx=0x7f69d402a410 comp rx=0 tx=0).stop 2024-09-21T02:40:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.582+0000 7f69e9212640 1 -- 172.21.15.29:0/1034552918 shutdown_connections 2024-09-21T02:40:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.582+0000 7f69e9212640 1 --2- 172.21.15.29:0/1034552918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69e4108d10 0x7f69e410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.582+0000 7f69e9212640 1 --2- 172.21.15.29:0/1034552918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69e41083c0 0x7f69e41087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.582+0000 7f69e9212640 1 --2- 172.21.15.29:0/1034552918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e4102a80 0x7f69e4102e90 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.582+0000 7f69e9212640 1 -- 172.21.15.29:0/1034552918 >> 172.21.15.29:0/1034552918 conn(0x7f69e4079740 msgr2=0x7f69e4079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.583+0000 7f69e9212640 1 -- 172.21.15.29:0/1034552918 shutdown_connections 2024-09-21T02:40:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.584+0000 7f69e9212640 1 -- 172.21.15.29:0/1034552918 wait complete. 2024-09-21T02:40:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.585+0000 7f69e9212640 1 Processor -- start 2024-09-21T02:40:00.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.585+0000 7f69e9212640 1 -- start start 2024-09-21T02:40:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.585+0000 7f69e9212640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e41083c0 0x7f69e41993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.585+0000 7f69e9212640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69e4108d10 0x7f69e4199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.585+0000 7f69e9212640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69e419eb50 0x7f69e419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.585+0000 7f69e9212640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69e4112880 con 0x7f69e41083c0 2024-09-21T02:40:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.585+0000 7f69e9212640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f69e4112700 con 0x7f69e419eb50 2024-09-21T02:40:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.585+0000 7f69e9212640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f69e4112a00 con 0x7f69e4108d10 2024-09-21T02:40:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e41083c0 0x7f69e41993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e359e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69e419eb50 0x7f69e419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69e4108d10 0x7f69e4199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e359e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69e419eb50 0x7f69e419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:40130/0 (socket says 172.21.15.29:40130) 2024-09-21T02:40:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69e4108d10 0x7f69e4199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34858/0 (socket says 172.21.15.29:34858) 2024-09-21T02:40:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e359e640 1 -- 172.21.15.29:0/963570189 learned_addr learned my addr 172.21.15.29:0/963570189 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e259c640 1 -- 172.21.15.29:0/963570189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69e419eb50 msgr2=0x7f69e419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e259c640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69e419eb50 0x7f69e419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e259c640 1 -- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e41083c0 msgr2=0x7f69e41993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e259c640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e41083c0 0x7f69e41993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e259c640 1 -- 172.21.15.29:0/963570189 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e419f730 con 0x7f69e4108d10 2024-09-21T02:40:00.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.586+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e41083c0 0x7f69e41993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:40:00.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.587+0000 7f69e259c640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69e4108d10 0x7f69e4199970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f69d8005fd0 tx=0x7f69d800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:00.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.587+0000 7f69cbfff640 1 -- 172.21.15.29:0/963570189 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8009c00 con 0x7f69e4108d10 2024-09-21T02:40:00.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.587+0000 7f69cbfff640 1 -- 172.21.15.29:0/963570189 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69d80044c0 con 0x7f69e4108d10 2024-09-21T02:40:00.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.587+0000 7f69e9212640 1 -- 172.21.15.29:0/963570189 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e406b820 con 0x7f69e4108d10 2024-09-21T02:40:00.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.587+0000 7f69cbfff640 1 -- 172.21.15.29:0/963570189 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8003da0 con 0x7f69e4108d10 2024-09-21T02:40:00.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.587+0000 7f69e9212640 1 -- 172.21.15.29:0/963570189 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e406bdb0 con 0x7f69e4108d10 2024-09-21T02:40:00.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69ef786740 1 Processor -- start 2024-09-21T02:40:00.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69ef786740 1 -- start start 2024-09-21T02:40:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69ef786740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820ca520 0x557582470510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69ef786740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582470a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69ef786740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 0x557582470f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69ef786740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55758235bdc0 con 0x5575820a1660 2024-09-21T02:40:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69ef786740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5575823e81f0 con 0x5575820a9840 2024-09-21T02:40:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69ef786740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5575823d7620 con 0x5575820ca520 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582470a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.591+0000 7f69e359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820ca520 0x557582470510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69e359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820ca520 0x557582470510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34860/0 (socket says 172.21.15.29:34860) 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69e359e640 1 -- 172.21.15.29:0/1636311017 learned_addr learned my addr 172.21.15.29:0/1636311017 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69e359e640 1 -- 172.21.15.29:0/1636311017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 msgr2=0x557582470f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69e359e640 1 --2- 172.21.15.29:0/1636311017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 0x557582470f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69e359e640 1 -- 172.21.15.29:0/1636311017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 msgr2=0x557582470a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69e359e640 1 --2- 172.21.15.29:0/1636311017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582470a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69e359e640 1 -- 172.21.15.29:0/1636311017 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55758239ffd0 con 0x5575820ca520 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/1636311017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582470a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.592+0000 7f69cbfff640 1 -- 172.21.15.29:0/963570189 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) ==== 154563+0+0 (secure 0 0 0) 0x7f69d8010460 con 0x7f69e4108d10 2024-09-21T02:40:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.594+0000 7f69e359e640 1 --2- 172.21.15.29:0/1636311017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820ca520 0x557582470510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f69dc009fb0 tx=0x7f69dc011870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eae7d9c60b790853 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:00.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.594+0000 7f69cbfff640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f69cc0b6620 0x7f69cc0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.594+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f69cc0b6620 0x7f69cc0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.594+0000 7f69cbfff640 1 -- 172.21.15.29:0/963570189 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f69d80e7350 con 0x7f69e4108d10 2024-09-21T02:40:00.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.594+0000 7f69c97fa640 1 -- 172.21.15.29:0/1636311017 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc0127d0 con 0x5575820ca520 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.594+0000 7f69c97fa640 1 -- 172.21.15.29:0/1636311017 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69dc012e00 con 0x5575820ca520 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.594+0000 7f69c97fa640 1 -- 172.21.15.29:0/1636311017 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc01ad20 con 0x5575820ca520 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.595+0000 7f69ef786740 1 -- 172.21.15.29:0/1636311017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820ca520 msgr2=0x557582470510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.595+0000 7f69ef786740 1 --2- 172.21.15.29:0/1636311017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820ca520 0x557582470510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f69dc009fb0 tx=0x7f69dc011870 comp rx=0 tx=0).stop 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.595+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f69cc0b6620 0x7f69cc0b8b30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009230 tx=0x7f69d4009560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.596+0000 7f69ef786740 1 -- 172.21.15.29:0/1636311017 shutdown_connections 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.596+0000 7f69ef786740 1 --2- 172.21.15.29:0/1636311017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 0x557582470f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.596+0000 7f69ef786740 1 --2- 172.21.15.29:0/1636311017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582470a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.596+0000 7f69ef786740 1 --2- 172.21.15.29:0/1636311017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820ca520 0x557582470510 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.596+0000 7f69ef786740 1 -- 172.21.15.29:0/1636311017 >> 172.21.15.29:0/1636311017 conn(0x5575820be910 msgr2=0x55758246e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:00.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.596+0000 7f69ef786740 1 -- 172.21.15.29:0/1636311017 shutdown_connections 2024-09-21T02:40:00.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.596+0000 7f69ef786740 1 -- 172.21.15.29:0/1636311017 wait complete. 2024-09-21T02:40:00.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.597+0000 7f69ef786740 1 -- start start 2024-09-21T02:40:00.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.597+0000 7f69ef786740 1 Processor -- start 2024-09-21T02:40:00.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.597+0000 7f69ef786740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x55758247bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.597+0000 7f69ef786740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a9840 0x55758247c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.597+0000 7f69ef786740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 0x557582487e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69ef786740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557582372300 con 0x5575820a9840 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69ef786740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5575823db430 con 0x5575820ca520 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69ef786740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x557582238bb0 con 0x5575820a1660 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x55758247bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x55758247bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34864/0 (socket says 172.21.15.29:34864) 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69e2d9d640 1 -- 172.21.15.29:0/1520872084 learned_addr learned my addr 172.21.15.29:0/1520872084 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69e359e640 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 0x557582487e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69ef786740 10 client.0 _ll_register_callbacks cb 0x55758238c120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69e2d9d640 1 -- 172.21.15.29:0/1520872084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 msgr2=0x557582487e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 0x557582487e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69e2d9d640 1 -- 172.21.15.29:0/1520872084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a9840 msgr2=0x55758247c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.598+0000 7f69e259c640 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a9840 0x55758247c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a9840 0x55758247c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69e2d9d640 1 -- 172.21.15.29:0/1520872084 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557582361780 con 0x5575820a1660 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69e259c640 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a9840 0x55758247c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x55758247bc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f69d40057c0 tx=0x7f69d402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10525aa1dc2f6b39 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69c6ff5640 1 -- 172.21.15.29:0/1520872084 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d404c020 con 0x5575820a1660 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69c6ff5640 1 -- 172.21.15.29:0/1520872084 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f69d4004ad0 con 0x5575820a1660 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69ef786740 1 -- 172.21.15.29:0/1520872084 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69dc012e00 con 0x5575820a1660 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69c6ff5640 1 -- 172.21.15.29:0/1520872084 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69d4057050 con 0x5575820a1660 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69ef786740 10 client.25396 fetch_fsmap learned FSMap version 29 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.599+0000 7f69ef786740 1 -- 172.21.15.29:0/1520872084 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557582135b00 con 0x5575820a1660 2024-09-21T02:40:00.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.600+0000 7f69c6ff5640 1 -- 172.21.15.29:0/1520872084 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69d40044c0 con 0x5575820a1660 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.601+0000 7f69c6ff5640 1 -- 172.21.15.29:0/1520872084 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f69d4048480 con 0x5575820a1660 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.601+0000 7f69ef786740 10 client.25396 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.601+0000 7f69ef786740 10 client.25396 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.601+0000 7f69ef786740 20 client.25396 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.601+0000 7f69ef786740 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557582088ea0 0x55758248df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.601+0000 7f69ef786740 4 client.25396 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.601+0000 7f69ef786740 1 -- 172.21.15.29:0/1520872084 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f69dc012e00 con 0x557582088ea0 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.601+0000 7f69e259c640 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557582088ea0 0x55758248df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.602+0000 7f69c6ff5640 10 client.25396 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:00.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.602+0000 7f69e259c640 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557582088ea0 0x55758248df70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=362fcb9f1f03f3d7 server_cookie=8d277fb883c8b12a in_seq=0 out_seq=0 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69c6ff5640 1 -- 172.21.15.29:0/1520872084 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f69dc012e00 con 0x557582088ea0 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69c6ff5640 10 client.25396 handle_command_reply: tid=0 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 client.25396 shutdown 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 -- 172.21.15.29:0/1520872084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 msgr2=0x55758247bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x55758247bc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f69d40057c0 tx=0x7f69d402a410 comp rx=0 tx=0).stop 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 -- 172.21.15.29:0/1520872084 shutdown_connections 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 0x557582487e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557582088ea0 0x55758248df70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69c6ff5640 0 client.25396 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a9840 0x55758247c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 --2- 172.21.15.29:0/1520872084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x55758247bc70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.603+0000 7f69ef786740 1 -- 172.21.15.29:0/1520872084 >> 172.21.15.29:0/1520872084 conn(0x5575820be910 msgr2=0x5575824736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.604+0000 7f69ef786740 1 -- 172.21.15.29:0/1520872084 shutdown_connections 2024-09-21T02:40:00.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.604+0000 7f69ef786740 1 -- 172.21.15.29:0/1520872084 wait complete. 2024-09-21T02:40:00.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.604+0000 7f69ef786740 20 client.25396 trim_cache size 0 max 16384 2024-09-21T02:40:00.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.638+0000 7f69ef786740 1 Processor -- start 2024-09-21T02:40:00.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.638+0000 7f69ef786740 1 -- start start 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69ef786740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582487d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69ef786740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a9840 0x557582471c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69ef786740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 0x5575824755f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69ef786740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41b3250 con 0x5575820a1660 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69ef786740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41b4d90 con 0x5575820ca520 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69ef786740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41b3920 con 0x5575820a9840 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582487d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582487d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48892/0 (socket says 172.21.15.29:48892) 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69e2d9d640 1 -- 172.21.15.29:0/580303843 learned_addr learned my addr 172.21.15.29:0/580303843 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.639+0000 7f69e259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a9840 0x557582471c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69e359e640 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 0x5575824755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69e259c640 1 -- 172.21.15.29:0/580303843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 msgr2=0x5575824755f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69e259c640 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 0x5575824755f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69e259c640 1 -- 172.21.15.29:0/580303843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 msgr2=0x557582487d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69e259c640 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582487d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69e259c640 1 -- 172.21.15.29:0/580303843 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575821ac960 con 0x5575820a9840 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582487d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69e259c640 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a9840 0x557582471c50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f69d80e4980 tx=0x7f69d8005bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a5ab066067b8c2b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.640+0000 7f69c8ff9640 1 -- 172.21.15.29:0/580303843 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8020040 con 0x5575820a9840 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.641+0000 7f69c8ff9640 1 -- 172.21.15.29:0/580303843 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69d80ad700 con 0x5575820a9840 2024-09-21T02:40:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.641+0000 7f69c8ff9640 1 -- 172.21.15.29:0/580303843 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d80b4e60 con 0x5575820a9840 2024-09-21T02:40:00.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.641+0000 7f69ef786740 1 -- 172.21.15.29:0/580303843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a9840 msgr2=0x557582471c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.641+0000 7f69ef786740 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a9840 0x557582471c50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f69d80e4980 tx=0x7f69d8005bb0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 -- 172.21.15.29:0/580303843 shutdown_connections 2024-09-21T02:40:00.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820ca520 0x5575824755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a9840 0x557582471c50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 --2- 172.21.15.29:0/580303843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820a1660 0x557582487d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 -- 172.21.15.29:0/580303843 >> 172.21.15.29:0/580303843 conn(0x5575820be910 msgr2=0x557582471540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 -- 172.21.15.29:0/580303843 shutdown_connections 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 -- 172.21.15.29:0/580303843 wait complete. 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 -- start start 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.642+0000 7f69ef786740 1 Processor -- start 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69ef786740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x557582483dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69ef786740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 0x557582484390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69ef786740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820ca520 0x55758247ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69ef786740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41b46c0 con 0x5575820ca520 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69ef786740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41b3ff0 con 0x5575820a9840 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69ef786740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41b1230 con 0x5575820a1660 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69ef786740 10 client.0 _ll_register_callbacks cb 0x7f69d403f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:00.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x557582483dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x557582483dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34870/0 (socket says 172.21.15.29:34870) 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69e2d9d640 1 -- 172.21.15.29:0/4138096086 learned_addr learned my addr 172.21.15.29:0/4138096086 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.643+0000 7f69e259c640 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 0x557582484390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69e359e640 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820ca520 0x55758247ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69e259c640 1 -- 172.21.15.29:0/4138096086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 msgr2=0x557582483dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69e259c640 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x557582483dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69e259c640 1 -- 172.21.15.29:0/4138096086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820ca520 msgr2=0x55758247ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69e259c640 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820ca520 0x55758247ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69e259c640 1 -- 172.21.15.29:0/4138096086 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55758247f2c0 con 0x5575820a9840 2024-09-21T02:40:00.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69e359e640 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820ca520 0x55758247ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69e259c640 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 0x557582484390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f69d80e4950 tx=0x7f69d80e4de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2c3370171961d6e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.644+0000 7f69c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.645+0000 7f69c6ff5640 1 -- 172.21.15.29:0/4138096086 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8020070 con 0x5575820a9840 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.645+0000 7f69c6ff5640 1 -- 172.21.15.29:0/4138096086 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f69d80a8950 con 0x5575820a9840 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.645+0000 7f69ef786740 1 -- 172.21.15.29:0/4138096086 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69d80ad700 con 0x5575820a9840 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.645+0000 7f69c6ff5640 1 -- 172.21.15.29:0/4138096086 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69d80b4e60 con 0x5575820a9840 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.645+0000 7f69ef786740 10 client.25334 fetch_fsmap learned FSMap version 29 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.645+0000 7f69ef786740 1 -- 172.21.15.29:0/4138096086 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557582501900 con 0x5575820a9840 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.645+0000 7f69c6ff5640 1 -- 172.21.15.29:0/4138096086 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69d80983d0 con 0x5575820a9840 2024-09-21T02:40:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.646+0000 7f69c6ff5640 1 -- 172.21.15.29:0/4138096086 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f69d80b2d00 con 0x5575820a9840 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.646+0000 7f69ef786740 10 client.25334 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.646+0000 7f69ef786740 10 client.25334 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.646+0000 7f69ef786740 20 client.25334 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.646+0000 7f69ef786740 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55758205e250 0x557582504020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.646+0000 7f69ef786740 4 client.25334 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b213b215", "release": true, "await": true}] 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.646+0000 7f69ef786740 1 -- 172.21.15.29:0/4138096086 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b213b215", "release": true, "await": true}) -- 0x7f69d80ad700 con 0x55758205e250 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.646+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55758205e250 0x557582504020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.647+0000 7f69e2d9d640 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55758205e250 0x557582504020 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b018f1e5ab0ba24 server_cookie=ea59c337acdb9710 in_seq=0 out_seq=0 2024-09-21T02:40:00.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.647+0000 7f69c6ff5640 10 client.25334 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:00.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.649+0000 7f69c6ff5640 1 -- 172.21.15.29:0/4138096086 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7f69d80ad700 con 0x55758205e250 2024-09-21T02:40:00.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.649+0000 7f69c6ff5640 10 client.25334 handle_command_reply: tid=0 2024-09-21T02:40:00.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.649+0000 7f69ef786740 1 client.25334 shutdown 2024-09-21T02:40:00.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 -- 172.21.15.29:0/4138096086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 msgr2=0x557582484390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 0x557582484390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f69d80e4950 tx=0x7f69d80e4de0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 -- 172.21.15.29:0/4138096086 shutdown_connections 2024-09-21T02:40:00.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55758205e250 0x557582504020 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.653 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 29, 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 4, 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "b213b215": { 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 4, 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:40:00.654 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 134.5, 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:00.655 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575820ca520 0x55758247ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5575820a9840 0x557582484390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69c6ff5640 0 client.25334 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 --2- 172.21.15.29:0/4138096086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5575820a1660 0x557582483dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 -- 172.21.15.29:0/4138096086 >> 172.21.15.29:0/4138096086 conn(0x5575820be910 msgr2=0x557582471540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 -- 172.21.15.29:0/4138096086 shutdown_connections 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 1 -- 172.21.15.29:0/4138096086 wait complete. 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.650+0000 7f69ef786740 20 client.25334 trim_cache size 0 max 16384 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 -- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f69cc0b6620 msgr2=0x7f69cc0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f69cc0b6620 0x7f69cc0b8b30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009230 tx=0x7f69d4009560 comp rx=0 tx=0).stop 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 -- 172.21.15.29:0/963570189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69e4108d10 msgr2=0x7f69e4199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69e4108d10 0x7f69e4199970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f69d8005fd0 tx=0x7f69d800dde0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 -- 172.21.15.29:0/963570189 shutdown_connections 2024-09-21T02:40:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f69cc0b6620 0x7f69cc0b8b30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69e419eb50 0x7f69e419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69e4108d10 0x7f69e4199970 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 --2- 172.21.15.29:0/963570189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e41083c0 0x7f69e41993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 -- 172.21.15.29:0/963570189 >> 172.21.15.29:0/963570189 conn(0x7f69e4079740 msgr2=0x7f69e4101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 -- 172.21.15.29:0/963570189 shutdown_connections 2024-09-21T02:40:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:00.651+0000 7f69e9212640 1 -- 172.21.15.29:0/963570189 wait complete. 2024-09-21T02:40:00.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:00 smithi066 ceph-mon[33436]: overall HEALTH_OK 2024-09-21T02:40:00.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:00 smithi190 ceph-mon[33871]: overall HEALTH_OK 2024-09-21T02:40:00.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:00 smithi029 ceph-mon[31006]: overall HEALTH_OK 2024-09-21T02:40:01.454 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b213b215', total seconds elapsed: 18.3 2024-09-21T02:40:01.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:01 smithi029 ceph-mon[31006]: pgmap v416: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-21T02:40:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:01 smithi066 ceph-mon[33436]: pgmap v416: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-21T02:40:01.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:01 smithi190 ceph-mon[33871]: pgmap v416: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-21T02:40:03.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:03 smithi066 ceph-mon[33436]: pgmap v417: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:03.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:03 smithi190 ceph-mon[33871]: pgmap v417: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:03 smithi029 ceph-mon[31006]: pgmap v417: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:05.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:05 smithi066 ceph-mon[33436]: pgmap v418: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:40:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:05 smithi190 ceph-mon[33871]: pgmap v418: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:40:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:05 smithi029 ceph-mon[31006]: pgmap v418: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:40:07.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:07 smithi066 ceph-mon[33436]: pgmap v419: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:07.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:07 smithi190 ceph-mon[33871]: pgmap v419: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:07 smithi029 ceph-mon[31006]: pgmap v419: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:09.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:09 smithi066 ceph-mon[33436]: pgmap v420: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:09.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:09 smithi190 ceph-mon[33871]: pgmap v420: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:09.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:09 smithi029 ceph-mon[31006]: pgmap v420: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:40:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 -- 172.21.15.29:0/1189993382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c106bd0 msgr2=0x7ffa8c108ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/1189993382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c106bd0 0x7ffa8c108ff0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009900 tx=0x7ffa7802a3f0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 -- 172.21.15.29:0/1189993382 shutdown_connections 2024-09-21T02:40:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/1189993382 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa8c109530 0x7ffa8c10b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/1189993382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c106bd0 0x7ffa8c108ff0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/1189993382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa8c101a80 0x7ffa8c101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 -- 172.21.15.29:0/1189993382 >> 172.21.15.29:0/1189993382 conn(0x7ffa8c0fd680 msgr2=0x7ffa8c0ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 -- 172.21.15.29:0/1189993382 shutdown_connections 2024-09-21T02:40:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.589+0000 7ffa92f4a640 1 -- 172.21.15.29:0/1189993382 wait complete. 2024-09-21T02:40:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa92f4a640 1 Processor -- start 2024-09-21T02:40:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa92f4a640 1 -- start start 2024-09-21T02:40:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa92f4a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c101a80 0x7ffa8c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa92f4a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa8c106bd0 0x7ffa8c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa92f4a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa8c109530 0x7ffa8c19b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa90ce6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c101a80 0x7ffa8c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa90ce6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c101a80 0x7ffa8c199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54412/0 (socket says 172.21.15.29:54412) 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa92f4a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c10f1b0 con 0x7ffa8c101a80 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa90ce6640 1 -- 172.21.15.29:0/558375890 learned_addr learned my addr 172.21.15.29:0/558375890 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c10f030 con 0x7ffa8c106bd0 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c10f330 con 0x7ffa8c109530 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.590+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa8c106bd0 0x7ffa8c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa914e7640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa8c109530 0x7ffa8c19b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa90ce6640 1 -- 172.21.15.29:0/558375890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa8c109530 msgr2=0x7ffa8c19b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa90ce6640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa8c109530 0x7ffa8c19b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa90ce6640 1 -- 172.21.15.29:0/558375890 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa8c106bd0 msgr2=0x7ffa8c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa90ce6640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa8c106bd0 0x7ffa8c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa90ce6640 1 -- 172.21.15.29:0/558375890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8c19b910 con 0x7ffa8c101a80 2024-09-21T02:40:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa90ce6640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c101a80 0x7ffa8c199ed0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000e9f0 tx=0x7ffa8000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa89ffb640 1 -- 172.21.15.29:0/558375890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000cd80 con 0x7ffa8c101a80 2024-09-21T02:40:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa8c19bc00 con 0x7ffa8c101a80 2024-09-21T02:40:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa89ffb640 1 -- 172.21.15.29:0/558375890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa800054b0 con 0x7ffa8c101a80 2024-09-21T02:40:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.591+0000 7ffa89ffb640 1 -- 172.21.15.29:0/558375890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80010430 con 0x7ffa8c101a80 2024-09-21T02:40:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.592+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa8c1a22f0 con 0x7ffa8c101a80 2024-09-21T02:40:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.594+0000 7ffa99eb4740 1 Processor -- start 2024-09-21T02:40:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa99eb4740 1 -- start start 2024-09-21T02:40:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5618c24656f0 con 0x5618c21222f0 2024-09-21T02:40:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5618c2454b20 con 0x5618c214b1b0 2024-09-21T02:40:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5618c2458930 con 0x5618c212a4d0 2024-09-21T02:40:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa914e7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa8bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa914e7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24f2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:54092/0 (socket says 172.21.15.29:54092) 2024-09-21T02:40:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa914e7640 1 -- 172.21.15.29:0/1739739395 learned_addr learned my addr 172.21.15.29:0/1739739395 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.595+0000 7ffa914e7640 1 -- 172.21.15.29:0/1739739395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 msgr2=0x5618c24f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.596+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1739739395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.596+0000 7ffa914e7640 1 -- 172.21.15.29:0/1739739395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 msgr2=0x5618c24f34c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:40:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.596+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1739739395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24f34c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.596+0000 7ffa914e7640 1 -- 172.21.15.29:0/1739739395 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618c241a530 con 0x5618c214b1b0 2024-09-21T02:40:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.596+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1739739395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24f2e50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa840039f0 tx=0x7ffa84003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95bc269610244e7e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.596+0000 7ffa89ffb640 1 -- 172.21.15.29:0/558375890 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ffa800105f0 con 0x7ffa8c101a80 2024-09-21T02:40:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.596+0000 7ffa76ffd640 1 -- 172.21.15.29:0/1739739395 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8401a470 con 0x5618c214b1b0 2024-09-21T02:40:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.596+0000 7ffa76ffd640 1 -- 172.21.15.29:0/1739739395 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa84015070 con 0x5618c214b1b0 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.597+0000 7ffa89ffb640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa7c0b6590 0x7ffa7c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.597+0000 7ffa76ffd640 1 -- 172.21.15.29:0/1739739395 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84010770 con 0x5618c214b1b0 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.597+0000 7ffa89ffb640 1 -- 172.21.15.29:0/558375890 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffa800e79d0 con 0x7ffa8c101a80 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.597+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1739739395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 msgr2=0x5618c24f2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.597+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1739739395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24f2e50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa840039f0 tx=0x7ffa84003eb0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.597+0000 7ffa90ce6640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa7c0b6590 0x7ffa7c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.598+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1739739395 shutdown_connections 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.598+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1739739395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.598+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1739739395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.598+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1739739395 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24f2e50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.598+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1739739395 >> 172.21.15.29:0/1739739395 conn(0x5618c213f300 msgr2=0x5618c24ee8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.598+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1739739395 shutdown_connections 2024-09-21T02:40:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.598+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1739739395 wait complete. 2024-09-21T02:40:10.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.598+0000 7ffa90ce6640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa7c0b6590 0x7ffa7c0b8aa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000e9c0 tx=0x7ffa800a9740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:10.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.599+0000 7ffa99eb4740 1 -- start start 2024-09-21T02:40:10.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.599+0000 7ffa99eb4740 1 Processor -- start 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c24fc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c212a4d0 0x5618c24fcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24fd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa914e7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c24fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa914e7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c24fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:59406/0 (socket says 172.21.15.29:59406) 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa914e7640 1 -- 172.21.15.29:0/1729003351 learned_addr learned my addr 172.21.15.29:0/1729003351 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5618c238cab0 con 0x5618c212a4d0 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c212a4d0 0x5618c24fcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5618c23fa730 con 0x5618c214b1b0 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5618c24f4bd0 con 0x5618c21222f0 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24fd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa8bfff640 1 -- 172.21.15.29:0/1729003351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 msgr2=0x5618c24fc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c24fc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa8bfff640 1 -- 172.21.15.29:0/1729003351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 msgr2=0x5618c24fd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24fd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa8bfff640 1 -- 172.21.15.29:0/1729003351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618c21b6790 con 0x5618c212a4d0 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.600+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c24fc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa99eb4740 10 client.0 _ll_register_callbacks cb 0x5618c244e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa84015070 con 0x5618c212a4d0 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c212a4d0 0x5618c24fcd20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffa78005760 tx=0x7ffa78031870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66cea3d775bf697 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa6bfff640 10 client.15750 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1729003351 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78035040 con 0x5618c212a4d0 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1729003351 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffa78004240 con 0x5618c212a4d0 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1729003351 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffa78004440 con 0x5618c212a4d0 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa99eb4740 10 client.15750 fetch_fsmap learned FSMap version 29 2024-09-21T02:40:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.601+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5618c222f620 con 0x5618c212a4d0 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.602+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1729003351 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ffa780035f0 con 0x5618c212a4d0 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.602+0000 7ffa99eb4740 10 client.15750 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.602+0000 7ffa99eb4740 10 client.15750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.602+0000 7ffa99eb4740 20 client.15750 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.602+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5618c2109d80 0x5618c2509c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.602+0000 7ffa99eb4740 4 client.15750 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.602+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618c2467640 con 0x5618c2109d80 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.602+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5618c2109d80 0x5618c2509c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.603+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5618c2109d80 0x5618c2509c40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63f0c1d916ce116 server_cookie=d94a3043c6282f08 in_seq=0 out_seq=0 2024-09-21T02:40:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.603+0000 7ffa6bfff640 10 client.15750 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1729003351 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5618c2467640 con 0x5618c2109d80 2024-09-21T02:40:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa6bfff640 10 client.15750 handle_command_reply: tid=0 2024-09-21T02:40:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 client.15750 shutdown 2024-09-21T02:40:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c212a4d0 msgr2=0x5618c24fcd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c212a4d0 0x5618c24fcd20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffa78005760 tx=0x7ffa78031870 comp rx=0 tx=0).stop 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 shutdown_connections 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24fd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5618c2109d80 0x5618c2509c40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c212a4d0 0x5618c24fcd20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1729003351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c24fc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 >> 172.21.15.29:0/1729003351 conn(0x5618c213f300 msgr2=0x5618c24ecea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa6bfff640 0 client.15750 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 shutdown_connections 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.604+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1729003351 wait complete. 2024-09-21T02:40:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.605+0000 7ffa99eb4740 20 client.15750 trim_cache size 0 max 16384 2024-09-21T02:40:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.619+0000 7ffa99eb4740 1 Processor -- start 2024-09-21T02:40:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.619+0000 7ffa99eb4740 1 -- start start 2024-09-21T02:40:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.619+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24fdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24ef090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1ae760 con 0x5618c21222f0 2024-09-21T02:40:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1b02a0 con 0x5618c214b1b0 2024-09-21T02:40:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1aee30 con 0x5618c212a4d0 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24fdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24fdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54436/0 (socket says 172.21.15.29:54436) 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 -- 172.21.15.29:0/3907152959 learned_addr learned my addr 172.21.15.29:0/3907152959 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24ef090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 -- 172.21.15.29:0/3907152959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 msgr2=0x5618c24f7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 -- 172.21.15.29:0/3907152959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 msgr2=0x5618c24ef090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24ef090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 -- 172.21.15.29:0/3907152959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618c24f4d50 con 0x5618c21222f0 2024-09-21T02:40:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.620+0000 7ffa914e7640 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24fdec0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ffa840039f0 tx=0x7ffa84003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a523b74f039fc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.621+0000 7ffa767fc640 1 -- 172.21.15.29:0/3907152959 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84004ab0 con 0x5618c21222f0 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.621+0000 7ffa767fc640 1 -- 172.21.15.29:0/3907152959 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa84011a90 con 0x5618c21222f0 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.621+0000 7ffa767fc640 1 -- 172.21.15.29:0/3907152959 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84005640 con 0x5618c21222f0 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 -- 172.21.15.29:0/3907152959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 msgr2=0x5618c24fdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24fdec0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ffa840039f0 tx=0x7ffa84003eb0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 -- 172.21.15.29:0/3907152959 shutdown_connections 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c214b1b0 0x5618c24ef090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c212a4d0 0x5618c24f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/3907152959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c21222f0 0x5618c24fdec0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 -- 172.21.15.29:0/3907152959 >> 172.21.15.29:0/3907152959 conn(0x5618c213f300 msgr2=0x5618c24f6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 -- 172.21.15.29:0/3907152959 shutdown_connections 2024-09-21T02:40:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 -- 172.21.15.29:0/3907152959 wait complete. 2024-09-21T02:40:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 -- start start 2024-09-21T02:40:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.622+0000 7ffa99eb4740 1 Processor -- start 2024-09-21T02:40:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c25042f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c212a4d0 0x5618c2502fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa99eb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c214b1b0 0x5618c2503550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1afbd0 con 0x5618c214b1b0 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1af500 con 0x5618c212a4d0 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa99eb4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c102e00 con 0x5618c21222f0 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa99eb4740 10 client.0 _ll_register_callbacks cb 0x7ffa78009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c214b1b0 0x5618c2503550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c214b1b0 0x5618c2503550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54438/0 (socket says 172.21.15.29:54438) 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 -- 172.21.15.29:0/1264731878 learned_addr learned my addr 172.21.15.29:0/1264731878 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c25042f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c212a4d0 0x5618c2502fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 -- 172.21.15.29:0/1264731878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 msgr2=0x5618c25042f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c25042f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 -- 172.21.15.29:0/1264731878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c212a4d0 msgr2=0x5618c2502fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c212a4d0 0x5618c2502fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.623+0000 7ffa914e7640 1 -- 172.21.15.29:0/1264731878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618c2503ce0 con 0x5618c214b1b0 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa914e7640 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c214b1b0 0x5618c2503550 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffa840051e0 tx=0x7ffa840096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a5cdd15a406d63 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1264731878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84038040 con 0x5618c214b1b0 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1264731878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa84011a90 con 0x5618c214b1b0 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1264731878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffa84005420 con 0x5618c214b1b0 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1264731878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffa840118e0 con 0x5618c214b1b0 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa99eb4740 10 client.15756 fetch_fsmap learned FSMap version 29 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1264731878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618c25745b0 con 0x5618c214b1b0 2024-09-21T02:40:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.624+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1264731878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa84011a90 con 0x5618c214b1b0 2024-09-21T02:40:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.625+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1264731878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ffa8403e040 con 0x5618c214b1b0 2024-09-21T02:40:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.625+0000 7ffa99eb4740 10 client.15756 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:40:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.625+0000 7ffa99eb4740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:40:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.625+0000 7ffa99eb4740 20 client.15756 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.625+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5618c20df1c0 0x5618c2576dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.625+0000 7ffa99eb4740 4 client.15756 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:40:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.625+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1264731878 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5618c24630f0 con 0x5618c20df1c0 2024-09-21T02:40:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.625+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5618c20df1c0 0x5618c2576dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.626+0000 7ffa6bfff640 10 client.15756 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.626+0000 7ffa8bfff640 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5618c20df1c0 0x5618c2576dc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=969b9924a85a99ba server_cookie=d0b26b8fa0a0ec8a in_seq=0 out_seq=0 2024-09-21T02:40:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa6bfff640 1 -- 172.21.15.29:0/1264731878 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5618c24630f0 con 0x5618c20df1c0 2024-09-21T02:40:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa6bfff640 10 client.15756 handle_command_reply: tid=0 2024-09-21T02:40:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa99eb4740 1 client.15756 shutdown 2024-09-21T02:40:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1264731878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c214b1b0 msgr2=0x5618c2503550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c214b1b0 0x5618c2503550 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffa840051e0 tx=0x7ffa840096a0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1264731878 shutdown_connections 2024-09-21T02:40:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5618c20df1c0 0x5618c2576dc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa6bfff640 0 client.15756 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.632+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618c214b1b0 0x5618c2503550 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.633+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5618c212a4d0 0x5618c2502fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.633+0000 7ffa99eb4740 1 --2- 172.21.15.29:0/1264731878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618c21222f0 0x5618c25042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.633+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1264731878 >> 172.21.15.29:0/1264731878 conn(0x5618c213f300 msgr2=0x5618c24fb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.633+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1264731878 shutdown_connections 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.633+0000 7ffa99eb4740 1 -- 172.21.15.29:0/1264731878 wait complete. 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.633+0000 7ffa99eb4740 20 client.15756 trim_cache size 0 max 16384 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:40:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:40:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa7c0b6590 msgr2=0x7ffa7c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa7c0b6590 0x7ffa7c0b8aa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000e9c0 tx=0x7ffa800a9740 comp rx=0 tx=0).stop 2024-09-21T02:40:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c101a80 msgr2=0x7ffa8c199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c101a80 0x7ffa8c199ed0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000e9f0 tx=0x7ffa8000edf0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 shutdown_connections 2024-09-21T02:40:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa7c0b6590 0x7ffa7c0b8aa0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa8c109530 0x7ffa8c19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa8c106bd0 0x7ffa8c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 --2- 172.21.15.29:0/558375890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa8c101a80 0x7ffa8c199ed0 secure :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000e9f0 tx=0x7ffa8000edf0 comp rx=0 tx=0).stop 2024-09-21T02:40:10.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 >> 172.21.15.29:0/558375890 conn(0x7ffa8c0fd680 msgr2=0x7ffa8c107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:10.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 shutdown_connections 2024-09-21T02:40:10.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:10.634+0000 7ffa92f4a640 1 -- 172.21.15.29:0/558375890 wait complete. 2024-09-21T02:40:11.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:11 smithi066 ceph-mon[33436]: pgmap v421: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:40:11.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:11 smithi066 ceph-mon[33436]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-09-21T02:40:11.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:11 smithi190 ceph-mon[33871]: pgmap v421: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:40:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:11 smithi190 ceph-mon[33871]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-09-21T02:40:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:11 smithi029 ceph-mon[31006]: pgmap v421: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T02:40:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:11 smithi029 ceph-mon[31006]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-09-21T02:40:13.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:13 smithi066 ceph-mon[33436]: pgmap v422: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:40:13.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:13 smithi190 ceph-mon[33871]: pgmap v422: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:40:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:13 smithi029 ceph-mon[31006]: pgmap v422: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:40:15.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:15 smithi066 ceph-mon[33436]: pgmap v423: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-21T02:40:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:15 smithi190 ceph-mon[33871]: pgmap v423: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-21T02:40:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:15 smithi029 ceph-mon[31006]: pgmap v423: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-21T02:40:17.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:17 smithi066 ceph-mon[33436]: pgmap v424: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:17.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:17 smithi190 ceph-mon[33871]: pgmap v424: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:17 smithi029 ceph-mon[31006]: pgmap v424: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:19.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:19 smithi066 ceph-mon[33436]: pgmap v425: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:19 smithi190 ceph-mon[33871]: pgmap v425: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:19.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:19 smithi029 ceph-mon[31006]: pgmap v425: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:20.656 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 92% (11787/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-21T02:40:21.486 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to '1843221895adccfb3b692060251dd6a0183bc410'. 2024-09-21T02:40:21.486 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-09-21T02:40:21.486 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T02:40:21.486 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T02:40:21.486 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T02:40:21.486 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-09-21T02:40:21.486 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-09-21T02:40:21.487 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at 1843221895a Merge PR #56776 into wip-pdonnell-testing-20240920.212106-debug 2024-09-21T02:40:21.491 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T02:40:21.562 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T02:40:21.564 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T02:40:21.564 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T02:40:21.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:21 smithi066 ceph-mon[33436]: pgmap v426: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-21T02:40:21.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:21 smithi190 ceph-mon[33871]: pgmap v426: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-21T02:40:21.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:21 smithi029 ceph-mon[31006]: pgmap v426: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-09-21T02:40:22.613 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T02:40:22.722 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T02:40:22.767 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T02:40:22.769 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T02:40:22.769 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T02:40:22.810 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T02:40:22.814 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T02:40:22.814 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T02:40:22.871 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-09-21T02:40:22.872 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-09-21T02:40:22.872 DEBUG:teuthology.orchestra.run.smithi029:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1843221895adccfb3b692060251dd6a0183bc410 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-09-21T02:40:22.943 INFO:tasks.workunit.client.0.smithi029.stderr:+ dbench 1 2024-09-21T02:40:22.954 INFO:tasks.workunit.client.0.smithi029.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-21T02:40:22.954 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-09-21T02:40:22.954 INFO:tasks.workunit.client.0.smithi029.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-21T02:40:22.964 INFO:tasks.workunit.client.0.smithi029.stdout:0 of 1 processes prepared for launch 0 sec 2024-09-21T02:40:23.064 INFO:tasks.workunit.client.0.smithi029.stdout:1 of 1 processes prepared for launch 0 sec 2024-09-21T02:40:23.064 INFO:tasks.workunit.client.0.smithi029.stdout:releasing clients 2024-09-21T02:40:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.744+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/1801992353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a881025a0 msgr2=0x7f5a88102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.744+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/1801992353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a881025a0 0x7f5a88102a10 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009900 tx=0x7f5a7c02a410 comp rx=0 tx=0).stop 2024-09-21T02:40:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.744+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/1801992353 shutdown_connections 2024-09-21T02:40:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.744+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/1801992353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a88103fc0 0x7f5a8810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.744+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/1801992353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a88103620 0x7f5a88103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.744+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/1801992353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a881025a0 0x7f5a88102a10 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.744+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/1801992353 >> 172.21.15.29:0/1801992353 conn(0x7f5a880fdee0 msgr2=0x7f5a88100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.745+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/1801992353 shutdown_connections 2024-09-21T02:40:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.745+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/1801992353 wait complete. 2024-09-21T02:40:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.745+0000 7f5a8f1e6640 1 Processor -- start 2024-09-21T02:40:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.745+0000 7f5a8f1e6640 1 -- start start 2024-09-21T02:40:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.745+0000 7f5a8f1e6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a881025a0 0x7f5a88199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8f1e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a88103620 0x7f5a8819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8f1e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a88103fc0 0x7f5a8819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8f1e6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a8810e0a0 con 0x7f5a88103fc0 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8f1e6640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5a8810df20 con 0x7f5a881025a0 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8f1e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a8810e220 con 0x7f5a88103620 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a88103fc0 0x7f5a8819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a87fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a88103620 0x7f5a8819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a88103fc0 0x7f5a8819b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58834/0 (socket says 172.21.15.29:58834) 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 -- 172.21.15.29:0/2512325039 learned_addr learned my addr 172.21.15.29:0/2512325039 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a881025a0 0x7f5a88199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 -- 172.21.15.29:0/2512325039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a88103620 msgr2=0x7f5a8819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a88103620 0x7f5a8819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 -- 172.21.15.29:0/2512325039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a881025a0 msgr2=0x7f5a88199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a881025a0 0x7f5a88199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 -- 172.21.15.29:0/2512325039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8819b8c0 con 0x7f5a88103fc0 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a87fff640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a88103620 0x7f5a8819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a8d783640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a88103fc0 0x7f5a8819b0a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000eb40 tx=0x7f5a8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.746+0000 7f5a85ffb640 1 -- 172.21.15.29:0/2512325039 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80010430 con 0x7f5a88103fc0 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.747+0000 7f5a85ffb640 1 -- 172.21.15.29:0/2512325039 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a80010a60 con 0x7f5a88103fc0 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.747+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/2512325039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a8819bb50 con 0x7f5a88103fc0 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.747+0000 7f5a85ffb640 1 -- 172.21.15.29:0/2512325039 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80018ac0 con 0x7f5a88103fc0 2024-09-21T02:40:23.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.747+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/2512325039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a881a22a0 con 0x7f5a88103fc0 2024-09-21T02:40:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.749+0000 7f5a9622c740 1 Processor -- start 2024-09-21T02:40:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a85ffb640 1 -- 172.21.15.29:0/2512325039 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f5a80010c10 con 0x7f5a88103fc0 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a9622c740 1 -- start start 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d258d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d258d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21c04d0 0x5565d258e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a9622c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5565d24fb6f0 con 0x5565d21c04d0 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a9622c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5565d24eab20 con 0x5565d21e11b0 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a9622c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5565d24ee930 con 0x5565d21b82f0 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a87fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d258d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a8cf82640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d258d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a87fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d258d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:56872/0 (socket says 172.21.15.29:56872) 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a87fff640 1 -- 172.21.15.29:0/3976671284 learned_addr learned my addr 172.21.15.29:0/3976671284 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.750+0000 7f5a87fff640 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21c04d0 0x5565d258e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a87fff640 1 -- 172.21.15.29:0/3976671284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 msgr2=0x5565d258d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a87fff640 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d258d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a87fff640 1 -- 172.21.15.29:0/3976671284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21c04d0 msgr2=0x5565d258e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a87fff640 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21c04d0 0x5565d258e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a87fff640 1 -- 172.21.15.29:0/3976671284 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565d24b0530 con 0x5565d21e11b0 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d258d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a85ffb640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a780b6510 0x7f5a780b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a780b6510 0x7f5a780b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a87fff640 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d258d260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a74011c30 tx=0x7f5a74009bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57288e3e72847991 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a85ffb640 1 -- 172.21.15.29:0/2512325039 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5a800e7c10 con 0x7f5a88103fc0 2024-09-21T02:40:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a72ffd640 1 -- 172.21.15.29:0/3976671284 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400fc50 con 0x5565d21e11b0 2024-09-21T02:40:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a72ffd640 1 -- 172.21.15.29:0/3976671284 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a7400fdf0 con 0x5565d21e11b0 2024-09-21T02:40:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.751+0000 7f5a72ffd640 1 -- 172.21.15.29:0/3976671284 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400e620 con 0x5565d21e11b0 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 -- 172.21.15.29:0/3976671284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 msgr2=0x5565d258d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a780b6510 0x7f5a780b8a20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0057c0 tx=0x7f5a7c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d258d260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a74011c30 tx=0x7f5a74009bc0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 -- 172.21.15.29:0/3976671284 shutdown_connections 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21c04d0 0x5565d258e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d258d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3976671284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d258d260 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 -- 172.21.15.29:0/3976671284 >> 172.21.15.29:0/3976671284 conn(0x5565d21d5300 msgr2=0x5565d25848e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 -- 172.21.15.29:0/3976671284 shutdown_connections 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.752+0000 7f5a9622c740 1 -- 172.21.15.29:0/3976671284 wait complete. 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 1 -- start start 2024-09-21T02:40:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 1 Processor -- start 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21b82f0 0x5565d2596a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d2597140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d2597cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5565d2422ab0 con 0x5565d21b82f0 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5565d2490730 con 0x5565d21e11b0 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5565d258ef40 con 0x5565d21c04d0 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a9622c740 10 client.0 _ll_register_callbacks cb 0x5565d24e46b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a8cf82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21b82f0 0x5565d2596a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a87fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d2597140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a8cf82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21b82f0 0x5565d2596a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58850/0 (socket says 172.21.15.29:58850) 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a8cf82640 1 -- 172.21.15.29:0/3743692390 learned_addr learned my addr 172.21.15.29:0/3743692390 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.753+0000 7f5a8d783640 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d2597cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a87fff640 1 -- 172.21.15.29:0/3743692390 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 msgr2=0x5565d2597cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a87fff640 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d2597cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a87fff640 1 -- 172.21.15.29:0/3743692390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21b82f0 msgr2=0x5565d2596a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a87fff640 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21b82f0 0x5565d2596a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a87fff640 1 -- 172.21.15.29:0/3743692390 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565d224c790 con 0x5565d21c04d0 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a87fff640 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d2597140 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5a7401e000 tx=0x7f5a740044a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d1b1d6ca9e65c8d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a67fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a67fff640 1 -- 172.21.15.29:0/3743692390 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400fc50 con 0x5565d21c04d0 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a67fff640 1 -- 172.21.15.29:0/3743692390 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5a74005440 con 0x5565d21c04d0 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a9622c740 1 -- 172.21.15.29:0/3743692390 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a7400fdf0 con 0x5565d21c04d0 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a67fff640 1 -- 172.21.15.29:0/3743692390 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5a74030050 con 0x5565d21c04d0 2024-09-21T02:40:23.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a9622c740 10 client.25432 fetch_fsmap learned FSMap version 29 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.754+0000 7f5a9622c740 1 -- 172.21.15.29:0/3743692390 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565d22c5620 con 0x5565d21c04d0 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a67fff640 1 -- 172.21.15.29:0/3743692390 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a7400fdf0 con 0x5565d21c04d0 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a67fff640 1 -- 172.21.15.29:0/3743692390 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5a7401d020 con 0x5565d21c04d0 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a9622c740 10 client.25432 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a9622c740 10 client.25432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a9622c740 20 client.25432 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5565d219fd80 0x5565d25a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a9622c740 4 client.25432 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a9622c740 1 -- 172.21.15.29:0/3743692390 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565d24fd640 con 0x5565d219fd80 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.755+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5565d219fd80 0x5565d25a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.756+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5565d219fd80 0x5565d25a3f70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=639b61674dc38b30 server_cookie=fe1e21895fe2e50b in_seq=0 out_seq=0 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.756+0000 7f5a67fff640 10 client.25432 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.756+0000 7f5a67fff640 1 -- 172.21.15.29:0/3743692390 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5565d24fd640 con 0x5565d219fd80 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.756+0000 7f5a67fff640 10 client.25432 handle_command_reply: tid=0 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.756+0000 7f5a9622c740 1 client.25432 shutdown 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 -- 172.21.15.29:0/3743692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 msgr2=0x5565d2597140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d2597140 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5a7401e000 tx=0x7f5a740044a0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 -- 172.21.15.29:0/3743692390 shutdown_connections 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21e11b0 0x5565d2597cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d2597140 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5565d219fd80 0x5565d25a3f70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 --2- 172.21.15.29:0/3743692390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21b82f0 0x5565d2596a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a67fff640 0 client.25432 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 -- 172.21.15.29:0/3743692390 >> 172.21.15.29:0/3743692390 conn(0x5565d21d5300 msgr2=0x5565d2582ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 -- 172.21.15.29:0/3743692390 shutdown_connections 2024-09-21T02:40:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 1 -- 172.21.15.29:0/3743692390 wait complete. 2024-09-21T02:40:23.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.757+0000 7f5a9622c740 20 client.25432 trim_cache size 0 max 16384 2024-09-21T02:40:23.764 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:23 smithi066 ceph-mon[33436]: pgmap v427: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:23.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.774+0000 7f5a9622c740 1 Processor -- start 2024-09-21T02:40:23.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.774+0000 7f5a9622c740 1 -- start start 2024-09-21T02:40:23.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d2594f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21b82f0 0x5565d259eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d25954d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a9622c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881aa3e0 con 0x5565d21d5300 2024-09-21T02:40:23.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a9622c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881abf20 con 0x5565d21b82f0 2024-09-21T02:40:23.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a9622c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881aaab0 con 0x5565d21c04d0 2024-09-21T02:40:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a87fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21b82f0 0x5565d259eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a87fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21b82f0 0x5565d259eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:56902/0 (socket says 172.21.15.29:56902) 2024-09-21T02:40:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a87fff640 1 -- 172.21.15.29:0/2171990851 learned_addr learned my addr 172.21.15.29:0/2171990851 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d2594f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a8d783640 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d25954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a8cf82640 1 -- 172.21.15.29:0/2171990851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 msgr2=0x5565d25954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d25954d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a8cf82640 1 -- 172.21.15.29:0/2171990851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21b82f0 msgr2=0x5565d259eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21b82f0 0x5565d259eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.775+0000 7f5a8cf82640 1 -- 172.21.15.29:0/2171990851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565d2584010 con 0x5565d21d5300 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.776+0000 7f5a8d783640 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d25954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.776+0000 7f5a87fff640 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21b82f0 0x5565d259eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.776+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d2594f90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x5565d258f090 tx=0x7f5a7c03f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0444395429290 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.776+0000 7f5a727fc640 1 -- 172.21.15.29:0/2171990851 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c035040 con 0x5565d21d5300 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.776+0000 7f5a727fc640 1 -- 172.21.15.29:0/2171990851 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a7c003a80 con 0x5565d21d5300 2024-09-21T02:40:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.776+0000 7f5a727fc640 1 -- 172.21.15.29:0/2171990851 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c052990 con 0x5565d21d5300 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 -- 172.21.15.29:0/2171990851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 msgr2=0x5565d2594f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d2594f90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x5565d258f090 tx=0x7f5a7c03f040 comp rx=0 tx=0).stop 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 -- 172.21.15.29:0/2171990851 shutdown_connections 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21c04d0 0x5565d25954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21b82f0 0x5565d259eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2171990851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d2594f90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 -- 172.21.15.29:0/2171990851 >> 172.21.15.29:0/2171990851 conn(0x5565d258e5d0 msgr2=0x5565d2595ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 -- 172.21.15.29:0/2171990851 shutdown_connections 2024-09-21T02:40:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.777+0000 7f5a9622c740 1 -- 172.21.15.29:0/2171990851 wait complete. 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 1 -- start start 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 1 Processor -- start 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d259e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21c04d0 0x5565d259d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d259d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881ab850 con 0x5565d21d5300 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881ab180 con 0x5565d21c04d0 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881a83c0 con 0x5565d21b82f0 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a9622c740 10 client.0 _ll_register_callbacks cb 0x7f5a74011720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a8d783640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d259d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.778+0000 7f5a8cf82640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d259e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a8d783640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d259d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58874/0 (socket says 172.21.15.29:58874) 2024-09-21T02:40:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a8d783640 1 -- 172.21.15.29:0/2805133315 learned_addr learned my addr 172.21.15.29:0/2805133315 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a87fff640 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21c04d0 0x5565d259d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a87fff640 1 -- 172.21.15.29:0/2805133315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 msgr2=0x5565d259e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a87fff640 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d259e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a87fff640 1 -- 172.21.15.29:0/2805133315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 msgr2=0x5565d259d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a87fff640 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d259d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a87fff640 1 -- 172.21.15.29:0/2805133315 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565d259e0f0 con 0x5565d21c04d0 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a8d783640 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d259d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a87fff640 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21c04d0 0x5565d259d470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5a74022040 tx=0x7f5a74011b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abc4331f0ce26157 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a67fff640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a67fff640 1 -- 172.21.15.29:0/2805133315 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a740118f0 con 0x5565d21c04d0 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a67fff640 1 -- 172.21.15.29:0/2805133315 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5a7400fc20 con 0x5565d21c04d0 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.779+0000 7f5a9622c740 1 -- 172.21.15.29:0/2805133315 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a7c003a80 con 0x5565d21c04d0 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a67fff640 1 -- 172.21.15.29:0/2805133315 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5a74005b90 con 0x5565d21c04d0 2024-09-21T02:40:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a9622c740 10 client.25370 fetch_fsmap learned FSMap version 29 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a9622c740 1 -- 172.21.15.29:0/2805133315 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565d260e970 con 0x5565d21c04d0 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a67fff640 1 -- 172.21.15.29:0/2805133315 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a7c003a80 con 0x5565d21c04d0 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a67fff640 1 -- 172.21.15.29:0/2805133315 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5a7401d040 con 0x5565d21c04d0 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a9622c740 10 client.25370 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a9622c740 10 client.25370 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a9622c740 20 client.25370 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5565d21e11b0 0x5565d26111a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a9622c740 4 client.25370 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a9622c740 1 -- 172.21.15.29:0/2805133315 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565d24f90f0 con 0x5565d21e11b0 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.780+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5565d21e11b0 0x5565d26111a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.781+0000 7f5a67fff640 10 client.25370 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.781+0000 7f5a8cf82640 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5565d21e11b0 0x5565d26111a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9342ddb6c8cb1e7f server_cookie=c352c598d10d806c in_seq=0 out_seq=0 2024-09-21T02:40:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a67fff640 1 -- 172.21.15.29:0/2805133315 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5565d24f90f0 con 0x5565d21e11b0 2024-09-21T02:40:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a67fff640 10 client.25370 handle_command_reply: tid=0 2024-09-21T02:40:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 client.25370 shutdown 2024-09-21T02:40:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 -- 172.21.15.29:0/2805133315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21c04d0 msgr2=0x5565d259d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21c04d0 0x5565d259d470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5a74022040 tx=0x7f5a74011b30 comp rx=0 tx=0).stop 2024-09-21T02:40:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 -- 172.21.15.29:0/2805133315 shutdown_connections 2024-09-21T02:40:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5565d21e11b0 0x5565d26111a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565d21d5300 0x5565d259d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5565d21c04d0 0x5565d259d470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 --2- 172.21.15.29:0/2805133315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565d21b82f0 0x5565d259e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a67fff640 0 client.25370 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 -- 172.21.15.29:0/2805133315 >> 172.21.15.29:0/2805133315 conn(0x5565d258e5d0 msgr2=0x5565d2596a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 -- 172.21.15.29:0/2805133315 shutdown_connections 2024-09-21T02:40:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 1 -- 172.21.15.29:0/2805133315 wait complete. 2024-09-21T02:40:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.821+0000 7f5a9622c740 20 client.25370 trim_cache size 0 max 16384 2024-09-21T02:40:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:40:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:40:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:40:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:40:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a780b6510 msgr2=0x7f5a780b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a780b6510 0x7f5a780b8a20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0057c0 tx=0x7f5a7c000bc0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a88103fc0 msgr2=0x7f5a8819b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a88103fc0 0x7f5a8819b0a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000eb40 tx=0x7f5a8000c2d0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/2512325039 shutdown_connections 2024-09-21T02:40:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a780b6510 0x7f5a780b8a20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a88103fc0 0x7f5a8819b0a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a88103620 0x7f5a8819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 --2- 172.21.15.29:0/2512325039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a881025a0 0x7f5a88199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/2512325039 >> 172.21.15.29:0/2512325039 conn(0x7f5a880fdee0 msgr2=0x7f5a880ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/2512325039 shutdown_connections 2024-09-21T02:40:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:23.822+0000 7f5a8f1e6640 1 -- 172.21.15.29:0/2512325039 wait complete. 2024-09-21T02:40:23.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:23 smithi029 ceph-mon[31006]: pgmap v427: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:23.974 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:23 smithi190 ceph-mon[33871]: pgmap v427: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-21T02:40:24.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 903 21.68 MB/sec warmup 1 sec latency 34.488 ms 2024-09-21T02:40:25.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1603 13.08 MB/sec warmup 2 sec latency 16.336 ms 2024-09-21T02:40:25.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:25 smithi066 ceph-mon[33436]: pgmap v428: 129 pgs: 129 active+clean; 5.2 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 384 KiB/s wr, 15 op/s 2024-09-21T02:40:25.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:25 smithi190 ceph-mon[33871]: pgmap v428: 129 pgs: 129 active+clean; 5.2 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 384 KiB/s wr, 15 op/s 2024-09-21T02:40:25.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:25 smithi029 ceph-mon[31006]: pgmap v428: 129 pgs: 129 active+clean; 5.2 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 384 KiB/s wr, 15 op/s 2024-09-21T02:40:26.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 2359 9.30 MB/sec warmup 3 sec latency 13.584 ms 2024-09-21T02:40:27.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 3275 8.84 MB/sec warmup 4 sec latency 18.836 ms 2024-09-21T02:40:27.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:27 smithi066 ceph-mon[33436]: pgmap v429: 129 pgs: 129 active+clean; 14 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 MiB/s wr, 21 op/s 2024-09-21T02:40:27.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:27 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:40:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:27 smithi190 ceph-mon[33871]: pgmap v429: 129 pgs: 129 active+clean; 14 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 MiB/s wr, 21 op/s 2024-09-21T02:40:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:27 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:40:27.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:27 smithi029 ceph-mon[31006]: pgmap v429: 129 pgs: 129 active+clean; 14 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 MiB/s wr, 21 op/s 2024-09-21T02:40:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:27 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:40:28.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 3779 7.95 MB/sec warmup 5 sec latency 14.041 ms 2024-09-21T02:40:28.384 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.0 2024-09-21T02:40:28.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:40:28.671 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:40:29.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 4026 6.68 MB/sec warmup 6 sec latency 12.152 ms 2024-09-21T02:40:29.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 -- 172.21.15.29:0/3050723650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe834102a60 msgr2=0x7fe834102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 --2- 172.21.15.29:0/3050723650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe834102a60 0x7fe834102e70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe824009900 tx=0x7fe82402a410 comp rx=0 tx=0).stop 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 -- 172.21.15.29:0/3050723650 shutdown_connections 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 --2- 172.21.15.29:0/3050723650 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe834108cf0 0x7fe83410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 --2- 172.21.15.29:0/3050723650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8341083a0 0x7fe8341087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 --2- 172.21.15.29:0/3050723650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe834102a60 0x7fe834102e70 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 -- 172.21.15.29:0/3050723650 >> 172.21.15.29:0/3050723650 conn(0x7fe834079730 msgr2=0x7fe834079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 -- 172.21.15.29:0/3050723650 shutdown_connections 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.221+0000 7fe838974640 1 -- 172.21.15.29:0/3050723650 wait complete. 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.222+0000 7fe838974640 1 Processor -- start 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.222+0000 7fe838974640 1 -- start start 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe838974640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe834102a60 0x7fe83419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe838974640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8341083a0 0x7fe83419b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe838974640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe834108cf0 0x7fe83419bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe838974640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341128c0 con 0x7fe8341083a0 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe838974640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe834112740 con 0x7fe834108cf0 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe838974640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe834112a40 con 0x7fe834102a60 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe82bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8341083a0 0x7fe83419b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe83259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe834102a60 0x7fe83419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe82bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8341083a0 0x7fe83419b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56818/0 (socket says 172.21.15.29:56818) 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe82bfff640 1 -- 172.21.15.29:0/326799652 learned_addr learned my addr 172.21.15.29:0/326799652 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe832d9d640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe834108cf0 0x7fe83419bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe82bfff640 1 -- 172.21.15.29:0/326799652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe834102a60 msgr2=0x7fe83419b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe82bfff640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe834102a60 0x7fe83419b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe82bfff640 1 -- 172.21.15.29:0/326799652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe834108cf0 msgr2=0x7fe83419bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe82bfff640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe834108cf0 0x7fe83419bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.223+0000 7fe82bfff640 1 -- 172.21.15.29:0/326799652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8341a6550 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.224+0000 7fe82bfff640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8341083a0 0x7fe83419b7d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00b450 tx=0x7fe81c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.224+0000 7fe82b7fe640 1 -- 172.21.15.29:0/326799652 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c015020 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.224+0000 7fe82b7fe640 1 -- 172.21.15.29:0/326799652 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe81c011d20 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.224+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8341a6840 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.224+0000 7fe82b7fe640 1 -- 172.21.15.29:0/326799652 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c010e10 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.224+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8341a6d80 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.225+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe834102e70 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.227+0000 7fe82b7fe640 1 -- 172.21.15.29:0/326799652 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fe81c018020 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.228+0000 7fe82b7fe640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe8200b6510 0x7fe8200b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.228+0000 7fe83259c640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe8200b6510 0x7fe8200b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.228+0000 7fe82b7fe640 1 -- 172.21.15.29:0/326799652 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe81c0e7370 con 0x7fe8341083a0 2024-09-21T02:40:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.228+0000 7fe83259c640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe8200b6510 0x7fe8200b8a20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe83406b9e0 tx=0x7fe824000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.231+0000 7fe82b7fe640 1 -- 172.21.15.29:0/326799652 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe81c01f020 con 0x7fe8341083a0 2024-09-21T02:40:29.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.491+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe83406b290 con 0x7fe8341083a0 2024-09-21T02:40:29.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.493+0000 7fe82b7fe640 1 -- 172.21.15.29:0/326799652 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11972 (secure 0 0 0) 0x7fe83406b290 con 0x7fe8341083a0 2024-09-21T02:40:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:40:29.494 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"btime":"2024-09-21T02:40:26:480768+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:40:26.304013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[3],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:40:29.495 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 30 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.495+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe8200b6510 msgr2=0x7fe8200b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.495+0000 7fe838974640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe8200b6510 0x7fe8200b8a20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe83406b9e0 tx=0x7fe824000bc0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.495+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8341083a0 msgr2=0x7fe83419b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.495+0000 7fe838974640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8341083a0 0x7fe83419b7d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00b450 tx=0x7fe81c00b910 comp rx=0 tx=0).stop 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.496+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 shutdown_connections 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.496+0000 7fe838974640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe8200b6510 0x7fe8200b8a20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.496+0000 7fe838974640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe834108cf0 0x7fe83419bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.496+0000 7fe838974640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8341083a0 0x7fe83419b7d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.496+0000 7fe838974640 1 --2- 172.21.15.29:0/326799652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe834102a60 0x7fe83419b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.496+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 >> 172.21.15.29:0/326799652 conn(0x7fe834079730 msgr2=0x7fe8341014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:29.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.496+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 shutdown_connections 2024-09-21T02:40:29.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:29.496+0000 7fe838974640 1 -- 172.21.15.29:0/326799652 wait complete. 2024-09-21T02:40:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:29 smithi066 ceph-mon[33436]: pgmap v430: 129 pgs: 129 active+clean; 21 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.8 MiB/s wr, 37 op/s 2024-09-21T02:40:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:29 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:40:29.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:29 smithi029 ceph-mon[31006]: pgmap v430: 129 pgs: 129 active+clean; 21 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.8 MiB/s wr, 37 op/s 2024-09-21T02:40:29.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:29 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:40:29.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:29 smithi190 ceph-mon[33871]: pgmap v430: 129 pgs: 129 active+clean; 21 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.8 MiB/s wr, 37 op/s 2024-09-21T02:40:29.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:29 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:40:30.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 4498 5.84 MB/sec warmup 7 sec latency 26.003 ms 2024-09-21T02:40:30.427 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 137.0 --await-for 45.0' 2024-09-21T02:40:30.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 137.0 --await-for 45.0 2024-09-21T02:40:30.728 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:40:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:30 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/326799652' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:40:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:40:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:40:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:40:30.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:30 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/326799652' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:40:30.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:40:30.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:40:30.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:40:30.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:30 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/326799652' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:40:30.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:40:30.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:40:30.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:40:31.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5129 5.65 MB/sec warmup 8 sec latency 13.407 ms 2024-09-21T02:40:31.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 -- 172.21.15.29:0/3690449758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248110200 msgr2=0x7f0248112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 --2- 172.21.15.29:0/3690449758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248110200 0x7f0248112620 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0240009ca0 tx=0x7f024002a950 comp rx=0 tx=0).stop 2024-09-21T02:40:31.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 -- 172.21.15.29:0/3690449758 shutdown_connections 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 --2- 172.21.15.29:0/3690449758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248110200 0x7f0248112620 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 --2- 172.21.15.29:0/3690449758 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0248102fa0 0x7f024810fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 --2- 172.21.15.29:0/3690449758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248102650 0x7f0248102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 -- 172.21.15.29:0/3690449758 >> 172.21.15.29:0/3690449758 conn(0x7f0248079760 msgr2=0x7f0248079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 -- 172.21.15.29:0/3690449758 shutdown_connections 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.460+0000 7f024dbf8640 1 -- 172.21.15.29:0/3690449758 wait complete. 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.461+0000 7f024dbf8640 1 Processor -- start 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f024dbf8640 1 -- start start 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f024dbf8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0248102650 0x7f024819c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f024dbf8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248102fa0 0x7f024819cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f024dbf8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248110200 0x7f02481a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f024dbf8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0248115b60 con 0x7f0248102fa0 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f024dbf8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f02481159e0 con 0x7f0248102650 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f024dbf8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0248115ce0 con 0x7f0248110200 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0246ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248102fa0 0x7f024819cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0246ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248102fa0 0x7f024819cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56836/0 (socket says 172.21.15.29:56836) 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0246ffd640 1 -- 172.21.15.29:0/2419170090 learned_addr learned my addr 172.21.15.29:0/2419170090 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f02477fe640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0248102650 0x7f024819c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0247fff640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248110200 0x7f02481a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0246ffd640 1 -- 172.21.15.29:0/2419170090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248110200 msgr2=0x7f02481a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0246ffd640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248110200 0x7f02481a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0246ffd640 1 -- 172.21.15.29:0/2419170090 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0248102650 msgr2=0x7f024819c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0246ffd640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0248102650 0x7f024819c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f0246ffd640 1 -- 172.21.15.29:0/2419170090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02481a2420 con 0x7f0248102fa0 2024-09-21T02:40:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.462+0000 7f02477fe640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0248102650 0x7f024819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.463+0000 7f0246ffd640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248102fa0 0x7f024819cb10 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f023c00b4a0 tx=0x7f023c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.463+0000 7f0244ff9640 1 -- 172.21.15.29:0/2419170090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c015020 con 0x7f0248102fa0 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.463+0000 7f0244ff9640 1 -- 172.21.15.29:0/2419170090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f023c011cc0 con 0x7f0248102fa0 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.463+0000 7f024dbf8640 1 -- 172.21.15.29:0/2419170090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02481a2710 con 0x7f0248102fa0 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.463+0000 7f0244ff9640 1 -- 172.21.15.29:0/2419170090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c010d00 con 0x7f0248102fa0 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.463+0000 7f024dbf8640 1 -- 172.21.15.29:0/2419170090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02481a8f30 con 0x7f0248102fa0 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.466+0000 7f0244ff9640 1 -- 172.21.15.29:0/2419170090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f023c018020 con 0x7f0248102fa0 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.466+0000 7f025416c740 1 Processor -- start 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.466+0000 7f025416c740 1 -- start start 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f025416c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803059840 0x555803400600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f025416c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803030980 0x555803400b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f025416c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f025416c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555803376b90 con 0x555803030980 2024-09-21T02:40:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f025416c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555803365fc0 con 0x555803038b60 2024-09-21T02:40:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f025416c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555803369dd0 con 0x555803059840 2024-09-21T02:40:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f02477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803030980 0x555803400b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f0244ff9640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02300b65c0 0x7f02300b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f0247fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803059840 0x555803400600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f02477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803030980 0x555803400b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56838/0 (socket says 172.21.15.29:56838) 2024-09-21T02:40:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f02477fe640 1 -- 172.21.15.29:0/450633314 learned_addr learned my addr 172.21.15.29:0/450633314 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f0246ffd640 1 --2- 172.21.15.29:0/450633314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f0247fff640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02300b65c0 0x7f02300b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f0244ff9640 1 -- 172.21.15.29:0/2419170090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f023c0e6cf0 con 0x7f0248102fa0 2024-09-21T02:40:31.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f02477fe640 1 -- 172.21.15.29:0/450633314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803059840 msgr2=0x555803400600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:31.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f02477fe640 1 --2- 172.21.15.29:0/450633314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803059840 0x555803400600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f02477fe640 1 -- 172.21.15.29:0/450633314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 msgr2=0x555803401080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f02477fe640 1 --2- 172.21.15.29:0/450633314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.467+0000 7f02477fe640 1 -- 172.21.15.29:0/450633314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558032ed060 con 0x555803030980 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.468+0000 7f02477fe640 1 --2- 172.21.15.29:0/450633314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803030980 0x555803400b40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f023800f4c0 tx=0x7f023800f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1be1d2b8a11030 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.468+0000 7f0247fff640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02300b65c0 0x7f02300b8ad0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f02400094e0 tx=0x7f024002a950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.468+0000 7f022dffb640 1 -- 172.21.15.29:0/450633314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238005220 con 0x555803030980 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.468+0000 7f022dffb640 1 -- 172.21.15.29:0/450633314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02380047a0 con 0x555803030980 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.468+0000 7f022dffb640 1 -- 172.21.15.29:0/450633314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238010a10 con 0x555803030980 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 -- 172.21.15.29:0/450633314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803030980 msgr2=0x555803400b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 --2- 172.21.15.29:0/450633314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803030980 0x555803400b40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f023800f4c0 tx=0x7f023800f980 comp rx=0 tx=0).stop 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 -- 172.21.15.29:0/450633314 shutdown_connections 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 --2- 172.21.15.29:0/450633314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 --2- 172.21.15.29:0/450633314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803030980 0x555803400b40 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 --2- 172.21.15.29:0/450633314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803059840 0x555803400600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 -- 172.21.15.29:0/450633314 >> 172.21.15.29:0/450633314 conn(0x55580304d990 msgr2=0x5558033fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 -- 172.21.15.29:0/450633314 shutdown_connections 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 -- 172.21.15.29:0/450633314 wait complete. 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.469+0000 7f025416c740 1 -- start start 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f025416c740 1 Processor -- start 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f025416c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x55580340c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f025416c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803038b60 0x55580340c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f025416c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803059840 0x555803418100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f025416c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5558031c77a0 con 0x555803059840 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f025416c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55580330b0a0 con 0x555803030980 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f025416c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55580340ede0 con 0x555803038b60 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f0247fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803059840 0x555803418100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f025416c740 10 client.0 _ll_register_callbacks cb 0x5558032e83c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f02477fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x55580340c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f0247fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803059840 0x555803418100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56850/0 (socket says 172.21.15.29:56850) 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f0247fff640 1 -- 172.21.15.29:0/4044041660 learned_addr learned my addr 172.21.15.29:0/4044041660 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.470+0000 7f0246ffd640 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803038b60 0x55580340c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f02477fe640 1 -- 172.21.15.29:0/4044041660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803038b60 msgr2=0x55580340c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f02477fe640 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803038b60 0x55580340c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f02477fe640 1 -- 172.21.15.29:0/4044041660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803059840 msgr2=0x555803418100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f02477fe640 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803059840 0x555803418100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f02477fe640 1 -- 172.21.15.29:0/4044041660 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558032f9880 con 0x555803030980 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f0247fff640 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803059840 0x555803418100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f02477fe640 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x55580340c190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f023801a040 tx=0x7f02380107d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9afc8ac40804d32d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f0226ffd640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f0226ffd640 1 -- 172.21.15.29:0/4044041660 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238026070 con 0x555803030980 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f0226ffd640 1 -- 172.21.15.29:0/4044041660 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0238010dd0 con 0x555803030980 2024-09-21T02:40:31.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.471+0000 7f025416c740 1 -- 172.21.15.29:0/4044041660 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02380047a0 con 0x555803030980 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f0226ffd640 1 -- 172.21.15.29:0/4044041660 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0238032050 con 0x555803030980 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f025416c740 10 client.25385 fetch_fsmap learned FSMap version 30 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f025416c740 1 -- 172.21.15.29:0/4044041660 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5558030d3b90 con 0x555803030980 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f0226ffd640 1 -- 172.21.15.29:0/4044041660 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02380047a0 con 0x555803030980 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f0226ffd640 1 -- 172.21.15.29:0/4044041660 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f023802b7b0 con 0x555803030980 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f025416c740 10 client.25385 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f025416c740 10 client.25385 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f025416c740 20 client.25385 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f025416c740 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5558030181c0 0x55580341df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f025416c740 4 client.25385 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f025416c740 1 -- 172.21.15.29:0/4044041660 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555803378ae0 con 0x5558030181c0 2024-09-21T02:40:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.472+0000 7f0246ffd640 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5558030181c0 0x55580341df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.473+0000 7f0226ffd640 10 client.25385 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.473+0000 7f0246ffd640 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5558030181c0 0x55580341df60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=971c736ebd4d91a2 server_cookie=151c30d1072166b8 in_seq=0 out_seq=0 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f0226ffd640 1 -- 172.21.15.29:0/4044041660 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555803378ae0 con 0x5558030181c0 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f0226ffd640 10 client.25385 handle_command_reply: tid=0 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 client.25385 shutdown 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 -- 172.21.15.29:0/4044041660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 msgr2=0x55580340c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x55580340c190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f023801a040 tx=0x7f02380107d0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 -- 172.21.15.29:0/4044041660 shutdown_connections 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5558030181c0 0x55580341df60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f0226ffd640 0 client.25385 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803059840 0x555803418100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803038b60 0x55580340c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 --2- 172.21.15.29:0/4044041660 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x55580340c190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 -- 172.21.15.29:0/4044041660 >> 172.21.15.29:0/4044041660 conn(0x55580304d990 msgr2=0x555803403790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 -- 172.21.15.29:0/4044041660 shutdown_connections 2024-09-21T02:40:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 1 -- 172.21.15.29:0/4044041660 wait complete. 2024-09-21T02:40:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.474+0000 7f025416c740 20 client.25385 trim_cache size 0 max 16384 2024-09-21T02:40:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.493+0000 7f025416c740 1 Processor -- start 2024-09-21T02:40:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.493+0000 7f025416c740 1 -- start start 2024-09-21T02:40:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f025416c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558030181c0 0x555803403d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f025416c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803030980 0x555803418000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f025416c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f025416c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f02481b2ba0 con 0x5558030181c0 2024-09-21T02:40:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f025416c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f02481b4710 con 0x555803038b60 2024-09-21T02:40:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f025416c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02481b3270 con 0x555803030980 2024-09-21T02:40:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0247fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0246ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803030980 0x555803418000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0247fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:48352/0 (socket says 172.21.15.29:48352) 2024-09-21T02:40:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f02477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558030181c0 0x555803403d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0247fff640 1 -- 172.21.15.29:0/3118180043 learned_addr learned my addr 172.21.15.29:0/3118180043 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0246ffd640 1 -- 172.21.15.29:0/3118180043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 msgr2=0x555803401ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0246ffd640 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0246ffd640 1 -- 172.21.15.29:0/3118180043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558030181c0 msgr2=0x555803403d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0246ffd640 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558030181c0 0x555803403d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0246ffd640 1 -- 172.21.15.29:0/3118180043 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558033fede0 con 0x555803030980 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f0247fff640 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.494+0000 7f02477fe640 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558030181c0 0x555803403d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.495+0000 7f0246ffd640 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803030980 0x555803418000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f023c0e44c0 tx=0x7f023c00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a44f034e6d8a9446 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.495+0000 7f022d7fa640 1 -- 172.21.15.29:0/3118180043 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c010ea0 con 0x555803030980 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.495+0000 7f022d7fa640 1 -- 172.21.15.29:0/3118180043 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f023c0ac610 con 0x555803030980 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.495+0000 7f022d7fa640 1 -- 172.21.15.29:0/3118180043 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c0b6520 con 0x555803030980 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 -- 172.21.15.29:0/3118180043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803030980 msgr2=0x555803418000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803030980 0x555803418000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f023c0e44c0 tx=0x7f023c00c840 comp rx=0 tx=0).stop 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 -- 172.21.15.29:0/3118180043 shutdown_connections 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803038b60 0x555803401ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555803030980 0x555803418000 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 --2- 172.21.15.29:0/3118180043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558030181c0 0x555803403d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 -- 172.21.15.29:0/3118180043 >> 172.21.15.29:0/3118180043 conn(0x55580304d990 msgr2=0x55580340d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 -- 172.21.15.29:0/3118180043 shutdown_connections 2024-09-21T02:40:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 -- 172.21.15.29:0/3118180043 wait complete. 2024-09-21T02:40:31.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.496+0000 7f025416c740 1 -- start start 2024-09-21T02:40:31.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f025416c740 1 Processor -- start 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f025416c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558030181c0 0x555803414520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f025416c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x5558034109e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f025416c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803038b60 0x555803410f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f025416c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f02481b4010 con 0x555803038b60 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f025416c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f02481b3940 con 0x555803030980 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f025416c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02481b0b80 con 0x5558030181c0 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f025416c740 10 client.0 _ll_register_callbacks cb 0x7f023c012f40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f0247fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803038b60 0x555803410f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f0247fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803038b60 0x555803410f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56870/0 (socket says 172.21.15.29:56870) 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f0247fff640 1 -- 172.21.15.29:0/3166814796 learned_addr learned my addr 172.21.15.29:0/3166814796 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f02477fe640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558030181c0 0x555803414520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.497+0000 7f0246ffd640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x5558034109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f02477fe640 1 -- 172.21.15.29:0/3166814796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 msgr2=0x5558034109e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f02477fe640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x5558034109e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f02477fe640 1 -- 172.21.15.29:0/3166814796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803038b60 msgr2=0x555803410f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f02477fe640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803038b60 0x555803410f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f02477fe640 1 -- 172.21.15.29:0/3166814796 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555803411460 con 0x5558030181c0 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f0246ffd640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x5558034109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f0247fff640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803038b60 0x555803410f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f02477fe640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558030181c0 0x555803414520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f023800f3e0 tx=0x7f0238005d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=106de3bffb4f6801 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f0226ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f0226ffd640 1 -- 172.21.15.29:0/3166814796 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238026070 con 0x5558030181c0 2024-09-21T02:40:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f0226ffd640 1 -- 172.21.15.29:0/3166814796 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f023802b640 con 0x5558030181c0 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f025416c740 1 -- 172.21.15.29:0/3166814796 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f023c0ac610 con 0x5558030181c0 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f0226ffd640 1 -- 172.21.15.29:0/3166814796 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0238044050 con 0x5558030181c0 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f025416c740 10 client.25456 fetch_fsmap learned FSMap version 30 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.498+0000 7f025416c740 1 -- 172.21.15.29:0/3166814796 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555803489370 con 0x5558030181c0 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f0226ffd640 1 -- 172.21.15.29:0/3166814796 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f023c0ac610 con 0x5558030181c0 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f0226ffd640 1 -- 172.21.15.29:0/3166814796 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f0238018ae0 con 0x5558030181c0 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f025416c740 10 client.25456 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f025416c740 10 client.25456 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f025416c740 20 client.25456 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f025416c740 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555803059840 0x55580348bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f025416c740 4 client.25456 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.0, "await_for": 45.0}] 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f025416c740 1 -- 172.21.15.29:0/3166814796 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.0, "await_for": 45.0}) -- 0x555803374590 con 0x555803059840 2024-09-21T02:40:31.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f0246ffd640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555803059840 0x55580348bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:31.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.499+0000 7f0226ffd640 10 client.25456 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:31.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.500+0000 7f0246ffd640 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555803059840 0x55580348bbc0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d674814f1bdc3a4d server_cookie=db6812fb2dcfa0bf in_seq=0 out_seq=0 2024-09-21T02:40:31.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f0226ffd640 1 -- 172.21.15.29:0/3166814796 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x555803374590 con 0x555803059840 2024-09-21T02:40:31.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f0226ffd640 10 client.25456 handle_command_reply: tid=0 2024-09-21T02:40:31.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 client.25456 shutdown 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 -- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558030181c0 msgr2=0x555803414520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558030181c0 0x555803414520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f023800f3e0 tx=0x7f0238005d60 comp rx=0 tx=0).stop 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 -- 172.21.15.29:0/3166814796 shutdown_connections 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555803059840 0x55580348bbc0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555803038b60 0x555803410f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555803030980 0x5558034109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 --2- 172.21.15.29:0/3166814796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558030181c0 0x555803414520 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f0226ffd640 0 client.25456 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 -- 172.21.15.29:0/3166814796 >> 172.21.15.29:0/3166814796 conn(0x55580304d990 msgr2=0x555803401dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 -- 172.21.15.29:0/3166814796 shutdown_connections 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 1 -- 172.21.15.29:0/3166814796 wait complete. 2024-09-21T02:40:31.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.590+0000 7f025416c740 20 client.25456 trim_cache size 0 max 16384 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 30, 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 6, 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "473fc997": { 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 6, 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:40:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 137.0, 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:31.595 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:40:31.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 -- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02300b65c0 msgr2=0x7f02300b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02300b65c0 0x7f02300b8ad0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f02400094e0 tx=0x7f024002a950 comp rx=0 tx=0).stop 2024-09-21T02:40:31.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 -- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248102fa0 msgr2=0x7f024819cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:31.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248102fa0 0x7f024819cb10 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f023c00b4a0 tx=0x7f023c00b8a0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 -- 172.21.15.29:0/2419170090 shutdown_connections 2024-09-21T02:40:31.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02300b65c0 0x7f02300b8ad0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248110200 0x7f02481a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0248102fa0 0x7f024819cb10 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 --2- 172.21.15.29:0/2419170090 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0248102650 0x7f024819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:31.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 -- 172.21.15.29:0/2419170090 >> 172.21.15.29:0/2419170090 conn(0x7f0248079760 msgr2=0x7f0248108c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:31.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 -- 172.21.15.29:0/2419170090 shutdown_connections 2024-09-21T02:40:31.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:31.591+0000 7f024dbf8640 1 -- 172.21.15.29:0/2419170090 wait complete. 2024-09-21T02:40:31.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:31 smithi029 ceph-mon[31006]: pgmap v431: 129 pgs: 129 active+clean; 23 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.1 MiB/s wr, 70 op/s 2024-09-21T02:40:31.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:31 smithi066 ceph-mon[33436]: pgmap v431: 129 pgs: 129 active+clean; 23 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.1 MiB/s wr, 70 op/s 2024-09-21T02:40:31.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:31 smithi190 ceph-mon[33871]: pgmap v431: 129 pgs: 129 active+clean; 23 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.1 MiB/s wr, 70 op/s 2024-09-21T02:40:32.064 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 5.05 MB/sec warmup 9 sec latency 469.332 ms 2024-09-21T02:40:32.535 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '473fc997', quiesce took 4.2 seconds. Will release after: 12.8 2024-09-21T02:40:33.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 4.55 MB/sec warmup 10 sec latency 1469.441 ms 2024-09-21T02:40:33.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:33 smithi066 ceph-mon[33436]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.4 MiB/s wr, 74 op/s 2024-09-21T02:40:33.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:33 smithi029 ceph-mon[31006]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.4 MiB/s wr, 74 op/s 2024-09-21T02:40:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:33 smithi190 ceph-mon[33871]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.4 MiB/s wr, 74 op/s 2024-09-21T02:40:34.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 4.13 MB/sec warmup 11 sec latency 2469.514 ms 2024-09-21T02:40:35.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 3.79 MB/sec warmup 12 sec latency 3469.626 ms 2024-09-21T02:40:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:35 smithi066 ceph-mon[33436]: pgmap v433: 129 pgs: 129 active+clean; 25 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 2.6 MiB/s wr, 108 op/s 2024-09-21T02:40:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:35 smithi190 ceph-mon[33871]: pgmap v433: 129 pgs: 129 active+clean; 25 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 2.6 MiB/s wr, 108 op/s 2024-09-21T02:40:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:35 smithi029 ceph-mon[31006]: pgmap v433: 129 pgs: 129 active+clean; 25 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 2.6 MiB/s wr, 108 op/s 2024-09-21T02:40:36.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 3.50 MB/sec warmup 13 sec latency 4469.719 ms 2024-09-21T02:40:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.936+0000 7eff1188b640 1 -- 172.21.15.29:0/1351171465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1036b0 msgr2=0x7eff0c103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.936+0000 7eff1188b640 1 --2- 172.21.15.29:0/1351171465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1036b0 0x7eff0c103b20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7efefc009900 tx=0x7efefc02a410 comp rx=0 tx=0).stop 2024-09-21T02:40:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 -- 172.21.15.29:0/1351171465 shutdown_connections 2024-09-21T02:40:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 --2- 172.21.15.29:0/1351171465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7eff0c1050d0 0x7eff0c10ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 --2- 172.21.15.29:0/1351171465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff0c104730 0x7eff0c104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 --2- 172.21.15.29:0/1351171465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1036b0 0x7eff0c103b20 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 -- 172.21.15.29:0/1351171465 >> 172.21.15.29:0/1351171465 conn(0x7eff0c0feff0 msgr2=0x7eff0c1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 -- 172.21.15.29:0/1351171465 shutdown_connections 2024-09-21T02:40:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 -- 172.21.15.29:0/1351171465 wait complete. 2024-09-21T02:40:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 Processor -- start 2024-09-21T02:40:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.937+0000 7eff1188b640 1 -- start start 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff1188b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7eff0c1036b0 0x7eff0c199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff1188b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff0c104730 0x7eff0c19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff1188b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1050d0 0x7eff0c19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff1188b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10f1a0 con 0x7eff0c1050d0 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff1188b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10f020 con 0x7eff0c1036b0 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff1188b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10f320 con 0x7eff0c104730 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1050d0 0x7eff0c19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1050d0 0x7eff0c19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51526/0 (socket says 172.21.15.29:51526) 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0b7fe640 1 -- 172.21.15.29:0/471291432 learned_addr learned my addr 172.21.15.29:0/471291432 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0affd640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7eff0c1036b0 0x7eff0c199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff027fc640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff0c104730 0x7eff0c19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0b7fe640 1 -- 172.21.15.29:0/471291432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff0c104730 msgr2=0x7eff0c19a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0b7fe640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff0c104730 0x7eff0c19a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0b7fe640 1 -- 172.21.15.29:0/471291432 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7eff0c1036b0 msgr2=0x7eff0c199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0b7fe640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7eff0c1036b0 0x7eff0c199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.938+0000 7eff0b7fe640 1 -- 172.21.15.29:0/471291432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0c19b8c0 con 0x7eff0c1050d0 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.939+0000 7eff0b7fe640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1050d0 0x7eff0c19b0a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7eff0400cad0 tx=0x7eff0400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.939+0000 7eff10889640 1 -- 172.21.15.29:0/471291432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0400ae20 con 0x7eff0c1050d0 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.939+0000 7eff10889640 1 -- 172.21.15.29:0/471291432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7eff04005020 con 0x7eff0c1050d0 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.939+0000 7eff1188b640 1 -- 172.21.15.29:0/471291432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0c19bbb0 con 0x7eff0c1050d0 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.939+0000 7eff10889640 1 -- 172.21.15.29:0/471291432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0400f6b0 con 0x7eff0c1050d0 2024-09-21T02:40:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.939+0000 7eff1188b640 1 -- 172.21.15.29:0/471291432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff0c1a22a0 con 0x7eff0c1050d0 2024-09-21T02:40:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff187f6740 1 Processor -- start 2024-09-21T02:40:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff10889640 1 -- 172.21.15.29:0/471291432 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7eff040051d0 con 0x7eff0c1050d0 2024-09-21T02:40:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff187f6740 1 -- start start 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff187f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04b041b0 0x55ea04eabe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff187f6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04eac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff187f6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 0x55ea04ead010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff187f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ea04e1e6f0 con 0x55ea04b041b0 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff187f6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55ea04e0db20 con 0x55ea04ae34d0 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff187f6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ea04e11930 con 0x55ea04adb2f0 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.942+0000 7eff027fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04b041b0 0x55ea04eabe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04b041b0 0x55ea04eabe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51536/0 (socket says 172.21.15.29:51536) 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 -- 172.21.15.29:0/3916968405 learned_addr learned my addr 172.21.15.29:0/3916968405 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff0affd640 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04eac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 0x55ea04ead010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff10889640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efef80b6590 0x7efef80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 -- 172.21.15.29:0/3916968405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 msgr2=0x55ea04eac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04eac4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 -- 172.21.15.29:0/3916968405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 msgr2=0x55ea04ead010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 0x55ea04ead010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 -- 172.21.15.29:0/3916968405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea04dd3530 con 0x55ea04b041b0 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff0b7fe640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efef80b6590 0x7efef80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff027fc640 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04b041b0 0x55ea04eabe50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7efef4013990 tx=0x7efef400b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578efc0b7a501e3c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff10889640 1 -- 172.21.15.29:0/471291432 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7eff040e7a40 con 0x7eff0c1050d0 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff01ffb640 1 -- 172.21.15.29:0/3916968405 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4011780 con 0x55ea04b041b0 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff01ffb640 1 -- 172.21.15.29:0/3916968405 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efef4011db0 con 0x55ea04b041b0 2024-09-21T02:40:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.943+0000 7eff01ffb640 1 -- 172.21.15.29:0/3916968405 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef400fa60 con 0x55ea04b041b0 2024-09-21T02:40:36.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff187f6740 1 -- 172.21.15.29:0/3916968405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04b041b0 msgr2=0x55ea04eabe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff0b7fe640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efef80b6590 0x7efef80b8aa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7eff040a9e20 tx=0x7eff0407ab80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff187f6740 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04b041b0 0x55ea04eabe50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7efef4013990 tx=0x7efef400b6f0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff187f6740 1 -- 172.21.15.29:0/3916968405 shutdown_connections 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff187f6740 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 0x55ea04ead010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff187f6740 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04eac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff187f6740 1 --2- 172.21.15.29:0/3916968405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04b041b0 0x55ea04eabe50 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff187f6740 1 -- 172.21.15.29:0/3916968405 >> 172.21.15.29:0/3916968405 conn(0x55ea04af8300 msgr2=0x55ea04ea78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.944+0000 7eff187f6740 1 -- 172.21.15.29:0/3916968405 shutdown_connections 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 -- 172.21.15.29:0/3916968405 wait complete. 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 -- start start 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 Processor -- start 2024-09-21T02:40:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04eb55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04eb5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04eb6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ea04d45ab0 con 0x55ea04ae34d0 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55ea04db3730 con 0x55ea04b041b0 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.945+0000 7eff187f6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ea04eadad0 con 0x55ea04adb2f0 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff187f6740 10 client.0 _ll_register_callbacks cb 0x55ea04e076b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04eb5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04eb5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51544/0 (socket says 172.21.15.29:51544) 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 -- 172.21.15.29:0/3431904880 learned_addr learned my addr 172.21.15.29:0/3431904880 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff027fc640 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04eb55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff027fc640 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04eb6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 -- 172.21.15.29:0/3431904880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 msgr2=0x55ea04eb55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04eb55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 -- 172.21.15.29:0/3431904880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 msgr2=0x55ea04eb6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04eb6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 -- 172.21.15.29:0/3431904880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea04b6f790 con 0x55ea04ae34d0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff027fc640 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04eb6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff0affd640 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04eb5ca0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7efefc0096c0 tx=0x7efefc031c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc672dbac23cb780 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7efeeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7efeeaffd640 1 -- 172.21.15.29:0/3431904880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc035040 con 0x55ea04ae34d0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7efeeaffd640 1 -- 172.21.15.29:0/3431904880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7efefc004450 con 0x55ea04ae34d0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.946+0000 7eff187f6740 1 -- 172.21.15.29:0/3431904880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efef4011db0 con 0x55ea04ae34d0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7efeeaffd640 1 -- 172.21.15.29:0/3431904880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efefc04e050 con 0x55ea04ae34d0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff187f6740 10 client.15810 fetch_fsmap learned FSMap version 30 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff187f6740 1 -- 172.21.15.29:0/3431904880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ea04be8620 con 0x55ea04ae34d0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7efeeaffd640 1 -- 172.21.15.29:0/3431904880 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efef4011db0 con 0x55ea04ae34d0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7efeeaffd640 1 -- 172.21.15.29:0/3431904880 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7efefc03d5e0 con 0x55ea04ae34d0 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff187f6740 10 client.15810 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff187f6740 10 client.15810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff187f6740 20 client.15810 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff187f6740 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ea04ac2d80 0x55ea04ec2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff187f6740 4 client.15810 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:40:36.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff187f6740 1 -- 172.21.15.29:0/3431904880 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea04e20640 con 0x55ea04ac2d80 2024-09-21T02:40:36.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.947+0000 7eff027fc640 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ea04ac2d80 0x55ea04ec2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.948+0000 7eff027fc640 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ea04ac2d80 0x55ea04ec2d50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfd561046456cee3 server_cookie=d320d6ba5562ea77 in_seq=0 out_seq=0 2024-09-21T02:40:36.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.948+0000 7efeeaffd640 10 client.15810 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:36.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7efeeaffd640 1 -- 172.21.15.29:0/3431904880 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ea04e20640 con 0x55ea04ac2d80 2024-09-21T02:40:36.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7efeeaffd640 10 client.15810 handle_command_reply: tid=0 2024-09-21T02:40:36.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 client.15810 shutdown 2024-09-21T02:40:36.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 -- 172.21.15.29:0/3431904880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 msgr2=0x55ea04eb5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:36.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04eb5ca0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7efefc0096c0 tx=0x7efefc031c60 comp rx=0 tx=0).stop 2024-09-21T02:40:36.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 -- 172.21.15.29:0/3431904880 shutdown_connections 2024-09-21T02:40:36.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04eb6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ea04ac2d80 0x55ea04ec2d50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04eb5ca0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 --2- 172.21.15.29:0/3431904880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04eb55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7efeeaffd640 0 client.15810 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:36.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 -- 172.21.15.29:0/3431904880 >> 172.21.15.29:0/3431904880 conn(0x55ea04af8300 msgr2=0x55ea04ea5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:36.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.949+0000 7eff187f6740 1 -- 172.21.15.29:0/3431904880 shutdown_connections 2024-09-21T02:40:36.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.950+0000 7eff187f6740 1 -- 172.21.15.29:0/3431904880 wait complete. 2024-09-21T02:40:36.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.950+0000 7eff187f6740 20 client.15810 trim_cache size 0 max 16384 2024-09-21T02:40:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.964+0000 7eff187f6740 1 Processor -- start 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.964+0000 7eff187f6740 1 -- start start 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff187f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04adb2f0 0x55ea04eac500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff187f6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 0x55ea04eb55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff187f6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04b041b0 0x55ea04eb5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff187f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c1ac590 con 0x55ea04adb2f0 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff187f6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c1ae100 con 0x55ea04ae34d0 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff187f6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c1acc60 con 0x55ea04b041b0 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04adb2f0 0x55ea04eac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04adb2f0 0x55ea04eac500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51558/0 (socket says 172.21.15.29:51558) 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 -- 172.21.15.29:0/2964902705 learned_addr learned my addr 172.21.15.29:0/2964902705 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff0affd640 1 --2- 172.21.15.29:0/2964902705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 0x55ea04eb55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 -- 172.21.15.29:0/2964902705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04b041b0 msgr2=0x55ea04eb5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 --2- 172.21.15.29:0/2964902705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04b041b0 0x55ea04eb5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 -- 172.21.15.29:0/2964902705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 msgr2=0x55ea04eb55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 --2- 172.21.15.29:0/2964902705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 0x55ea04eb55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 -- 172.21.15.29:0/2964902705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea04ea6fd0 con 0x55ea04adb2f0 2024-09-21T02:40:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff027fc640 1 --2- 172.21.15.29:0/2964902705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04adb2f0 0x55ea04eac500 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7efef4013990 tx=0x7efef4013f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1491903b6ee27a73 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.965+0000 7eff017fa640 1 -- 172.21.15.29:0/2964902705 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef402f070 con 0x55ea04adb2f0 2024-09-21T02:40:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff017fa640 1 -- 172.21.15.29:0/2964902705 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efef4011750 con 0x55ea04adb2f0 2024-09-21T02:40:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff017fa640 1 -- 172.21.15.29:0/2964902705 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef40048a0 con 0x55ea04adb2f0 2024-09-21T02:40:36.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 -- 172.21.15.29:0/2964902705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04adb2f0 msgr2=0x55ea04eac500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:36.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 --2- 172.21.15.29:0/2964902705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04adb2f0 0x55ea04eac500 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7efef4013990 tx=0x7efef4013f10 comp rx=0 tx=0).stop 2024-09-21T02:40:36.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 -- 172.21.15.29:0/2964902705 shutdown_connections 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 --2- 172.21.15.29:0/2964902705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04b041b0 0x55ea04eb5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 --2- 172.21.15.29:0/2964902705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04ae34d0 0x55ea04eb55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 --2- 172.21.15.29:0/2964902705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04adb2f0 0x55ea04eac500 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 -- 172.21.15.29:0/2964902705 >> 172.21.15.29:0/2964902705 conn(0x55ea04af8300 msgr2=0x55ea04eabdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 -- 172.21.15.29:0/2964902705 shutdown_connections 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.966+0000 7eff187f6740 1 -- 172.21.15.29:0/2964902705 wait complete. 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 1 -- start start 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 1 Processor -- start 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04ebcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04ebbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04ebc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c1ada00 con 0x55ea04ae34d0 2024-09-21T02:40:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c1ad330 con 0x55ea04b041b0 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c1aa570 con 0x55ea04adb2f0 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff187f6740 10 client.0 _ll_register_callbacks cb 0x7efefc005e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff0affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04ebbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff0affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04ebbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51572/0 (socket says 172.21.15.29:51572) 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.967+0000 7eff0affd640 1 -- 172.21.15.29:0/3529621251 learned_addr learned my addr 172.21.15.29:0/3529621251 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff027fc640 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04ebc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff027fc640 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04ebcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff0affd640 1 -- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 msgr2=0x55ea04ebcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff0affd640 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04ebcf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff0affd640 1 -- 172.21.15.29:0/3529621251 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 msgr2=0x55ea04ebc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff0affd640 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04ebc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff0affd640 1 -- 172.21.15.29:0/3529621251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea04ebc930 con 0x55ea04ae34d0 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff027fc640 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04ebcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:40:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff0affd640 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04ebbbe0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7efefc009850 tx=0x7efefc03dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4b0c12fee0eed8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7efeeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7efeeaffd640 1 -- 172.21.15.29:0/3529621251 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc056020 con 0x55ea04ae34d0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7efeeaffd640 1 -- 172.21.15.29:0/3529621251 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7efefc004460 con 0x55ea04ae34d0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff187f6740 1 -- 172.21.15.29:0/3529621251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efef4011750 con 0x55ea04ae34d0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7efeeaffd640 1 -- 172.21.15.29:0/3529621251 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efefc061050 con 0x55ea04ae34d0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff187f6740 10 client.15816 fetch_fsmap learned FSMap version 30 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.968+0000 7eff187f6740 1 -- 172.21.15.29:0/3529621251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ea04f2d770 con 0x55ea04ae34d0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7efeeaffd640 1 -- 172.21.15.29:0/3529621251 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efef4011750 con 0x55ea04ae34d0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7efeeaffd640 1 -- 172.21.15.29:0/3529621251 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7efefc003a10 con 0x55ea04ae34d0 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7eff187f6740 10 client.15816 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7eff187f6740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7eff187f6740 20 client.15816 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7eff187f6740 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ea04a981c0 0x55ea04f30170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7eff187f6740 4 client.15816 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:40:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7eff187f6740 1 -- 172.21.15.29:0/3529621251 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ea04e1c0f0 con 0x55ea04a981c0 2024-09-21T02:40:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7eff027fc640 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ea04a981c0 0x55ea04f30170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7efeeaffd640 10 client.15816 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:36.969+0000 7eff027fc640 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ea04a981c0 0x55ea04f30170 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c74b864341a8275 server_cookie=398ac96fa73348b3 in_seq=0 out_seq=0 2024-09-21T02:40:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7efeeaffd640 1 -- 172.21.15.29:0/3529621251 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ea04e1c0f0 con 0x55ea04a981c0 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7efeeaffd640 10 client.15816 handle_command_reply: tid=0 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 client.15816 shutdown 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 -- 172.21.15.29:0/3529621251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 msgr2=0x55ea04ebbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04ebbbe0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7efefc009850 tx=0x7efefc03dda0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 -- 172.21.15.29:0/3529621251 shutdown_connections 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ea04b041b0 0x55ea04ebc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ea04a981c0 0x55ea04f30170 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea04ae34d0 0x55ea04ebbbe0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 --2- 172.21.15.29:0/3529621251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04adb2f0 0x55ea04ebcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7efeeaffd640 0 client.15816 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 -- 172.21.15.29:0/3529621251 >> 172.21.15.29:0/3529621251 conn(0x55ea04af8300 msgr2=0x55ea04eb55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 -- 172.21.15.29:0/3529621251 shutdown_connections 2024-09-21T02:40:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 1 -- 172.21.15.29:0/3529621251 wait complete. 2024-09-21T02:40:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.001+0000 7eff187f6740 20 client.15816 trim_cache size 0 max 16384 2024-09-21T02:40:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:40:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:40:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:40:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 -- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efef80b6590 msgr2=0x7efef80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efef80b6590 0x7efef80b8aa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7eff040a9e20 tx=0x7eff0407ab80 comp rx=0 tx=0).stop 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 -- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1050d0 msgr2=0x7eff0c19b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1050d0 0x7eff0c19b0a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7eff0400cad0 tx=0x7eff0400cf90 comp rx=0 tx=0).stop 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 -- 172.21.15.29:0/471291432 shutdown_connections 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7efef80b6590 0x7efef80b8aa0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c1050d0 0x7eff0c19b0a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff0c104730 0x7eff0c19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 --2- 172.21.15.29:0/471291432 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7eff0c1036b0 0x7eff0c199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 -- 172.21.15.29:0/471291432 >> 172.21.15.29:0/471291432 conn(0x7eff0c0feff0 msgr2=0x7eff0c107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 -- 172.21.15.29:0/471291432 shutdown_connections 2024-09-21T02:40:37.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:37.002+0000 7eff1188b640 1 -- 172.21.15.29:0/471291432 wait complete. 2024-09-21T02:40:37.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 3.25 MB/sec warmup 14 sec latency 5469.833 ms 2024-09-21T02:40:37.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:37 smithi066 ceph-mon[33436]: pgmap v434: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-21T02:40:37.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:37 smithi190 ceph-mon[33871]: pgmap v434: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-21T02:40:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:37 smithi029 ceph-mon[31006]: pgmap v434: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-21T02:40:38.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 3.03 MB/sec warmup 15 sec latency 6469.911 ms 2024-09-21T02:40:39.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 2.84 MB/sec warmup 16 sec latency 7469.969 ms 2024-09-21T02:40:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:39 smithi066 ceph-mon[33436]: pgmap v435: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-09-21T02:40:39.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:39 smithi190 ceph-mon[33871]: pgmap v435: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-09-21T02:40:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:39 smithi029 ceph-mon[31006]: pgmap v435: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-09-21T02:40:40.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 2.67 MB/sec warmup 17 sec latency 8470.097 ms 2024-09-21T02:40:41.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 2.53 MB/sec warmup 18 sec latency 9470.182 ms 2024-09-21T02:40:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:41 smithi066 ceph-mon[33436]: pgmap v436: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 872 KiB/s wr, 91 op/s 2024-09-21T02:40:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:41 smithi190 ceph-mon[33871]: pgmap v436: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 872 KiB/s wr, 91 op/s 2024-09-21T02:40:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:41 smithi029 ceph-mon[31006]: pgmap v436: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 872 KiB/s wr, 91 op/s 2024-09-21T02:40:42.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 2.39 MB/sec warmup 19 sec latency 10470.286 ms 2024-09-21T02:40:43.065 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 2.27 MB/sec warmup 20 sec latency 11470.360 ms 2024-09-21T02:40:43.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:43 smithi190 ceph-mon[33871]: pgmap v437: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 495 KiB/s wr, 61 op/s 2024-09-21T02:40:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:43 smithi029 ceph-mon[31006]: pgmap v437: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 495 KiB/s wr, 61 op/s 2024-09-21T02:40:44.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:43 smithi066 ceph-mon[33436]: pgmap v437: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 495 KiB/s wr, 61 op/s 2024-09-21T02:40:44.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 2.17 MB/sec warmup 21 sec latency 12470.455 ms 2024-09-21T02:40:45.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 2.07 MB/sec warmup 22 sec latency 13470.557 ms 2024-09-21T02:40:45.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:40:45.612 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:40:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:45 smithi029 ceph-mon[31006]: pgmap v438: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 210 KiB/s wr, 58 op/s 2024-09-21T02:40:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:45 smithi190 ceph-mon[33871]: pgmap v438: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 210 KiB/s wr, 58 op/s 2024-09-21T02:40:46.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:45 smithi066 ceph-mon[33436]: pgmap v438: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 210 KiB/s wr, 58 op/s 2024-09-21T02:40:46.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 1.98 MB/sec warmup 23 sec latency 14470.657 ms 2024-09-21T02:40:47.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 1.89 MB/sec warmup 24 sec latency 15470.747 ms 2024-09-21T02:40:47.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:47 smithi190 ceph-mon[33871]: pgmap v439: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 60 KiB/s wr, 24 op/s 2024-09-21T02:40:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:47 smithi029 ceph-mon[31006]: pgmap v439: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 60 KiB/s wr, 24 op/s 2024-09-21T02:40:48.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:47 smithi066 ceph-mon[33436]: pgmap v439: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 60 KiB/s wr, 24 op/s 2024-09-21T02:40:48.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 1.82 MB/sec warmup 25 sec latency 16470.797 ms 2024-09-21T02:40:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 -- 172.21.15.29:0/1000876301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8108cf0 msgr2=0x7f02f810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 --2- 172.21.15.29:0/1000876301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8108cf0 0x7f02f810f520 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009ca0 tx=0x7f02f002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:40:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 -- 172.21.15.29:0/1000876301 shutdown_connections 2024-09-21T02:40:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 --2- 172.21.15.29:0/1000876301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8108cf0 0x7f02f810f520 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 --2- 172.21.15.29:0/1000876301 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02f81083a0 0x7f02f81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 --2- 172.21.15.29:0/1000876301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02f8102a60 0x7f02f8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 -- 172.21.15.29:0/1000876301 >> 172.21.15.29:0/1000876301 conn(0x7f02f8079730 msgr2=0x7f02f8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 -- 172.21.15.29:0/1000876301 shutdown_connections 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.968+0000 7f02fc817640 1 -- 172.21.15.29:0/1000876301 wait complete. 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.969+0000 7f02fc817640 1 Processor -- start 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.969+0000 7f02fc817640 1 -- start start 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02fc817640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8102a60 0x7f02f819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02fc817640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02f81083a0 0x7f02f819b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02fc817640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02f8108cf0 0x7f02f819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02fc817640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8112940 con 0x7f02f8102a60 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02fc817640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81127c0 con 0x7f02f8108cf0 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02fc817640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8112ac0 con 0x7f02f81083a0 2024-09-21T02:40:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8102a60 0x7f02f819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8102a60 0x7f02f819b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35692/0 (socket says 172.21.15.29:35692) 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f659c640 1 -- 172.21.15.29:0/2874245454 learned_addr learned my addr 172.21.15.29:0/2874245454 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f5d9b640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02f81083a0 0x7f02f819b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f6d9d640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02f8108cf0 0x7f02f819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f659c640 1 -- 172.21.15.29:0/2874245454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02f81083a0 msgr2=0x7f02f819b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f659c640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02f81083a0 0x7f02f819b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f659c640 1 -- 172.21.15.29:0/2874245454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02f8108cf0 msgr2=0x7f02f819bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f659c640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02f8108cf0 0x7f02f819bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f659c640 1 -- 172.21.15.29:0/2874245454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f81a6520 con 0x7f02f8102a60 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.970+0000 7f02f6d9d640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02f8108cf0 0x7f02f819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.971+0000 7f02f659c640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8102a60 0x7f02f819b290 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f02e800b4a0 tx=0x7f02e800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.971+0000 7f02e77fe640 1 -- 172.21.15.29:0/2874245454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8015020 con 0x7f02f8102a60 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.971+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f81a6810 con 0x7f02f8102a60 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.971+0000 7f02e77fe640 1 -- 172.21.15.29:0/2874245454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e8011cc0 con 0x7f02f8102a60 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.971+0000 7f02e77fe640 1 -- 172.21.15.29:0/2874245454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8010d00 con 0x7f02f8102a60 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.971+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f81a6d00 con 0x7f02f8102a60 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.972+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f8102e70 con 0x7f02f8102a60 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.974+0000 7f02e77fe640 1 -- 172.21.15.29:0/2874245454 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f02e8018020 con 0x7f02f8102a60 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.974+0000 7f02e77fe640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02e00b6540 0x7f02e00b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.975+0000 7f02f5d9b640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02e00b6540 0x7f02e00b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.975+0000 7f02e77fe640 1 -- 172.21.15.29:0/2874245454 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f02e80e7420 con 0x7f02f8102a60 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.975+0000 7f02f5d9b640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02e00b6540 0x7f02e00b8a50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f02f807a060 tx=0x7f02ec009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:48.978+0000 7f02e77fe640 1 -- 172.21.15.29:0/2874245454 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f02e801f020 con 0x7f02f8102a60 2024-09-21T02:40:49.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 1.75 MB/sec warmup 26 sec latency 17470.849 ms 2024-09-21T02:40:49.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.206+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02f806b270 con 0x7f02f8102a60 2024-09-21T02:40:49.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.206+0000 7f02e77fe640 1 -- 172.21.15.29:0/2874245454 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11972 (secure 0 0 0) 0x7f02f806b270 con 0x7f02f8102a60 2024-09-21T02:40:49.208 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:40:49.208 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"btime":"2024-09-21T02:40:26:480768+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:40:26.304013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[3],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:40:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 30 2024-09-21T02:40:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02e00b6540 msgr2=0x7f02e00b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02e00b6540 0x7f02e00b8a50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f02f807a060 tx=0x7f02ec009110 comp rx=0 tx=0).stop 2024-09-21T02:40:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8102a60 msgr2=0x7f02f819b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8102a60 0x7f02f819b290 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f02e800b4a0 tx=0x7f02e800b8a0 comp rx=0 tx=0).stop 2024-09-21T02:40:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 shutdown_connections 2024-09-21T02:40:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02e00b6540 0x7f02e00b8a50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02f8108cf0 0x7f02f819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02f81083a0 0x7f02f819b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 --2- 172.21.15.29:0/2874245454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f8102a60 0x7f02f819b290 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 >> 172.21.15.29:0/2874245454 conn(0x7f02f8079730 msgr2=0x7f02f8105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 shutdown_connections 2024-09-21T02:40:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:49.209+0000 7f02fc817640 1 -- 172.21.15.29:0/2874245454 wait complete. 2024-09-21T02:40:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:49 smithi190 ceph-mon[33871]: pgmap v440: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-09-21T02:40:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:49 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2874245454' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:40:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:49 smithi029 ceph-mon[31006]: pgmap v440: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-09-21T02:40:49.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:49 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2874245454' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:40:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:49 smithi066 ceph-mon[33436]: pgmap v440: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-09-21T02:40:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:49 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2874245454' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:40:50.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5410 1.68 MB/sec warmup 27 sec latency 18470.898 ms 2024-09-21T02:40:50.078 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 473fc997 --release --await' 2024-09-21T02:40:50.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 473fc997 --release --await 2024-09-21T02:40:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 -- 172.21.15.29:0/4133400854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103620 msgr2=0x7f0980103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 --2- 172.21.15.29:0/4133400854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103620 0x7f0980103a30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0974009900 tx=0x7f097402a410 comp rx=0 tx=0).stop 2024-09-21T02:40:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 -- 172.21.15.29:0/4133400854 shutdown_connections 2024-09-21T02:40:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 --2- 172.21.15.29:0/4133400854 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0980103fc0 0x7f098010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 --2- 172.21.15.29:0/4133400854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103620 0x7f0980103a30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 --2- 172.21.15.29:0/4133400854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09801025a0 0x7f0980102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 -- 172.21.15.29:0/4133400854 >> 172.21.15.29:0/4133400854 conn(0x7f09800fdee0 msgr2=0x7f0980100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 -- 172.21.15.29:0/4133400854 shutdown_connections 2024-09-21T02:40:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.108+0000 7f098677b640 1 -- 172.21.15.29:0/4133400854 wait complete. 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f098677b640 1 Processor -- start 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f098677b640 1 -- start start 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f098677b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09801025a0 0x7f0980199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f098677b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0980103620 0x7f098019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f098677b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103fc0 0x7f098019b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f098677b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f098010e090 con 0x7f0980103fc0 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f098677b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f098010df10 con 0x7f0980103620 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f098677b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f098010e210 con 0x7f09801025a0 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f0984d18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103fc0 0x7f098019b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0980103620 0x7f098019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0980103620 0x7f098019a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:55252/0 (socket says 172.21.15.29:55252) 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.109+0000 7f097f7fe640 1 -- 172.21.15.29:0/4205224696 learned_addr learned my addr 172.21.15.29:0/4205224696 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f097ffff640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09801025a0 0x7f0980199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f0984d18640 1 -- 172.21.15.29:0/4205224696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09801025a0 msgr2=0x7f0980199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f0984d18640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09801025a0 0x7f0980199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f0984d18640 1 -- 172.21.15.29:0/4205224696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0980103620 msgr2=0x7f098019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f0984d18640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0980103620 0x7f098019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f0984d18640 1 -- 172.21.15.29:0/4205224696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f098019b8b0 con 0x7f0980103fc0 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f097f7fe640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0980103620 0x7f098019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f0984d18640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103fc0 0x7f098019b090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f097800eb20 tx=0x7f097800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f097d7fa640 1 -- 172.21.15.29:0/4205224696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09780039c0 con 0x7f0980103fc0 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f097d7fa640 1 -- 172.21.15.29:0/4205224696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0978004020 con 0x7f0980103fc0 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f098677b640 1 -- 172.21.15.29:0/4205224696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f098019bba0 con 0x7f0980103fc0 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f097d7fa640 1 -- 172.21.15.29:0/4205224696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0978010690 con 0x7f0980103fc0 2024-09-21T02:40:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.110+0000 7f098677b640 1 -- 172.21.15.29:0/4205224696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09801a2290 con 0x7f0980103fc0 2024-09-21T02:40:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.113+0000 7f098d7c3740 1 Processor -- start 2024-09-21T02:40:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.113+0000 7f097d7fa640 1 -- 172.21.15.29:0/4205224696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f0978003b60 con 0x7f0980103fc0 2024-09-21T02:40:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.113+0000 7f098d7c3740 1 -- start start 2024-09-21T02:40:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89871b0 0x5617c8d2ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89664d0 0x5617c8d30010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f098d7c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5617c8ca16f0 con 0x5617c89871b0 2024-09-21T02:40:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f098d7c3740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5617c8c90b20 con 0x5617c895e2f0 2024-09-21T02:40:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f098d7c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5617c8c94930 con 0x5617c89664d0 2024-09-21T02:40:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89871b0 0x5617c8d2ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89871b0 0x5617c8d2ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35718/0 (socket says 172.21.15.29:35718) 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 -- 172.21.15.29:0/3966434919 learned_addr learned my addr 172.21.15.29:0/3966434919 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097ffff640 1 --2- 172.21.15.29:0/3966434919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 -- 172.21.15.29:0/3966434919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89664d0 msgr2=0x5617c8d30010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3966434919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89664d0 0x5617c8d30010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 -- 172.21.15.29:0/3966434919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 msgr2=0x5617c8d2f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3966434919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 -- 172.21.15.29:0/3966434919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617c8c56530 con 0x5617c89871b0 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3966434919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89871b0 0x5617c8d2ee50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0974009a20 tx=0x7f0974009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62496025f56aa866 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.114+0000 7f09667fc640 1 -- 172.21.15.29:0/3966434919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974035070 con 0x5617c89871b0 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.115+0000 7f09667fc640 1 -- 172.21.15.29:0/3966434919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0974041070 con 0x5617c89871b0 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.115+0000 7f097d7fa640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09680b6510 0x7f09680b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.115+0000 7f09667fc640 1 -- 172.21.15.29:0/3966434919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974003c00 con 0x5617c89871b0 2024-09-21T02:40:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.115+0000 7f097ffff640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09680b6510 0x7f09680b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.115+0000 7f097d7fa640 1 -- 172.21.15.29:0/4205224696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f09780e7cc0 con 0x7f0980103fc0 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.115+0000 7f098d7c3740 1 -- 172.21.15.29:0/3966434919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89871b0 msgr2=0x5617c8d2ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.115+0000 7f097ffff640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09680b6510 0x7f09680b8a20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f097000bc00 tx=0x7f097000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.115+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3966434919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89871b0 0x5617c8d2ee50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0974009a20 tx=0x7f0974009670 comp rx=0 tx=0).stop 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 -- 172.21.15.29:0/3966434919 shutdown_connections 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3966434919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89664d0 0x5617c8d30010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3966434919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3966434919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89871b0 0x5617c8d2ee50 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 -- 172.21.15.29:0/3966434919 >> 172.21.15.29:0/3966434919 conn(0x5617c897b300 msgr2=0x5617c8d2a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 -- 172.21.15.29:0/3966434919 shutdown_connections 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 -- 172.21.15.29:0/3966434919 wait complete. 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 -- start start 2024-09-21T02:40:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 Processor -- start 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d385d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d38c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.116+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d39800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f098d7c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5617c8bc8ab0 con 0x5617c89664d0 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f098d7c3740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5617c8c36730 con 0x5617c895e2f0 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f098d7c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5617c8d30ad0 con 0x5617c89871b0 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f098d7c3740 10 client.0 _ll_register_callbacks cb 0x5617c8c8a6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d38c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d38c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35734/0 (socket says 172.21.15.29:35734) 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 -- 172.21.15.29:0/3537739370 learned_addr learned my addr 172.21.15.29:0/3537739370 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f0984d18640 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d39800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097ffff640 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d385d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 -- 172.21.15.29:0/3537739370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 msgr2=0x5617c8d39800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d39800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 -- 172.21.15.29:0/3537739370 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 msgr2=0x5617c8d385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d385d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 -- 172.21.15.29:0/3537739370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617c89f2790 con 0x5617c89664d0 2024-09-21T02:40:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d38c90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0974005ab0 tx=0x7f097403c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7baa7fb47ea40ec5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f095f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:40:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f095f7fe640 1 -- 172.21.15.29:0/3537739370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974035050 con 0x5617c89664d0 2024-09-21T02:40:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f095f7fe640 1 -- 172.21.15.29:0/3537739370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f097403cc70 con 0x5617c89664d0 2024-09-21T02:40:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.117+0000 7f098d7c3740 1 -- 172.21.15.29:0/3537739370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0974041070 con 0x5617c89664d0 2024-09-21T02:40:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f095f7fe640 1 -- 172.21.15.29:0/3537739370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0974058050 con 0x5617c89664d0 2024-09-21T02:40:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f098d7c3740 10 client.15834 fetch_fsmap learned FSMap version 30 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f098d7c3740 1 -- 172.21.15.29:0/3537739370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5617c8a6b620 con 0x5617c89664d0 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f095f7fe640 1 -- 172.21.15.29:0/3537739370 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0974041070 con 0x5617c89664d0 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f095f7fe640 1 -- 172.21.15.29:0/3537739370 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f0974046460 con 0x5617c89664d0 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f098d7c3740 10 client.15834 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f098d7c3740 10 client.15834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f098d7c3740 20 client.15834 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617c8945d80 0x5617c8d458e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f098d7c3740 4 client.15834 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f098d7c3740 1 -- 172.21.15.29:0/3537739370 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617c8ca3640 con 0x5617c8945d80 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.118+0000 7f097ffff640 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617c8945d80 0x5617c8d458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.119+0000 7f097ffff640 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617c8945d80 0x5617c8d458e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c86a2b5a43096b6 server_cookie=513e55fb6205e722 in_seq=0 out_seq=0 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.119+0000 7f095f7fe640 10 client.15834 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f095f7fe640 1 -- 172.21.15.29:0/3537739370 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5617c8ca3640 con 0x5617c8945d80 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f095f7fe640 10 client.15834 handle_command_reply: tid=0 2024-09-21T02:40:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 client.15834 shutdown 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 -- 172.21.15.29:0/3537739370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 msgr2=0x5617c8d38c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d38c90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0974005ab0 tx=0x7f097403c670 comp rx=0 tx=0).stop 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 -- 172.21.15.29:0/3537739370 shutdown_connections 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d39800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617c8945d80 0x5617c8d458e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d38c90 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3537739370 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d385d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f095f7fe640 0 client.15834 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 -- 172.21.15.29:0/3537739370 >> 172.21.15.29:0/3537739370 conn(0x5617c897b300 msgr2=0x5617c8d28ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 -- 172.21.15.29:0/3537739370 shutdown_connections 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 1 -- 172.21.15.29:0/3537739370 wait complete. 2024-09-21T02:40:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.120+0000 7f098d7c3740 20 client.15834 trim_cache size 0 max 16384 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.135+0000 7f098d7c3740 1 Processor -- start 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.135+0000 7f098d7c3740 1 -- start start 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.135+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.135+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d406e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.135+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d36b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.135+0000 7f098d7c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09801ac4a0 con 0x5617c89664d0 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.135+0000 7f098d7c3740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f09801ae010 con 0x5617c895e2f0 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.135+0000 7f098d7c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f09801acb70 con 0x5617c89871b0 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d406e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35736/0 (socket says 172.21.15.29:35736) 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 -- 172.21.15.29:0/4198009802 learned_addr learned my addr 172.21.15.29:0/4198009802 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f0984d18640 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d36b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097ffff640 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 -- 172.21.15.29:0/4198009802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 msgr2=0x5617c8d36b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d36b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 -- 172.21.15.29:0/4198009802 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 msgr2=0x5617c8d2fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 -- 172.21.15.29:0/4198009802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617c8d30c50 con 0x5617c89664d0 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097ffff640 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f097f7fe640 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d406e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f09740467d0 tx=0x7f0974005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506b3d111f40b300 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f0965ffb640 1 -- 172.21.15.29:0/4198009802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974035040 con 0x5617c89664d0 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f0965ffb640 1 -- 172.21.15.29:0/4198009802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0974041020 con 0x5617c89664d0 2024-09-21T02:40:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.136+0000 7f0965ffb640 1 -- 172.21.15.29:0/4198009802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974003900 con 0x5617c89664d0 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 -- 172.21.15.29:0/4198009802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 msgr2=0x5617c8d406e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d406e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f09740467d0 tx=0x7f0974005720 comp rx=0 tx=0).stop 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 -- 172.21.15.29:0/4198009802 shutdown_connections 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d36b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d406e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 --2- 172.21.15.29:0/4198009802 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d2fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 -- 172.21.15.29:0/4198009802 >> 172.21.15.29:0/4198009802 conn(0x5617c897b300 msgr2=0x5617c8d2a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 -- 172.21.15.29:0/4198009802 shutdown_connections 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.137+0000 7f098d7c3740 1 -- 172.21.15.29:0/4198009802 wait complete. 2024-09-21T02:40:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 1 -- start start 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 1 Processor -- start 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d404b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d3f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d3f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09801ad910 con 0x5617c89664d0 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f09801ad240 con 0x5617c895e2f0 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f09801aa480 con 0x5617c89871b0 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f098d7c3740 10 client.0 _ll_register_callbacks cb 0x7f09740095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d3f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f097f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d3f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35744/0 (socket says 172.21.15.29:35744) 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f097f7fe640 1 -- 172.21.15.29:0/3664593026 learned_addr learned my addr 172.21.15.29:0/3664593026 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.138+0000 7f097ffff640 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f0984d18640 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d3f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f097f7fe640 1 -- 172.21.15.29:0/3664593026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 msgr2=0x5617c8d3f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d3f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f097f7fe640 1 -- 172.21.15.29:0/3664593026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 msgr2=0x5617c8d404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d404b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f097f7fe640 1 -- 172.21.15.29:0/3664593026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617c8d3ff80 con 0x5617c89664d0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f097f7fe640 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d3f170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f09740468f0 tx=0x7f09740319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d7a8a813a88af9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f095f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f095f7fe640 1 -- 172.21.15.29:0/3664593026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974035070 con 0x5617c89664d0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f095f7fe640 1 -- 172.21.15.29:0/3664593026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f09740469c0 con 0x5617c89664d0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f098d7c3740 1 -- 172.21.15.29:0/3664593026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0974041020 con 0x5617c89664d0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f095f7fe640 1 -- 172.21.15.29:0/3664593026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f097406d050 con 0x5617c89664d0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f098d7c3740 10 client.15840 fetch_fsmap learned FSMap version 30 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.139+0000 7f098d7c3740 1 -- 172.21.15.29:0/3664593026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5617c8db02d0 con 0x5617c89664d0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f095f7fe640 1 -- 172.21.15.29:0/3664593026 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0974041020 con 0x5617c89664d0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f095f7fe640 1 -- 172.21.15.29:0/3664593026 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f097404fc00 con 0x5617c89664d0 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f098d7c3740 10 client.15840 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:40:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f098d7c3740 10 client.15840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:40:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f098d7c3740 20 client.15840 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617c891b1c0 0x5617c8db2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f098d7c3740 4 client.15840 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:40:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f098d7c3740 1 -- 172.21.15.29:0/3664593026 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5617c8c9f0f0 con 0x5617c891b1c0 2024-09-21T02:40:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.140+0000 7f097ffff640 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617c891b1c0 0x5617c8db2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.141+0000 7f095f7fe640 10 client.15840 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.141+0000 7f097ffff640 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617c891b1c0 0x5617c8db2d20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=548003f6ef09e7ac server_cookie=85370b302ca52010 in_seq=0 out_seq=0 2024-09-21T02:40:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.156+0000 7f095f7fe640 1 -- 172.21.15.29:0/3664593026 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5617c8c9f0f0 con 0x5617c891b1c0 2024-09-21T02:40:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f095f7fe640 10 client.15840 handle_command_reply: tid=0 2024-09-21T02:40:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 client.15840 shutdown 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 -- 172.21.15.29:0/3664593026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 msgr2=0x5617c8d3f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d3f170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f09740468f0 tx=0x7f09740319f0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 -- 172.21.15.29:0/3664593026 shutdown_connections 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617c89871b0 0x5617c8d3f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617c891b1c0 0x5617c8db2d20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617c89664d0 0x5617c8d3f170 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 --2- 172.21.15.29:0/3664593026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617c895e2f0 0x5617c8d404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f095f7fe640 0 client.15840 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 -- 172.21.15.29:0/3664593026 >> 172.21.15.29:0/3664593026 conn(0x5617c897b300 msgr2=0x5617c8d366e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 -- 172.21.15.29:0/3664593026 shutdown_connections 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 1 -- 172.21.15.29:0/3664593026 wait complete. 2024-09-21T02:40:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.157+0000 7f098d7c3740 20 client.15840 trim_cache size 0 max 16384 2024-09-21T02:40:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:40:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:40:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:40:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 -- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09680b6510 msgr2=0x7f09680b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09680b6510 0x7f09680b8a20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f097000bc00 tx=0x7f097000b390 comp rx=0 tx=0).stop 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 -- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103fc0 msgr2=0x7f098019b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103fc0 0x7f098019b090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f097800eb20 tx=0x7f097800c2d0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 -- 172.21.15.29:0/4205224696 shutdown_connections 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09680b6510 0x7f09680b8a20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980103fc0 0x7f098019b090 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0980103620 0x7f098019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 --2- 172.21.15.29:0/4205224696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09801025a0 0x7f0980199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 -- 172.21.15.29:0/4205224696 >> 172.21.15.29:0/4205224696 conn(0x7f09800fdee0 msgr2=0x7f09800ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 -- 172.21.15.29:0/4205224696 shutdown_connections 2024-09-21T02:40:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:40:50.158+0000 7f098677b640 1 -- 172.21.15.29:0/4205224696 wait complete. 2024-09-21T02:40:50.348 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 -- 172.21.15.29:0/1540548003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 msgr2=0x7f0990110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 --2- 172.21.15.29:0/1540548003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 0x7f0990110620 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0988009ca0 tx=0x7f098802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 -- 172.21.15.29:0/1540548003 shutdown_connections 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 --2- 172.21.15.29:0/1540548003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 0x7f0990110620 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 --2- 172.21.15.29:0/1540548003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09901094a0 0x7f09901098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 --2- 172.21.15.29:0/1540548003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0990103b60 0x7f0990103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 -- 172.21.15.29:0/1540548003 >> 172.21.15.29:0/1540548003 conn(0x7f099007b850 msgr2=0x7f099007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 -- 172.21.15.29:0/1540548003 shutdown_connections 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.917+0000 7f099661f640 1 -- 172.21.15.29:0/1540548003 wait complete. 2024-09-21T02:40:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f099661f640 1 Processor -- start 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f099661f640 1 -- start start 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f099661f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0990103b60 0x7f099019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f099661f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09901094a0 0x7f099019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f099661f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 0x7f099019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f099661f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0990113a10 con 0x7f0990109df0 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f099661f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0990113890 con 0x7f09901094a0 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f099661f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0990113b90 con 0x7f0990103b60 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f0994bbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 0x7f099019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f0994bbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 0x7f099019fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35754/0 (socket says 172.21.15.29:35754) 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f0994bbc640 1 -- 172.21.15.29:0/2927626064 learned_addr learned my addr 172.21.15.29:0/2927626064 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.918+0000 7f0987fff640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09901094a0 0x7f099019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f098ffff640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0990103b60 0x7f099019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f0994bbc640 1 -- 172.21.15.29:0/2927626064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0990103b60 msgr2=0x7f099019a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f0994bbc640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0990103b60 0x7f099019a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f0994bbc640 1 -- 172.21.15.29:0/2927626064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09901094a0 msgr2=0x7f099019a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f0994bbc640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09901094a0 0x7f099019a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f0994bbc640 1 -- 172.21.15.29:0/2927626064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09901a0330 con 0x7f0990109df0 2024-09-21T02:40:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f098ffff640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0990103b60 0x7f099019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:40:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f0994bbc640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 0x7f099019fc00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0988005cc0 tx=0x7f0988032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f098dffb640 1 -- 172.21.15.29:0/2927626064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988032e00 con 0x7f0990109df0 2024-09-21T02:40:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f098dffb640 1 -- 172.21.15.29:0/2927626064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09880044d0 con 0x7f0990109df0 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f098dffb640 1 -- 172.21.15.29:0/2927626064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988030b70 con 0x7f0990109df0 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f099661f640 1 -- 172.21.15.29:0/2927626064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09901a05c0 con 0x7f0990109df0 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.919+0000 7f099661f640 1 -- 172.21.15.29:0/2927626064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09901a6d80 con 0x7f0990109df0 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f099cb93740 1 Processor -- start 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f098dffb640 1 -- 172.21.15.29:0/2927626064 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f0988004680 con 0x7f0990109df0 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f099cb93740 1 -- start start 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f099cb93740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff8a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f099cb93740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff8aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f099cb93740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 0x56352ff8af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f099cb93740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56352fe75dc0 con 0x56352fbe4520 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f099cb93740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56352ff021f0 con 0x56352fbc3840 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f099cb93740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56352fef1620 con 0x56352fbbb660 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f0987fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff8a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f0987fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff8a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35762/0 (socket says 172.21.15.29:35762) 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f0987fff640 1 -- 172.21.15.29:0/3580146379 learned_addr learned my addr 172.21.15.29:0/3580146379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.922+0000 7f098ffff640 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff8aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f0987fff640 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 0x56352ff8af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f0987fff640 1 -- 172.21.15.29:0/3580146379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 msgr2=0x56352ff8aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f0987fff640 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff8aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f0987fff640 1 -- 172.21.15.29:0/3580146379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 msgr2=0x56352ff8af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f0987fff640 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 0x56352ff8af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f0987fff640 1 -- 172.21.15.29:0/3580146379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352feb9fd0 con 0x56352fbe4520 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f098dffb640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f097c0b6510 0x7f097c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f098ffff640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f097c0b6510 0x7f097c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f0987fff640 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff8a510 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0978005b90 tx=0x7f097800b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9310530fdfed26 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f098dffb640 1 -- 172.21.15.29:0/2927626064 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0988108960 con 0x7f0990109df0 2024-09-21T02:40:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f09867fc640 1 -- 172.21.15.29:0/3580146379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09780120d0 con 0x56352fbe4520 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f09867fc640 1 -- 172.21.15.29:0/3580146379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0978005020 con 0x56352fbe4520 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.923+0000 7f09867fc640 1 -- 172.21.15.29:0/3580146379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09780049e0 con 0x56352fbe4520 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 -- 172.21.15.29:0/3580146379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 msgr2=0x56352ff8a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f098ffff640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f097c0b6510 0x7f097c0b8a20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x56352ff89190 tx=0x7f0980008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff8a510 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0978005b90 tx=0x7f097800b820 comp rx=0 tx=0).stop 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 -- 172.21.15.29:0/3580146379 shutdown_connections 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 0x56352ff8af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff8aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 --2- 172.21.15.29:0/3580146379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff8a510 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 -- 172.21.15.29:0/3580146379 >> 172.21.15.29:0/3580146379 conn(0x56352fbd8910 msgr2=0x56352ff88400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 -- 172.21.15.29:0/3580146379 shutdown_connections 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.924+0000 7f099cb93740 1 -- 172.21.15.29:0/3580146379 wait complete. 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 1 -- start start 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 1 Processor -- start 2024-09-21T02:40:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff99fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbc3840 0x56352ff9a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbe4520 0x56352ffa61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56352fe8c300 con 0x56352fbc3840 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56352fef5430 con 0x56352fbe4520 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56352fd52bb0 con 0x56352fbbb660 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f099cb93740 10 client.0 _ll_register_callbacks cb 0x56352fea6120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f0987fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbc3840 0x56352ff9a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.925+0000 7f0987fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbc3840 0x56352ff9a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35778/0 (socket says 172.21.15.29:35778) 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0987fff640 1 -- 172.21.15.29:0/2549611245 learned_addr learned my addr 172.21.15.29:0/2549611245 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f098ffff640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff99fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0987fff640 1 -- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 msgr2=0x56352ff99fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0987fff640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff99fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0987fff640 1 -- 172.21.15.29:0/2549611245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbe4520 msgr2=0x56352ffa61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0994bbc640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbe4520 0x56352ffa61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0987fff640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbe4520 0x56352ffa61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0987fff640 1 -- 172.21.15.29:0/2549611245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56352fe7b780 con 0x56352fbc3840 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f098ffff640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff99fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0994bbc640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbe4520 0x56352ffa61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f0987fff640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbc3840 0x56352ff9a510 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f097800b3d0 tx=0x7f097800bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3227fa9cf6109d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f096f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f096f7fe640 1 -- 172.21.15.29:0/2549611245 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09780046c0 con 0x56352fbc3840 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f099cb93740 1 -- 172.21.15.29:0/2549611245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0978005020 con 0x56352fbc3840 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f096f7fe640 1 -- 172.21.15.29:0/2549611245 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0978004860 con 0x56352fbc3840 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f096f7fe640 1 -- 172.21.15.29:0/2549611245 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0978030050 con 0x56352fbc3840 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f099cb93740 10 client.15852 fetch_fsmap learned FSMap version 30 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.926+0000 7f099cb93740 1 -- 172.21.15.29:0/2549611245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56352fc4fb00 con 0x56352fbc3840 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f096f7fe640 1 -- 172.21.15.29:0/2549611245 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0978005020 con 0x56352fbc3840 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f096f7fe640 1 -- 172.21.15.29:0/2549611245 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f0978010bc0 con 0x56352fbc3840 2024-09-21T02:40:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f099cb93740 10 client.15852 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f099cb93740 10 client.15852 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f099cb93740 20 client.15852 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f099cb93740 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56352fba2ea0 0x56352ffac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f099cb93740 4 client.15852 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f099cb93740 1 -- 172.21.15.29:0/2549611245 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56352ff04140 con 0x56352fba2ea0 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.927+0000 7f098ffff640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56352fba2ea0 0x56352ffac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.928+0000 7f096f7fe640 10 client.15852 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.928+0000 7f098ffff640 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56352fba2ea0 0x56352ffac1d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86a0939215f0b3d7 server_cookie=1da809a38f82157b in_seq=0 out_seq=0 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.928+0000 7f096f7fe640 1 -- 172.21.15.29:0/2549611245 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56352ff04140 con 0x56352fba2ea0 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.928+0000 7f096f7fe640 10 client.15852 handle_command_reply: tid=0 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 client.15852 shutdown 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 -- 172.21.15.29:0/2549611245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbc3840 msgr2=0x56352ff9a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbc3840 0x56352ff9a510 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f097800b3d0 tx=0x7f097800bb50 comp rx=0 tx=0).stop 2024-09-21T02:40:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 -- 172.21.15.29:0/2549611245 shutdown_connections 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbe4520 0x56352ffa61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56352fba2ea0 0x56352ffac1d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbc3840 0x56352ff9a510 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 --2- 172.21.15.29:0/2549611245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbbb660 0x56352ff99fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 -- 172.21.15.29:0/2549611245 >> 172.21.15.29:0/2549611245 conn(0x56352fbd8910 msgr2=0x56352ff86930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f096f7fe640 0 client.15852 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 -- 172.21.15.29:0/2549611245 shutdown_connections 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 1 -- 172.21.15.29:0/2549611245 wait complete. 2024-09-21T02:40:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.929+0000 7f099cb93740 20 client.15852 trim_cache size 0 max 16384 2024-09-21T02:40:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.946+0000 7f099cb93740 1 Processor -- start 2024-09-21T02:40:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.946+0000 7f099cb93740 1 -- start start 2024-09-21T02:40:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f099cb93740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbbb660 0x56352ff87040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f099cb93740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbc3840 0x56352ff971c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f099cb93740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff97700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f099cb93740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09901ae6f0 con 0x56352fbe4520 2024-09-21T02:40:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f099cb93740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f09901b0260 con 0x56352fbbb660 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f099cb93740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f09901aedc0 con 0x56352fbc3840 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff97700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff97700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35792/0 (socket says 172.21.15.29:35792) 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 -- 172.21.15.29:0/2588710414 learned_addr learned my addr 172.21.15.29:0/2588710414 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0987fff640 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbc3840 0x56352ff971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f098ffff640 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbbb660 0x56352ff87040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 -- 172.21.15.29:0/2588710414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbc3840 msgr2=0x56352ff971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbc3840 0x56352ff971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 -- 172.21.15.29:0/2588710414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbbb660 msgr2=0x56352ff87040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbbb660 0x56352ff87040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 -- 172.21.15.29:0/2588710414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352fcc6960 con 0x56352fbe4520 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0994bbc640 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff97700 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f09880cad40 tx=0x7f09880ca700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107c7b1b76fafb03 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0985ffb640 1 -- 172.21.15.29:0/2588710414 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09880ce280 con 0x56352fbe4520 2024-09-21T02:40:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0985ffb640 1 -- 172.21.15.29:0/2588710414 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0988036040 con 0x56352fbe4520 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.947+0000 7f0985ffb640 1 -- 172.21.15.29:0/2588710414 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09880d7d00 con 0x56352fbe4520 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 -- 172.21.15.29:0/2588710414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 msgr2=0x56352ff97700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff97700 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f09880cad40 tx=0x7f09880ca700 comp rx=0 tx=0).stop 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 -- 172.21.15.29:0/2588710414 shutdown_connections 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbe4520 0x56352ff97700 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbc3840 0x56352ff971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 --2- 172.21.15.29:0/2588710414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbbb660 0x56352ff87040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 -- 172.21.15.29:0/2588710414 >> 172.21.15.29:0/2588710414 conn(0x56352fbd8910 msgr2=0x56352ff86930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 -- 172.21.15.29:0/2588710414 shutdown_connections 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.948+0000 7f099cb93740 1 -- 172.21.15.29:0/2588710414 wait complete. 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 1 -- start start 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 1 Processor -- start 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbbb660 0x56352ffa2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 0x56352ffa2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbe4520 0x56352ff9d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09901afb60 con 0x56352fbbb660 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f09901af490 con 0x56352fbc3840 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f09901ac6d0 con 0x56352fbe4520 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f099cb93740 10 client.0 _ll_register_callbacks cb 0x7f098000a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f098ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbbb660 0x56352ffa2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f098ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbbb660 0x56352ffa2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35796/0 (socket says 172.21.15.29:35796) 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.949+0000 7f098ffff640 1 -- 172.21.15.29:0/1946642227 learned_addr learned my addr 172.21.15.29:0/1946642227 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f0987fff640 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 0x56352ffa2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f0994bbc640 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbe4520 0x56352ff9d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f098ffff640 1 -- 172.21.15.29:0/1946642227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbe4520 msgr2=0x56352ff9d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f098ffff640 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbe4520 0x56352ff9d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f098ffff640 1 -- 172.21.15.29:0/1946642227 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 msgr2=0x56352ffa2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:40:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f098ffff640 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 0x56352ffa2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f098ffff640 1 -- 172.21.15.29:0/1946642227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56352ff9d900 con 0x56352fbbb660 2024-09-21T02:40:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f098ffff640 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbbb660 0x56352ffa2480 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f098000fc30 tx=0x7f0980012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981bf1b2c4c4de50 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:40:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f096f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:40:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f096f7fe640 1 -- 172.21.15.29:0/1946642227 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098002e070 con 0x56352fbbb660 2024-09-21T02:40:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f099cb93740 1 -- 172.21.15.29:0/1946642227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0988036040 con 0x56352fbbb660 2024-09-21T02:40:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f096f7fe640 1 -- 172.21.15.29:0/1946642227 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0980015860 con 0x56352fbbb660 2024-09-21T02:40:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f096f7fe640 1 -- 172.21.15.29:0/1946642227 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f098003a050 con 0x56352fbbb660 2024-09-21T02:40:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f099cb93740 10 client.15858 fetch_fsmap learned FSMap version 30 2024-09-21T02:40:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.950+0000 7f099cb93740 1 -- 172.21.15.29:0/1946642227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563530017550 con 0x56352fbbb660 2024-09-21T02:40:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f096f7fe640 1 -- 172.21.15.29:0/1946642227 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0988036040 con 0x56352fbbb660 2024-09-21T02:40:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f096f7fe640 1 -- 172.21.15.29:0/1946642227 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f0980013ad0 con 0x56352fbbb660 2024-09-21T02:40:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f099cb93740 10 client.15858 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:40:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f099cb93740 10 client.15858 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:40:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f099cb93740 20 client.15858 populate_metadata read hostname 'smithi029' 2024-09-21T02:40:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f099cb93740 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56352fb78250 0x563530019cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:40:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f099cb93740 4 client.15858 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "473fc997", "release": true, "await": true}] 2024-09-21T02:40:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f099cb93740 1 -- 172.21.15.29:0/1946642227 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "473fc997", "release": true, "await": true}) -- 0x56352feffbf0 con 0x56352fb78250 2024-09-21T02:40:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.951+0000 7f0987fff640 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56352fb78250 0x563530019cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:40:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.952+0000 7f0987fff640 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56352fb78250 0x563530019cc0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab6883660775ea7f server_cookie=74eac7f27030a3c5 in_seq=0 out_seq=0 2024-09-21T02:40:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.952+0000 7f096f7fe640 10 client.15858 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.961+0000 7f096f7fe640 1 -- 172.21.15.29:0/1946642227 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56352feffbf0 con 0x56352fb78250 2024-09-21T02:40:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.961+0000 7f096f7fe640 10 client.15858 handle_command_reply: tid=0 2024-09-21T02:40:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 client.15858 shutdown 2024-09-21T02:40:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 -- 172.21.15.29:0/1946642227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbbb660 msgr2=0x56352ffa2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbbb660 0x56352ffa2480 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f098000fc30 tx=0x7f0980012000 comp rx=0 tx=0).stop 2024-09-21T02:40:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 -- 172.21.15.29:0/1946642227 shutdown_connections 2024-09-21T02:40:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56352fbe4520 0x56352ff9d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56352fbc3840 0x56352ffa2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56352fb78250 0x563530019cc0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 --2- 172.21.15.29:0/1946642227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56352fbbb660 0x56352ffa2480 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f096f7fe640 0 client.15858 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:40:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 -- 172.21.15.29:0/1946642227 >> 172.21.15.29:0/1946642227 conn(0x56352fbd8910 msgr2=0x56352ff870c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 -- 172.21.15.29:0/1946642227 shutdown_connections 2024-09-21T02:40:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 1 -- 172.21.15.29:0/1946642227 wait complete. 2024-09-21T02:40:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.962+0000 7f099cb93740 20 client.15858 trim_cache size 0 max 16384 2024-09-21T02:40:50.964 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 30, 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 8, 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "473fc997": { 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 8, 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 137.0, 2024-09-21T02:40:50.965 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:40:50.966 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:40:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 -- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f097c0b6510 msgr2=0x7f097c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f097c0b6510 0x7f097c0b8a20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x56352ff89190 tx=0x7f0980008040 comp rx=0 tx=0).stop 2024-09-21T02:40:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 -- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 msgr2=0x7f099019fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:40:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 0x7f099019fc00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0988005cc0 tx=0x7f0988032a10 comp rx=0 tx=0).stop 2024-09-21T02:40:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 -- 172.21.15.29:0/2927626064 shutdown_connections 2024-09-21T02:40:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f097c0b6510 0x7f097c0b8a20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0990109df0 0x7f099019fc00 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09901094a0 0x7f099019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 --2- 172.21.15.29:0/2927626064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0990103b60 0x7f099019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:40:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 -- 172.21.15.29:0/2927626064 >> 172.21.15.29:0/2927626064 conn(0x7f099007b850 msgr2=0x7f09901074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:40:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 -- 172.21.15.29:0/2927626064 shutdown_connections 2024-09-21T02:40:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:40:50.963+0000 7f099661f640 1 -- 172.21.15.29:0/2927626064 wait complete. 2024-09-21T02:40:51.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 5468 1.63 MB/sec warmup 28 sec latency 19366.613 ms 2024-09-21T02:40:51.835 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:51 smithi029 ceph-mon[31006]: pgmap v441: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-09-21T02:40:51.863 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '473fc997', total seconds elapsed: 23.5 2024-09-21T02:40:51.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:51 smithi190 ceph-mon[33871]: pgmap v441: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-09-21T02:40:52.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:51 smithi066 ceph-mon[33436]: pgmap v441: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-09-21T02:40:52.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 6058 1.65 MB/sec warmup 29 sec latency 15.355 ms 2024-09-21T02:40:53.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 6873 1.82 MB/sec warmup 30 sec latency 12.346 ms 2024-09-21T02:40:53.812 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:53 smithi066 ceph-mon[33436]: pgmap v442: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-09-21T02:40:53.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:53 smithi190 ceph-mon[33871]: pgmap v442: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-09-21T02:40:53.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:53 smithi029 ceph-mon[31006]: pgmap v442: 129 pgs: 129 active+clean; 23 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-09-21T02:40:54.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 7379 1.90 MB/sec warmup 31 sec latency 15.017 ms 2024-09-21T02:40:55.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 7693 1.85 MB/sec warmup 32 sec latency 12.060 ms 2024-09-21T02:40:55.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:55 smithi190 ceph-mon[33871]: pgmap v443: 129 pgs: 129 active+clean; 25 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 150 KiB/s wr, 24 op/s 2024-09-21T02:40:55.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:55 smithi029 ceph-mon[31006]: pgmap v443: 129 pgs: 129 active+clean; 25 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 150 KiB/s wr, 24 op/s 2024-09-21T02:40:56.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:55 smithi066 ceph-mon[33436]: pgmap v443: 129 pgs: 129 active+clean; 25 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 150 KiB/s wr, 24 op/s 2024-09-21T02:40:56.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 8276 1.86 MB/sec warmup 33 sec latency 18.784 ms 2024-09-21T02:40:57.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 8948 1.90 MB/sec warmup 34 sec latency 15.183 ms 2024-09-21T02:40:57.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:57 smithi190 ceph-mon[33871]: pgmap v444: 129 pgs: 129 active+clean; 26 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 482 KiB/s wr, 26 op/s 2024-09-21T02:40:57.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:57 smithi029 ceph-mon[31006]: pgmap v444: 129 pgs: 129 active+clean; 26 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 482 KiB/s wr, 26 op/s 2024-09-21T02:40:58.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:57 smithi066 ceph-mon[33436]: pgmap v444: 129 pgs: 129 active+clean; 26 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 482 KiB/s wr, 26 op/s 2024-09-21T02:40:58.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 9557 1.90 MB/sec warmup 35 sec latency 16.089 ms 2024-09-21T02:40:59.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 10404 2.04 MB/sec warmup 36 sec latency 38.067 ms 2024-09-21T02:40:59.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:40:59 smithi190 ceph-mon[33871]: pgmap v445: 129 pgs: 129 active+clean; 26 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 576 KiB/s wr, 40 op/s 2024-09-21T02:40:59.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:40:59 smithi029 ceph-mon[31006]: pgmap v445: 129 pgs: 129 active+clean; 26 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 576 KiB/s wr, 40 op/s 2024-09-21T02:41:00.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:40:59 smithi066 ceph-mon[33436]: pgmap v445: 129 pgs: 129 active+clean; 26 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 576 KiB/s wr, 40 op/s 2024-09-21T02:41:00.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 10901 2.11 MB/sec warmup 37 sec latency 15.323 ms 2024-09-21T02:41:01.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 11172 2.06 MB/sec warmup 38 sec latency 14.733 ms 2024-09-21T02:41:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:01 smithi190 ceph-mon[33871]: pgmap v446: 129 pgs: 129 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 992 KiB/s wr, 69 op/s 2024-09-21T02:41:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:01 smithi029 ceph-mon[31006]: pgmap v446: 129 pgs: 129 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 992 KiB/s wr, 69 op/s 2024-09-21T02:41:02.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:01 smithi066 ceph-mon[33436]: pgmap v446: 129 pgs: 129 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 992 KiB/s wr, 69 op/s 2024-09-21T02:41:02.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 11790 2.06 MB/sec warmup 39 sec latency 15.805 ms 2024-09-21T02:41:03.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 12428 2.09 MB/sec warmup 40 sec latency 21.437 ms 2024-09-21T02:41:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.261+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/3532107546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb4103fc0 msgr2=0x7fdbb410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.261+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/3532107546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb4103fc0 0x7fdbb410a960 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009c90 tx=0x7fdbac02a860 comp rx=0 tx=0).stop 2024-09-21T02:41:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.261+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/3532107546 shutdown_connections 2024-09-21T02:41:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.261+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/3532107546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb4103fc0 0x7fdbb410a960 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.261+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/3532107546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbb4103620 0x7fdbb4103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.261+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/3532107546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbb41025a0 0x7fdbb4102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.261+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/3532107546 >> 172.21.15.29:0/3532107546 conn(0x7fdbb40fdee0 msgr2=0x7fdbb4100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.262+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/3532107546 shutdown_connections 2024-09-21T02:41:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.262+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/3532107546 wait complete. 2024-09-21T02:41:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.262+0000 7fdbbb9cf640 1 Processor -- start 2024-09-21T02:41:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.262+0000 7fdbbb9cf640 1 -- start start 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbbb9cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb41025a0 0x7fdbb4199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbbb9cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbb4103620 0x7fdbb419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbbb9cf640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbb4103fc0 0x7fdbb419b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbbb9cf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb410e0b0 con 0x7fdbb41025a0 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbbb9cf640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb410df30 con 0x7fdbb4103fc0 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbbb9cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb410e230 con 0x7fdbb4103620 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb976b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb41025a0 0x7fdbb4199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb976b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb41025a0 0x7fdbb4199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35278/0 (socket says 172.21.15.29:35278) 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb976b640 1 -- 172.21.15.29:0/1742956792 learned_addr learned my addr 172.21.15.29:0/1742956792 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbb4103620 0x7fdbb419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbb4103fc0 0x7fdbb419b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb976b640 1 -- 172.21.15.29:0/1742956792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbb4103620 msgr2=0x7fdbb419a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb976b640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbb4103620 0x7fdbb419a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb976b640 1 -- 172.21.15.29:0/1742956792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbb4103fc0 msgr2=0x7fdbb419b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb976b640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbb4103fc0 0x7fdbb419b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb976b640 1 -- 172.21.15.29:0/1742956792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb419b8a0 con 0x7fdbb41025a0 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.263+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbb4103fc0 0x7fdbb419b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.264+0000 7fdbb976b640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb41025a0 0x7fdbb4199e60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000e9f0 tx=0x7fdbb000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.264+0000 7fdbaa7fc640 1 -- 172.21.15.29:0/1742956792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb000cd80 con 0x7fdbb41025a0 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.264+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/1742956792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb419bb90 con 0x7fdbb41025a0 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.264+0000 7fdbaa7fc640 1 -- 172.21.15.29:0/1742956792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbb00054b0 con 0x7fdbb41025a0 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.264+0000 7fdbaa7fc640 1 -- 172.21.15.29:0/1742956792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0010430 con 0x7fdbb41025a0 2024-09-21T02:41:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.264+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/1742956792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb410ec60 con 0x7fdbb41025a0 2024-09-21T02:41:03.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.267+0000 7fdbc2a1d740 1 Processor -- start 2024-09-21T02:41:03.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.267+0000 7fdbc2a1d740 1 -- start start 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.267+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e007660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.267+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e007cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.267+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e008820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55736df716f0 con 0x55736dc571b0 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55736df60b20 con 0x55736dc2e2f0 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55736df64930 con 0x55736dc364d0 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e007660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e007660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35290/0 (socket says 172.21.15.29:35290) 2024-09-21T02:41:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/4126407036 learned_addr learned my addr 172.21.15.29:0/4126407036 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbaa7fc640 1 -- 172.21.15.29:0/1742956792 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fdbb00046c0 con 0x7fdbb41025a0 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb976b640 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e007cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/4126407036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 msgr2=0x55736e008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e008820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/4126407036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 msgr2=0x55736e007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e007cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/4126407036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55736df26530 con 0x55736dc571b0 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb976b640 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e007cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.268+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e007660 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009c90 tx=0x7fdbac031830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba8220ba80669e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.269+0000 7fdb9f7fe640 1 -- 172.21.15.29:0/4126407036 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac004c90 con 0x55736dc571b0 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.269+0000 7fdb9f7fe640 1 -- 172.21.15.29:0/4126407036 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbac036070 con 0x55736dc571b0 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.269+0000 7fdb9f7fe640 1 -- 172.21.15.29:0/4126407036 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac004320 con 0x55736dc571b0 2024-09-21T02:41:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.269+0000 7fdbaa7fc640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdba00b6510 0x7fdba00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.269+0000 7fdbb976b640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdba00b6510 0x7fdba00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.269+0000 7fdbaa7fc640 1 -- 172.21.15.29:0/1742956792 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdbb00e7a20 con 0x7fdbb41025a0 2024-09-21T02:41:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/4126407036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 msgr2=0x55736e007660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbb976b640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdba00b6510 0x7fdba00b8a20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00aaaa0 tx=0x7fdbb0009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e007660 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009c90 tx=0x7fdbac031830 comp rx=0 tx=0).stop 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/4126407036 shutdown_connections 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e008820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e007cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/4126407036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e007660 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/4126407036 >> 172.21.15.29:0/4126407036 conn(0x55736dc4b300 msgr2=0x55736dffa8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/4126407036 shutdown_connections 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/4126407036 wait complete. 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 -- start start 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.270+0000 7fdbc2a1d740 1 Processor -- start 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e010d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e011430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e011fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55736de98ab0 con 0x55736dc571b0 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55736df06730 con 0x55736dc2e2f0 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55736e0092e0 con 0x55736dc364d0 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbc2a1d740 10 client.0 _ll_register_callbacks cb 0x55736df5a6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb9f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e011fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb8f6a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e011430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb8f6a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e011430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45642/0 (socket says 172.21.15.29:45642) 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb8f6a640 1 -- 172.21.15.29:0/3983468180 learned_addr learned my addr 172.21.15.29:0/3983468180 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb9f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e011fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35292/0 (socket says 172.21.15.29:35292) 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e010d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb8f6a640 1 -- 172.21.15.29:0/3983468180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 msgr2=0x55736e010d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e010d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb8f6a640 1 -- 172.21.15.29:0/3983468180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 msgr2=0x55736e011fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e011fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.271+0000 7fdbb8f6a640 1 -- 172.21.15.29:0/3983468180 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55736dcc2790 con 0x55736dc364d0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.272+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e011430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdba4012840 tx=0x7fdba4012d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d44657555b0c3ba server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.272+0000 7fdb9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.272+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3983468180 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba400f8a0 con 0x55736dc364d0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.272+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3983468180 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdba400fa40 con 0x55736dc364d0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.272+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3983468180 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbac036070 con 0x55736dc364d0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.272+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3983468180 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdba400fc40 con 0x55736dc364d0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.272+0000 7fdbc2a1d740 10 client.25510 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.272+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3983468180 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55736dd3b620 con 0x55736dc364d0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3983468180 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbac036070 con 0x55736dc364d0 2024-09-21T02:41:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3983468180 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fdba401c420 con 0x55736dc364d0 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdbc2a1d740 10 client.25510 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdbc2a1d740 10 client.25510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdbc2a1d740 20 client.25510 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55736dc15d80 0x55736e01e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdbc2a1d740 4 client.25510 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3983468180 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55736df73640 con 0x55736dc15d80 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.273+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55736dc15d80 0x55736e01e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.274+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55736dc15d80 0x55736e01e560 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=225cb5655f9fc4bd server_cookie=1d6f630881e3f0f in_seq=0 out_seq=0 2024-09-21T02:41:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.274+0000 7fdb9cff9640 10 client.25510 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3983468180 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55736df73640 con 0x55736dc15d80 2024-09-21T02:41:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdb9cff9640 10 client.25510 handle_command_reply: tid=0 2024-09-21T02:41:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 client.25510 shutdown 2024-09-21T02:41:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3983468180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 msgr2=0x55736e011430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e011430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdba4012840 tx=0x7fdba4012d00 comp rx=0 tx=0).stop 2024-09-21T02:41:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3983468180 shutdown_connections 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55736dc15d80 0x55736e01e560 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc571b0 0x55736e011fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdb9cff9640 0 client.25510 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc364d0 0x55736e011430 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3983468180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e010d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3983468180 >> 172.21.15.29:0/3983468180 conn(0x55736dc4b300 msgr2=0x55736e0086d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3983468180 shutdown_connections 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3983468180 wait complete. 2024-09-21T02:41:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.275+0000 7fdbc2a1d740 20 client.25510 trim_cache size 0 max 16384 2024-09-21T02:41:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbc2a1d740 1 Processor -- start 2024-09-21T02:41:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbc2a1d740 1 -- start start 2024-09-21T02:41:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc15d80 0x55736e011480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc2e2f0 0x55736e00f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e011fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41aa300 con 0x55736dc364d0 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41abe70 con 0x55736dc15d80 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41aa9d0 con 0x55736dc2e2f0 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbb9f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e011fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbb8f6a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc2e2f0 0x55736e00f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbb9f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e011fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35294/0 (socket says 172.21.15.29:35294) 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.293+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/3345178313 learned_addr learned my addr 172.21.15.29:0/3345178313 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc15d80 0x55736e011480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/3345178313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc2e2f0 msgr2=0x55736e00f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc2e2f0 0x55736e00f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/3345178313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc15d80 msgr2=0x55736e011480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc15d80 0x55736e011480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/3345178313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55736e00c560 con 0x55736dc364d0 2024-09-21T02:41:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc2e2f0 0x55736e00f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e011fa0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fdbac02a8a0 tx=0x7fdbac031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb1d2e64a534937 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdb9effd640 1 -- 172.21.15.29:0/3345178313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac003be0 con 0x55736dc364d0 2024-09-21T02:41:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdb9effd640 1 -- 172.21.15.29:0/3345178313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbac036050 con 0x55736dc364d0 2024-09-21T02:41:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.294+0000 7fdb9effd640 1 -- 172.21.15.29:0/3345178313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac030560 con 0x55736dc364d0 2024-09-21T02:41:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3345178313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 msgr2=0x55736e011fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e011fa0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fdbac02a8a0 tx=0x7fdbac031000 comp rx=0 tx=0).stop 2024-09-21T02:41:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3345178313 shutdown_connections 2024-09-21T02:41:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e011fa0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc2e2f0 0x55736e00f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3345178313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc15d80 0x55736e011480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3345178313 >> 172.21.15.29:0/3345178313 conn(0x55736dc4b300 msgr2=0x55736e010d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3345178313 shutdown_connections 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3345178313 wait complete. 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 -- start start 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.295+0000 7fdbc2a1d740 1 Processor -- start 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc15d80 0x55736e015d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e016450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbc2a1d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e016fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41ab770 con 0x55736dc364d0 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41ab0a0 con 0x55736dc2e2f0 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbc2a1d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41a82e0 con 0x55736dc15d80 2024-09-21T02:41:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbc2a1d740 10 client.0 _ll_register_callbacks cb 0x7fdba4009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb9f6c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc15d80 0x55736e015d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb9f6c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc15d80 0x55736e015d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45654/0 (socket says 172.21.15.29:45654) 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb8f6a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e016450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb9f6c640 1 -- 172.21.15.29:0/3661385444 learned_addr learned my addr 172.21.15.29:0/3661385444 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e016fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb8f6a640 1 -- 172.21.15.29:0/3661385444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc15d80 msgr2=0x55736e015d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc15d80 0x55736e015d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb8f6a640 1 -- 172.21.15.29:0/3661385444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 msgr2=0x55736e016fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e016fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.296+0000 7fdbb8f6a640 1 -- 172.21.15.29:0/3661385444 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55736e017720 con 0x55736dc2e2f0 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdbb8f6a640 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e016450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdba401c860 tx=0x7fdba4005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38c60cbdc1c629d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdb9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:41:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3661385444 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba401cce0 con 0x55736dc2e2f0 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3661385444 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbac036050 con 0x55736dc2e2f0 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3661385444 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdba401c420 con 0x55736dc2e2f0 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3661385444 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdba401c640 con 0x55736dc2e2f0 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdbc2a1d740 10 client.25442 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3661385444 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55736e018a80 con 0x55736dc2e2f0 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.297+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3661385444 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbac036050 con 0x55736dc2e2f0 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3661385444 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fdba4003880 con 0x55736dc2e2f0 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdbc2a1d740 10 client.25442 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdbc2a1d740 10 client.25442 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdbc2a1d740 20 client.25442 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55736dc571b0 0x55736e08b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdbc2a1d740 4 client.25442 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:41:03.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3661385444 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55736df6f0f0 con 0x55736dc571b0 2024-09-21T02:41:03.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55736dc571b0 0x55736e08b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:03.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdb9cff9640 10 client.25442 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:03.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.298+0000 7fdbb9f6c640 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55736dc571b0 0x55736e08b060 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a984a9efe357d3 server_cookie=1b0a77fcac6d8386 in_seq=0 out_seq=0 2024-09-21T02:41:03.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdb9cff9640 1 -- 172.21.15.29:0/3661385444 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55736df6f0f0 con 0x55736dc571b0 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdb9cff9640 10 client.25442 handle_command_reply: tid=0 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 client.25442 shutdown 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3661385444 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 msgr2=0x55736e016450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e016450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdba401c860 tx=0x7fdba4005d30 comp rx=0 tx=0).stop 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3661385444 shutdown_connections 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55736dc571b0 0x55736e08b060 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55736dc364d0 0x55736e016fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55736dc2e2f0 0x55736e016450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 --2- 172.21.15.29:0/3661385444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55736dc15d80 0x55736e015d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdb9cff9640 0 client.25442 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3661385444 >> 172.21.15.29:0/3661385444 conn(0x55736dc4b300 msgr2=0x55736dff8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3661385444 shutdown_connections 2024-09-21T02:41:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 1 -- 172.21.15.29:0/3661385444 wait complete. 2024-09-21T02:41:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.339+0000 7fdbc2a1d740 20 client.25442 trim_cache size 0 max 16384 2024-09-21T02:41:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:41:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:41:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:41:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:41:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdba00b6510 msgr2=0x7fdba00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdba00b6510 0x7fdba00b8a20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00aaaa0 tx=0x7fdbb0009720 comp rx=0 tx=0).stop 2024-09-21T02:41:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb41025a0 msgr2=0x7fdbb4199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb41025a0 0x7fdbb4199e60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000e9f0 tx=0x7fdbb000edf0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/1742956792 shutdown_connections 2024-09-21T02:41:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdba00b6510 0x7fdba00b8a20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbb4103fc0 0x7fdbb419b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbb4103620 0x7fdbb419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 --2- 172.21.15.29:0/1742956792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb41025a0 0x7fdbb4199e60 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/1742956792 >> 172.21.15.29:0/1742956792 conn(0x7fdbb40fdee0 msgr2=0x7fdbb40ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/1742956792 shutdown_connections 2024-09-21T02:41:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:03.340+0000 7fdbbb9cf640 1 -- 172.21.15.29:0/1742956792 wait complete. 2024-09-21T02:41:03.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:03 smithi190 ceph-mon[33871]: pgmap v447: 129 pgs: 129 active+clean; 28 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 1.2 MiB/s wr, 78 op/s 2024-09-21T02:41:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:03 smithi029 ceph-mon[31006]: pgmap v447: 129 pgs: 129 active+clean; 28 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 1.2 MiB/s wr, 78 op/s 2024-09-21T02:41:04.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:03 smithi066 ceph-mon[33436]: pgmap v447: 129 pgs: 129 active+clean; 28 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 1.2 MiB/s wr, 78 op/s 2024-09-21T02:41:04.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 13018 2.08 MB/sec warmup 41 sec latency 46.154 ms 2024-09-21T02:41:05.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 13929 2.21 MB/sec warmup 42 sec latency 14.275 ms 2024-09-21T02:41:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:05 smithi190 ceph-mon[33871]: pgmap v448: 129 pgs: 129 active+clean; 28 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 109 op/s 2024-09-21T02:41:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:05 smithi029 ceph-mon[31006]: pgmap v448: 129 pgs: 129 active+clean; 28 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 109 op/s 2024-09-21T02:41:06.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:05 smithi066 ceph-mon[33436]: pgmap v448: 129 pgs: 129 active+clean; 28 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 109 op/s 2024-09-21T02:41:06.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 14412 2.26 MB/sec warmup 43 sec latency 14.023 ms 2024-09-21T02:41:07.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 14662 2.21 MB/sec warmup 44 sec latency 14.885 ms 2024-09-21T02:41:07.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:07 smithi190 ceph-mon[33871]: pgmap v449: 129 pgs: 129 active+clean; 28 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.5 MiB/s wr, 109 op/s 2024-09-21T02:41:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:07 smithi029 ceph-mon[31006]: pgmap v449: 129 pgs: 129 active+clean; 28 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.5 MiB/s wr, 109 op/s 2024-09-21T02:41:08.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:07 smithi066 ceph-mon[33436]: pgmap v449: 129 pgs: 129 active+clean; 28 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.5 MiB/s wr, 109 op/s 2024-09-21T02:41:08.066 INFO:tasks.workunit.client.0.smithi029.stdout: 1 15232 2.21 MB/sec warmup 45 sec latency 16.863 ms 2024-09-21T02:41:09.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 15887 2.23 MB/sec warmup 46 sec latency 13.210 ms 2024-09-21T02:41:09.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:09 smithi190 ceph-mon[33871]: pgmap v450: 129 pgs: 129 active+clean; 29 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:41:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:09 smithi029 ceph-mon[31006]: pgmap v450: 129 pgs: 129 active+clean; 29 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:41:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:09 smithi066 ceph-mon[33436]: pgmap v450: 129 pgs: 129 active+clean; 29 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:41:10.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 16558 2.22 MB/sec warmup 47 sec latency 11.820 ms 2024-09-21T02:41:11.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 17472 2.33 MB/sec warmup 48 sec latency 15.394 ms 2024-09-21T02:41:11.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:11 smithi190 ceph-mon[33871]: pgmap v451: 129 pgs: 129 active+clean; 26 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T02:41:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:11 smithi029 ceph-mon[31006]: pgmap v451: 129 pgs: 129 active+clean; 26 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T02:41:12.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:11 smithi066 ceph-mon[33436]: pgmap v451: 129 pgs: 129 active+clean; 26 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T02:41:12.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 17951 2.37 MB/sec warmup 49 sec latency 16.630 ms 2024-09-21T02:41:13.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 18199 2.33 MB/sec warmup 50 sec latency 13.744 ms 2024-09-21T02:41:13.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:13 smithi190 ceph-mon[33871]: pgmap v452: 129 pgs: 129 active+clean; 26 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:41:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:13 smithi029 ceph-mon[31006]: pgmap v452: 129 pgs: 129 active+clean; 26 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:41:14.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:13 smithi066 ceph-mon[33436]: pgmap v452: 129 pgs: 129 active+clean; 26 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:41:14.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 18722 2.30 MB/sec warmup 51 sec latency 19.617 ms 2024-09-21T02:41:15.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 19391 2.34 MB/sec warmup 52 sec latency 15.682 ms 2024-09-21T02:41:15.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:15 smithi190 ceph-mon[33871]: pgmap v453: 129 pgs: 129 active+clean; 27 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-09-21T02:41:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:15 smithi029 ceph-mon[31006]: pgmap v453: 129 pgs: 129 active+clean; 27 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-09-21T02:41:16.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:15 smithi066 ceph-mon[33436]: pgmap v453: 129 pgs: 129 active+clean; 27 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-09-21T02:41:16.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 20119 2.33 MB/sec warmup 53 sec latency 16.183 ms 2024-09-21T02:41:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 -- 172.21.15.29:0/1121219764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc103650 msgr2=0x7fe5fc103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 --2- 172.21.15.29:0/1121219764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc103650 0x7fe5fc103a60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009900 tx=0x7fe5f002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 -- 172.21.15.29:0/1121219764 shutdown_connections 2024-09-21T02:41:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 --2- 172.21.15.29:0/1121219764 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe5fc103ff0 0x7fe5fc10a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 --2- 172.21.15.29:0/1121219764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc103650 0x7fe5fc103a60 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 --2- 172.21.15.29:0/1121219764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5fc1025d0 0x7fe5fc102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 -- 172.21.15.29:0/1121219764 >> 172.21.15.29:0/1121219764 conn(0x7fe5fc0fdf30 msgr2=0x7fe5fc1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 -- 172.21.15.29:0/1121219764 shutdown_connections 2024-09-21T02:41:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 -- 172.21.15.29:0/1121219764 wait complete. 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.451+0000 7fe60164a640 1 Processor -- start 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe60164a640 1 -- start start 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe60164a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc1025d0 0x7fe5fc197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe60164a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5fc103650 0x7fe5fc1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe60164a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe5fc103ff0 0x7fe5fc198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe60164a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc10d000 con 0x7fe5fc1025d0 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe60164a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc10ce80 con 0x7fe5fc103ff0 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe60164a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc10d180 con 0x7fe5fc103650 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc1025d0 0x7fe5fc197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc1025d0 0x7fe5fc197c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50324/0 (socket says 172.21.15.29:50324) 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5faffd640 1 -- 172.21.15.29:0/3830697367 learned_addr learned my addr 172.21.15.29:0/3830697367 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5fc103650 0x7fe5fc1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe5fc103ff0 0x7fe5fc198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5faffd640 1 -- 172.21.15.29:0/3830697367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5fc103650 msgr2=0x7fe5fc1982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5faffd640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5fc103650 0x7fe5fc1982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5faffd640 1 -- 172.21.15.29:0/3830697367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe5fc103ff0 msgr2=0x7fe5fc198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5faffd640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe5fc103ff0 0x7fe5fc198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5faffd640 1 -- 172.21.15.29:0/3830697367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5fc1996a0 con 0x7fe5fc1025d0 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.452+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5fc103650 0x7fe5fc1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.453+0000 7fe5faffd640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc1025d0 0x7fe5fc197c60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009990 tx=0x7fe5ec009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.453+0000 7fe5e3fff640 1 -- 172.21.15.29:0/3830697367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec01b070 con 0x7fe5fc1025d0 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.453+0000 7fe5e3fff640 1 -- 172.21.15.29:0/3830697367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5ec016410 con 0x7fe5fc1025d0 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.453+0000 7fe60164a640 1 -- 172.21.15.29:0/3830697367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5fc199990 con 0x7fe5fc1025d0 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.453+0000 7fe5e3fff640 1 -- 172.21.15.29:0/3830697367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec015610 con 0x7fe5fc1025d0 2024-09-21T02:41:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.453+0000 7fe60164a640 1 -- 172.21.15.29:0/3830697367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5fc1a0080 con 0x7fe5fc1025d0 2024-09-21T02:41:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe608691740 1 Processor -- start 2024-09-21T02:41:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe608691740 1 -- start start 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe608691740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f5be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe608691740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b8b2f0 0x55dab8f5c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe608691740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 0x55dab8f5d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe608691740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dab8ece6f0 con 0x55dab8b8b2f0 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe608691740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55dab8ebdb20 con 0x55dab8b934d0 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe608691740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55dab8ec1930 con 0x55dab8bb41b0 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f5be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f5be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48020/0 (socket says 172.21.15.29:48020) 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fa7fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 0x55dab8f5d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/3875609226 learned_addr learned my addr 172.21.15.29:0/3875609226 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5e3fff640 1 -- 172.21.15.29:0/3830697367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fe5ec01e070 con 0x7fe5fc1025d0 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/3875609226 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 msgr2=0x55dab8f5d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/3875609226 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 0x55dab8f5d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/3875609226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b8b2f0 msgr2=0x55dab8f5c4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/3875609226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b8b2f0 0x55dab8f5c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.456+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/3875609226 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dab8e83530 con 0x55dab8bb41b0 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.457+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/3875609226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f5be50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009fc0 tx=0x7fe5f4009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5a92444907650d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.457+0000 7fe5e17fa640 1 -- 172.21.15.29:0/3875609226 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4003b70 con 0x55dab8bb41b0 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.457+0000 7fe5e17fa640 1 -- 172.21.15.29:0/3875609226 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5f4003d10 con 0x55dab8bb41b0 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.457+0000 7fe5e17fa640 1 -- 172.21.15.29:0/3875609226 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4007960 con 0x55dab8bb41b0 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.457+0000 7fe5e3fff640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5e40b6590 0x7fe5e40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.457+0000 7fe5faffd640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5e40b6590 0x7fe5e40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe5e3fff640 1 -- 172.21.15.29:0/3830697367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe5ec0e69c0 con 0x7fe5fc1025d0 2024-09-21T02:41:16.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 -- 172.21.15.29:0/3875609226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 msgr2=0x55dab8f5be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe5faffd640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5e40b6590 0x7fe5e40b8aa0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0ac580 tx=0x7fe5ec070e60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 --2- 172.21.15.29:0/3875609226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f5be50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009fc0 tx=0x7fe5f4009500 comp rx=0 tx=0).stop 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 -- 172.21.15.29:0/3875609226 shutdown_connections 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 --2- 172.21.15.29:0/3875609226 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 0x55dab8f5d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 --2- 172.21.15.29:0/3875609226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b8b2f0 0x55dab8f5c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 --2- 172.21.15.29:0/3875609226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f5be50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 -- 172.21.15.29:0/3875609226 >> 172.21.15.29:0/3875609226 conn(0x55dab8ba8300 msgr2=0x55dab8f578e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 -- 172.21.15.29:0/3875609226 shutdown_connections 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 -- 172.21.15.29:0/3875609226 wait complete. 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.458+0000 7fe608691740 1 -- start start 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe608691740 1 Processor -- start 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe608691740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe608691740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe608691740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f66800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe608691740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dab8df5ab0 con 0x55dab8b934d0 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe608691740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55dab8e63730 con 0x55dab8b8b2f0 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe608691740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55dab8f5db30 con 0x55dab8bb41b0 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe608691740 10 client.0 _ll_register_callbacks cb 0x55dab8eb76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fa7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fa7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50338/0 (socket says 172.21.15.29:50338) 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fa7fc640 1 -- 172.21.15.29:0/4013498275 learned_addr learned my addr 172.21.15.29:0/4013498275 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f66800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fa7fc640 1 -- 172.21.15.29:0/4013498275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 msgr2=0x55dab8f66800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f66800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fa7fc640 1 -- 172.21.15.29:0/4013498275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b8b2f0 msgr2=0x55dab8f655d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.459+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe5fa7fc640 1 -- 172.21.15.29:0/4013498275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dab8c1f790 con 0x55dab8b934d0 2024-09-21T02:41:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65c90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe5f002a910 tx=0x7fe5f0031cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7188dd6bdae001ec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe5deff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe5deff5640 1 -- 172.21.15.29:0/4013498275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0035040 con 0x55dab8b934d0 2024-09-21T02:41:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe608691740 1 -- 172.21.15.29:0/4013498275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5f4003d10 con 0x55dab8b934d0 2024-09-21T02:41:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe5deff5640 1 -- 172.21.15.29:0/4013498275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe5f0004450 con 0x55dab8b934d0 2024-09-21T02:41:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe5deff5640 1 -- 172.21.15.29:0/4013498275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe5f004e050 con 0x55dab8b934d0 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe608691740 10 client.15885 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe608691740 1 -- 172.21.15.29:0/4013498275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dab8c98620 con 0x55dab8b934d0 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.460+0000 7fe5deff5640 1 -- 172.21.15.29:0/4013498275 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5f4003d10 con 0x55dab8b934d0 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe5deff5640 1 -- 172.21.15.29:0/4013498275 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fe5f003c740 con 0x55dab8b934d0 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe608691740 10 client.15885 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe608691740 10 client.15885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe608691740 20 client.15885 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe608691740 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dab8b72d80 0x55dab8f72db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe608691740 4 client.15885 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:41:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe608691740 1 -- 172.21.15.29:0/4013498275 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dab8ed0640 con 0x55dab8b72d80 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dab8b72d80 0x55dab8f72db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe5deff5640 10 client.15885 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.461+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dab8b72d80 0x55dab8f72db0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e5e688e76cd8245 server_cookie=a99d051460ffb099 in_seq=0 out_seq=0 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe5deff5640 1 -- 172.21.15.29:0/4013498275 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dab8ed0640 con 0x55dab8b72d80 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe5deff5640 10 client.15885 handle_command_reply: tid=0 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 client.15885 shutdown 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 -- 172.21.15.29:0/4013498275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 msgr2=0x55dab8f65c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65c90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe5f002a910 tx=0x7fe5f0031cd0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 -- 172.21.15.29:0/4013498275 shutdown_connections 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8bb41b0 0x55dab8f66800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dab8b72d80 0x55dab8f72db0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65c90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe5deff5640 0 client.15885 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 --2- 172.21.15.29:0/4013498275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.462+0000 7fe608691740 1 -- 172.21.15.29:0/4013498275 >> 172.21.15.29:0/4013498275 conn(0x55dab8ba8300 msgr2=0x55dab8f55ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.463+0000 7fe608691740 1 -- 172.21.15.29:0/4013498275 shutdown_connections 2024-09-21T02:41:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.463+0000 7fe608691740 1 -- 172.21.15.29:0/4013498275 wait complete. 2024-09-21T02:41:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.463+0000 7fe608691740 20 client.15885 trim_cache size 0 max 16384 2024-09-21T02:41:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe608691740 1 Processor -- start 2024-09-21T02:41:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe608691740 1 -- start start 2024-09-21T02:41:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe608691740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b72d80 0x55dab8f5cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe608691740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe608691740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe608691740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1aa2f0 con 0x55dab8b934d0 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe608691740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1abe60 con 0x55dab8b72d80 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe608691740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1aa9c0 con 0x55dab8b8b2f0 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe5fb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe5fa7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe5fb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50354/0 (socket says 172.21.15.29:50354) 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/4077913674 learned_addr learned my addr 172.21.15.29:0/4077913674 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe5fa7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48042/0 (socket says 172.21.15.29:48042) 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.481+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b72d80 0x55dab8f5cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5fa7fc640 1 -- 172.21.15.29:0/4077913674 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b72d80 msgr2=0x55dab8f5cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b72d80 0x55dab8f5cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5fa7fc640 1 -- 172.21.15.29:0/4077913674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 msgr2=0x55dab8f65b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5fa7fc640 1 -- 172.21.15.29:0/4077913674 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dab8f5c8e0 con 0x55dab8b8b2f0 2024-09-21T02:41:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0005ab0 tx=0x7fe5f002a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b580329d90f80164 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5e0ff9640 1 -- 172.21.15.29:0/4077913674 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0035070 con 0x55dab8b8b2f0 2024-09-21T02:41:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5e0ff9640 1 -- 172.21.15.29:0/4077913674 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5f0004450 con 0x55dab8b8b2f0 2024-09-21T02:41:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.482+0000 7fe5e0ff9640 1 -- 172.21.15.29:0/4077913674 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0003950 con 0x55dab8b8b2f0 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 -- 172.21.15.29:0/4077913674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 msgr2=0x55dab8f655d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0005ab0 tx=0x7fe5f002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 -- 172.21.15.29:0/4077913674 shutdown_connections 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b934d0 0x55dab8f65b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f655d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 --2- 172.21.15.29:0/4077913674 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b72d80 0x55dab8f5cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 -- 172.21.15.29:0/4077913674 >> 172.21.15.29:0/4077913674 conn(0x55dab8ba8300 msgr2=0x55dab8f647d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 -- 172.21.15.29:0/4077913674 shutdown_connections 2024-09-21T02:41:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 -- 172.21.15.29:0/4077913674 wait complete. 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 -- start start 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.483+0000 7fe608691740 1 Processor -- start 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe608691740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b72d80 0x55dab8f69e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe608691740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f6a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe608691740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 0x55dab8f6b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe608691740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1ab760 con 0x55dab8b72d80 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe608691740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1ab090 con 0x55dab8b934d0 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe608691740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1a82d0 con 0x55dab8b8b2f0 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe608691740 10 client.0 _ll_register_callbacks cb 0x7fe5f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b72d80 0x55dab8f69e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b72d80 0x55dab8f69e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50366/0 (socket says 172.21.15.29:50366) 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/1378361588 learned_addr learned my addr 172.21.15.29:0/1378361588 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f6a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 0x55dab8f6b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/1378361588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 msgr2=0x55dab8f6a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f6a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/1378361588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 msgr2=0x55dab8f6b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 0x55dab8f6b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 -- 172.21.15.29:0/1378361588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dab8f6b820 con 0x55dab8b72d80 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5fb7fe640 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b72d80 0x55dab8f69e90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe5f401d000 tx=0x7fe5f4010040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73416443c7384b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5deff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5deff5640 1 -- 172.21.15.29:0/1378361588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4037030 con 0x55dab8b72d80 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe5deff5640 1 -- 172.21.15.29:0/1378361588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe5f4005300 con 0x55dab8b72d80 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.484+0000 7fe608691740 1 -- 172.21.15.29:0/1378361588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5f0004450 con 0x55dab8b72d80 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe5deff5640 1 -- 172.21.15.29:0/1378361588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe5f4041050 con 0x55dab8b72d80 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe608691740 10 client.15891 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe608691740 1 -- 172.21.15.29:0/1378361588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dab8f6cb40 con 0x55dab8b72d80 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe5deff5640 1 -- 172.21.15.29:0/1378361588 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5f0004450 con 0x55dab8b72d80 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe5deff5640 1 -- 172.21.15.29:0/1378361588 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fe5f4007bb0 con 0x55dab8b72d80 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe608691740 10 client.15891 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe608691740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe608691740 20 client.15891 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe608691740 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dab8bb41b0 0x55dab8fdf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe608691740 4 client.15891 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe608691740 1 -- 172.21.15.29:0/1378361588 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dab8ecc0f0 con 0x55dab8bb41b0 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.485+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dab8bb41b0 0x55dab8fdf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.486+0000 7fe5deff5640 10 client.15891 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.486+0000 7fe5fa7fc640 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dab8bb41b0 0x55dab8fdf7c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38cbd87a3ab40078 server_cookie=9a326f1ae1715884 in_seq=0 out_seq=0 2024-09-21T02:41:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe5deff5640 1 -- 172.21.15.29:0/1378361588 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dab8ecc0f0 con 0x55dab8bb41b0 2024-09-21T02:41:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe5deff5640 10 client.15891 handle_command_reply: tid=0 2024-09-21T02:41:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 client.15891 shutdown 2024-09-21T02:41:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 -- 172.21.15.29:0/1378361588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b72d80 msgr2=0x55dab8f69e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b72d80 0x55dab8f69e90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe5f401d000 tx=0x7fe5f4010040 comp rx=0 tx=0).stop 2024-09-21T02:41:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 -- 172.21.15.29:0/1378361588 shutdown_connections 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dab8b934d0 0x55dab8f6b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dab8b8b2f0 0x55dab8f6a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dab8bb41b0 0x55dab8fdf7c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 --2- 172.21.15.29:0/1378361588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dab8b72d80 0x55dab8f69e90 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 -- 172.21.15.29:0/1378361588 >> 172.21.15.29:0/1378361588 conn(0x55dab8ba8300 msgr2=0x55dab8f647d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe5deff5640 0 client.15891 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 -- 172.21.15.29:0/1378361588 shutdown_connections 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.528+0000 7fe608691740 1 -- 172.21.15.29:0/1378361588 wait complete. 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe608691740 20 client.15891 trim_cache size 0 max 16384 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:41:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:41:16.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:41:16.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:41:16.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 -- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5e40b6590 msgr2=0x7fe5e40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5e40b6590 0x7fe5e40b8aa0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0ac580 tx=0x7fe5ec070e60 comp rx=0 tx=0).stop 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 -- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc1025d0 msgr2=0x7fe5fc197c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc1025d0 0x7fe5fc197c60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009990 tx=0x7fe5ec009e50 comp rx=0 tx=0).stop 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 -- 172.21.15.29:0/3830697367 shutdown_connections 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe5e40b6590 0x7fe5e40b8aa0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe5fc103ff0 0x7fe5fc198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5fc103650 0x7fe5fc1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 --2- 172.21.15.29:0/3830697367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5fc1025d0 0x7fe5fc197c60 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 -- 172.21.15.29:0/3830697367 >> 172.21.15.29:0/3830697367 conn(0x7fe5fc0fdf30 msgr2=0x7fe5fc106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 -- 172.21.15.29:0/3830697367 shutdown_connections 2024-09-21T02:41:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:16.529+0000 7fe60164a640 1 -- 172.21.15.29:0/3830697367 wait complete. 2024-09-21T02:41:17.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 20949 2.40 MB/sec warmup 54 sec latency 28.201 ms 2024-09-21T02:41:17.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:17 smithi190 ceph-mon[33871]: pgmap v454: 129 pgs: 129 active+clean; 27 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-09-21T02:41:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:17 smithi029 ceph-mon[31006]: pgmap v454: 129 pgs: 129 active+clean; 27 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-09-21T02:41:18.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:17 smithi066 ceph-mon[33436]: pgmap v454: 129 pgs: 129 active+clean; 27 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-09-21T02:41:18.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 21456 2.44 MB/sec warmup 55 sec latency 16.619 ms 2024-09-21T02:41:19.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 21759 2.42 MB/sec warmup 56 sec latency 15.363 ms 2024-09-21T02:41:19.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:19 smithi190 ceph-mon[33871]: pgmap v455: 129 pgs: 129 active+clean; 28 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T02:41:19.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:19 smithi029 ceph-mon[31006]: pgmap v455: 129 pgs: 129 active+clean; 28 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T02:41:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:19 smithi066 ceph-mon[33436]: pgmap v455: 129 pgs: 129 active+clean; 28 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T02:41:20.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 22263 2.40 MB/sec warmup 57 sec latency 16.347 ms 2024-09-21T02:41:21.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 22921 2.43 MB/sec warmup 58 sec latency 14.450 ms 2024-09-21T02:41:21.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:21 smithi190 ceph-mon[33871]: pgmap v456: 129 pgs: 129 active+clean; 28 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:41:21.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:21 smithi029 ceph-mon[31006]: pgmap v456: 129 pgs: 129 active+clean; 28 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:41:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:21 smithi066 ceph-mon[33436]: pgmap v456: 129 pgs: 129 active+clean; 28 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:41:22.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 23575 2.40 MB/sec warmup 59 sec latency 21.342 ms 2024-09-21T02:41:23.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 24317 2.47 MB/sec warmup 60 sec latency 14.981 ms 2024-09-21T02:41:23.435 INFO:tasks.quiescer.fs.[cephfs]:Total duration (55.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:41:23.436 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.8 2024-09-21T02:41:23.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:41:23.714 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:41:23.811 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:23 smithi066 ceph-mon[33436]: pgmap v457: 129 pgs: 129 active+clean; 29 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:41:23.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:23 smithi029 ceph-mon[31006]: pgmap v457: 129 pgs: 129 active+clean; 29 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:41:23.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:23 smithi190 ceph-mon[33871]: pgmap v457: 129 pgs: 129 active+clean; 29 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:41:24.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 24960 2.52 MB/sec warmup 61 sec latency 14.309 ms 2024-09-21T02:41:25.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 25265 2.50 MB/sec warmup 62 sec latency 80.423 ms 2024-09-21T02:41:25.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:25 smithi190 ceph-mon[33871]: pgmap v458: 129 pgs: 129 active+clean; 30 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:41:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:25 smithi029 ceph-mon[31006]: pgmap v458: 129 pgs: 129 active+clean; 30 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:41:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:25 smithi066 ceph-mon[33436]: pgmap v458: 129 pgs: 129 active+clean; 30 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:41:26.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 25491 2.46 MB/sec warmup 63 sec latency 218.172 ms 2024-09-21T02:41:27.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 26029 2.45 MB/sec warmup 64 sec latency 122.663 ms 2024-09-21T02:41:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:27 smithi190 ceph-mon[33871]: pgmap v459: 129 pgs: 129 active+clean; 30 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-09-21T02:41:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:27 smithi029 ceph-mon[31006]: pgmap v459: 129 pgs: 129 active+clean; 30 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-09-21T02:41:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:27 smithi066 ceph-mon[33436]: pgmap v459: 129 pgs: 129 active+clean; 30 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-09-21T02:41:28.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 26500 2.46 MB/sec warmup 65 sec latency 131.851 ms 2024-09-21T02:41:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.877+0000 7fef1f8d4640 1 -- 172.21.15.29:0/2873728975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18104840 msgr2=0x7fef18104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.877+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/2873728975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18104840 0x7fef18104c50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fef14009900 tx=0x7fef1402a410 comp rx=0 tx=0).stop 2024-09-21T02:41:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.877+0000 7fef1f8d4640 1 -- 172.21.15.29:0/2873728975 shutdown_connections 2024-09-21T02:41:28.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.877+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/2873728975 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef1810ddf0 0x7fef18110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.877+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/2873728975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef18105270 0x7fef1810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.877+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/2873728975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18104840 0x7fef18104c50 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.877+0000 7fef1f8d4640 1 -- 172.21.15.29:0/2873728975 >> 172.21.15.29:0/2873728975 conn(0x7fef18079730 msgr2=0x7fef18079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.877+0000 7fef1f8d4640 1 -- 172.21.15.29:0/2873728975 shutdown_connections 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.878+0000 7fef1f8d4640 1 -- 172.21.15.29:0/2873728975 wait complete. 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.878+0000 7fef1f8d4640 1 Processor -- start 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.878+0000 7fef1f8d4640 1 -- start start 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1f8d4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef18104840 0x7fef1819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1f8d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18105270 0x7fef1819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1f8d4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef1810ddf0 0x7fef1819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1f8d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef18113920 con 0x7fef18105270 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1f8d4640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fef181137a0 con 0x7fef1810ddf0 2024-09-21T02:41:28.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1f8d4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fef18113aa0 con 0x7fef18104840 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1ce6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18105270 0x7fef1819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1ce6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18105270 0x7fef1819a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36050/0 (socket says 172.21.15.29:36050) 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1ce6f640 1 -- 172.21.15.29:0/1170092853 learned_addr learned my addr 172.21.15.29:0/1170092853 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1de71640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef1810ddf0 0x7fef1819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1d670640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef18104840 0x7fef1819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1ce6f640 1 -- 172.21.15.29:0/1170092853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef18104840 msgr2=0x7fef1819a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1ce6f640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef18104840 0x7fef1819a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1ce6f640 1 -- 172.21.15.29:0/1170092853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef1810ddf0 msgr2=0x7fef1819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1ce6f640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef1810ddf0 0x7fef1819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1ce6f640 1 -- 172.21.15.29:0/1170092853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef181a0380 con 0x7fef18105270 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.879+0000 7fef1de71640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef1810ddf0 0x7fef1819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.880+0000 7fef1ce6f640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18105270 0x7fef1819a9b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fef08009ec0 tx=0x7fef0800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.880+0000 7fef0e7fc640 1 -- 172.21.15.29:0/1170092853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08020070 con 0x7fef18105270 2024-09-21T02:41:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.880+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef181a0670 con 0x7fef18105270 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.880+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef18102630 con 0x7fef18105270 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.880+0000 7fef0e7fc640 1 -- 172.21.15.29:0/1170092853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef0800cd80 con 0x7fef18105270 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.880+0000 7fef0e7fc640 1 -- 172.21.15.29:0/1170092853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0801b950 con 0x7fef18105270 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.881+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef18104c50 con 0x7fef18105270 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.883+0000 7fef0e7fc640 1 -- 172.21.15.29:0/1170092853 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fef08023070 con 0x7fef18105270 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.884+0000 7fef0e7fc640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef040b6590 0x7fef040b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.884+0000 7fef1d670640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef040b6590 0x7fef040b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.884+0000 7fef0e7fc640 1 -- 172.21.15.29:0/1170092853 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fef080eb7f0 con 0x7fef18105270 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.885+0000 7fef1d670640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef040b6590 0x7fef040b8aa0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fef14004770 tx=0x7fef140058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:28.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:28.888+0000 7fef0e7fc640 1 -- 172.21.15.29:0/1170092853 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fef080b60e0 con 0x7fef18105270 2024-09-21T02:41:29.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 27094 2.43 MB/sec warmup 66 sec latency 91.700 ms 2024-09-21T02:41:29.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.180+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef1802e640 con 0x7fef18105270 2024-09-21T02:41:29.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.181+0000 7fef0e7fc640 1 -- 172.21.15.29:0/1170092853 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11972 (secure 0 0 0) 0x7fef080bb020 con 0x7fef18105270 2024-09-21T02:41:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:41:29.183 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"btime":"2024-09-21T02:40:26:480768+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:40:26.304013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[3],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:41:29.183 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 30 2024-09-21T02:41:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef040b6590 msgr2=0x7fef040b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef040b6590 0x7fef040b8aa0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fef14004770 tx=0x7fef140058d0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18105270 msgr2=0x7fef1819a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18105270 0x7fef1819a9b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fef08009ec0 tx=0x7fef0800edc0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 shutdown_connections 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fef040b6590 0x7fef040b8aa0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef1810ddf0 0x7fef1819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef18105270 0x7fef1819a9b0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 --2- 172.21.15.29:0/1170092853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef18104840 0x7fef1819a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 >> 172.21.15.29:0/1170092853 conn(0x7fef18079730 msgr2=0x7fef180ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 shutdown_connections 2024-09-21T02:41:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:29.184+0000 7fef1f8d4640 1 -- 172.21.15.29:0/1170092853 wait complete. 2024-09-21T02:41:29.563 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:29 smithi029 ceph-mon[31006]: pgmap v460: 129 pgs: 129 active+clean; 30 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-09-21T02:41:29.563 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:29 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1170092853' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:29.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 -- 172.21.15.29:0/448806998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f01025a0 msgr2=0x7fe0f0102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 --2- 172.21.15.29:0/448806998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f01025a0 0x7fe0f0102a10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009900 tx=0x7fe0e002a410 comp rx=0 tx=0).stop 2024-09-21T02:41:29.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 -- 172.21.15.29:0/448806998 shutdown_connections 2024-09-21T02:41:29.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 --2- 172.21.15.29:0/448806998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0f0103fc0 0x7fe0f010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 --2- 172.21.15.29:0/448806998 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe0f0103620 0x7fe0f0103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 --2- 172.21.15.29:0/448806998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f01025a0 0x7fe0f0102a10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 -- 172.21.15.29:0/448806998 >> 172.21.15.29:0/448806998 conn(0x7fe0f00fdee0 msgr2=0x7fe0f0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:29.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 -- 172.21.15.29:0/448806998 shutdown_connections 2024-09-21T02:41:29.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.645+0000 7fe0f5339640 1 -- 172.21.15.29:0/448806998 wait complete. 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0f5339640 1 Processor -- start 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0f5339640 1 -- start start 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0f5339640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe0f01025a0 0x7fe0f0106190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0f5339640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f0103620 0x7fe0f0104a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0f5339640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0f0103fc0 0x7fe0f0104f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0f5339640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f010e080 con 0x7fe0f0103620 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0f5339640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f010df00 con 0x7fe0f01025a0 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0f5339640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f010e200 con 0x7fe0f0103fc0 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0ee7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f0103620 0x7fe0f0104a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0eeffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe0f01025a0 0x7fe0f0106190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0eeffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe0f01025a0 0x7fe0f0106190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:38400/0 (socket says 172.21.15.29:38400) 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0ef7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0f0103fc0 0x7fe0f0104f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.646+0000 7fe0eeffd640 1 -- 172.21.15.29:0/1655972783 learned_addr learned my addr 172.21.15.29:0/1655972783 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0eeffd640 1 -- 172.21.15.29:0/1655972783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0f0103fc0 msgr2=0x7fe0f0104f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0f0103fc0 0x7fe0f0104f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0eeffd640 1 -- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f0103620 msgr2=0x7fe0f0104a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f0103620 0x7fe0f0104a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0eeffd640 1 -- 172.21.15.29:0/1655972783 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f01056c0 con 0x7fe0f01025a0 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0ef7fe640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0f0103fc0 0x7fe0f0104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe0f01025a0 0x7fe0f0106190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009230 tx=0x7fe0e0004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0d7fff640 1 -- 172.21.15.29:0/1655972783 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e003f030 con 0x7fe0f01025a0 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0d7fff640 1 -- 172.21.15.29:0/1655972783 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0e0004440 con 0x7fe0f01025a0 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0f5339640 1 -- 172.21.15.29:0/1655972783 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f01a3ee0 con 0x7fe0f01025a0 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0d7fff640 1 -- 172.21.15.29:0/1655972783 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0035070 con 0x7fe0f01025a0 2024-09-21T02:41:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.647+0000 7fe0f5339640 1 -- 172.21.15.29:0/1655972783 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f01a4290 con 0x7fe0f01025a0 2024-09-21T02:41:29.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.650+0000 7fe0d7fff640 1 -- 172.21.15.29:0/1655972783 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fe0e0042030 con 0x7fe0f01025a0 2024-09-21T02:41:29.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0fc2a9740 1 Processor -- start 2024-09-21T02:41:29.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0fc2a9740 1 -- start start 2024-09-21T02:41:29.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe481b0 0x559b801efe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe1f2f0 0x559b801f04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0d7fff640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0d80b6590 0x7fe0d80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801f1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559b801626f0 con 0x559b7fe1f2f0 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559b80151b20 con 0x559b7fe274d0 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559b80155930 con 0x559b7fe481b0 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0d80b6590 0x7fe0d80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0d7fff640 1 -- 172.21.15.29:0/1655972783 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe0e0107a50 con 0x7fe0f01025a0 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.651+0000 7fe0eeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe1f2f0 0x559b801f04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0ef7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0eeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe1f2f0 0x559b801f04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36084/0 (socket says 172.21.15.29:36084) 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0eeffd640 1 -- 172.21.15.29:0/1997157051 learned_addr learned my addr 172.21.15.29:0/1997157051 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0ef7fe640 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe481b0 0x559b801efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0eeffd640 1 -- 172.21.15.29:0/1997157051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe481b0 msgr2=0x559b801efe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe481b0 0x559b801efe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0eeffd640 1 -- 172.21.15.29:0/1997157051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 msgr2=0x559b801f1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801f1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0eeffd640 1 -- 172.21.15.29:0/1997157051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b80117530 con 0x559b7fe1f2f0 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0d80b6590 0x7fe0d80b8aa0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009bd0 tx=0x7fe0e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0ef7fe640 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe1f2f0 0x559b801f04c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00c9ee0 tx=0x7fe0e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5af54b85d0c0073 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0d57fa640 1 -- 172.21.15.29:0/1997157051 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0049040 con 0x559b7fe1f2f0 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0d57fa640 1 -- 172.21.15.29:0/1997157051 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0e00d2340 con 0x559b7fe1f2f0 2024-09-21T02:41:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.652+0000 7fe0d57fa640 1 -- 172.21.15.29:0/1997157051 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e00d7030 con 0x559b7fe1f2f0 2024-09-21T02:41:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/1997157051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe1f2f0 msgr2=0x559b801f04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe1f2f0 0x559b801f04c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00c9ee0 tx=0x7fe0e0009260 comp rx=0 tx=0).stop 2024-09-21T02:41:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/1997157051 shutdown_connections 2024-09-21T02:41:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe1f2f0 0x559b801f04c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/1997157051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe481b0 0x559b801efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/1997157051 >> 172.21.15.29:0/1997157051 conn(0x559b7fe3c300 msgr2=0x559b801eb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/1997157051 shutdown_connections 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.653+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/1997157051 wait complete. 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 1 -- start start 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 1 Processor -- start 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 0x559b801f9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801fa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559b80089ab0 con 0x559b7fe481b0 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559b800f7730 con 0x559b7fe1f2f0 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559b801f18f0 con 0x559b7fe274d0 2024-09-21T02:41:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0fc2a9740 10 client.0 _ll_register_callbacks cb 0x559b8014b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801fa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.654+0000 7fe0ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801fa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36100/0 (socket says 172.21.15.29:36100) 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0ef7fe640 1 -- 172.21.15.29:0/3138129615 learned_addr learned my addr 172.21.15.29:0/3138129615 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 0x559b801f9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0ef7fe640 1 -- 172.21.15.29:0/3138129615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 msgr2=0x559b801f9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0ef7fe640 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0ef7fe640 1 -- 172.21.15.29:0/3138129615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 msgr2=0x559b801f9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0ef7fe640 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 0x559b801f9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0ef7fe640 1 -- 172.21.15.29:0/3138129615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b7feb3790 con 0x559b7fe481b0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0ef7fe640 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801fa570 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8004440 tx=0x7fe0e8004900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab5182363b48867 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/3138129615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8019070 con 0x559b7fe481b0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/3138129615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0e00d2340 con 0x559b7fe481b0 2024-09-21T02:41:29.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.655+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/3138129615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe0e8010430 con 0x559b7fe481b0 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/3138129615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe0e8028050 con 0x559b7fe481b0 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0fc2a9740 10 client.15909 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/3138129615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559b7ff2c620 con 0x559b7fe481b0 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/3138129615 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0e00d2340 con 0x559b7fe481b0 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/3138129615 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fe0e801d520 con 0x559b7fe481b0 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0fc2a9740 10 client.15909 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0fc2a9740 10 client.15909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0fc2a9740 20 client.15909 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559b7fe06d80 0x559b80206c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0fc2a9740 4 client.15909 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/3138129615 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b80164640 con 0x559b7fe06d80 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.656+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559b7fe06d80 0x559b80206c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.657+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559b7fe06d80 0x559b80206c20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52da164a8c73ec9e server_cookie=cc086a5ec8e07c97 in_seq=0 out_seq=0 2024-09-21T02:41:29.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.657+0000 7fe0d2ff5640 10 client.15909 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:29.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.658+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/3138129615 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559b80164640 con 0x559b7fe06d80 2024-09-21T02:41:29.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.658+0000 7fe0d2ff5640 10 client.15909 handle_command_reply: tid=0 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.658+0000 7fe0fc2a9740 1 client.15909 shutdown 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.658+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/3138129615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 msgr2=0x559b801fa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.658+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801fa570 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8004440 tx=0x7fe0e8004900 comp rx=0 tx=0).stop 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/3138129615 shutdown_connections 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559b7fe06d80 0x559b80206c20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801fa570 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/3138129615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 0x559b801f9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/3138129615 >> 172.21.15.29:0/3138129615 conn(0x559b7fe3c300 msgr2=0x559b801e9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0d2ff5640 0 client.15909 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/3138129615 shutdown_connections 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/3138129615 wait complete. 2024-09-21T02:41:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.659+0000 7fe0fc2a9740 20 client.15909 trim_cache size 0 max 16384 2024-09-21T02:41:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.677+0000 7fe0fc2a9740 1 Processor -- start 2024-09-21T02:41:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.677+0000 7fe0fc2a9740 1 -- start start 2024-09-21T02:41:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.677+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 0x559b801e6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.677+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.677+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801eb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.677+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f01ae6a0 con 0x559b7fe481b0 2024-09-21T02:41:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.677+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f01b0210 con 0x559b7fe1f2f0 2024-09-21T02:41:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.677+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f01aed70 con 0x559b7fe274d0 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801eb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49092/0 (socket says 172.21.15.29:49092) 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801eb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36114/0 (socket says 172.21.15.29:36114) 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 -- 172.21.15.29:0/2232686761 learned_addr learned my addr 172.21.15.29:0/2232686761 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/2232686761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 0x559b801e6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 -- 172.21.15.29:0/2232686761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 msgr2=0x559b801e6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/2232686761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 0x559b801e6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 -- 172.21.15.29:0/2232686761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 msgr2=0x559b801eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/2232686761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801eb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 -- 172.21.15.29:0/2232686761 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b7fe06fb0 con 0x559b7fe274d0 2024-09-21T02:41:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/2232686761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f0c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4004490 tx=0x7fe0e4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2bde02cb49d4d14 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.678+0000 7fe0d4ff9640 1 -- 172.21.15.29:0/2232686761 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4003a70 con 0x559b7fe274d0 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.679+0000 7fe0d4ff9640 1 -- 172.21.15.29:0/2232686761 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0e4003c10 con 0x559b7fe274d0 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.679+0000 7fe0d4ff9640 1 -- 172.21.15.29:0/2232686761 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e401c670 con 0x559b7fe274d0 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.679+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/2232686761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 msgr2=0x559b801f0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.679+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/2232686761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f0c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4004490 tx=0x7fe0e4014000 comp rx=0 tx=0).stop 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/2232686761 shutdown_connections 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/2232686761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801eb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/2232686761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe274d0 0x559b801f0c90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/2232686761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe1f2f0 0x559b801e6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/2232686761 >> 172.21.15.29:0/2232686761 conn(0x559b7fe3c300 msgr2=0x559b80206bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/2232686761 shutdown_connections 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/2232686761 wait complete. 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 -- start start 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 Processor -- start 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe1f2f0 0x559b801fe350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801fea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801ff530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f01afb10 con 0x559b7fe481b0 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f01af440 con 0x559b7fe274d0 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.680+0000 7fe0fc2a9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f01ac680 con 0x559b7fe1f2f0 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0fc2a9740 10 client.0 _ll_register_callbacks cb 0x7fe0e8003aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe1f2f0 0x559b801fe350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801ff530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe1f2f0 0x559b801fe350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49106/0 (socket says 172.21.15.29:49106) 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 -- 172.21.15.29:0/298874281 learned_addr learned my addr 172.21.15.29:0/298874281 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801fea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 -- 172.21.15.29:0/298874281 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 msgr2=0x559b801fea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801fea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 -- 172.21.15.29:0/298874281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 msgr2=0x559b801ff530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801ff530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 -- 172.21.15.29:0/298874281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b801ffc90 con 0x559b7fe1f2f0 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0ef7fe640 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801ff530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0eeffd640 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe1f2f0 0x559b801fe350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00cd610 tx=0x7fe0e00d7350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7078ebbfe399211d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:41:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/298874281 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0049040 con 0x559b7fe1f2f0 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/298874281 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe0e00d6ff0 con 0x559b7fe1f2f0 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.681+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/298874281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0e4003c10 con 0x559b7fe1f2f0 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/298874281 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe0e00ceb90 con 0x559b7fe1f2f0 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0fc2a9740 10 client.25558 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/298874281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559b80201020 con 0x559b7fe1f2f0 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/298874281 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0e4003c10 con 0x559b7fe1f2f0 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/298874281 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fe0e00c9260 con 0x559b7fe1f2f0 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0fc2a9740 10 client.25558 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0fc2a9740 10 client.25558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0fc2a9740 20 client.25558 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559b7fddc1c0 0x559b802735f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0fc2a9740 4 client.25558 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/298874281 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559b801600f0 con 0x559b7fddc1c0 2024-09-21T02:41:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.682+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559b7fddc1c0 0x559b802735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.683+0000 7fe0d2ff5640 10 client.25558 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.683+0000 7fe0ee7fc640 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559b7fddc1c0 0x559b802735f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13688f741da810 server_cookie=84a7b5f22a97c191 in_seq=0 out_seq=0 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0d2ff5640 1 -- 172.21.15.29:0/298874281 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559b801600f0 con 0x559b7fddc1c0 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0d2ff5640 10 client.25558 handle_command_reply: tid=0 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 client.25558 shutdown 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/298874281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe1f2f0 msgr2=0x559b801fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe1f2f0 0x559b801fe350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00cd610 tx=0x7fe0e00d7350 comp rx=0 tx=0).stop 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/298874281 shutdown_connections 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559b7fddc1c0 0x559b802735f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b7fe481b0 0x559b801ff530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0d2ff5640 0 client.25558 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559b7fe274d0 0x559b801fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 --2- 172.21.15.29:0/298874281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b7fe1f2f0 0x559b801fe350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/298874281 >> 172.21.15.29:0/298874281 conn(0x559b7fe3c300 msgr2=0x559b801e66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/298874281 shutdown_connections 2024-09-21T02:41:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 1 -- 172.21.15.29:0/298874281 wait complete. 2024-09-21T02:41:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.723+0000 7fe0fc2a9740 20 client.25558 trim_cache size 0 max 16384 2024-09-21T02:41:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:41:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:41:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:41:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:41:29.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 -- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0d80b6590 msgr2=0x7fe0d80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0d80b6590 0x7fe0d80b8aa0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009bd0 tx=0x7fe0e4005f90 comp rx=0 tx=0).stop 2024-09-21T02:41:29.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 -- 172.21.15.29:0/1655972783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe0f01025a0 msgr2=0x7fe0f0106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:29.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe0f01025a0 0x7fe0f0106190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009230 tx=0x7fe0e0004040 comp rx=0 tx=0).stop 2024-09-21T02:41:29.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 -- 172.21.15.29:0/1655972783 shutdown_connections 2024-09-21T02:41:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe0d80b6590 0x7fe0d80b8aa0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0f0103fc0 0x7fe0f0104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f0103620 0x7fe0f0104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 --2- 172.21.15.29:0/1655972783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe0f01025a0 0x7fe0f0106190 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 -- 172.21.15.29:0/1655972783 >> 172.21.15.29:0/1655972783 conn(0x7fe0f00fdee0 msgr2=0x7fe0f00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 -- 172.21.15.29:0/1655972783 shutdown_connections 2024-09-21T02:41:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:29.724+0000 7fe0f5339640 1 -- 172.21.15.29:0/1655972783 wait complete. 2024-09-21T02:41:29.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:29 smithi190 ceph-mon[33871]: pgmap v460: 129 pgs: 129 active+clean; 30 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-09-21T02:41:29.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:29 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1170092853' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:29 smithi066 ceph-mon[33436]: pgmap v460: 129 pgs: 129 active+clean; 30 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-09-21T02:41:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:29 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1170092853' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:30.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 27793 2.50 MB/sec warmup 67 sec latency 19.033 ms 2024-09-21T02:41:30.084 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0' 2024-09-21T02:41:30.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0 2024-09-21T02:41:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:41:30.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:41:30.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:41:30.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:41:30.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:41:30.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:41:30.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:41:30.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:41:30.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:41:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/2811564409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 msgr2=0x7f1c58103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/2811564409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 0x7f1c58103fa0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009900 tx=0x7f1c4802a410 comp rx=0 tx=0).stop 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/2811564409 shutdown_connections 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/2811564409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c58109e20 0x7f1c58110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/2811564409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c581094d0 0x7f1c581098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/2811564409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 0x7f1c58103fa0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/2811564409 >> 172.21.15.29:0/2811564409 conn(0x7f1c580ff5c0 msgr2=0x7f1c58101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/2811564409 shutdown_connections 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.954+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/2811564409 wait complete. 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.955+0000 7f1c5cf0d640 1 Processor -- start 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.955+0000 7f1c5cf0d640 1 -- start start 2024-09-21T02:41:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5cf0d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 0x7f1c5819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5cf0d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c581094d0 0x7f1c5819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5cf0d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c58109e20 0x7f1c5819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5cf0d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581139e0 con 0x7f1c58103b90 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5cf0d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1c58113860 con 0x7f1c58109e20 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5cf0d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c58113b60 con 0x7f1c581094d0 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 0x7f1c5819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 0x7f1c5819a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36154/0 (socket says 172.21.15.29:36154) 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5659c640 1 -- 172.21.15.29:0/1718960479 learned_addr learned my addr 172.21.15.29:0/1718960479 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c58109e20 0x7f1c5819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c581094d0 0x7f1c5819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5659c640 1 -- 172.21.15.29:0/1718960479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c581094d0 msgr2=0x7f1c5819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5659c640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c581094d0 0x7f1c5819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5659c640 1 -- 172.21.15.29:0/1718960479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c58109e20 msgr2=0x7f1c5819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5659c640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c58109e20 0x7f1c5819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.956+0000 7f1c5659c640 1 -- 172.21.15.29:0/1718960479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c581a0380 con 0x7f1c58103b90 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.957+0000 7f1c5659c640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 0x7f1c5819a410 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1c4802a930 tx=0x7f1c48003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.957+0000 7f1c477fe640 1 -- 172.21.15.29:0/1718960479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4803f070 con 0x7f1c58103b90 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.957+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/1718960479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c581a0610 con 0x7f1c58103b90 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.957+0000 7f1c477fe640 1 -- 172.21.15.29:0/1718960479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c4803baf0 con 0x7f1c58103b90 2024-09-21T02:41:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.957+0000 7f1c477fe640 1 -- 172.21.15.29:0/1718960479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48035030 con 0x7f1c58103b90 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.957+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/1718960479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c581a6dd0 con 0x7f1c58103b90 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c63481740 1 Processor -- start 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c63481740 1 -- start start 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c477fe640 1 -- 172.21.15.29:0/1718960479 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f1c48042070 con 0x7f1c58103b90 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c63481740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d103600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c63481740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d103b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c63481740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd3bb60 0x561a4d104080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c63481740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561a4d079b90 con 0x561a4cd3bb60 2024-09-21T02:41:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c63481740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561a4d068fc0 con 0x561a4cd5c840 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c63481740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561a4d06cdd0 con 0x561a4cd33980 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c55d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd3bb60 0x561a4d104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c5659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c55d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd3bb60 0x561a4d104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36156/0 (socket says 172.21.15.29:36156) 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c55d9b640 1 -- 172.21.15.29:0/3788635392 learned_addr learned my addr 172.21.15.29:0/3788635392 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.960+0000 7f1c56d9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d103600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c56d9d640 1 -- 172.21.15.29:0/3788635392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 msgr2=0x561a4d103b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/3788635392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d103b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c56d9d640 1 -- 172.21.15.29:0/3788635392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd3bb60 msgr2=0x561a4d104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/3788635392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd3bb60 0x561a4d104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c56d9d640 1 -- 172.21.15.29:0/3788635392 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a4cff0060 con 0x561a4cd5c840 2024-09-21T02:41:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/3788635392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd3bb60 0x561a4d104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/3788635392 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d103600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009f00 tx=0x7f1c50011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3c6026eea91512 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c44ff9640 1 -- 172.21.15.29:0/3788635392 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50004d20 con 0x561a4cd5c840 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c44ff9640 1 -- 172.21.15.29:0/3788635392 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c50004050 con 0x561a4cd5c840 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.961+0000 7f1c44ff9640 1 -- 172.21.15.29:0/3788635392 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50005280 con 0x561a4cd5c840 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c477fe640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c400b6510 0x7f1c400b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c5659c640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c400b6510 0x7f1c400b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c477fe640 1 -- 172.21.15.29:0/1718960479 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1c48107d30 con 0x7f1c58103b90 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c63481740 1 -- 172.21.15.29:0/3788635392 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 msgr2=0x561a4d103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c63481740 1 --2- 172.21.15.29:0/3788635392 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d103600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009f00 tx=0x7f1c50011810 comp rx=0 tx=0).stop 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c63481740 1 -- 172.21.15.29:0/3788635392 shutdown_connections 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c63481740 1 --2- 172.21.15.29:0/3788635392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd3bb60 0x561a4d104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c63481740 1 --2- 172.21.15.29:0/3788635392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c63481740 1 --2- 172.21.15.29:0/3788635392 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d103600 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.962+0000 7f1c63481740 1 -- 172.21.15.29:0/3788635392 >> 172.21.15.29:0/3788635392 conn(0x561a4cd50990 msgr2=0x561a4d1014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.963+0000 7f1c63481740 1 -- 172.21.15.29:0/3788635392 shutdown_connections 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.963+0000 7f1c5659c640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c400b6510 0x7f1c400b8a20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x561a4d102280 tx=0x7f1c4810b810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.963+0000 7f1c63481740 1 -- 172.21.15.29:0/3788635392 wait complete. 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.963+0000 7f1c63481740 1 -- start start 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.963+0000 7f1c63481740 1 Processor -- start 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.963+0000 7f1c63481740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd33980 0x561a4d1135d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c63481740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d113b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c63481740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 0x561a4d11f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c63481740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561a4ceca7a0 con 0x561a4cd5c840 2024-09-21T02:41:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c63481740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561a4d00e0a0 con 0x561a4cd33980 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c63481740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561a4d116080 con 0x561a4cd3bb60 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c63481740 10 client.0 _ll_register_callbacks cb 0x561a4cfeb3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd33980 0x561a4d1135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c55d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd33980 0x561a4d1135d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:38476/0 (socket says 172.21.15.29:38476) 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 -- 172.21.15.29:0/3563593186 learned_addr learned my addr 172.21.15.29:0/3563593186 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49148/0 (socket says 172.21.15.29:49148) 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 0x561a4d11f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 -- 172.21.15.29:0/3563593186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 msgr2=0x561a4d113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 -- 172.21.15.29:0/3563593186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 msgr2=0x561a4d11f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 0x561a4d11f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 -- 172.21.15.29:0/3563593186 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a4cffc880 con 0x561a4cd33980 2024-09-21T02:41:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd33980 0x561a4d1135d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009370 tx=0x7f1c50009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46b554313a80b84a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.964+0000 7f1c3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.965+0000 7f1c3dffb640 1 -- 172.21.15.29:0/3563593186 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50005200 con 0x561a4cd33980 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.965+0000 7f1c3dffb640 1 -- 172.21.15.29:0/3563593186 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1c500053a0 con 0x561a4cd33980 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.965+0000 7f1c63481740 1 -- 172.21.15.29:0/3563593186 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c50004050 con 0x561a4cd33980 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.965+0000 7f1c3dffb640 1 -- 172.21.15.29:0/3563593186 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1c50035050 con 0x561a4cd33980 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.965+0000 7f1c63481740 10 client.25496 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.965+0000 7f1c63481740 1 -- 172.21.15.29:0/3563593186 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561a4cdd6b90 con 0x561a4cd33980 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.965+0000 7f1c3dffb640 1 -- 172.21.15.29:0/3563593186 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c50004050 con 0x561a4cd33980 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.965+0000 7f1c3dffb640 1 -- 172.21.15.29:0/3563593186 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f1c500197d0 con 0x561a4cd33980 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c63481740 10 client.25496 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c63481740 10 client.25496 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c63481740 20 client.25496 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c63481740 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561a4cd1b1c0 0x561a4d125310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c63481740 4 client.25496 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c63481740 1 -- 172.21.15.29:0/3563593186 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a4d07bae0 con 0x561a4cd1b1c0 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561a4cd1b1c0 0x561a4d125310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561a4cd1b1c0 0x561a4d125310 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3504a310b2fddff server_cookie=a724d275a859af1a in_seq=0 out_seq=0 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.966+0000 7f1c3dffb640 10 client.25496 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.967+0000 7f1c3dffb640 1 -- 172.21.15.29:0/3563593186 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561a4d07bae0 con 0x561a4cd1b1c0 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.967+0000 7f1c3dffb640 10 client.25496 handle_command_reply: tid=0 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.967+0000 7f1c63481740 1 client.25496 shutdown 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 -- 172.21.15.29:0/3563593186 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd33980 msgr2=0x561a4d1135d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd33980 0x561a4d1135d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009370 tx=0x7f1c50009c40 comp rx=0 tx=0).stop 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 -- 172.21.15.29:0/3563593186 shutdown_connections 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561a4cd1b1c0 0x561a4d125310 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 0x561a4d11f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 --2- 172.21.15.29:0/3563593186 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd33980 0x561a4d1135d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 -- 172.21.15.29:0/3563593186 >> 172.21.15.29:0/3563593186 conn(0x561a4cd50990 msgr2=0x561a4d104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c3dffb640 0 client.25496 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 -- 172.21.15.29:0/3563593186 shutdown_connections 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 1 -- 172.21.15.29:0/3563593186 wait complete. 2024-09-21T02:41:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.968+0000 7f1c63481740 20 client.25496 trim_cache size 0 max 16384 2024-09-21T02:41:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.989+0000 7f1c63481740 1 Processor -- start 2024-09-21T02:41:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.989+0000 7f1c63481740 1 -- start start 2024-09-21T02:41:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.989+0000 7f1c63481740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd33980 0x561a4d11f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.989+0000 7f1c63481740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d111c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.989+0000 7f1c63481740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d112190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.989+0000 7f1c63481740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581ae800 con 0x561a4cd33980 2024-09-21T02:41:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.989+0000 7f1c63481740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581b0370 con 0x561a4cd5c840 2024-09-21T02:41:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.989+0000 7f1c63481740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581aeed0 con 0x561a4cd3bb60 2024-09-21T02:41:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c56d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd33980 0x561a4d11f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c56d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd33980 0x561a4d11f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36164/0 (socket says 172.21.15.29:36164) 2024-09-21T02:41:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c56d9d640 1 -- 172.21.15.29:0/2508038493 learned_addr learned my addr 172.21.15.29:0/2508038493 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d111c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d112190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c55d9b640 1 -- 172.21.15.29:0/2508038493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 msgr2=0x561a4d112190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d112190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c55d9b640 1 -- 172.21.15.29:0/2508038493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd33980 msgr2=0x561a4d11f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd33980 0x561a4d11f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c55d9b640 1 -- 172.21.15.29:0/2508038493 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a4d1063c0 con 0x561a4cd3bb60 2024-09-21T02:41:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d111c50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x561a4d125930 tx=0x7f1c4c00b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f3fc05127db034 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c3ffff640 1 -- 172.21.15.29:0/2508038493 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c005220 con 0x561a4cd3bb60 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c3ffff640 1 -- 172.21.15.29:0/2508038493 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c4c004c30 con 0x561a4cd3bb60 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.990+0000 7f1c3ffff640 1 -- 172.21.15.29:0/2508038493 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c01d8b0 con 0x561a4cd3bb60 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.991+0000 7f1c63481740 1 -- 172.21.15.29:0/2508038493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 msgr2=0x561a4d111c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.991+0000 7f1c63481740 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d111c50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x561a4d125930 tx=0x7f1c4c00b410 comp rx=0 tx=0).stop 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.991+0000 7f1c63481740 1 -- 172.21.15.29:0/2508038493 shutdown_connections 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.991+0000 7f1c63481740 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd5c840 0x561a4d112190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.991+0000 7f1c63481740 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd3bb60 0x561a4d111c50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.991+0000 7f1c63481740 1 --2- 172.21.15.29:0/2508038493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd33980 0x561a4d11f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.991+0000 7f1c63481740 1 -- 172.21.15.29:0/2508038493 >> 172.21.15.29:0/2508038493 conn(0x561a4cd50990 msgr2=0x561a4d10e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.992+0000 7f1c63481740 1 -- 172.21.15.29:0/2508038493 shutdown_connections 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.992+0000 7f1c63481740 1 -- 172.21.15.29:0/2508038493 wait complete. 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.992+0000 7f1c63481740 1 -- start start 2024-09-21T02:41:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.992+0000 7f1c63481740 1 Processor -- start 2024-09-21T02:41:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c63481740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d1183b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c63481740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd3bb60 0x561a4d1188f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c63481740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 0x561a4d118e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c63481740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581afc70 con 0x561a4cd5c840 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c63481740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581af5a0 con 0x561a4cd3bb60 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c63481740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581ac7e0 con 0x561a4cd33980 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c63481740 10 client.0 _ll_register_callbacks cb 0x7f1c4c00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c56d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d1183b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c56d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d1183b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49170/0 (socket says 172.21.15.29:49170) 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c56d9d640 1 -- 172.21.15.29:0/2152500343 learned_addr learned my addr 172.21.15.29:0/2152500343 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd3bb60 0x561a4d1188f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 0x561a4d118e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c55d9b640 1 -- 172.21.15.29:0/2152500343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 msgr2=0x561a4d1183b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d1183b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c55d9b640 1 -- 172.21.15.29:0/2152500343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 msgr2=0x561a4d118e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 0x561a4d118e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c55d9b640 1 -- 172.21.15.29:0/2152500343 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a4d104440 con 0x561a4cd3bb60 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d1183b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.993+0000 7f1c55d9b640 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd3bb60 0x561a4d1188f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c01d4f0 tx=0x7f1c4c004de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aea844049774b725 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.994+0000 7f1c3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.994+0000 7f1c3dffb640 1 -- 172.21.15.29:0/2152500343 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c004940 con 0x561a4cd3bb60 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.994+0000 7f1c3dffb640 1 -- 172.21.15.29:0/2152500343 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1c4c004020 con 0x561a4cd3bb60 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.994+0000 7f1c63481740 1 -- 172.21.15.29:0/2152500343 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c4c004c30 con 0x561a4cd3bb60 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.994+0000 7f1c3dffb640 1 -- 172.21.15.29:0/2152500343 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1c4c004220 con 0x561a4cd3bb60 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.994+0000 7f1c63481740 10 client.25502 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.994+0000 7f1c63481740 1 -- 172.21.15.29:0/2152500343 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561a4d190ce0 con 0x561a4cd3bb60 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.994+0000 7f1c3dffb640 1 -- 172.21.15.29:0/2152500343 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c4c004c30 con 0x561a4cd3bb60 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.995+0000 7f1c3dffb640 1 -- 172.21.15.29:0/2152500343 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f1c4c003680 con 0x561a4cd3bb60 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.995+0000 7f1c63481740 10 client.25502 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.995+0000 7f1c63481740 10 client.25502 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:41:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.995+0000 7f1c63481740 20 client.25502 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.995+0000 7f1c63481740 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561a4ccf0570 0x561a4d193500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.995+0000 7f1c63481740 4 client.25502 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}] 2024-09-21T02:41:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.995+0000 7f1c63481740 1 -- 172.21.15.29:0/2152500343 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}) -- 0x561a4d077590 con 0x561a4ccf0570 2024-09-21T02:41:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.995+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561a4ccf0570 0x561a4d193500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.996+0000 7f1c56d9d640 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561a4ccf0570 0x561a4d193500 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8720d5a9dcd5d5a5 server_cookie=6ec6012fd7d9a1bf in_seq=0 out_seq=0 2024-09-21T02:41:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:30.996+0000 7f1c3dffb640 10 client.25502 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:41:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:41:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:41:31.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:41:31.067 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28248 2.49 MB/sec warmup 68 sec latency 55.765 ms 2024-09-21T02:41:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c3dffb640 1 -- 172.21.15.29:0/2152500343 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561a4d077590 con 0x561a4ccf0570 2024-09-21T02:41:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c3dffb640 10 client.25502 handle_command_reply: tid=0 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 client.25502 shutdown 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 -- 172.21.15.29:0/2152500343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd3bb60 msgr2=0x561a4d1188f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd3bb60 0x561a4d1188f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c01d4f0 tx=0x7f1c4c004de0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 -- 172.21.15.29:0/2152500343 shutdown_connections 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561a4ccf0570 0x561a4d193500 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561a4cd5c840 0x561a4d118e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561a4cd3bb60 0x561a4d1188f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c3dffb640 0 client.25502 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 --2- 172.21.15.29:0/2152500343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4cd33980 0x561a4d1183b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.119+0000 7f1c63481740 1 -- 172.21.15.29:0/2152500343 >> 172.21.15.29:0/2152500343 conn(0x561a4cd50990 msgr2=0x561a4d104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c63481740 1 -- 172.21.15.29:0/2152500343 shutdown_connections 2024-09-21T02:41:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c63481740 1 -- 172.21.15.29:0/2152500343 wait complete. 2024-09-21T02:41:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c63481740 20 client.25502 trim_cache size 0 max 16384 2024-09-21T02:41:31.122 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:41:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 30, 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 10, 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "6b96cd99": { 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 10, 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:41:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 147.8, 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:31.124 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:41:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c400b6510 msgr2=0x7f1c400b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c400b6510 0x7f1c400b8a20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x561a4d102280 tx=0x7f1c4810b810 comp rx=0 tx=0).stop 2024-09-21T02:41:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 msgr2=0x7f1c5819a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 0x7f1c5819a410 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1c4802a930 tx=0x7f1c48003b90 comp rx=0 tx=0).stop 2024-09-21T02:41:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/1718960479 shutdown_connections 2024-09-21T02:41:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1c400b6510 0x7f1c400b8a20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c58109e20 0x7f1c5819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c581094d0 0x7f1c5819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 --2- 172.21.15.29:0/1718960479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c58103b90 0x7f1c5819a410 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.120+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/1718960479 >> 172.21.15.29:0/1718960479 conn(0x7f1c580ff5c0 msgr2=0x7f1c58108b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.121+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/1718960479 shutdown_connections 2024-09-21T02:41:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:31.121+0000 7f1c5cf0d640 1 -- 172.21.15.29:0/1718960479 wait complete. 2024-09-21T02:41:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:31 smithi190 ceph-mon[33871]: pgmap v461: 129 pgs: 129 active+clean; 29 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-09-21T02:41:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:31 smithi029 ceph-mon[31006]: pgmap v461: 129 pgs: 129 active+clean; 29 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-09-21T02:41:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:31 smithi066 ceph-mon[33436]: pgmap v461: 129 pgs: 129 active+clean; 29 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-09-21T02:41:32.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.45 MB/sec warmup 69 sec latency 948.215 ms 2024-09-21T02:41:32.120 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6b96cd99', quiesce took 8.7 seconds. Will release after: 19.1 2024-09-21T02:41:33.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.42 MB/sec warmup 70 sec latency 1948.308 ms 2024-09-21T02:41:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:33 smithi190 ceph-mon[33871]: pgmap v462: 129 pgs: 129 active+clean; 25 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:41:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:33 smithi029 ceph-mon[31006]: pgmap v462: 129 pgs: 129 active+clean; 25 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:41:34.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:33 smithi066 ceph-mon[33436]: pgmap v462: 129 pgs: 129 active+clean; 25 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:41:34.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.39 MB/sec warmup 71 sec latency 2948.409 ms 2024-09-21T02:41:35.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.35 MB/sec warmup 72 sec latency 3948.536 ms 2024-09-21T02:41:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:35 smithi190 ceph-mon[33871]: pgmap v463: 129 pgs: 129 active+clean; 26 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-09-21T02:41:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:35 smithi029 ceph-mon[31006]: pgmap v463: 129 pgs: 129 active+clean; 26 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-09-21T02:41:36.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:35 smithi066 ceph-mon[33436]: pgmap v463: 129 pgs: 129 active+clean; 26 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-09-21T02:41:36.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.32 MB/sec warmup 73 sec latency 4948.587 ms 2024-09-21T02:41:37.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.29 MB/sec warmup 74 sec latency 5948.670 ms 2024-09-21T02:41:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:37 smithi190 ceph-mon[33871]: pgmap v464: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.1 MiB/s wr, 105 op/s 2024-09-21T02:41:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:37 smithi029 ceph-mon[31006]: pgmap v464: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.1 MiB/s wr, 105 op/s 2024-09-21T02:41:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:37 smithi066 ceph-mon[33436]: pgmap v464: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.1 MiB/s wr, 105 op/s 2024-09-21T02:41:38.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.26 MB/sec warmup 75 sec latency 6948.778 ms 2024-09-21T02:41:39.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.23 MB/sec warmup 76 sec latency 7948.887 ms 2024-09-21T02:41:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:39 smithi190 ceph-mon[33871]: pgmap v465: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 838 KiB/s wr, 101 op/s 2024-09-21T02:41:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:39 smithi029 ceph-mon[31006]: pgmap v465: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 838 KiB/s wr, 101 op/s 2024-09-21T02:41:40.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:39 smithi066 ceph-mon[33436]: pgmap v465: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 838 KiB/s wr, 101 op/s 2024-09-21T02:41:40.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.20 MB/sec warmup 77 sec latency 8948.940 ms 2024-09-21T02:41:41.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.17 MB/sec warmup 78 sec latency 9949.049 ms 2024-09-21T02:41:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:41 smithi190 ceph-mon[33871]: pgmap v466: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 637 KiB/s wr, 71 op/s 2024-09-21T02:41:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:41 smithi029 ceph-mon[31006]: pgmap v466: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 637 KiB/s wr, 71 op/s 2024-09-21T02:41:42.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:41 smithi066 ceph-mon[33436]: pgmap v466: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 637 KiB/s wr, 71 op/s 2024-09-21T02:41:42.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.14 MB/sec warmup 79 sec latency 10949.163 ms 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 -- 172.21.15.29:0/748750684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe241025d0 msgr2=0x7fbe24102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/748750684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe241025d0 0x7fbe24102a40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009900 tx=0x7fbe2002a410 comp rx=0 tx=0).stop 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 -- 172.21.15.29:0/748750684 shutdown_connections 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/748750684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe24103ff0 0x7fbe2410a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/748750684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe24103650 0x7fbe24103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/748750684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe241025d0 0x7fbe24102a40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 -- 172.21.15.29:0/748750684 >> 172.21.15.29:0/748750684 conn(0x7fbe240fdf30 msgr2=0x7fbe241003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 -- 172.21.15.29:0/748750684 shutdown_connections 2024-09-21T02:41:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.838+0000 7fbe2c40a640 1 -- 172.21.15.29:0/748750684 wait complete. 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe2c40a640 1 Processor -- start 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe2c40a640 1 -- start start 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe2c40a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe241025d0 0x7fbe24199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe2c40a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe24103650 0x7fbe2419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe2c40a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe24103ff0 0x7fbe2419b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe2c40a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2410e0c0 con 0x7fbe24103650 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe2c40a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2410df40 con 0x7fbe24103ff0 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe2c40a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2410e240 con 0x7fbe241025d0 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe299a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe24103650 0x7fbe2419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe299a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe24103650 0x7fbe2419a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51888/0 (socket says 172.21.15.29:51888) 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.839+0000 7fbe299a5640 1 -- 172.21.15.29:0/106396875 learned_addr learned my addr 172.21.15.29:0/106396875 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe2a9a7640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe24103ff0 0x7fbe2419b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe241025d0 0x7fbe24199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe299a5640 1 -- 172.21.15.29:0/106396875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe241025d0 msgr2=0x7fbe24199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe299a5640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe241025d0 0x7fbe24199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe299a5640 1 -- 172.21.15.29:0/106396875 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe24103ff0 msgr2=0x7fbe2419b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe299a5640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe24103ff0 0x7fbe2419b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe299a5640 1 -- 172.21.15.29:0/106396875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2419b8a0 con 0x7fbe24103650 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe241025d0 0x7fbe24199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe299a5640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe24103650 0x7fbe2419a4d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400e9f0 tx=0x7fbe1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/106396875 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1400cd80 con 0x7fbe24103650 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/106396875 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe140108c0 con 0x7fbe24103650 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/106396875 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe140187c0 con 0x7fbe24103650 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe2c40a640 1 -- 172.21.15.29:0/106396875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2419bb90 con 0x7fbe24103650 2024-09-21T02:41:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.840+0000 7fbe2c40a640 1 -- 172.21.15.29:0/106396875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe241a2280 con 0x7fbe24103650 2024-09-21T02:41:42.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.843+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/106396875 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fbe14010430 con 0x7fbe24103650 2024-09-21T02:41:42.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.843+0000 7fbe33453740 1 Processor -- start 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.843+0000 7fbe33453740 1 -- start start 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe33453740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd481b0 0x562a3c0f4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe33453740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c0f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe33453740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd274d0 0x562a3c0f5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe33453740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562a3c0626f0 con 0x562a3bd1f2f0 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe33453740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562a3c051b20 con 0x562a3bd481b0 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe33453740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562a3c055930 con 0x562a3bd274d0 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a1a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c0f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a1a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c0f48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51896/0 (socket says 172.21.15.29:51896) 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a1a6640 1 -- 172.21.15.29:0/205461400 learned_addr learned my addr 172.21.15.29:0/205461400 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe299a5640 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd274d0 0x562a3c0f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a9a7640 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd481b0 0x562a3c0f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe1b7fe640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe100b6590 0x7fbe100b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a1a6640 1 -- 172.21.15.29:0/205461400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd274d0 msgr2=0x562a3c0f5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd274d0 0x562a3c0f5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a1a6640 1 -- 172.21.15.29:0/205461400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd481b0 msgr2=0x562a3c0f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd481b0 0x562a3c0f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.844+0000 7fbe2a1a6640 1 -- 172.21.15.29:0/205461400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3c017530 con 0x562a3bd1f2f0 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.845+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/106396875 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbe140e7a00 con 0x7fbe24103650 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.845+0000 7fbe2a9a7640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe100b6590 0x7fbe100b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.845+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c0f48d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbe20006010 tx=0x7fbe20031580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0146eb1fa902952 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.845+0000 7fbe18bfe640 1 -- 172.21.15.29:0/205461400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20035030 con 0x562a3bd1f2f0 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.845+0000 7fbe18bfe640 1 -- 172.21.15.29:0/205461400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe20004030 con 0x562a3bd1f2f0 2024-09-21T02:41:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.845+0000 7fbe18bfe640 1 -- 172.21.15.29:0/205461400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20003a20 con 0x562a3bd1f2f0 2024-09-21T02:41:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 -- 172.21.15.29:0/205461400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 msgr2=0x562a3c0f48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c0f48d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbe20006010 tx=0x7fbe20031580 comp rx=0 tx=0).stop 2024-09-21T02:41:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe2a9a7640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe100b6590 0x7fbe100b8aa0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009fd0 tx=0x7fbe1c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 -- 172.21.15.29:0/205461400 shutdown_connections 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd274d0 0x562a3c0f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c0f48d0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 --2- 172.21.15.29:0/205461400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd481b0 0x562a3c0f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 -- 172.21.15.29:0/205461400 >> 172.21.15.29:0/205461400 conn(0x562a3bd3c300 msgr2=0x562a3c0eb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 -- 172.21.15.29:0/205461400 shutdown_connections 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 -- 172.21.15.29:0/205461400 wait complete. 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.846+0000 7fbe33453740 1 -- start start 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe33453740 1 Processor -- start 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe33453740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 0x562a3c0fd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe33453740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0fdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe33453740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c0feb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe33453740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562a3bf89ab0 con 0x562a3bd274d0 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe33453740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562a3bff7730 con 0x562a3bd1f2f0 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe33453740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562a3c0f5ee0 con 0x562a3bd481b0 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe33453740 10 client.0 _ll_register_callbacks cb 0x562a3c04b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe299a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0fdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe299a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0fdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51904/0 (socket says 172.21.15.29:51904) 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe299a5640 1 -- 172.21.15.29:0/1122647359 learned_addr learned my addr 172.21.15.29:0/1122647359 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 0x562a3c0fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe2a9a7640 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c0feb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe299a5640 1 -- 172.21.15.29:0/1122647359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 msgr2=0x562a3c0feb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe299a5640 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c0feb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe299a5640 1 -- 172.21.15.29:0/1122647359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 msgr2=0x562a3c0fd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe299a5640 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 0x562a3c0fd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.847+0000 7fbe299a5640 1 -- 172.21.15.29:0/1122647359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3bdb3790 con 0x562a3bd274d0 2024-09-21T02:41:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe299a5640 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0fdff0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbe140ad510 tx=0x7fbe140eb0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782aadd6934cc702 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe0dffb640 1 -- 172.21.15.29:0/1122647359 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe140ad160 con 0x562a3bd274d0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe0dffb640 1 -- 172.21.15.29:0/1122647359 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbe140a9360 con 0x562a3bd274d0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe33453740 1 -- 172.21.15.29:0/1122647359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe20004030 con 0x562a3bd274d0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe0dffb640 1 -- 172.21.15.29:0/1122647359 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbe14010720 con 0x562a3bd274d0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe33453740 10 client.15945 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe33453740 1 -- 172.21.15.29:0/1122647359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562a3be2c620 con 0x562a3bd274d0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe0dffb640 1 -- 172.21.15.29:0/1122647359 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe20004030 con 0x562a3bd274d0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe0dffb640 1 -- 172.21.15.29:0/1122647359 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fbe140b3480 con 0x562a3bd274d0 2024-09-21T02:41:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe33453740 10 client.15945 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe33453740 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.848+0000 7fbe33453740 20 client.15945 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.849+0000 7fbe33453740 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562a3bd06d80 0x562a3c10b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.849+0000 7fbe33453740 4 client.15945 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.849+0000 7fbe33453740 1 -- 172.21.15.29:0/1122647359 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a3c064640 con 0x562a3bd06d80 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.849+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562a3bd06d80 0x562a3c10b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.849+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562a3bd06d80 0x562a3c10b0a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6d0317e418480b8 server_cookie=8d71e75c2c535139 in_seq=0 out_seq=0 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.849+0000 7fbe0dffb640 10 client.15945 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe0dffb640 1 -- 172.21.15.29:0/1122647359 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562a3c064640 con 0x562a3bd06d80 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe0dffb640 10 client.15945 handle_command_reply: tid=0 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 client.15945 shutdown 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 -- 172.21.15.29:0/1122647359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 msgr2=0x562a3c0fdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0fdff0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbe140ad510 tx=0x7fbe140eb0d0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 -- 172.21.15.29:0/1122647359 shutdown_connections 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c0feb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562a3bd06d80 0x562a3c10b0a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0fdff0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe0dffb640 0 client.15945 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 --2- 172.21.15.29:0/1122647359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 0x562a3c0fd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 -- 172.21.15.29:0/1122647359 >> 172.21.15.29:0/1122647359 conn(0x562a3bd3c300 msgr2=0x562a3c0e9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 -- 172.21.15.29:0/1122647359 shutdown_connections 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 1 -- 172.21.15.29:0/1122647359 wait complete. 2024-09-21T02:41:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.850+0000 7fbe33453740 20 client.15945 trim_cache size 0 max 16384 2024-09-21T02:41:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.865+0000 7fbe33453740 1 Processor -- start 2024-09-21T02:41:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.865+0000 7fbe33453740 1 -- start start 2024-09-21T02:41:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.865+0000 7fbe33453740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 0x562a3c0eb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.865+0000 7fbe33453740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0f4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.865+0000 7fbe33453740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c0f4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.865+0000 7fbe33453740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241aa360 con 0x562a3bd274d0 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.865+0000 7fbe33453740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241abed0 con 0x562a3bd1f2f0 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.865+0000 7fbe33453740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241aaa30 con 0x562a3bd481b0 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0f4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51918/0 (socket says 172.21.15.29:51918) 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 -- 172.21.15.29:0/256325552 learned_addr learned my addr 172.21.15.29:0/256325552 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe2a9a7640 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c0f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 -- 172.21.15.29:0/256325552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 msgr2=0x562a3c0f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 0x562a3c0eb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c0f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 -- 172.21.15.29:0/256325552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 msgr2=0x562a3c0eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 0x562a3c0eb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 -- 172.21.15.29:0/256325552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3c0f6020 con 0x562a3bd274d0 2024-09-21T02:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe299a5640 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0f4200 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400e9c0 tx=0x7fbe14010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7525d0d536cd5ea0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe0ffff640 1 -- 172.21.15.29:0/256325552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe140acf80 con 0x562a3bd274d0 2024-09-21T02:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe0ffff640 1 -- 172.21.15.29:0/256325552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe140b20d0 con 0x562a3bd274d0 2024-09-21T02:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.866+0000 7fbe0ffff640 1 -- 172.21.15.29:0/256325552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe140a93c0 con 0x562a3bd274d0 2024-09-21T02:41:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 -- 172.21.15.29:0/256325552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 msgr2=0x562a3c0f4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0f4200 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400e9c0 tx=0x7fbe14010cc0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 -- 172.21.15.29:0/256325552 shutdown_connections 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c0f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd274d0 0x562a3c0f4200 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 --2- 172.21.15.29:0/256325552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd1f2f0 0x562a3c0eb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 -- 172.21.15.29:0/256325552 >> 172.21.15.29:0/256325552 conn(0x562a3bd3c300 msgr2=0x562a3c0e65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 -- 172.21.15.29:0/256325552 shutdown_connections 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 -- 172.21.15.29:0/256325552 wait complete. 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 -- start start 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.867+0000 7fbe33453740 1 Processor -- start 2024-09-21T02:41:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe33453740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c1054b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe33453740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd274d0 0x562a3c104170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe33453740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c1046b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe33453740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241ab7d0 con 0x562a3bd1f2f0 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe33453740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241ab100 con 0x562a3bd274d0 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe33453740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241a8340 con 0x562a3bd481b0 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe33453740 10 client.0 _ll_register_callbacks cb 0x7fbe140eb1d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c1054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c1054b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51920/0 (socket says 172.21.15.29:51920) 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 -- 172.21.15.29:0/3709359752 learned_addr learned my addr 172.21.15.29:0/3709359752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe299a5640 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd274d0 0x562a3c104170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a9a7640 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c1046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 -- 172.21.15.29:0/3709359752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 msgr2=0x562a3c1046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c1046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 -- 172.21.15.29:0/3709359752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd274d0 msgr2=0x562a3c104170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd274d0 0x562a3c104170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 -- 172.21.15.29:0/3709359752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3c104bf0 con 0x562a3bd1f2f0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe2a1a6640 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c1054b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbe20005760 tx=0x7fbe20005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a261db41037e578b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.868+0000 7fbe0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe0dffb640 1 -- 172.21.15.29:0/3709359752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2005b060 con 0x562a3bd1f2f0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe0dffb640 1 -- 172.21.15.29:0/3709359752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbe20031890 con 0x562a3bd1f2f0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 1 -- 172.21.15.29:0/3709359752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562a3c0600f0 con 0x562a3bd1f2f0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe0dffb640 1 -- 172.21.15.29:0/3709359752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbe20062050 con 0x562a3bd1f2f0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 10 client.15951 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 1 -- 172.21.15.29:0/3709359752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562a3c103a80 con 0x562a3bd1f2f0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe0dffb640 1 -- 172.21.15.29:0/3709359752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x562a3c0600f0 con 0x562a3bd1f2f0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe0dffb640 1 -- 172.21.15.29:0/3709359752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fbe2003a040 con 0x562a3bd1f2f0 2024-09-21T02:41:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 10 client.15951 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 10 client.15951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 20 client.15951 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562a3bcdc1c0 0x562a3c177d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 4 client.15951 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:41:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.869+0000 7fbe33453740 1 -- 172.21.15.29:0/3709359752 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a3c05d520 con 0x562a3bcdc1c0 2024-09-21T02:41:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.870+0000 7fbe299a5640 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562a3bcdc1c0 0x562a3c177d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.870+0000 7fbe0dffb640 10 client.15951 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.870+0000 7fbe299a5640 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562a3bcdc1c0 0x562a3c177d70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c437f6565b3e909e server_cookie=cd262f4ed7427cc2 in_seq=0 out_seq=0 2024-09-21T02:41:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.898+0000 7fbe0dffb640 1 -- 172.21.15.29:0/3709359752 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a3c05d520 con 0x562a3bcdc1c0 2024-09-21T02:41:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.898+0000 7fbe0dffb640 10 client.15951 handle_command_reply: tid=0 2024-09-21T02:41:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.898+0000 7fbe33453740 1 client.15951 shutdown 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 -- 172.21.15.29:0/3709359752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 msgr2=0x562a3c1054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c1054b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbe20005760 tx=0x7fbe20005de0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 -- 172.21.15.29:0/3709359752 shutdown_connections 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a3bd481b0 0x562a3c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562a3bd274d0 0x562a3c104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562a3bcdc1c0 0x562a3c177d70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 --2- 172.21.15.29:0/3709359752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a3bd1f2f0 0x562a3c1054b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe0dffb640 0 client.15951 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 -- 172.21.15.29:0/3709359752 >> 172.21.15.29:0/3709359752 conn(0x562a3bd3c300 msgr2=0x562a3c0fba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 -- 172.21.15.29:0/3709359752 shutdown_connections 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 1 -- 172.21.15.29:0/3709359752 wait complete. 2024-09-21T02:41:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe33453740 20 client.15951 trim_cache size 0 max 16384 2024-09-21T02:41:42.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:41:42.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:41:42.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:41:42.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.899+0000 7fbe2c40a640 1 -- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe100b6590 msgr2=0x7fbe100b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe100b6590 0x7fbe100b8aa0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009fd0 tx=0x7fbe1c004a40 comp rx=0 tx=0).stop 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 -- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe24103650 msgr2=0x7fbe2419a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe24103650 0x7fbe2419a4d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400e9f0 tx=0x7fbe1400eeb0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 -- 172.21.15.29:0/106396875 shutdown_connections 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe100b6590 0x7fbe100b8aa0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe24103ff0 0x7fbe2419b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe24103650 0x7fbe2419a4d0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 --2- 172.21.15.29:0/106396875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe241025d0 0x7fbe24199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 -- 172.21.15.29:0/106396875 >> 172.21.15.29:0/106396875 conn(0x7fbe240fdf30 msgr2=0x7fbe2406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 -- 172.21.15.29:0/106396875 shutdown_connections 2024-09-21T02:41:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:42.900+0000 7fbe2c40a640 1 -- 172.21.15.29:0/106396875 wait complete. 2024-09-21T02:41:43.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.12 MB/sec warmup 80 sec latency 11949.253 ms 2024-09-21T02:41:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:43 smithi190 ceph-mon[33871]: pgmap v467: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 487 KiB/s wr, 63 op/s 2024-09-21T02:41:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:43 smithi029 ceph-mon[31006]: pgmap v467: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 487 KiB/s wr, 63 op/s 2024-09-21T02:41:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:43 smithi066 ceph-mon[33436]: pgmap v467: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 487 KiB/s wr, 63 op/s 2024-09-21T02:41:44.068 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.09 MB/sec warmup 81 sec latency 12949.323 ms 2024-09-21T02:41:45.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.07 MB/sec warmup 82 sec latency 13949.441 ms 2024-09-21T02:41:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:45 smithi190 ceph-mon[33871]: pgmap v468: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 305 KiB/s wr, 37 op/s 2024-09-21T02:41:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:45 smithi029 ceph-mon[31006]: pgmap v468: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 305 KiB/s wr, 37 op/s 2024-09-21T02:41:46.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:45 smithi066 ceph-mon[33436]: pgmap v468: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 305 KiB/s wr, 37 op/s 2024-09-21T02:41:46.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.04 MB/sec warmup 83 sec latency 14949.528 ms 2024-09-21T02:41:47.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 2.02 MB/sec warmup 84 sec latency 15949.650 ms 2024-09-21T02:41:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:47 smithi190 ceph-mon[33871]: pgmap v469: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 161 KiB/s wr, 22 op/s 2024-09-21T02:41:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:47 smithi029 ceph-mon[31006]: pgmap v469: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 161 KiB/s wr, 22 op/s 2024-09-21T02:41:48.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:47 smithi066 ceph-mon[33436]: pgmap v469: 129 pgs: 129 active+clean; 27 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 161 KiB/s wr, 22 op/s 2024-09-21T02:41:48.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 1.99 MB/sec warmup 85 sec latency 16949.773 ms 2024-09-21T02:41:49.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 1.97 MB/sec warmup 86 sec latency 17949.874 ms 2024-09-21T02:41:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:49 smithi190 ceph-mon[33871]: pgmap v470: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 KiB/s wr, 22 op/s 2024-09-21T02:41:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:49 smithi029 ceph-mon[31006]: pgmap v470: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 KiB/s wr, 22 op/s 2024-09-21T02:41:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:49 smithi066 ceph-mon[33436]: pgmap v470: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 KiB/s wr, 22 op/s 2024-09-21T02:41:50.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 1.95 MB/sec warmup 87 sec latency 18950.004 ms 2024-09-21T02:41:51.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 1.92 MB/sec warmup 88 sec latency 19950.133 ms 2024-09-21T02:41:51.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:41:51.495 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:41:51.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:51 smithi190 ceph-mon[33871]: pgmap v471: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.7 KiB/s wr, 21 op/s 2024-09-21T02:41:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:51 smithi029 ceph-mon[31006]: pgmap v471: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.7 KiB/s wr, 21 op/s 2024-09-21T02:41:52.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:51 smithi066 ceph-mon[33436]: pgmap v471: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.7 KiB/s wr, 21 op/s 2024-09-21T02:41:52.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 -- 172.21.15.29:0/3461248938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720108cf0 msgr2=0x7f572010f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:52.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 --2- 172.21.15.29:0/3461248938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720108cf0 0x7f572010f520 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5718009ca0 tx=0x7f571802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 -- 172.21.15.29:0/3461248938 shutdown_connections 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 --2- 172.21.15.29:0/3461248938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720108cf0 0x7f572010f520 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 --2- 172.21.15.29:0/3461248938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57201083a0 0x7f57201087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 --2- 172.21.15.29:0/3461248938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720102a60 0x7f5720102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 -- 172.21.15.29:0/3461248938 >> 172.21.15.29:0/3461248938 conn(0x7f5720079730 msgr2=0x7f5720079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 -- 172.21.15.29:0/3461248938 shutdown_connections 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.016+0000 7f572659f640 1 -- 172.21.15.29:0/3461248938 wait complete. 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f572659f640 1 Processor -- start 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f572659f640 1 -- start start 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f572659f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720102a60 0x7f572019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f572659f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201083a0 0x7f572019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f572659f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5720108cf0 0x7f57201a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f572659f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5720112860 con 0x7f57201083a0 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f572659f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f57201126e0 con 0x7f5720108cf0 2024-09-21T02:41:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f572659f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f57201129e0 con 0x7f5720102a60 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f5717fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201083a0 0x7f572019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f5717fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201083a0 0x7f572019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53444/0 (socket says 172.21.15.29:53444) 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f5717fff640 1 -- 172.21.15.29:0/1209023918 learned_addr learned my addr 172.21.15.29:0/1209023918 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f571ffff640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720102a60 0x7f572019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.017+0000 7f5724b3c640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5720108cf0 0x7f57201a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f5717fff640 1 -- 172.21.15.29:0/1209023918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720102a60 msgr2=0x7f572019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f5717fff640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720102a60 0x7f572019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f5717fff640 1 -- 172.21.15.29:0/1209023918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5720108cf0 msgr2=0x7f57201a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f5717fff640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5720108cf0 0x7f57201a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f5717fff640 1 -- 172.21.15.29:0/1209023918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57201a2470 con 0x7f57201083a0 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f5717fff640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201083a0 0x7f572019caa0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f570800d770 tx=0x7f570800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f571dffb640 1 -- 172.21.15.29:0/1209023918 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708017070 con 0x7f57201083a0 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f571dffb640 1 -- 172.21.15.29:0/1209023918 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5708009c30 con 0x7f57201083a0 2024-09-21T02:41:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57201a2760 con 0x7f57201083a0 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f571dffb640 1 -- 172.21.15.29:0/1209023918 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570800f620 con 0x7f57201083a0 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.018+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57201a8ec0 con 0x7f57201083a0 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.020+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5720102e70 con 0x7f57201083a0 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.021+0000 7f571dffb640 1 -- 172.21.15.29:0/1209023918 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f570801a070 con 0x7f57201083a0 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.022+0000 7f571dffb640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f570c0b6590 0x7f570c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.022+0000 7f571ffff640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f570c0b6590 0x7f570c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.022+0000 7f571dffb640 1 -- 172.21.15.29:0/1209023918 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f57080e7080 con 0x7f57201083a0 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.022+0000 7f571ffff640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f570c0b6590 0x7f570c0b8aa0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f57100058c0 tx=0x7f571000a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.026+0000 7f571dffb640 1 -- 172.21.15.29:0/1209023918 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5708009de0 con 0x7f57201083a0 2024-09-21T02:41:52.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 1.90 MB/sec warmup 89 sec latency 20950.189 ms 2024-09-21T02:41:52.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.251+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57201a91a0 con 0x7f57201083a0 2024-09-21T02:41:52.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.251+0000 7f571dffb640 1 -- 172.21.15.29:0/1209023918 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11972 (secure 0 0 0) 0x7f57080b1910 con 0x7f57201083a0 2024-09-21T02:41:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:41:52.253 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"btime":"2024-09-21T02:40:26:480768+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:40:26.304013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[3],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:41:52.253 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 30 2024-09-21T02:41:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f570c0b6590 msgr2=0x7f570c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f570c0b6590 0x7f570c0b8aa0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f57100058c0 tx=0x7f571000a250 comp rx=0 tx=0).stop 2024-09-21T02:41:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201083a0 msgr2=0x7f572019caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201083a0 0x7f572019caa0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f570800d770 tx=0x7f570800dc30 comp rx=0 tx=0).stop 2024-09-21T02:41:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 shutdown_connections 2024-09-21T02:41:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f570c0b6590 0x7f570c0b8aa0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5720108cf0 0x7f57201a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201083a0 0x7f572019caa0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 --2- 172.21.15.29:0/1209023918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5720102a60 0x7f572019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:52.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 >> 172.21.15.29:0/1209023918 conn(0x7f5720079730 msgr2=0x7f5720104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:52.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 shutdown_connections 2024-09-21T02:41:52.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:52.254+0000 7f572659f640 1 -- 172.21.15.29:0/1209023918 wait complete. 2024-09-21T02:41:52.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:52 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1209023918' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:52.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:52 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1209023918' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:53.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:52 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1209023918' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:53.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 1.88 MB/sec warmup 90 sec latency 21950.278 ms 2024-09-21T02:41:53.193 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 6b96cd99 --release --await' 2024-09-21T02:41:53.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 6b96cd99 --release --await 2024-09-21T02:41:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:41:53.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:53 smithi190 ceph-mon[33871]: pgmap v472: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.7 KiB/s wr, 19 op/s 2024-09-21T02:41:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:53 smithi029 ceph-mon[31006]: pgmap v472: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.7 KiB/s wr, 19 op/s 2024-09-21T02:41:53.911 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:53 smithi066 ceph-mon[33436]: pgmap v472: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.7 KiB/s wr, 19 op/s 2024-09-21T02:41:54.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.032+0000 7f5368569640 1 -- 172.21.15.29:0/2498659470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5360105300 msgr2=0x7f536010d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.032+0000 7f5368569640 1 --2- 172.21.15.29:0/2498659470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5360105300 0x7f536010d850 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5350009900 tx=0x7f535002a410 comp rx=0 tx=0).stop 2024-09-21T02:41:54.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 -- 172.21.15.29:0/2498659470 shutdown_connections 2024-09-21T02:41:54.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 --2- 172.21.15.29:0/2498659470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f536010de10 0x7f5360110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 --2- 172.21.15.29:0/2498659470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5360105300 0x7f536010d850 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 --2- 172.21.15.29:0/2498659470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53601049b0 0x7f5360104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 -- 172.21.15.29:0/2498659470 >> 172.21.15.29:0/2498659470 conn(0x7f5360079730 msgr2=0x7f5360079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 -- 172.21.15.29:0/2498659470 shutdown_connections 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 -- 172.21.15.29:0/2498659470 wait complete. 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 Processor -- start 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.033+0000 7f5368569640 1 -- start start 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5368569640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53601049b0 0x7f536019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5368569640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5360105300 0x7f536019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5368569640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f536010de10 0x7f536019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5368569640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5360113920 con 0x7f53601049b0 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5368569640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f53601137a0 con 0x7f536010de10 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5368569640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5360113aa0 con 0x7f5360105300 2024-09-21T02:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53601049b0 0x7f536019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53601049b0 0x7f536019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53464/0 (socket says 172.21.15.29:53464) 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 -- 172.21.15.29:0/3855454297 learned_addr learned my addr 172.21.15.29:0/3855454297 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366b06640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f536010de10 0x7f536019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5365b04640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5360105300 0x7f536019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 -- 172.21.15.29:0/3855454297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5360105300 msgr2=0x7f536019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5360105300 0x7f536019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 -- 172.21.15.29:0/3855454297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f536010de10 msgr2=0x7f536019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f536010de10 0x7f536019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 -- 172.21.15.29:0/3855454297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53601a0340 con 0x7f53601049b0 2024-09-21T02:41:54.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.034+0000 7f5366305640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53601049b0 0x7f536019a3d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f535c00d950 tx=0x7f535c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:54.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.035+0000 7f53577fe640 1 -- 172.21.15.29:0/3855454297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c017070 con 0x7f53601049b0 2024-09-21T02:41:54.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.035+0000 7f53577fe640 1 -- 172.21.15.29:0/3855454297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f535c009c30 con 0x7f53601049b0 2024-09-21T02:41:54.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.035+0000 7f53577fe640 1 -- 172.21.15.29:0/3855454297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c00f620 con 0x7f53601049b0 2024-09-21T02:41:54.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.035+0000 7f5368569640 1 -- 172.21.15.29:0/3855454297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53601a0630 con 0x7f53601049b0 2024-09-21T02:41:54.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.035+0000 7f5368569640 1 -- 172.21.15.29:0/3855454297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53601a6d90 con 0x7f53601049b0 2024-09-21T02:41:54.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.038+0000 7f536eadd740 1 Processor -- start 2024-09-21T02:41:54.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.038+0000 7f536eadd740 1 -- start start 2024-09-21T02:41:54.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f53577fe640 1 -- 172.21.15.29:0/3855454297 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f535c01a070 con 0x7f53601049b0 2024-09-21T02:41:54.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f536eadd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd40510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f536eadd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd40a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f536eadd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd40f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f536eadd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556dbdc2bdc0 con 0x556dbd979840 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f536eadd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556dbdcb81f0 con 0x556dbd971660 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f536eadd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556dbdca7620 con 0x556dbd99a520 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5365b04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd40f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5365b04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd40f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53466/0 (socket says 172.21.15.29:53466) 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5365b04640 1 -- 172.21.15.29:0/1940133578 learned_addr learned my addr 172.21.15.29:0/1940133578 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5366305640 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd40a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5366b06640 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd40510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5365b04640 1 -- 172.21.15.29:0/1940133578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 msgr2=0x556dbdd40510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5365b04640 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd40510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5365b04640 1 -- 172.21.15.29:0/1940133578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 msgr2=0x556dbdd40a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5365b04640 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd40a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5365b04640 1 -- 172.21.15.29:0/1940133578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbdc6ffd0 con 0x556dbd979840 2024-09-21T02:41:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.039+0000 7f5366305640 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd40a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:54.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.040+0000 7f5365b04640 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd40f90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f53500057c0 tx=0x7f5350031ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7ab2ae98ec73e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:54.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.040+0000 7f5354ff9640 1 -- 172.21.15.29:0/1940133578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350040070 con 0x556dbd979840 2024-09-21T02:41:54.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.040+0000 7f5354ff9640 1 -- 172.21.15.29:0/1940133578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5350031e50 con 0x556dbd979840 2024-09-21T02:41:54.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.040+0000 7f5354ff9640 1 -- 172.21.15.29:0/1940133578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350035040 con 0x556dbd979840 2024-09-21T02:41:54.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.040+0000 7f53577fe640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f534c0b6590 0x7f534c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.040+0000 7f5366b06640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f534c0b6590 0x7f534c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.040+0000 7f53577fe640 1 -- 172.21.15.29:0/3855454297 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f535c0e6fa0 con 0x7f53601049b0 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 -- 172.21.15.29:0/1940133578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 msgr2=0x556dbdd40f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd40f90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f53500057c0 tx=0x7f5350031ba0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f5366b06640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f534c0b6590 0x7f534c0b8aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5358005fd0 tx=0x7f5358009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 -- 172.21.15.29:0/1940133578 shutdown_connections 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd40f90 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd40a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 --2- 172.21.15.29:0/1940133578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd40510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 -- 172.21.15.29:0/1940133578 >> 172.21.15.29:0/1940133578 conn(0x556dbd98e910 msgr2=0x556dbdd3e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 -- 172.21.15.29:0/1940133578 shutdown_connections 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 -- 172.21.15.29:0/1940133578 wait complete. 2024-09-21T02:41:54.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.041+0000 7f536eadd740 1 -- start start 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f536eadd740 1 Processor -- start 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f536eadd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd4bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f536eadd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f536eadd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd4c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f536eadd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556dbdc42300 con 0x556dbd979840 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f536eadd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556dbdcab430 con 0x556dbd971660 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f536eadd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556dbdb08bb0 con 0x556dbd99a520 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f536eadd740 10 client.0 _ll_register_callbacks cb 0x556dbdc5c120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5365b04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5365b04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53468/0 (socket says 172.21.15.29:53468) 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5365b04640 1 -- 172.21.15.29:0/3710031659 learned_addr learned my addr 172.21.15.29:0/3710031659 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5366305640 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd4bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5366b06640 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd4c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5365b04640 1 -- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 msgr2=0x556dbdd4c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5365b04640 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd4c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5365b04640 1 -- 172.21.15.29:0/3710031659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 msgr2=0x556dbdd4bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5365b04640 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd4bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.042+0000 7f5365b04640 1 -- 172.21.15.29:0/3710031659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbdc31780 con 0x556dbd979840 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f5366b06640 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd4c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f5365b04640 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4c010 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5350003630 tx=0x7f5350031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a4fcb5d105972b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f5349ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f5349ffb640 1 -- 172.21.15.29:0/3710031659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350040070 con 0x556dbd979840 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f5349ffb640 1 -- 172.21.15.29:0/3710031659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f535003c850 con 0x556dbd979840 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f536eadd740 1 -- 172.21.15.29:0/3710031659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5350031e50 con 0x556dbd979840 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f5349ffb640 1 -- 172.21.15.29:0/3710031659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5350056050 con 0x556dbd979840 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f536eadd740 10 client.15969 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:54.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.043+0000 7f536eadd740 1 -- 172.21.15.29:0/3710031659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556dbda05b00 con 0x556dbd979840 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f5349ffb640 1 -- 172.21.15.29:0/3710031659 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5350031e50 con 0x556dbd979840 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f5349ffb640 1 -- 172.21.15.29:0/3710031659 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f5350044890 con 0x556dbd979840 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f536eadd740 10 client.15969 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f536eadd740 10 client.15969 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f536eadd740 20 client.15969 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f536eadd740 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556dbd958ea0 0x556dbdd5dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f536eadd740 4 client.15969 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f536eadd740 1 -- 172.21.15.29:0/3710031659 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dbdcba140 con 0x556dbd958ea0 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.044+0000 7f5366305640 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556dbd958ea0 0x556dbdd5dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.045+0000 7f5349ffb640 10 client.15969 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:54.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.045+0000 7f5366305640 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556dbd958ea0 0x556dbdd5dc70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aba152932a020a1 server_cookie=fd39934b29080c35 in_seq=0 out_seq=0 2024-09-21T02:41:54.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.045+0000 7f5349ffb640 1 -- 172.21.15.29:0/3710031659 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556dbdcba140 con 0x556dbd958ea0 2024-09-21T02:41:54.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.045+0000 7f5349ffb640 10 client.15969 handle_command_reply: tid=0 2024-09-21T02:41:54.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.045+0000 7f536eadd740 1 client.15969 shutdown 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 -- 172.21.15.29:0/3710031659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 msgr2=0x556dbdd4c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4c010 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5350003630 tx=0x7f5350031630 comp rx=0 tx=0).stop 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 -- 172.21.15.29:0/3710031659 shutdown_connections 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbdd4c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556dbd958ea0 0x556dbdd5dc70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4c010 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 --2- 172.21.15.29:0/3710031659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd971660 0x556dbdd4bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 -- 172.21.15.29:0/3710031659 >> 172.21.15.29:0/3710031659 conn(0x556dbd98e910 msgr2=0x556dbdd436a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f5349ffb640 0 client.15969 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 -- 172.21.15.29:0/3710031659 shutdown_connections 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 1 -- 172.21.15.29:0/3710031659 wait complete. 2024-09-21T02:41:54.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.046+0000 7f536eadd740 20 client.15969 trim_cache size 0 max 16384 2024-09-21T02:41:54.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f536eadd740 1 Processor -- start 2024-09-21T02:41:54.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f536eadd740 1 -- start start 2024-09-21T02:41:54.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f536eadd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd971660 0x556dbdd41bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f536eadd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f536eadd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd99a520 0x556dbdd574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f536eadd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53601b0940 con 0x556dbd979840 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f536eadd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f53601b24b0 con 0x556dbd99a520 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f536eadd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53601b1010 con 0x556dbd971660 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f5365b04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f5365b04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53474/0 (socket says 172.21.15.29:53474) 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.063+0000 7f5365b04640 1 -- 172.21.15.29:0/1162342720 learned_addr learned my addr 172.21.15.29:0/1162342720 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5366305640 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd971660 0x556dbdd41bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5366b06640 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd99a520 0x556dbdd574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5365b04640 1 -- 172.21.15.29:0/1162342720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd971660 msgr2=0x556dbdd41bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5365b04640 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd971660 0x556dbdd41bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5365b04640 1 -- 172.21.15.29:0/1162342720 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd99a520 msgr2=0x556dbdd574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5365b04640 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd99a520 0x556dbdd574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5365b04640 1 -- 172.21.15.29:0/1162342720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbda7c960 con 0x556dbd979840 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5366b06640 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd99a520 0x556dbdd574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f5365b04640 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4d9b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5350044c60 tx=0x7f5350031ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9423da792b5e7d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f534bfff640 1 -- 172.21.15.29:0/1162342720 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535005e020 con 0x556dbd979840 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f534bfff640 1 -- 172.21.15.29:0/1162342720 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f535004d070 con 0x556dbd979840 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.064+0000 7f534bfff640 1 -- 172.21.15.29:0/1162342720 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350040070 con 0x556dbd979840 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 -- 172.21.15.29:0/1162342720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 msgr2=0x556dbdd4d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4d9b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5350044c60 tx=0x7f5350031ba0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 -- 172.21.15.29:0/1162342720 shutdown_connections 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd99a520 0x556dbdd574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd979840 0x556dbdd4d9b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 --2- 172.21.15.29:0/1162342720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd971660 0x556dbdd41bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 -- 172.21.15.29:0/1162342720 >> 172.21.15.29:0/1162342720 conn(0x556dbd98e910 msgr2=0x556dbdd56d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 -- 172.21.15.29:0/1162342720 shutdown_connections 2024-09-21T02:41:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.065+0000 7f536eadd740 1 -- 172.21.15.29:0/1162342720 wait complete. 2024-09-21T02:41:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 1 -- start start 2024-09-21T02:41:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 1 Processor -- start 2024-09-21T02:41:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd971660 0x556dbddbc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd979840 0x556dbddbc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbddbce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53601b1db0 con 0x556dbd971660 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f53601b16e0 con 0x556dbd979840 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53601ae920 con 0x556dbd99a520 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f536eadd740 10 client.0 _ll_register_callbacks cb 0x7f535c0ea950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366305640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd971660 0x556dbddbc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366305640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd971660 0x556dbddbc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53488/0 (socket says 172.21.15.29:53488) 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366305640 1 -- 172.21.15.29:0/4011879094 learned_addr learned my addr 172.21.15.29:0/4011879094 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5365b04640 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd979840 0x556dbddbc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366b06640 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbddbce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366305640 1 -- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 msgr2=0x556dbddbce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366305640 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbddbce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366305640 1 -- 172.21.15.29:0/4011879094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd979840 msgr2=0x556dbddbc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366305640 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd979840 0x556dbddbc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366305640 1 -- 172.21.15.29:0/4011879094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbddc7400 con 0x556dbd971660 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.066+0000 7f5366b06640 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbddbce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f5366305640 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd971660 0x556dbddbc380 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f535c0e4720 tx=0x7f535c0e71a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ce68a2ad36629c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f5349ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f5349ffb640 1 -- 172.21.15.29:0/4011879094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c021040 con 0x556dbd971660 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f5349ffb640 1 -- 172.21.15.29:0/4011879094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f535c0e4be0 con 0x556dbd971660 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f536eadd740 1 -- 172.21.15.29:0/4011879094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f535004d070 con 0x556dbd971660 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f5349ffb640 1 -- 172.21.15.29:0/4011879094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f535c0e92b0 con 0x556dbd971660 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f536eadd740 10 client.15975 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f536eadd740 1 -- 172.21.15.29:0/4011879094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556dbddc8590 con 0x556dbd971660 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f5349ffb640 1 -- 172.21.15.29:0/4011879094 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f535004d070 con 0x556dbd971660 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.067+0000 7f5349ffb640 1 -- 172.21.15.29:0/4011879094 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f535c0d41a0 con 0x556dbd971660 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f536eadd740 10 client.15975 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f536eadd740 10 client.15975 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:41:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f536eadd740 20 client.15975 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f536eadd740 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556dbd92e250 0x556dbddcad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f536eadd740 4 client.15975 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6b96cd99", "release": true, "await": true}] 2024-09-21T02:41:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f536eadd740 1 -- 172.21.15.29:0/4011879094 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6b96cd99", "release": true, "await": true}) -- 0x556dbdcb5bf0 con 0x556dbd92e250 2024-09-21T02:41:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f5365b04640 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556dbd92e250 0x556dbddcad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f5365b04640 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556dbd92e250 0x556dbddcad30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8f827edf42e2650 server_cookie=305d86a5d6f1164 in_seq=0 out_seq=0 2024-09-21T02:41:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.068+0000 7f5349ffb640 10 client.15975 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:54.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28249 1.86 MB/sec warmup 91 sec latency 22950.325 ms 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f5349ffb640 1 -- 172.21.15.29:0/4011879094 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x556dbdcb5bf0 con 0x556dbd92e250 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f5349ffb640 10 client.15975 handle_command_reply: tid=0 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 client.15975 shutdown 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 -- 172.21.15.29:0/4011879094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd971660 msgr2=0x556dbddbc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd971660 0x556dbddbc380 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f535c0e4720 tx=0x7f535c0e71a0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 -- 172.21.15.29:0/4011879094 shutdown_connections 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dbd99a520 0x556dbddbce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556dbd979840 0x556dbddbc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556dbd92e250 0x556dbddcad30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 --2- 172.21.15.29:0/4011879094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dbd971660 0x556dbddbc380 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f5349ffb640 0 client.15975 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.085+0000 7f536eadd740 1 -- 172.21.15.29:0/4011879094 >> 172.21.15.29:0/4011879094 conn(0x556dbd98e910 msgr2=0x556dbdd56d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.086+0000 7f536eadd740 1 -- 172.21.15.29:0/4011879094 shutdown_connections 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.086+0000 7f536eadd740 1 -- 172.21.15.29:0/4011879094 wait complete. 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.086+0000 7f536eadd740 20 client.15975 trim_cache size 0 max 16384 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 30, 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 12, 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:41:54.088 INFO:teuthology.orchestra.run.smithi029.stdout: "6b96cd99": { 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 12, 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 147.8, 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:41:54.089 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:54.090 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:41:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.086+0000 7f5368569640 1 -- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f534c0b6590 msgr2=0x7f534c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.086+0000 7f5368569640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f534c0b6590 0x7f534c0b8aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5358005fd0 tx=0x7f5358009600 comp rx=0 tx=0).stop 2024-09-21T02:41:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.086+0000 7f5368569640 1 -- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53601049b0 msgr2=0x7f536019a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.086+0000 7f5368569640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53601049b0 0x7f536019a3d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f535c00d950 tx=0x7f535c00de10 comp rx=0 tx=0).stop 2024-09-21T02:41:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.087+0000 7f5368569640 1 -- 172.21.15.29:0/3855454297 shutdown_connections 2024-09-21T02:41:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.087+0000 7f5368569640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f534c0b6590 0x7f534c0b8aa0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.087+0000 7f5368569640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f536010de10 0x7f536019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.087+0000 7f5368569640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5360105300 0x7f536019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.087+0000 7f5368569640 1 --2- 172.21.15.29:0/3855454297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53601049b0 0x7f536019a3d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.087+0000 7f5368569640 1 -- 172.21.15.29:0/3855454297 >> 172.21.15.29:0/3855454297 conn(0x7f5360079730 msgr2=0x7f5360100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.087+0000 7f5368569640 1 -- 172.21.15.29:0/3855454297 shutdown_connections 2024-09-21T02:41:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:54.087+0000 7f5368569640 1 -- 172.21.15.29:0/3855454297 wait complete. 2024-09-21T02:41:55.057 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6b96cd99', total seconds elapsed: 31.6 2024-09-21T02:41:55.058 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.8 2024-09-21T02:41:55.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:41:55.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 28759 1.89 MB/sec warmup 92 sec latency 22963.316 ms 2024-09-21T02:41:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:41:55.769 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:55 smithi029 ceph-mon[31006]: pgmap v473: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.2 KiB/s wr, 20 op/s 2024-09-21T02:41:55.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:55 smithi190 ceph-mon[33871]: pgmap v473: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.2 KiB/s wr, 20 op/s 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 -- 172.21.15.29:0/4158234978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c109e20 msgr2=0x7f043c110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 --2- 172.21.15.29:0/4158234978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c109e20 0x7f043c110650 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0434009ca0 tx=0x7f043402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 -- 172.21.15.29:0/4158234978 shutdown_connections 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 --2- 172.21.15.29:0/4158234978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c109e20 0x7f043c110650 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 --2- 172.21.15.29:0/4158234978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f043c1094d0 0x7f043c1098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 --2- 172.21.15.29:0/4158234978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f043c103b90 0x7f043c103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 -- 172.21.15.29:0/4158234978 >> 172.21.15.29:0/4158234978 conn(0x7f043c0ff5e0 msgr2=0x7f043c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 -- 172.21.15.29:0/4158234978 shutdown_connections 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.862+0000 7f0441117640 1 -- 172.21.15.29:0/4158234978 wait complete. 2024-09-21T02:41:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.863+0000 7f0441117640 1 Processor -- start 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f0441117640 1 -- start start 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f0441117640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c103b90 0x7f043c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f0441117640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f043c1094d0 0x7f043c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f0441117640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f043c109e20 0x7f043c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f0441117640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f043c1139e0 con 0x7f043c103b90 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f0441117640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f043c113860 con 0x7f043c1094d0 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f0441117640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f043c113b60 con 0x7f043c109e20 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f043ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c103b90 0x7f043c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f043ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c103b90 0x7f043c19a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44294/0 (socket says 172.21.15.29:44294) 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f043ad9d640 1 -- 172.21.15.29:0/1025583131 learned_addr learned my addr 172.21.15.29:0/1025583131 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f043a59c640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f043c1094d0 0x7f043c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f043b59e640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f043c109e20 0x7f043c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f043ad9d640 1 -- 172.21.15.29:0/1025583131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f043c109e20 msgr2=0x7f043c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.864+0000 7f043ad9d640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f043c109e20 0x7f043c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f043ad9d640 1 -- 172.21.15.29:0/1025583131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f043c1094d0 msgr2=0x7f043c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f043ad9d640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f043c1094d0 0x7f043c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f043ad9d640 1 -- 172.21.15.29:0/1025583131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043c1a02e0 con 0x7f043c103b90 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f043ad9d640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c103b90 0x7f043c19a370 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f042c00d950 tx=0x7f042c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f0423fff640 1 -- 172.21.15.29:0/1025583131 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c017030 con 0x7f043c103b90 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043c1a05d0 con 0x7f043c103b90 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f0423fff640 1 -- 172.21.15.29:0/1025583131 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f042c004020 con 0x7f043c103b90 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f0423fff640 1 -- 172.21.15.29:0/1025583131 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c0055d0 con 0x7f043c103b90 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.865+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043c1a6d30 con 0x7f043c103b90 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.866+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043c103fa0 con 0x7f043c103b90 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.868+0000 7f0423fff640 1 -- 172.21.15.29:0/1025583131 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f042c009a30 con 0x7f043c103b90 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.869+0000 7f0423fff640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f04240b6510 0x7f04240b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.869+0000 7f043a59c640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f04240b6510 0x7f04240b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.869+0000 7f0423fff640 1 -- 172.21.15.29:0/1025583131 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f042c0e7110 con 0x7f043c103b90 2024-09-21T02:41:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.870+0000 7f043a59c640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f04240b6510 0x7f04240b8a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f043c108f90 tx=0x7f0430005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:55.874+0000 7f0423fff640 1 -- 172.21.15.29:0/1025583131 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f042c0ad620 con 0x7f043c103b90 2024-09-21T02:41:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 -- 172.21.15.29:0/2270977778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34141025c0 msgr2=0x7f3414102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2270977778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34141025c0 0x7f3414102a30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3410009860 tx=0x7f341002a340 comp rx=0 tx=0).stop 2024-09-21T02:41:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 -- 172.21.15.29:0/2270977778 shutdown_connections 2024-09-21T02:41:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2270977778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3414103fe0 0x7f341410a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2270977778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3414103640 0x7f3414103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2270977778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34141025c0 0x7f3414102a30 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 -- 172.21.15.29:0/2270977778 >> 172.21.15.29:0/2270977778 conn(0x7f34140fdf00 msgr2=0x7f34141003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 -- 172.21.15.29:0/2270977778 shutdown_connections 2024-09-21T02:41:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.006+0000 7f341bc4d640 1 -- 172.21.15.29:0/2270977778 wait complete. 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.007+0000 7f341bc4d640 1 Processor -- start 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.007+0000 7f341bc4d640 1 -- start start 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.007+0000 7f341bc4d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34141025c0 0x7f341406c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.007+0000 7f341bc4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3414103640 0x7f341419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.007+0000 7f341bc4d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3414103fe0 0x7f34141a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.007+0000 7f341bc4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f341410e0b0 con 0x7f3414103640 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.007+0000 7f341bc4d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f341410df30 con 0x7f34141025c0 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.007+0000 7f341bc4d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f341410e230 con 0x7f3414103fe0 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3414103640 0x7f341419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3414103640 0x7f341419ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44302/0 (socket says 172.21.15.29:44302) 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 -- 172.21.15.29:0/2383317533 learned_addr learned my addr 172.21.15.29:0/2383317533 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f341a1ea640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3414103fe0 0x7f34141a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34199e9640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34141025c0 0x7f341406c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 -- 172.21.15.29:0/2383317533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3414103fe0 msgr2=0x7f34141a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3414103fe0 0x7f34141a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 -- 172.21.15.29:0/2383317533 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34141025c0 msgr2=0x7f341406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34141025c0 0x7f341406c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 -- 172.21.15.29:0/2383317533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34141a1950 con 0x7f3414103640 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f34191e8640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3414103640 0x7f341419ec60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f340400d950 tx=0x7f340400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f340affd640 1 -- 172.21.15.29:0/2383317533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404017070 con 0x7f3414103640 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f340affd640 1 -- 172.21.15.29:0/2383317533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3404009c30 con 0x7f3414103640 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f340affd640 1 -- 172.21.15.29:0/2383317533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340400f620 con 0x7f3414103640 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.008+0000 7f341bc4d640 1 -- 172.21.15.29:0/2383317533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34141a1c40 con 0x7f3414103640 2024-09-21T02:41:56.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.009+0000 7f341bc4d640 1 -- 172.21.15.29:0/2383317533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34141a2190 con 0x7f3414103640 2024-09-21T02:41:56.012 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:55 smithi066 ceph-mon[33436]: pgmap v473: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.2 KiB/s wr, 20 op/s 2024-09-21T02:41:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.012+0000 7f3422bc0740 1 Processor -- start 2024-09-21T02:41:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.012+0000 7f3422bc0740 1 -- start start 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801da8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801da9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f3422bc0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560801d1b6f0 con 0x5608019e04d0 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f3422bc0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560801d0ab20 con 0x560801a011b0 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f3422bc0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560801d0e930 con 0x5608019d82f0 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f341a1ea640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801da8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f341a1ea640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801da8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:47498/0 (socket says 172.21.15.29:47498) 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f341a1ea640 1 -- 172.21.15.29:0/2454236608 learned_addr learned my addr 172.21.15.29:0/2454236608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f34199e9640 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f34191e8640 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801da9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.013+0000 7f340affd640 1 -- 172.21.15.29:0/2383317533 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f3404010460 con 0x7f3414103640 2024-09-21T02:41:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f341a1ea640 1 -- 172.21.15.29:0/2454236608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 msgr2=0x560801da9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f341a1ea640 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f341a1ea640 1 -- 172.21.15.29:0/2454236608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 msgr2=0x560801da9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f341a1ea640 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801da9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f341a1ea640 1 -- 172.21.15.29:0/2454236608 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560801cd0530 con 0x560801a011b0 2024-09-21T02:41:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f341a1ea640 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801da8e10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f340c00e580 tx=0x7f340c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=336005b07cabae84 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f33fffff640 1 -- 172.21.15.29:0/2454236608 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c003970 con 0x560801a011b0 2024-09-21T02:41:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f33fffff640 1 -- 172.21.15.29:0/2454236608 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340c00d830 con 0x560801a011b0 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f33fffff640 1 -- 172.21.15.29:0/2454236608 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c004490 con 0x560801a011b0 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.014+0000 7f340affd640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34000b65c0 0x7f34000b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.015+0000 7f34199e9640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34000b65c0 0x7f34000b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.015+0000 7f340affd640 1 -- 172.21.15.29:0/2383317533 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f34040e7230 con 0x7f3414103640 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.015+0000 7f3422bc0740 1 -- 172.21.15.29:0/2454236608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 msgr2=0x560801da8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.015+0000 7f3422bc0740 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801da8e10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f340c00e580 tx=0x7f340c0095c0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.015+0000 7f34199e9640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34000b65c0 0x7f34000b8ad0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3410005940 tx=0x7f34100058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 -- 172.21.15.29:0/2454236608 shutdown_connections 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801da9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 --2- 172.21.15.29:0/2454236608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801da8e10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 -- 172.21.15.29:0/2454236608 >> 172.21.15.29:0/2454236608 conn(0x5608019f5300 msgr2=0x560801da48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 -- 172.21.15.29:0/2454236608 shutdown_connections 2024-09-21T02:41:56.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 -- 172.21.15.29:0/2454236608 wait complete. 2024-09-21T02:41:56.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 -- start start 2024-09-21T02:41:56.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.016+0000 7f3422bc0740 1 Processor -- start 2024-09-21T02:41:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801dbae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801dbb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801dbc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f3422bc0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560801c42ab0 con 0x5608019e04d0 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f3422bc0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560801cb0730 con 0x560801a011b0 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f3422bc0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560801daaaf0 con 0x5608019d82f0 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f3422bc0740 10 client.0 _ll_register_callbacks cb 0x560801d046b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34191e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801dbb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34191e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801dbb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44316/0 (socket says 172.21.15.29:44316) 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34191e8640 1 -- 172.21.15.29:0/3735619270 learned_addr learned my addr 172.21.15.29:0/3735619270 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34199e9640 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801dbae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f341a1ea640 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801dbc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34191e8640 1 -- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 msgr2=0x560801dbae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34191e8640 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801dbae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34191e8640 1 -- 172.21.15.29:0/3735619270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 msgr2=0x560801dbc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34191e8640 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801dbc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34191e8640 1 -- 172.21.15.29:0/3735619270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560801a6c790 con 0x5608019e04d0 2024-09-21T02:41:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.017+0000 7f34199e9640 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801dbae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:56.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f34191e8640 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801dbb4f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f34040acd90 tx=0x7f34040a96e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7bd3e5d73d57cd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:56.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f33fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:56.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3735619270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404020070 con 0x5608019e04d0 2024-09-21T02:41:56.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3735619270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f34040a89e0 con 0x5608019e04d0 2024-09-21T02:41:56.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f3422bc0740 1 -- 172.21.15.29:0/3735619270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f340c00d830 con 0x5608019e04d0 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3735619270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f34040ac9e0 con 0x5608019e04d0 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f3422bc0740 10 client.15993 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f3422bc0740 1 -- 172.21.15.29:0/3735619270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560801ae5620 con 0x5608019e04d0 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3735619270 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f340c00d830 con 0x5608019e04d0 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3735619270 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f34040a8be0 con 0x5608019e04d0 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.018+0000 7f3422bc0740 10 client.15993 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.019+0000 7f3422bc0740 10 client.15993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.019+0000 7f3422bc0740 20 client.15993 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.019+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608019bfd80 0x560801dc85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.019+0000 7f3422bc0740 4 client.15993 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.019+0000 7f3422bc0740 1 -- 172.21.15.29:0/3735619270 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560801d1d640 con 0x5608019bfd80 2024-09-21T02:41:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.019+0000 7f34199e9640 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608019bfd80 0x560801dc85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.019+0000 7f33fd7fa640 10 client.15993 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.019+0000 7f34199e9640 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608019bfd80 0x560801dc85f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2748f979b2fc2b7 server_cookie=eeaacf18655d2d88 in_seq=0 out_seq=0 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3735619270 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560801d1d640 con 0x5608019bfd80 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f33fd7fa640 10 client.15993 handle_command_reply: tid=0 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 client.15993 shutdown 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 -- 172.21.15.29:0/3735619270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 msgr2=0x560801dbb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801dbb4f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f34040acd90 tx=0x7f34040a96e0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 -- 172.21.15.29:0/3735619270 shutdown_connections 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801dbc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608019bfd80 0x560801dc85f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801dbb4f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3735619270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801dbae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 -- 172.21.15.29:0/3735619270 >> 172.21.15.29:0/3735619270 conn(0x5608019f5300 msgr2=0x560801da2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f33fd7fa640 0 client.15993 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 -- 172.21.15.29:0/3735619270 shutdown_connections 2024-09-21T02:41:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.020+0000 7f3422bc0740 1 -- 172.21.15.29:0/3735619270 wait complete. 2024-09-21T02:41:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.021+0000 7f3422bc0740 20 client.15993 trim_cache size 0 max 16384 2024-09-21T02:41:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f3422bc0740 1 Processor -- start 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f3422bc0740 1 -- start start 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801d9f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801db93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f3422bc0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34141aa390 con 0x5608019e04d0 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f3422bc0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f34141abf00 con 0x560801a011b0 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f3422bc0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34141aaa60 con 0x5608019d82f0 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f34191e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801d9f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f34199e9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f34199e9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44694/0 (socket says 172.21.15.29:44694) 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.037+0000 7f34199e9640 1 -- 172.21.15.29:0/3329182916 learned_addr learned my addr 172.21.15.29:0/3329182916 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f341a1ea640 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801db93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f34199e9640 1 -- 172.21.15.29:0/3329182916 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 msgr2=0x560801db93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f34199e9640 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801db93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f34199e9640 1 -- 172.21.15.29:0/3329182916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 msgr2=0x560801d9f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f34199e9640 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801d9f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f34199e9640 1 -- 172.21.15.29:0/3329182916 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560801dbabc0 con 0x5608019d82f0 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f34191e8640 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801d9f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f34199e9640 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da5d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3410030840 tx=0x7f3410005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17df897a49f7c70d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f33ff7fe640 1 -- 172.21.15.29:0/3329182916 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410037040 con 0x5608019d82f0 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f33ff7fe640 1 -- 172.21.15.29:0/3329182916 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3410045040 con 0x5608019d82f0 2024-09-21T02:41:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.038+0000 7f33ff7fe640 1 -- 172.21.15.29:0/3329182916 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410003d10 con 0x5608019d82f0 2024-09-21T02:41:56.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 -- 172.21.15.29:0/3329182916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 msgr2=0x560801da5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da5d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3410030840 tx=0x7f3410005f90 comp rx=0 tx=0).stop 2024-09-21T02:41:56.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 -- 172.21.15.29:0/3329182916 shutdown_connections 2024-09-21T02:41:56.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560801a011b0 0x560801db93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608019e04d0 0x560801d9f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3329182916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019d82f0 0x560801da5d90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 -- 172.21.15.29:0/3329182916 >> 172.21.15.29:0/3329182916 conn(0x5608019f5300 msgr2=0x560801da9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 -- 172.21.15.29:0/3329182916 shutdown_connections 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.039+0000 7f3422bc0740 1 -- 172.21.15.29:0/3329182916 wait complete. 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 1 -- start start 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 1 Processor -- start 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608019d82f0 0x560801dc0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019e04d0 0x560801dc1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560801a011b0 0x560801dc1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34141ab800 con 0x560801a011b0 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f34141ab130 con 0x5608019d82f0 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34141a8370 con 0x5608019e04d0 2024-09-21T02:41:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f3422bc0740 10 client.0 _ll_register_callbacks cb 0x7f34040ea9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f341a1ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560801a011b0 0x560801dc1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f341a1ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560801a011b0 0x560801dc1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44326/0 (socket says 172.21.15.29:44326) 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f341a1ea640 1 -- 172.21.15.29:0/3438428379 learned_addr learned my addr 172.21.15.29:0/3438428379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f34199e9640 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608019d82f0 0x560801dc0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.040+0000 7f34191e8640 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019e04d0 0x560801dc1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f34199e9640 1 -- 172.21.15.29:0/3438428379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019e04d0 msgr2=0x560801dc1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f34199e9640 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019e04d0 0x560801dc1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f34199e9640 1 -- 172.21.15.29:0/3438428379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560801a011b0 msgr2=0x560801dc1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f34199e9640 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560801a011b0 0x560801dc1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f34199e9640 1 -- 172.21.15.29:0/3438428379 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560801dc21e0 con 0x5608019d82f0 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f341a1ea640 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560801a011b0 0x560801dc1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f34199e9640 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608019d82f0 0x560801dc0ff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f34100095f0 tx=0x7f3410003670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c27eaefe8c2e290f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f33fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3438428379 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410037040 con 0x5608019d82f0 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f3422bc0740 1 -- 172.21.15.29:0/3438428379 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3410045040 con 0x5608019d82f0 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.041+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3438428379 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3410032440 con 0x5608019d82f0 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3438428379 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f341006a050 con 0x5608019d82f0 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f3422bc0740 10 client.25562 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f3422bc0740 1 -- 172.21.15.29:0/3438428379 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560801e32ff0 con 0x5608019d82f0 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3438428379 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3410045040 con 0x5608019d82f0 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3438428379 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f3410065030 con 0x5608019d82f0 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f3422bc0740 10 client.25562 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f3422bc0740 10 client.25562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f3422bc0740 20 client.25562 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608019951c0 0x560801e359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f3422bc0740 4 client.25562 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:41:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f3422bc0740 1 -- 172.21.15.29:0/3438428379 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560801d190f0 con 0x5608019951c0 2024-09-21T02:41:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.042+0000 7f34191e8640 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608019951c0 0x560801e359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.043+0000 7f33fd7fa640 10 client.25562 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.043+0000 7f34191e8640 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608019951c0 0x560801e359e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3052ff92f0c6ce5c server_cookie=14ba4e4353e780ed in_seq=0 out_seq=0 2024-09-21T02:41:56.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 29082 1.87 MB/sec warmup 93 sec latency 30.301 ms 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f33fd7fa640 1 -- 172.21.15.29:0/3438428379 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560801d190f0 con 0x5608019951c0 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f33fd7fa640 10 client.25562 handle_command_reply: tid=0 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 client.25562 shutdown 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 -- 172.21.15.29:0/3438428379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608019d82f0 msgr2=0x560801dc0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608019d82f0 0x560801dc0ff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f34100095f0 tx=0x7f3410003670 comp rx=0 tx=0).stop 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 -- 172.21.15.29:0/3438428379 shutdown_connections 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608019951c0 0x560801e359e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560801a011b0 0x560801dc1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f33fd7fa640 0 client.25562 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608019e04d0 0x560801dc1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 --2- 172.21.15.29:0/3438428379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608019d82f0 0x560801dc0ff0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 -- 172.21.15.29:0/3438428379 >> 172.21.15.29:0/3438428379 conn(0x5608019f5300 msgr2=0x560801da9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 -- 172.21.15.29:0/3438428379 shutdown_connections 2024-09-21T02:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 1 -- 172.21.15.29:0/3438428379 wait complete. 2024-09-21T02:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.078+0000 7f3422bc0740 20 client.25562 trim_cache size 0 max 16384 2024-09-21T02:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:41:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 -- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34000b65c0 msgr2=0x7f34000b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34000b65c0 0x7f34000b8ad0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3410005940 tx=0x7f34100058b0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 -- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3414103640 msgr2=0x7f341419ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3414103640 0x7f341419ec60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f340400d950 tx=0x7f340400de10 comp rx=0 tx=0).stop 2024-09-21T02:41:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 -- 172.21.15.29:0/2383317533 shutdown_connections 2024-09-21T02:41:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34000b65c0 0x7f34000b8ad0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3414103fe0 0x7f34141a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3414103640 0x7f341419ec60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 --2- 172.21.15.29:0/2383317533 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34141025c0 0x7f341406c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 -- 172.21.15.29:0/2383317533 >> 172.21.15.29:0/2383317533 conn(0x7f34140fdf00 msgr2=0x7f3414106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 -- 172.21.15.29:0/2383317533 shutdown_connections 2024-09-21T02:41:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:41:56.079+0000 7f341bc4d640 1 -- 172.21.15.29:0/2383317533 wait complete. 2024-09-21T02:41:56.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.163+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f043c19b4a0 con 0x7f043c103b90 2024-09-21T02:41:56.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.164+0000 7f0423fff640 1 -- 172.21.15.29:0/1025583131 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11972 (secure 0 0 0) 0x7f042c0b1a00 con 0x7f043c103b90 2024-09-21T02:41:56.166 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:41:56.166 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"btime":"2024-09-21T02:40:26:480768+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:40:26.304013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[3],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:41:56.166 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 30 2024-09-21T02:41:56.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f04240b6510 msgr2=0x7f04240b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f04240b6510 0x7f04240b8a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f043c108f90 tx=0x7f0430005f90 comp rx=0 tx=0).stop 2024-09-21T02:41:56.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c103b90 msgr2=0x7f043c19a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c103b90 0x7f043c19a370 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f042c00d950 tx=0x7f042c00de10 comp rx=0 tx=0).stop 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 shutdown_connections 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f04240b6510 0x7f04240b8a20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f043c109e20 0x7f043c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f043c1094d0 0x7f043c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 --2- 172.21.15.29:0/1025583131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c103b90 0x7f043c19a370 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 >> 172.21.15.29:0/1025583131 conn(0x7f043c0ff5e0 msgr2=0x7f043c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 shutdown_connections 2024-09-21T02:41:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:56.166+0000 7f0441117640 1 -- 172.21.15.29:0/1025583131 wait complete. 2024-09-21T02:41:56.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:56 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1025583131' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:56.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:56 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1025583131' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:57.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:56 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1025583131' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:41:57.051 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0' 2024-09-21T02:41:57.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0 2024-09-21T02:41:57.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 29705 1.87 MB/sec warmup 94 sec latency 15.278 ms 2024-09-21T02:41:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:41:57.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:57 smithi029 ceph-mon[31006]: pgmap v474: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 79 KiB/s wr, 14 op/s 2024-09-21T02:41:57.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:57 smithi190 ceph-mon[33871]: pgmap v474: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 79 KiB/s wr, 14 op/s 2024-09-21T02:41:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.920+0000 7f34c71aa640 1 -- 172.21.15.29:0/4191947794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c010bff0 msgr2=0x7f34c010c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.920+0000 7f34c71aa640 1 --2- 172.21.15.29:0/4191947794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c010bff0 0x7f34c010c400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f34b800d960 tx=0x7f34b802c2b0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.920+0000 7f34c71aa640 1 -- 172.21.15.29:0/4191947794 shutdown_connections 2024-09-21T02:41:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.920+0000 7f34c71aa640 1 --2- 172.21.15.29:0/4191947794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c0115720 0x7f34c0117b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.920+0000 7f34c71aa640 1 --2- 172.21.15.29:0/4191947794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c0112dc0 0x7f34c01151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.920+0000 7f34c71aa640 1 --2- 172.21.15.29:0/4191947794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c010bff0 0x7f34c010c400 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f34b800d960 tx=0x7f34b802c2b0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.921+0000 7f34c71aa640 1 -- 172.21.15.29:0/4191947794 >> 172.21.15.29:0/4191947794 conn(0x7f34c00712b0 msgr2=0x7f34c0073e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.921+0000 7f34c71aa640 1 -- 172.21.15.29:0/4191947794 shutdown_connections 2024-09-21T02:41:57.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.923+0000 7f34c71aa640 1 -- 172.21.15.29:0/4191947794 wait complete. 2024-09-21T02:41:57.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.924+0000 7f34c71aa640 1 Processor -- start 2024-09-21T02:41:57.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.924+0000 7f34c71aa640 1 -- start start 2024-09-21T02:41:57.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.924+0000 7f34c71aa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c0112dc0 0x7f34c010cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.924+0000 7f34c71aa640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c0115720 0x7f34c010d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.924+0000 7f34c71aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c010d970 0x7f34c01acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.924+0000 7f34c71aa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0119f70 con 0x7f34c010d970 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.924+0000 7f34c71aa640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0119df0 con 0x7f34c0115720 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.924+0000 7f34c71aa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34c011a0f0 con 0x7f34c0112dc0 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34bffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c0115720 0x7f34c010d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34c5747640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c010d970 0x7f34c01acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34bffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c0115720 0x7f34c010d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:47538/0 (socket says 172.21.15.29:47538) 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34bffff640 1 -- 172.21.15.29:0/951045149 learned_addr learned my addr 172.21.15.29:0/951045149 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34c4f46640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c0112dc0 0x7f34c010cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34bffff640 1 -- 172.21.15.29:0/951045149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c0112dc0 msgr2=0x7f34c010cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34bffff640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c0112dc0 0x7f34c010cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34bffff640 1 -- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c010d970 msgr2=0x7f34c01acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34bffff640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c010d970 0x7f34c01acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.925+0000 7f34bffff640 1 -- 172.21.15.29:0/951045149 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c0110e50 con 0x7f34c0115720 2024-09-21T02:41:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.926+0000 7f34bffff640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c0115720 0x7f34c010d430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f34b000c420 tx=0x7f34b000c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.926+0000 7f34bdffb640 1 -- 172.21.15.29:0/951045149 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b001b070 con 0x7f34c0115720 2024-09-21T02:41:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.926+0000 7f34bdffb640 1 -- 172.21.15.29:0/951045149 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34b000bb20 con 0x7f34c0115720 2024-09-21T02:41:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.926+0000 7f34bdffb640 1 -- 172.21.15.29:0/951045149 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0016b70 con 0x7f34c0115720 2024-09-21T02:41:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.927+0000 7f34c71aa640 1 -- 172.21.15.29:0/951045149 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c01be330 con 0x7f34c0115720 2024-09-21T02:41:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.927+0000 7f34c71aa640 1 -- 172.21.15.29:0/951045149 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c01be7f0 con 0x7f34c0115720 2024-09-21T02:41:57.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.932+0000 7f34cd71e740 1 Processor -- start 2024-09-21T02:41:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.933+0000 7f34bdffb640 1 -- 172.21.15.29:0/951045149 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f34b001e070 con 0x7f34c0115720 2024-09-21T02:41:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.934+0000 7f34cd71e740 1 -- start start 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b113d840 0x5564b14e4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 0x5564b14e5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34cd71e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5564b145ab90 con 0x5564b1114980 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34cd71e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5564b1449fc0 con 0x5564b111cb60 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34cd71e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5564b144ddd0 con 0x5564b113d840 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34c5747640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b113d840 0x5564b14e4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34c5747640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b113d840 0x5564b14e4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44724/0 (socket says 172.21.15.29:44724) 2024-09-21T02:41:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.935+0000 7f34c5747640 1 -- 172.21.15.29:0/2288012925 learned_addr learned my addr 172.21.15.29:0/2288012925 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:57.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34c5747640 1 -- 172.21.15.29:0/2288012925 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 msgr2=0x5564b14e5080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:41:57.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34c5747640 1 --2- 172.21.15.29:0/2288012925 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 0x5564b14e5080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34c5747640 1 -- 172.21.15.29:0/2288012925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 msgr2=0x5564b14e4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:57.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34c5747640 1 --2- 172.21.15.29:0/2288012925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34c5747640 1 -- 172.21.15.29:0/2288012925 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564b13d1060 con 0x5564b113d840 2024-09-21T02:41:57.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34c5747640 1 --2- 172.21.15.29:0/2288012925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b113d840 0x5564b14e4600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f34a800bc70 tx=0x7f34a800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b629bce903be9e65 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:57.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34b6ffd640 1 -- 172.21.15.29:0/2288012925 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8015070 con 0x5564b113d840 2024-09-21T02:41:57.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34b6ffd640 1 -- 172.21.15.29:0/2288012925 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34a801a8f0 con 0x5564b113d840 2024-09-21T02:41:57.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.936+0000 7f34b6ffd640 1 -- 172.21.15.29:0/2288012925 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8011810 con 0x5564b113d840 2024-09-21T02:41:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.937+0000 7f34cd71e740 1 -- 172.21.15.29:0/2288012925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b113d840 msgr2=0x5564b14e4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34cd71e740 1 --2- 172.21.15.29:0/2288012925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b113d840 0x5564b14e4600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f34a800bc70 tx=0x7f34a800f730 comp rx=0 tx=0).stop 2024-09-21T02:41:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34bdffb640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34ac0b6610 0x7f34ac0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34cd71e740 1 -- 172.21.15.29:0/2288012925 shutdown_connections 2024-09-21T02:41:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34cd71e740 1 --2- 172.21.15.29:0/2288012925 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 0x5564b14e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34cd71e740 1 --2- 172.21.15.29:0/2288012925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34cd71e740 1 --2- 172.21.15.29:0/2288012925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b113d840 0x5564b14e4600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34cd71e740 1 -- 172.21.15.29:0/2288012925 >> 172.21.15.29:0/2288012925 conn(0x5564b1131990 msgr2=0x5564b14e24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:57.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34bdffb640 1 -- 172.21.15.29:0/951045149 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f34b00e7680 con 0x7f34c0115720 2024-09-21T02:41:57.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34c4f46640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34ac0b6610 0x7f34ac0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34cd71e740 1 -- 172.21.15.29:0/2288012925 shutdown_connections 2024-09-21T02:41:57.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.938+0000 7f34cd71e740 1 -- 172.21.15.29:0/2288012925 wait complete. 2024-09-21T02:41:57.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.939+0000 7f34c4f46640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34ac0b6610 0x7f34ac0b8b20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f34c0110960 tx=0x7f34b8005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:57.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.939+0000 7f34cd71e740 1 -- start start 2024-09-21T02:41:57.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.939+0000 7f34cd71e740 1 Processor -- start 2024-09-21T02:41:57.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14f0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b111cb60 0x5564b14f0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b113d840 0x5564b14fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34c4f46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34c4f46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14f0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44354/0 (socket says 172.21.15.29:44354) 2024-09-21T02:41:57.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34c4f46640 1 -- 172.21.15.29:0/197113534 learned_addr learned my addr 172.21.15.29:0/197113534 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:57.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34bffff640 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b111cb60 0x5564b14f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34c5747640 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b113d840 0x5564b14fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5564b12ab7a0 con 0x5564b1114980 2024-09-21T02:41:57.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5564b13ef0a0 con 0x5564b113d840 2024-09-21T02:41:57.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.940+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5564b14f2e80 con 0x5564b111cb60 2024-09-21T02:41:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34c4f46640 1 -- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b111cb60 msgr2=0x5564b14f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34c4f46640 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b111cb60 0x5564b14f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34c4f46640 1 -- 172.21.15.29:0/197113534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b113d840 msgr2=0x5564b14fc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34c4f46640 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b113d840 0x5564b14fc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34c4f46640 1 -- 172.21.15.29:0/197113534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564b13dd880 con 0x5564b1114980 2024-09-21T02:41:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34bffff640 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b111cb60 0x5564b14f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34c4f46640 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14f0120 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f34b800d960 tx=0x7f34b802c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58d3fdda2858588 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f349ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f349ffff640 1 -- 172.21.15.29:0/197113534 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b8050060 con 0x5564b1114980 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f349ffff640 1 -- 172.21.15.29:0/197113534 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f34b8004e20 con 0x5564b1114980 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34cd71e740 10 client.0 _ll_register_callbacks cb 0x5564b13cc3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.941+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34a801a8f0 con 0x5564b1114980 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f349ffff640 1 -- 172.21.15.29:0/197113534 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f34b8004250 con 0x5564b1114980 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34cd71e740 10 client.16008 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5564b11b7b90 con 0x5564b1114980 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f349ffff640 1 -- 172.21.15.29:0/197113534 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f34b80035f0 con 0x5564b1114980 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34cd71e740 10 client.16008 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34cd71e740 10 client.16008 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34cd71e740 20 client.16008 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34cd71e740 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5564b10fc1c0 0x5564b1501ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34cd71e740 4 client.16008 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:41:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564b145cae0 con 0x5564b10fc1c0 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.942+0000 7f34bffff640 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5564b10fc1c0 0x5564b1501ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.943+0000 7f349ffff640 10 client.16008 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.943+0000 7f34bffff640 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5564b10fc1c0 0x5564b1501ea0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c19bc1b3ab282f78 server_cookie=c0c780aa5085855f in_seq=0 out_seq=0 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.943+0000 7f349ffff640 1 -- 172.21.15.29:0/197113534 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5564b145cae0 con 0x5564b10fc1c0 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.943+0000 7f349ffff640 10 client.16008 handle_command_reply: tid=0 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.943+0000 7f34cd71e740 1 client.16008 shutdown 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 msgr2=0x5564b14f0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14f0120 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f34b800d960 tx=0x7f34b802c2b0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 shutdown_connections 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b113d840 0x5564b14fc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b111cb60 0x5564b14f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5564b10fc1c0 0x5564b1501ea0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 --2- 172.21.15.29:0/197113534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14f0120 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f349ffff640 0 client.16008 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:57.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 >> 172.21.15.29:0/197113534 conn(0x5564b1131990 msgr2=0x5564b14e7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:57.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 shutdown_connections 2024-09-21T02:41:57.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 1 -- 172.21.15.29:0/197113534 wait complete. 2024-09-21T02:41:57.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.944+0000 7f34cd71e740 20 client.16008 trim_cache size 0 max 16384 2024-09-21T02:41:57.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.965+0000 7f34cd71e740 1 Processor -- start 2024-09-21T02:41:57.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.965+0000 7f34cd71e740 1 -- start start 2024-09-21T02:41:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b10fc1c0 0x5564b14e26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 0x5564b14fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34cd71e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01c8620 con 0x5564b1114980 2024-09-21T02:41:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34cd71e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01ca190 con 0x5564b111cb60 2024-09-21T02:41:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34cd71e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01c8cf0 con 0x5564b10fc1c0 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44358/0 (socket says 172.21.15.29:44358) 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34bffff640 1 -- 172.21.15.29:0/1789798144 learned_addr learned my addr 172.21.15.29:0/1789798144 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34c4f46640 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b10fc1c0 0x5564b14e26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34c5747640 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 0x5564b14fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34bffff640 1 -- 172.21.15.29:0/1789798144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b10fc1c0 msgr2=0x5564b14e26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34bffff640 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b10fc1c0 0x5564b14e26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34bffff640 1 -- 172.21.15.29:0/1789798144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 msgr2=0x5564b14fc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34bffff640 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 0x5564b14fc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.966+0000 7f34bffff640 1 -- 172.21.15.29:0/1789798144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564b14ee420 con 0x5564b1114980 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.967+0000 7f34bffff640 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e7e10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f34b00aa6a0 tx=0x7f34b00a9a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875de5b92db468df server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.967+0000 7f34b67fc640 1 -- 172.21.15.29:0/1789798144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00b6e50 con 0x5564b1114980 2024-09-21T02:41:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.967+0000 7f34b67fc640 1 -- 172.21.15.29:0/1789798144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34b0016410 con 0x5564b1114980 2024-09-21T02:41:57.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.967+0000 7f34b67fc640 1 -- 172.21.15.29:0/1789798144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0094e20 con 0x5564b1114980 2024-09-21T02:41:57.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 -- 172.21.15.29:0/1789798144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 msgr2=0x5564b14e7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:57.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e7e10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f34b00aa6a0 tx=0x7f34b00a9a80 comp rx=0 tx=0).stop 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 -- 172.21.15.29:0/1789798144 shutdown_connections 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b111cb60 0x5564b14fc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b1114980 0x5564b14e7e10 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 --2- 172.21.15.29:0/1789798144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b10fc1c0 0x5564b14e26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 -- 172.21.15.29:0/1789798144 >> 172.21.15.29:0/1789798144 conn(0x5564b1131990 msgr2=0x5564b14e7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 -- 172.21.15.29:0/1789798144 shutdown_connections 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 -- 172.21.15.29:0/1789798144 wait complete. 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.968+0000 7f34cd71e740 1 -- start start 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34cd71e740 1 Processor -- start 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b10fc1c0 0x5564b14f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b1114980 0x5564b14f3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34cd71e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b111cb60 0x5564b14f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34cd71e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01c9a90 con 0x5564b111cb60 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34cd71e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01c93c0 con 0x5564b10fc1c0 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34cd71e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01c6600 con 0x5564b1114980 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34cd71e740 10 client.0 _ll_register_callbacks cb 0x7f34b8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34bffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b1114980 0x5564b14f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34c4f46640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b10fc1c0 0x5564b14f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34bffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b1114980 0x5564b14f3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44762/0 (socket says 172.21.15.29:44762) 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34c4f46640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b10fc1c0 0x5564b14f30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:47574/0 (socket says 172.21.15.29:47574) 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34bffff640 1 -- 172.21.15.29:0/3661758676 learned_addr learned my addr 172.21.15.29:0/3661758676 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.969+0000 7f34c5747640 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b111cb60 0x5564b14f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34bffff640 1 -- 172.21.15.29:0/3661758676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b10fc1c0 msgr2=0x5564b14f30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34bffff640 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b10fc1c0 0x5564b14f30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34bffff640 1 -- 172.21.15.29:0/3661758676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b111cb60 msgr2=0x5564b14f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34bffff640 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b111cb60 0x5564b14f3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34bffff640 1 -- 172.21.15.29:0/3661758676 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564b156c8a0 con 0x5564b1114980 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34c4f46640 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b10fc1c0 0x5564b14f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34bffff640 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b1114980 0x5564b14f3620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f34b00a99b0 tx=0x7f34b00e50c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d50cca45213731f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f349ffff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f349ffff640 1 -- 172.21.15.29:0/3661758676 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0094bc0 con 0x5564b1114980 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34cd71e740 1 -- 172.21.15.29:0/3661758676 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34b0016410 con 0x5564b1114980 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f349ffff640 1 -- 172.21.15.29:0/3661758676 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f34b0094d60 con 0x5564b1114980 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f349ffff640 1 -- 172.21.15.29:0/3661758676 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f34b00a7b30 con 0x5564b1114980 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34cd71e740 10 client.25645 fetch_fsmap learned FSMap version 30 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.970+0000 7f34cd71e740 1 -- 172.21.15.29:0/3661758676 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5564b156da60 con 0x5564b1114980 2024-09-21T02:41:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f349ffff640 1 -- 172.21.15.29:0/3661758676 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34b0016410 con 0x5564b1114980 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f349ffff640 1 -- 172.21.15.29:0/3661758676 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f34b000b040 con 0x5564b1114980 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f34cd71e740 10 client.25645 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f34cd71e740 10 client.25645 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f34cd71e740 20 client.25645 populate_metadata read hostname 'smithi029' 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f34cd71e740 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5564b113d840 0x5564b1570160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f34cd71e740 4 client.25645 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}] 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f34cd71e740 1 -- 172.21.15.29:0/3661758676 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}) -- 0x5564b1458590 con 0x5564b113d840 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.971+0000 7f34c4f46640 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5564b113d840 0x5564b1570160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.972+0000 7f349ffff640 10 client.25645 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:57.972+0000 7f34c4f46640 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5564b113d840 0x5564b1570160 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57340cb5d78e2ea8 server_cookie=2254ea67607f1d49 in_seq=0 out_seq=0 2024-09-21T02:41:58.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:57 smithi066 ceph-mon[33436]: pgmap v474: 129 pgs: 129 active+clean; 27 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 79 KiB/s wr, 14 op/s 2024-09-21T02:41:58.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.058+0000 7f349ffff640 1 -- 172.21.15.29:0/3661758676 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5564b1458590 con 0x5564b113d840 2024-09-21T02:41:58.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.058+0000 7f349ffff640 10 client.25645 handle_command_reply: tid=0 2024-09-21T02:41:58.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.058+0000 7f34cd71e740 1 client.25645 shutdown 2024-09-21T02:41:58.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 -- 172.21.15.29:0/3661758676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b1114980 msgr2=0x5564b14f3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b1114980 0x5564b14f3620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f34b00a99b0 tx=0x7f34b00e50c0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 -- 172.21.15.29:0/3661758676 shutdown_connections 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5564b113d840 0x5564b1570160 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5564b111cb60 0x5564b14f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f349ffff640 0 client.25645 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5564b1114980 0x5564b14f3620 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 --2- 172.21.15.29:0/3661758676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5564b10fc1c0 0x5564b14f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 -- 172.21.15.29:0/3661758676 >> 172.21.15.29:0/3661758676 conn(0x5564b1131990 msgr2=0x5564b14e2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 -- 172.21.15.29:0/3661758676 shutdown_connections 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 1 -- 172.21.15.29:0/3661758676 wait complete. 2024-09-21T02:41:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.059+0000 7f34cd71e740 20 client.25645 trim_cache size 0 max 16384 2024-09-21T02:41:58.063 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:41:58.063 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 30, 2024-09-21T02:41:58.063 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:41:58.063 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 14, 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "50b9a31b": { 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 14, 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 147.8, 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:41:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:41:58.065 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:41:58.065 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:41:58.065 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:58.065 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:58.065 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:58.065 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:58.065 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:41:58.065 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 -- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34ac0b6610 msgr2=0x7f34ac0b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34ac0b6610 0x7f34ac0b8b20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f34c0110960 tx=0x7f34b8005c90 comp rx=0 tx=0).stop 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 -- 172.21.15.29:0/951045149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c0115720 msgr2=0x7f34c010d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c0115720 0x7f34c010d430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f34b000c420 tx=0x7f34b000c8e0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 -- 172.21.15.29:0/951045149 shutdown_connections 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f34ac0b6610 0x7f34ac0b8b20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c010d970 0x7f34c01acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34c0115720 0x7f34c010d430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 --2- 172.21.15.29:0/951045149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c0112dc0 0x7f34c010cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 -- 172.21.15.29:0/951045149 >> 172.21.15.29:0/951045149 conn(0x7f34c00712b0 msgr2=0x7f34c0115ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 -- 172.21.15.29:0/951045149 shutdown_connections 2024-09-21T02:41:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:41:58.060+0000 7f34c71aa640 1 -- 172.21.15.29:0/951045149 wait complete. 2024-09-21T02:41:58.069 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30289 1.89 MB/sec warmup 95 sec latency 87.410 ms 2024-09-21T02:41:59.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.87 MB/sec warmup 96 sec latency 995.815 ms 2024-09-21T02:41:59.072 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '50b9a31b', quiesce took 4.0 seconds. Will release after: 23.8 2024-09-21T02:41:59.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:41:59 smithi190 ceph-mon[33871]: pgmap v475: 129 pgs: 129 active+clean; 28 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 181 KiB/s wr, 36 op/s 2024-09-21T02:41:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:41:59 smithi029 ceph-mon[31006]: pgmap v475: 129 pgs: 129 active+clean; 28 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 181 KiB/s wr, 36 op/s 2024-09-21T02:42:00.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:41:59 smithi066 ceph-mon[33436]: pgmap v475: 129 pgs: 129 active+clean; 28 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 181 KiB/s wr, 36 op/s 2024-09-21T02:42:00.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.85 MB/sec warmup 97 sec latency 1995.852 ms 2024-09-21T02:42:01.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.83 MB/sec warmup 98 sec latency 2995.963 ms 2024-09-21T02:42:01.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:01 smithi190 ceph-mon[33871]: pgmap v476: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 274 KiB/s wr, 50 op/s 2024-09-21T02:42:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:01 smithi029 ceph-mon[31006]: pgmap v476: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 274 KiB/s wr, 50 op/s 2024-09-21T02:42:02.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:01 smithi066 ceph-mon[33436]: pgmap v476: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 274 KiB/s wr, 50 op/s 2024-09-21T02:42:02.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.81 MB/sec warmup 99 sec latency 3996.009 ms 2024-09-21T02:42:03.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.79 MB/sec warmup 100 sec latency 4996.128 ms 2024-09-21T02:42:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:03 smithi029 ceph-mon[31006]: pgmap v477: 129 pgs: 129 active+clean; 27 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 362 KiB/s wr, 64 op/s 2024-09-21T02:42:04.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:03 smithi066 ceph-mon[33436]: pgmap v477: 129 pgs: 129 active+clean; 27 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 362 KiB/s wr, 64 op/s 2024-09-21T02:42:04.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.78 MB/sec warmup 101 sec latency 5996.234 ms 2024-09-21T02:42:04.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:03 smithi190 ceph-mon[33871]: pgmap v477: 129 pgs: 129 active+clean; 27 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 362 KiB/s wr, 64 op/s 2024-09-21T02:42:05.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.76 MB/sec warmup 102 sec latency 6996.292 ms 2024-09-21T02:42:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:05 smithi029 ceph-mon[31006]: pgmap v478: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 363 KiB/s wr, 68 op/s 2024-09-21T02:42:06.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:05 smithi066 ceph-mon[33436]: pgmap v478: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 363 KiB/s wr, 68 op/s 2024-09-21T02:42:06.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.74 MB/sec warmup 103 sec latency 7996.364 ms 2024-09-21T02:42:06.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:05 smithi190 ceph-mon[33871]: pgmap v478: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 363 KiB/s wr, 68 op/s 2024-09-21T02:42:07.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.72 MB/sec warmup 104 sec latency 8996.452 ms 2024-09-21T02:42:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:07 smithi029 ceph-mon[31006]: pgmap v479: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 363 KiB/s wr, 66 op/s 2024-09-21T02:42:08.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:07 smithi066 ceph-mon[33436]: pgmap v479: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 363 KiB/s wr, 66 op/s 2024-09-21T02:42:08.070 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.71 MB/sec warmup 105 sec latency 9996.558 ms 2024-09-21T02:42:08.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:07 smithi190 ceph-mon[33871]: pgmap v479: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 363 KiB/s wr, 66 op/s 2024-09-21T02:42:09.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.69 MB/sec warmup 106 sec latency 10996.659 ms 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 -- 172.21.15.29:0/1812157778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8104760 msgr2=0x7f51d8104b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 --2- 172.21.15.29:0/1812157778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8104760 0x7f51d8104b70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009900 tx=0x7f51c002a410 comp rx=0 tx=0).stop 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 -- 172.21.15.29:0/1812157778 shutdown_connections 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 --2- 172.21.15.29:0/1812157778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51d8105100 0x7f51d810baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 --2- 172.21.15.29:0/1812157778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8104760 0x7f51d8104b70 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 --2- 172.21.15.29:0/1812157778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d81036e0 0x7f51d8103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 -- 172.21.15.29:0/1812157778 >> 172.21.15.29:0/1812157778 conn(0x7f51d80ff020 msgr2=0x7f51d81014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 -- 172.21.15.29:0/1812157778 shutdown_connections 2024-09-21T02:42:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.196+0000 7f51dcb50640 1 -- 172.21.15.29:0/1812157778 wait complete. 2024-09-21T02:42:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.197+0000 7f51dcb50640 1 Processor -- start 2024-09-21T02:42:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.197+0000 7f51dcb50640 1 -- start start 2024-09-21T02:42:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.197+0000 7f51dcb50640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51d81036e0 0x7f51d8199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.197+0000 7f51dcb50640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d8104760 0x7f51d819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.197+0000 7f51dcb50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8105100 0x7f51d819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.197+0000 7f51dcb50640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f51d810f1c0 con 0x7f51d8105100 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.197+0000 7f51dcb50640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f51d810f040 con 0x7f51d81036e0 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.197+0000 7f51dcb50640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f51d810f340 con 0x7f51d8104760 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8105100 0x7f51d819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8105100 0x7f51d819b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40906/0 (socket says 172.21.15.29:40906) 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 -- 172.21.15.29:0/2057976912 learned_addr learned my addr 172.21.15.29:0/2057976912 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51cffff640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d8104760 0x7f51d819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d659c640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51d81036e0 0x7f51d8199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 -- 172.21.15.29:0/2057976912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d8104760 msgr2=0x7f51d819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d8104760 0x7f51d819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 -- 172.21.15.29:0/2057976912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51d81036e0 msgr2=0x7f51d8199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51d81036e0 0x7f51d8199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 -- 172.21.15.29:0/2057976912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d819b8c0 con 0x7f51d8105100 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d659c640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51d81036e0 0x7f51d8199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:42:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51d6d9d640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8105100 0x7f51d819b0a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f51d000eb10 tx=0x7f51d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51cf7fe640 1 -- 172.21.15.29:0/2057976912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d0004b20 con 0x7f51d8105100 2024-09-21T02:42:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51dcb50640 1 -- 172.21.15.29:0/2057976912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d819bbb0 con 0x7f51d8105100 2024-09-21T02:42:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51cf7fe640 1 -- 172.21.15.29:0/2057976912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51d0004cc0 con 0x7f51d8105100 2024-09-21T02:42:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51cf7fe640 1 -- 172.21.15.29:0/2057976912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d0010690 con 0x7f51d8105100 2024-09-21T02:42:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.198+0000 7f51dcb50640 1 -- 172.21.15.29:0/2057976912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d81a22a0 con 0x7f51d8105100 2024-09-21T02:42:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.201+0000 7f51cf7fe640 1 -- 172.21.15.29:0/2057976912 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f51d0005020 con 0x7f51d8105100 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.201+0000 7f51e3abb740 1 Processor -- start 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.201+0000 7f51e3abb740 1 -- start start 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b60260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b61420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51e3abb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dfa0ace6f0 con 0x55dfa07b41b0 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51e3abb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55dfa0abdb20 con 0x55dfa07934d0 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51e3abb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55dfa0ac1930 con 0x55dfa078b2f0 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b60260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b60260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40918/0 (socket says 172.21.15.29:40918) 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 -- 172.21.15.29:0/4021916512 learned_addr learned my addr 172.21.15.29:0/4021916512 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51d659c640 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b61420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cf7fe640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51c40b6510 0x7f51c40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 -- 172.21.15.29:0/4021916512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 msgr2=0x55dfa0b608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b608d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 -- 172.21.15.29:0/4021916512 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 msgr2=0x55dfa0b61420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b61420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.202+0000 7f51cffff640 1 -- 172.21.15.29:0/4021916512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa0a83530 con 0x55dfa07b41b0 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.203+0000 7f51cf7fe640 1 -- 172.21.15.29:0/2057976912 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f51d00e7c60 con 0x7f51d8105100 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.203+0000 7f51d6d9d640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51c40b6510 0x7f51c40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.203+0000 7f51cffff640 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b60260 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f51c0031460 tx=0x7f51c0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32f1433c2f5ec8e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.203+0000 7f51ccff9640 1 -- 172.21.15.29:0/4021916512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0035030 con 0x55dfa07b41b0 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.203+0000 7f51ccff9640 1 -- 172.21.15.29:0/4021916512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51c0004030 con 0x55dfa07b41b0 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.203+0000 7f51ccff9640 1 -- 172.21.15.29:0/4021916512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0003810 con 0x55dfa07b41b0 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 -- 172.21.15.29:0/4021916512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 msgr2=0x55dfa0b60260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b60260 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f51c0031460 tx=0x7f51c0005fd0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51d6d9d640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51c40b6510 0x7f51c40b8a20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f51d000cf70 tx=0x7f51d0009200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 -- 172.21.15.29:0/4021916512 shutdown_connections 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b61420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 --2- 172.21.15.29:0/4021916512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b60260 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 -- 172.21.15.29:0/4021916512 >> 172.21.15.29:0/4021916512 conn(0x55dfa07a8300 msgr2=0x55dfa0b578e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 -- 172.21.15.29:0/4021916512 shutdown_connections 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 -- 172.21.15.29:0/4021916512 wait complete. 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 -- start start 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.204+0000 7f51e3abb740 1 Processor -- start 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa078b2f0 0x55dfa0b69a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b6a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa07b41b0 0x55dfa0b6ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51e3abb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dfa09f5ab0 con 0x55dfa07934d0 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51e3abb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55dfa0a63730 con 0x55dfa078b2f0 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51e3abb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55dfa0b61ee0 con 0x55dfa07b41b0 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51e3abb740 10 client.0 _ll_register_callbacks cb 0x55dfa0ab76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b6a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b6a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40928/0 (socket says 172.21.15.29:40928) 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 -- 172.21.15.29:0/1817719601 learned_addr learned my addr 172.21.15.29:0/1817719601 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51cffff640 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa07b41b0 0x55dfa0b6ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51cffff640 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa078b2f0 0x55dfa0b69a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 -- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa07b41b0 msgr2=0x55dfa0b6ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa07b41b0 0x55dfa0b6ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 -- 172.21.15.29:0/1817719601 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa078b2f0 msgr2=0x55dfa0b69a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa078b2f0 0x55dfa0b69a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 -- 172.21.15.29:0/1817719601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa081f790 con 0x55dfa07934d0 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51cffff640 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa07b41b0 0x55dfa0b6ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51d659c640 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b6a0f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f51c800bad0 tx=0x7f51c800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e248b04c9b40e9b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.205+0000 7f51bdffb640 1 -- 172.21.15.29:0/1817719601 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c801b640 con 0x55dfa07934d0 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51bdffb640 1 -- 172.21.15.29:0/1817719601 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f51c801b7e0 con 0x55dfa07934d0 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 1 -- 172.21.15.29:0/1817719601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c0004030 con 0x55dfa07934d0 2024-09-21T02:42:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51bdffb640 1 -- 172.21.15.29:0/1817719601 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f51c8024050 con 0x55dfa07934d0 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 10 client.16026 fetch_fsmap learned FSMap version 30 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 1 -- 172.21.15.29:0/1817719601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dfa0898620 con 0x55dfa07934d0 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51bdffb640 1 -- 172.21.15.29:0/1817719601 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51c0004030 con 0x55dfa07934d0 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51bdffb640 1 -- 172.21.15.29:0/1817719601 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f51c8019020 con 0x55dfa07934d0 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 10 client.16026 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 10 client.16026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 20 client.16026 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dfa0772d80 0x55dfa0b77240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 4 client.16026 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.206+0000 7f51e3abb740 1 -- 172.21.15.29:0/1817719601 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfa0ad0640 con 0x55dfa0772d80 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.207+0000 7f51cffff640 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dfa0772d80 0x55dfa0b77240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.207+0000 7f51cffff640 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dfa0772d80 0x55dfa0b77240 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4e6bf8edebc5316 server_cookie=11c185b9295ea657 in_seq=0 out_seq=0 2024-09-21T02:42:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.207+0000 7f51bdffb640 10 client.16026 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51bdffb640 1 -- 172.21.15.29:0/1817719601 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dfa0ad0640 con 0x55dfa0772d80 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51bdffb640 10 client.16026 handle_command_reply: tid=0 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 client.16026 shutdown 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 -- 172.21.15.29:0/1817719601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 msgr2=0x55dfa0b6a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b6a0f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f51c800bad0 tx=0x7f51c800bf90 comp rx=0 tx=0).stop 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 -- 172.21.15.29:0/1817719601 shutdown_connections 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa07b41b0 0x55dfa0b6ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dfa0772d80 0x55dfa0b77240 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b6a0f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1817719601 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa078b2f0 0x55dfa0b69a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51bdffb640 0 client.16026 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 -- 172.21.15.29:0/1817719601 >> 172.21.15.29:0/1817719601 conn(0x55dfa07a8300 msgr2=0x55dfa0b55ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 -- 172.21.15.29:0/1817719601 shutdown_connections 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 1 -- 172.21.15.29:0/1817719601 wait complete. 2024-09-21T02:42:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.208+0000 7f51e3abb740 20 client.16026 trim_cache size 0 max 16384 2024-09-21T02:42:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51e3abb740 1 Processor -- start 2024-09-21T02:42:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51e3abb740 1 -- start start 2024-09-21T02:42:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b67f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07b41b0 0x55dfa0b684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51e3abb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81aa330 con 0x55dfa07934d0 2024-09-21T02:42:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51e3abb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81abea0 con 0x55dfa07b41b0 2024-09-21T02:42:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51e3abb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81aaa00 con 0x55dfa078b2f0 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.223+0000 7f51d659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b67f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51d659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b67f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40942/0 (socket says 172.21.15.29:40942) 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51d659c640 1 -- 172.21.15.29:0/419527547 learned_addr learned my addr 172.21.15.29:0/419527547 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51d659c640 1 -- 172.21.15.29:0/419527547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 msgr2=0x55dfa0b610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51d659c640 1 --2- 172.21.15.29:0/419527547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51d659c640 1 -- 172.21.15.29:0/419527547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07b41b0 msgr2=0x55dfa0b684c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51d659c640 1 --2- 172.21.15.29:0/419527547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07b41b0 0x55dfa0b684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51d659c640 1 -- 172.21.15.29:0/419527547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa0b697c0 con 0x55dfa07934d0 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51d659c640 1 --2- 172.21.15.29:0/419527547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b67f80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f51c800b430 tx=0x7f51c8003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca290b9170cc2c6a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51bffff640 1 -- 172.21.15.29:0/419527547 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8011440 con 0x55dfa07934d0 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51bffff640 1 -- 172.21.15.29:0/419527547 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51c80054c0 con 0x55dfa07934d0 2024-09-21T02:42:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.224+0000 7f51bffff640 1 -- 172.21.15.29:0/419527547 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c80048d0 con 0x55dfa07934d0 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 -- 172.21.15.29:0/419527547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 msgr2=0x55dfa0b67f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 --2- 172.21.15.29:0/419527547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b67f80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f51c800b430 tx=0x7f51c8003fe0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 -- 172.21.15.29:0/419527547 shutdown_connections 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 --2- 172.21.15.29:0/419527547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07b41b0 0x55dfa0b684c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 --2- 172.21.15.29:0/419527547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07934d0 0x55dfa0b67f80 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 --2- 172.21.15.29:0/419527547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 -- 172.21.15.29:0/419527547 >> 172.21.15.29:0/419527547 conn(0x55dfa07a8300 msgr2=0x55dfa0b525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 -- 172.21.15.29:0/419527547 shutdown_connections 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 -- 172.21.15.29:0/419527547 wait complete. 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 -- start start 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.225+0000 7f51e3abb740 1 Processor -- start 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b71a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51e3abb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b70130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51e3abb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ab7a0 con 0x55dfa07b41b0 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51e3abb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ab0d0 con 0x55dfa07934d0 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51e3abb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81a8310 con 0x55dfa078b2f0 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51e3abb740 10 client.0 _ll_register_callbacks cb 0x7f51c80040c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b70130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b70130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40954/0 (socket says 172.21.15.29:40954) 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 -- 172.21.15.29:0/1663768677 learned_addr learned my addr 172.21.15.29:0/1663768677 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51d659c640 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b71a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 -- 172.21.15.29:0/1663768677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 msgr2=0x55dfa0b713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b713b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 -- 172.21.15.29:0/1663768677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 msgr2=0x55dfa0b71a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b71a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 -- 172.21.15.29:0/1663768677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa0b70910 con 0x55dfa07b41b0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51d659c640 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b71a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51cffff640 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b70130 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f51c002a930 tx=0x7f51c0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7702be80d1a8c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51bdffb640 1 -- 172.21.15.29:0/1663768677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0035040 con 0x55dfa07b41b0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51bdffb640 1 -- 172.21.15.29:0/1663768677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f51c003b460 con 0x55dfa07b41b0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.226+0000 7f51e3abb740 1 -- 172.21.15.29:0/1663768677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c80054c0 con 0x55dfa07b41b0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51bdffb640 1 -- 172.21.15.29:0/1663768677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f51c0005850 con 0x55dfa07b41b0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51e3abb740 10 client.16032 fetch_fsmap learned FSMap version 30 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51e3abb740 1 -- 172.21.15.29:0/1663768677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dfa0b6f750 con 0x55dfa07b41b0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51bdffb640 1 -- 172.21.15.29:0/1663768677 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51c80054c0 con 0x55dfa07b41b0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51bdffb640 1 -- 172.21.15.29:0/1663768677 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f51c00035f0 con 0x55dfa07b41b0 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51e3abb740 10 client.16032 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51e3abb740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:42:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51e3abb740 20 client.16032 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dfa07481c0 0x55dfa0be3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51e3abb740 4 client.16032 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:42:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51e3abb740 1 -- 172.21.15.29:0/1663768677 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dfa0acc0f0 con 0x55dfa07481c0 2024-09-21T02:42:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.227+0000 7f51d659c640 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dfa07481c0 0x55dfa0be3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.228+0000 7f51d659c640 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dfa07481c0 0x55dfa0be3c80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41ea50eef526baa6 server_cookie=c7e81b298f01740a in_seq=0 out_seq=0 2024-09-21T02:42:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.228+0000 7f51bdffb640 10 client.16032 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.253+0000 7f51bdffb640 1 -- 172.21.15.29:0/1663768677 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dfa0acc0f0 con 0x55dfa07481c0 2024-09-21T02:42:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.253+0000 7f51bdffb640 10 client.16032 handle_command_reply: tid=0 2024-09-21T02:42:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.253+0000 7f51e3abb740 1 client.16032 shutdown 2024-09-21T02:42:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.253+0000 7f51e3abb740 1 -- 172.21.15.29:0/1663768677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 msgr2=0x55dfa0b70130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.253+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b70130 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f51c002a930 tx=0x7f51c0000bc0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 1 -- 172.21.15.29:0/1663768677 shutdown_connections 2024-09-21T02:42:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dfa07481c0 0x55dfa0be3c80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfa07b41b0 0x55dfa0b70130 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51bdffb640 0 client.16032 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dfa07934d0 0x55dfa0b71a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 1 --2- 172.21.15.29:0/1663768677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfa078b2f0 0x55dfa0b713b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 1 -- 172.21.15.29:0/1663768677 >> 172.21.15.29:0/1663768677 conn(0x55dfa07a8300 msgr2=0x55dfa0b60200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 1 -- 172.21.15.29:0/1663768677 shutdown_connections 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 1 -- 172.21.15.29:0/1663768677 wait complete. 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51e3abb740 20 client.16032 trim_cache size 0 max 16384 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:42:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 -- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51c40b6510 msgr2=0x7f51c40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51c40b6510 0x7f51c40b8a20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f51d000cf70 tx=0x7f51d0009200 comp rx=0 tx=0).stop 2024-09-21T02:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 -- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8105100 msgr2=0x7f51d819b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8105100 0x7f51d819b0a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f51d000eb10 tx=0x7f51d000c2d0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 -- 172.21.15.29:0/2057976912 shutdown_connections 2024-09-21T02:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51c40b6510 0x7f51c40b8a20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51d8105100 0x7f51d819b0a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d8104760 0x7f51d819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.254+0000 7f51dcb50640 1 --2- 172.21.15.29:0/2057976912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51d81036e0 0x7f51d8199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.255+0000 7f51dcb50640 1 -- 172.21.15.29:0/2057976912 >> 172.21.15.29:0/2057976912 conn(0x7f51d80ff020 msgr2=0x7f51d8107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.255+0000 7f51dcb50640 1 -- 172.21.15.29:0/2057976912 shutdown_connections 2024-09-21T02:42:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:09.255+0000 7f51dcb50640 1 -- 172.21.15.29:0/2057976912 wait complete. 2024-09-21T02:42:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:09 smithi066 ceph-mon[33436]: pgmap v480: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 292 KiB/s wr, 68 op/s 2024-09-21T02:42:10.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.68 MB/sec warmup 107 sec latency 11996.780 ms 2024-09-21T02:42:10.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:09 smithi190 ceph-mon[33871]: pgmap v480: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 292 KiB/s wr, 68 op/s 2024-09-21T02:42:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:09 smithi029 ceph-mon[31006]: pgmap v480: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 292 KiB/s wr, 68 op/s 2024-09-21T02:42:11.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.66 MB/sec warmup 108 sec latency 12996.870 ms 2024-09-21T02:42:12.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:11 smithi066 ceph-mon[33436]: pgmap v481: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 189 KiB/s wr, 45 op/s 2024-09-21T02:42:12.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.65 MB/sec warmup 109 sec latency 13997.007 ms 2024-09-21T02:42:12.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:11 smithi190 ceph-mon[33871]: pgmap v481: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 189 KiB/s wr, 45 op/s 2024-09-21T02:42:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:11 smithi029 ceph-mon[31006]: pgmap v481: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 189 KiB/s wr, 45 op/s 2024-09-21T02:42:13.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.63 MB/sec warmup 110 sec latency 14997.114 ms 2024-09-21T02:42:14.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:13 smithi066 ceph-mon[33436]: pgmap v482: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 32 op/s 2024-09-21T02:42:14.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.62 MB/sec warmup 111 sec latency 15997.211 ms 2024-09-21T02:42:14.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:13 smithi190 ceph-mon[33871]: pgmap v482: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 32 op/s 2024-09-21T02:42:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:13 smithi029 ceph-mon[31006]: pgmap v482: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 32 op/s 2024-09-21T02:42:15.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.60 MB/sec warmup 112 sec latency 16997.279 ms 2024-09-21T02:42:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:15 smithi029 ceph-mon[31006]: pgmap v483: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-09-21T02:42:16.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:15 smithi066 ceph-mon[33436]: pgmap v483: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-09-21T02:42:16.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.59 MB/sec warmup 113 sec latency 17997.352 ms 2024-09-21T02:42:16.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:15 smithi190 ceph-mon[33871]: pgmap v483: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-09-21T02:42:17.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.57 MB/sec warmup 114 sec latency 18997.441 ms 2024-09-21T02:42:18.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:17 smithi066 ceph-mon[33436]: pgmap v484: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-09-21T02:42:18.071 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.56 MB/sec warmup 115 sec latency 19997.546 ms 2024-09-21T02:42:18.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:17 smithi190 ceph-mon[33871]: pgmap v484: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-09-21T02:42:18.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:17 smithi029 ceph-mon[31006]: pgmap v484: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-09-21T02:42:19.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.55 MB/sec warmup 116 sec latency 20997.656 ms 2024-09-21T02:42:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:19 smithi066 ceph-mon[33436]: pgmap v485: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-09-21T02:42:20.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.53 MB/sec warmup 117 sec latency 21997.762 ms 2024-09-21T02:42:20.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:19 smithi190 ceph-mon[33871]: pgmap v485: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-09-21T02:42:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:19 smithi029 ceph-mon[31006]: pgmap v485: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 19 op/s 2024-09-21T02:42:21.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.52 MB/sec warmup 118 sec latency 22997.890 ms 2024-09-21T02:42:21.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:21 smithi190 ceph-mon[33871]: pgmap v486: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 KiB/s wr, 19 op/s 2024-09-21T02:42:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:21 smithi066 ceph-mon[33436]: pgmap v486: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 KiB/s wr, 19 op/s 2024-09-21T02:42:22.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 1.51 MB/sec warmup 119 sec latency 23997.966 ms 2024-09-21T02:42:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:21 smithi029 ceph-mon[31006]: pgmap v486: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 KiB/s wr, 19 op/s 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 -- 172.21.15.29:0/1072950563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5081025c0 msgr2=0x7fc508102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/1072950563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5081025c0 0x7fc508102a30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009900 tx=0x7fc4f802a410 comp rx=0 tx=0).stop 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 -- 172.21.15.29:0/1072950563 shutdown_connections 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/1072950563 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc508103fe0 0x7fc50810a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/1072950563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc508103640 0x7fc508103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/1072950563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5081025c0 0x7fc508102a30 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 -- 172.21.15.29:0/1072950563 >> 172.21.15.29:0/1072950563 conn(0x7fc5080fdf00 msgr2=0x7fc5081003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 -- 172.21.15.29:0/1072950563 shutdown_connections 2024-09-21T02:42:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.368+0000 7fc50ca1f640 1 -- 172.21.15.29:0/1072950563 wait complete. 2024-09-21T02:42:22.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.369+0000 7fc50ca1f640 1 Processor -- start 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.369+0000 7fc50ca1f640 1 -- start start 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.369+0000 7fc50ca1f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5081025c0 0x7fc50806c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.369+0000 7fc50ca1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc508103640 0x7fc50819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.369+0000 7fc50ca1f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc508103fe0 0x7fc5081a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.369+0000 7fc50ca1f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc50810e0b0 con 0x7fc508103640 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.369+0000 7fc50ca1f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc50810df30 con 0x7fc508103fe0 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.369+0000 7fc50ca1f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc50810e230 con 0x7fc5081025c0 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc508103640 0x7fc50819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc508103640 0x7fc50819ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37502/0 (socket says 172.21.15.29:37502) 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 -- 172.21.15.29:0/68908697 learned_addr learned my addr 172.21.15.29:0/68908697 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc506d9d640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc508103fe0 0x7fc5081a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc50659c640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5081025c0 0x7fc50806c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 -- 172.21.15.29:0/68908697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5081025c0 msgr2=0x7fc50806c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5081025c0 0x7fc50806c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 -- 172.21.15.29:0/68908697 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc508103fe0 msgr2=0x7fc5081a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc508103fe0 0x7fc5081a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 -- 172.21.15.29:0/68908697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5081a1930 con 0x7fc508103640 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc50659c640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5081025c0 0x7fc50806c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc505d9b640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc508103640 0x7fc50819ebe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00d770 tx=0x7fc4fc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc4f77fe640 1 -- 172.21.15.29:0/68908697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc017070 con 0x7fc508103640 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc4f77fe640 1 -- 172.21.15.29:0/68908697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4fc0054b0 con 0x7fc508103640 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc4f77fe640 1 -- 172.21.15.29:0/68908697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc004c60 con 0x7fc508103640 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.370+0000 7fc50ca1f640 1 -- 172.21.15.29:0/68908697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5081a1c20 con 0x7fc508103640 2024-09-21T02:42:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.371+0000 7fc50ca1f640 1 -- 172.21.15.29:0/68908697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5081a2170 con 0x7fc508103640 2024-09-21T02:42:22.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.373+0000 7fc513a65740 1 Processor -- start 2024-09-21T02:42:22.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc513a65740 1 -- start start 2024-09-21T02:42:22.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc4f77fe640 1 -- 172.21.15.29:0/68908697 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc4fc004e00 con 0x7fc508103640 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc513a65740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6d61b0 0x558efaa7de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc513a65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 0x558efaa7e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc513a65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa7f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc513a65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558efa9f06f0 con 0x558efa6b54d0 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc513a65740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x558efa9dfb20 con 0x558efa6d61b0 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc513a65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558efa9e3930 con 0x558efa6ad2f0 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc505d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa7f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc505d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa7f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37508/0 (socket says 172.21.15.29:37508) 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc50659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 0x558efaa7e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc505d9b640 1 -- 172.21.15.29:0/869034232 learned_addr learned my addr 172.21.15.29:0/869034232 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.374+0000 7fc506d9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6d61b0 0x558efaa7de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc505d9b640 1 -- 172.21.15.29:0/869034232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 msgr2=0x558efaa7e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc505d9b640 1 --2- 172.21.15.29:0/869034232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 0x558efaa7e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc505d9b640 1 -- 172.21.15.29:0/869034232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6d61b0 msgr2=0x558efaa7de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc505d9b640 1 --2- 172.21.15.29:0/869034232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6d61b0 0x558efaa7de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc505d9b640 1 -- 172.21.15.29:0/869034232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558efa9a5530 con 0x558efa6b54d0 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc506d9d640 1 --2- 172.21.15.29:0/869034232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6d61b0 0x558efaa7de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc4f77fe640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4f00b6590 0x7fc4f00b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc505d9b640 1 --2- 172.21.15.29:0/869034232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa7f010 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0ef2a0 tx=0x7fc4fc0ef760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c583f0899c494cd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc506d9d640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4f00b6590 0x7fc4f00b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc4f77fe640 1 -- 172.21.15.29:0/68908697 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc4fc0e70b0 con 0x7fc508103640 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc4f4ff9640 1 -- 172.21.15.29:0/869034232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0a9510 con 0x558efa6b54d0 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc4f4ff9640 1 -- 172.21.15.29:0/869034232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4fc0ac9d0 con 0x558efa6b54d0 2024-09-21T02:42:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.375+0000 7fc4f4ff9640 1 -- 172.21.15.29:0/869034232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0ac630 con 0x558efa6b54d0 2024-09-21T02:42:22.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 -- 172.21.15.29:0/869034232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 msgr2=0x558efaa7f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc506d9d640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4f00b6590 0x7fc4f00b8aa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc500009fd0 tx=0x7fc500009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 --2- 172.21.15.29:0/869034232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa7f010 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0ef2a0 tx=0x7fc4fc0ef760 comp rx=0 tx=0).stop 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 -- 172.21.15.29:0/869034232 shutdown_connections 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 --2- 172.21.15.29:0/869034232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa7f010 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 --2- 172.21.15.29:0/869034232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 0x558efaa7e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 --2- 172.21.15.29:0/869034232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6d61b0 0x558efaa7de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 -- 172.21.15.29:0/869034232 >> 172.21.15.29:0/869034232 conn(0x558efa6ca300 msgr2=0x558efaa798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 -- 172.21.15.29:0/869034232 shutdown_connections 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.376+0000 7fc513a65740 1 -- 172.21.15.29:0/869034232 wait complete. 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 1 -- start start 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 1 Processor -- start 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 0x558efaa87620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa87ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6d61b0 0x558efaa887e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558efa917ab0 con 0x558efa6d61b0 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x558efa985730 con 0x558efa6b54d0 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558efaa7fad0 con 0x558efa6ad2f0 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc513a65740 10 client.0 _ll_register_callbacks cb 0x558efa9d96b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc506d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6d61b0 0x558efaa887e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc506d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6d61b0 0x558efaa887e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37520/0 (socket says 172.21.15.29:37520) 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc506d9d640 1 -- 172.21.15.29:0/2915558262 learned_addr learned my addr 172.21.15.29:0/2915558262 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc50659c640 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 0x558efaa87620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.377+0000 7fc505d9b640 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa87ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc506d9d640 1 -- 172.21.15.29:0/2915558262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 msgr2=0x558efaa87620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc506d9d640 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 0x558efaa87620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc506d9d640 1 -- 172.21.15.29:0/2915558262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 msgr2=0x558efaa87ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc506d9d640 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa87ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc506d9d640 1 -- 172.21.15.29:0/2915558262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558efa741790 con 0x558efa6d61b0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc505d9b640 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa87ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc506d9d640 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6d61b0 0x558efaa887e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc500011e30 tx=0x7fc500004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441e73a445fba5e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc4edffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc4edffb640 1 -- 172.21.15.29:0/2915558262 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50001f9b0 con 0x558efa6d61b0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc4edffb640 1 -- 172.21.15.29:0/2915558262 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc50001fb50 con 0x558efa6d61b0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc513a65740 1 -- 172.21.15.29:0/2915558262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4fc0ac9d0 con 0x558efa6d61b0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc4edffb640 1 -- 172.21.15.29:0/2915558262 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc50001fd50 con 0x558efa6d61b0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc513a65740 10 client.16044 fetch_fsmap learned FSMap version 30 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc513a65740 1 -- 172.21.15.29:0/2915558262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558efa7ba620 con 0x558efa6d61b0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.378+0000 7fc4edffb640 1 -- 172.21.15.29:0/2915558262 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4fc0ac9d0 con 0x558efa6d61b0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.379+0000 7fc4edffb640 1 -- 172.21.15.29:0/2915558262 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fc500026070 con 0x558efa6d61b0 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.379+0000 7fc513a65740 10 client.16044 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.379+0000 7fc513a65740 10 client.16044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:42:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.379+0000 7fc513a65740 20 client.16044 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.379+0000 7fc513a65740 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558efa694d80 0x558efaa94f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.379+0000 7fc513a65740 4 client.16044 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.379+0000 7fc513a65740 1 -- 172.21.15.29:0/2915558262 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558efa9f2640 con 0x558efa694d80 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.379+0000 7fc50659c640 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558efa694d80 0x558efaa94f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.380+0000 7fc50659c640 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558efa694d80 0x558efaa94f20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2de4aaebc7909cf2 server_cookie=1b4cb89726a33ff3 in_seq=0 out_seq=0 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.380+0000 7fc4edffb640 10 client.16044 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.380+0000 7fc4edffb640 1 -- 172.21.15.29:0/2915558262 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558efa9f2640 con 0x558efa694d80 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.380+0000 7fc4edffb640 10 client.16044 handle_command_reply: tid=0 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.380+0000 7fc513a65740 1 client.16044 shutdown 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 -- 172.21.15.29:0/2915558262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6d61b0 msgr2=0x558efaa887e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6d61b0 0x558efaa887e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc500011e30 tx=0x7fc500004ab0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 -- 172.21.15.29:0/2915558262 shutdown_connections 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558efa694d80 0x558efaa94f20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6d61b0 0x558efaa887e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa87ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc4edffb640 0 client.16044 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 --2- 172.21.15.29:0/2915558262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6ad2f0 0x558efaa87620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 -- 172.21.15.29:0/2915558262 >> 172.21.15.29:0/2915558262 conn(0x558efa6ca300 msgr2=0x558efaa77ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 -- 172.21.15.29:0/2915558262 shutdown_connections 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 1 -- 172.21.15.29:0/2915558262 wait complete. 2024-09-21T02:42:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.381+0000 7fc513a65740 20 client.16044 trim_cache size 0 max 16384 2024-09-21T02:42:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc513a65740 1 Processor -- start 2024-09-21T02:42:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc513a65740 1 -- start start 2024-09-21T02:42:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc513a65740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6ad2f0 0x558efaa94e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc513a65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa79220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc513a65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa86820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc513a65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc5081ac4b0 con 0x558efa6b54d0 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc513a65740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc5081ae020 con 0x558efa6ad2f0 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc513a65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5081acb80 con 0x558efa6d61b0 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc505d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa79220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc505d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa79220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37524/0 (socket says 172.21.15.29:37524) 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc505d9b640 1 -- 172.21.15.29:0/4063164753 learned_addr learned my addr 172.21.15.29:0/4063164753 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc50659c640 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6ad2f0 0x558efaa94e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.396+0000 7fc506d9d640 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa86820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc505d9b640 1 -- 172.21.15.29:0/4063164753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 msgr2=0x558efaa86820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc505d9b640 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa86820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc505d9b640 1 -- 172.21.15.29:0/4063164753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6ad2f0 msgr2=0x558efaa94e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc505d9b640 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6ad2f0 0x558efaa94e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc505d9b640 1 -- 172.21.15.29:0/4063164753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558efaa857f0 con 0x558efa6b54d0 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc505d9b640 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa79220 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00d740 tx=0x7fc4fc0eff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a124cde86a1c8dd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc4effff640 1 -- 172.21.15.29:0/4063164753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc020070 con 0x558efa6b54d0 2024-09-21T02:42:22.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc4effff640 1 -- 172.21.15.29:0/4063164753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4fc0ac9d0 con 0x558efa6b54d0 2024-09-21T02:42:22.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.397+0000 7fc4effff640 1 -- 172.21.15.29:0/4063164753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0ac810 con 0x558efa6b54d0 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- 172.21.15.29:0/4063164753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 msgr2=0x558efaa79220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa79220 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00d740 tx=0x7fc4fc0eff90 comp rx=0 tx=0).stop 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- 172.21.15.29:0/4063164753 shutdown_connections 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa86820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6b54d0 0x558efaa79220 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 --2- 172.21.15.29:0/4063164753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6ad2f0 0x558efaa94e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- 172.21.15.29:0/4063164753 >> 172.21.15.29:0/4063164753 conn(0x558efa6ca300 msgr2=0x558efaa77ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- 172.21.15.29:0/4063164753 shutdown_connections 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- 172.21.15.29:0/4063164753 wait complete. 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- start start 2024-09-21T02:42:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 Processor -- start 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6ad2f0 0x558efaa8c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa8cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa8d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc5081ad920 con 0x558efa6ad2f0 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc5081ad250 con 0x558efa6b54d0 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.398+0000 7fc513a65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5081aa490 con 0x558efa6d61b0 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6ad2f0 0x558efaa8c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc513a65740 10 client.0 _ll_register_callbacks cb 0x7fc500009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6ad2f0 0x558efaa8c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37534/0 (socket says 172.21.15.29:37534) 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 -- 172.21.15.29:0/2121687589 learned_addr learned my addr 172.21.15.29:0/2121687589 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc506d9d640 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa8d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc505d9b640 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa8cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 -- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 msgr2=0x558efaa8d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa8d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 -- 172.21.15.29:0/2121687589 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 msgr2=0x558efaa8cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa8cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 -- 172.21.15.29:0/2121687589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558efaa8df40 con 0x558efa6ad2f0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc506d9d640 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa8d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc50659c640 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6ad2f0 0x558efaa8c5b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8005760 tx=0x7fc4f8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee47b4ab92313f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc4edffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc4edffb640 1 -- 172.21.15.29:0/2121687589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8035040 con 0x558efa6ad2f0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc4edffb640 1 -- 172.21.15.29:0/2121687589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc4f8031940 con 0x558efa6ad2f0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc513a65740 1 -- 172.21.15.29:0/2121687589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4fc0ac9d0 con 0x558efa6ad2f0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc4edffb640 1 -- 172.21.15.29:0/2121687589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc4f8055050 con 0x558efa6ad2f0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc513a65740 10 client.16050 fetch_fsmap learned FSMap version 30 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.399+0000 7fc513a65740 1 -- 172.21.15.29:0/2121687589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558efaa8f330 con 0x558efa6ad2f0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc4edffb640 1 -- 172.21.15.29:0/2121687589 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4fc0ac9d0 con 0x558efa6ad2f0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc4edffb640 1 -- 172.21.15.29:0/2121687589 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fc4f803c440 con 0x558efa6ad2f0 2024-09-21T02:42:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc513a65740 10 client.16050 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:42:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc513a65740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:42:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc513a65740 20 client.16050 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc513a65740 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558efa66a1c0 0x558efab01a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc513a65740 4 client.16050 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:42:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc513a65740 1 -- 172.21.15.29:0/2121687589 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558efa9ee0f0 con 0x558efa66a1c0 2024-09-21T02:42:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.400+0000 7fc505d9b640 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558efa66a1c0 0x558efab01a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.401+0000 7fc4edffb640 10 client.16050 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.401+0000 7fc505d9b640 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558efa66a1c0 0x558efab01a70 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a2745c3a80fe053 server_cookie=f9574fcfc1185406 in_seq=0 out_seq=0 2024-09-21T02:42:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.414+0000 7fc4edffb640 1 -- 172.21.15.29:0/2121687589 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558efa9ee0f0 con 0x558efa66a1c0 2024-09-21T02:42:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.414+0000 7fc4edffb640 10 client.16050 handle_command_reply: tid=0 2024-09-21T02:42:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 client.16050 shutdown 2024-09-21T02:42:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 -- 172.21.15.29:0/2121687589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6ad2f0 msgr2=0x558efaa8c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6ad2f0 0x558efaa8c5b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8005760 tx=0x7fc4f8004040 comp rx=0 tx=0).stop 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 -- 172.21.15.29:0/2121687589 shutdown_connections 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558efa6d61b0 0x558efaa8d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558efa6b54d0 0x558efaa8cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558efa66a1c0 0x558efab01a70 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 --2- 172.21.15.29:0/2121687589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558efa6ad2f0 0x558efaa8c5b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc4edffb640 0 client.16050 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 -- 172.21.15.29:0/2121687589 >> 172.21.15.29:0/2121687589 conn(0x558efa6ca300 msgr2=0x558efaa79220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 -- 172.21.15.29:0/2121687589 shutdown_connections 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 1 -- 172.21.15.29:0/2121687589 wait complete. 2024-09-21T02:42:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.415+0000 7fc513a65740 20 client.16050 trim_cache size 0 max 16384 2024-09-21T02:42:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:42:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:42:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:42:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 -- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4f00b6590 msgr2=0x7fc4f00b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4f00b6590 0x7fc4f00b8aa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc500009fd0 tx=0x7fc500009500 comp rx=0 tx=0).stop 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 -- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc508103640 msgr2=0x7fc50819ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc508103640 0x7fc50819ebe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00d770 tx=0x7fc4fc00dc30 comp rx=0 tx=0).stop 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 -- 172.21.15.29:0/68908697 shutdown_connections 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4f00b6590 0x7fc4f00b8aa0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc508103fe0 0x7fc5081a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc508103640 0x7fc50819ebe0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 --2- 172.21.15.29:0/68908697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5081025c0 0x7fc50806c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 -- 172.21.15.29:0/68908697 >> 172.21.15.29:0/68908697 conn(0x7fc5080fdf00 msgr2=0x7fc508106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 -- 172.21.15.29:0/68908697 shutdown_connections 2024-09-21T02:42:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:22.416+0000 7fc50ca1f640 1 -- 172.21.15.29:0/68908697 wait complete. 2024-09-21T02:42:22.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:42:23.138 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:42:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1534418914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b081083a0 msgr2=0x7f3b081087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1534418914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b081083a0 0x7f3b081087b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009980 tx=0x7f3afc02c620 comp rx=0 tx=0).stop 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1534418914 shutdown_connections 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1534418914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b08108cf0 0x7f3b0810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1534418914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b081083a0 0x7f3b081087b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1534418914 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b08102a60 0x7f3b08102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1534418914 >> 172.21.15.29:0/1534418914 conn(0x7f3b08079730 msgr2=0x7f3b08079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1534418914 shutdown_connections 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.676+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1534418914 wait complete. 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.677+0000 7f3b0ea9e640 1 Processor -- start 2024-09-21T02:42:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.677+0000 7f3b0ea9e640 1 -- start start 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.677+0000 7f3b0ea9e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b08102a60 0x7f3b081982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.677+0000 7f3b0ea9e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b081083a0 0x7f3b08198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.677+0000 7f3b0ea9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b08108cf0 0x7f3b0819db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0ea9e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b08111710 con 0x7f3b08108cf0 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0ea9e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3b08111590 con 0x7f3b08102a60 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0ea9e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3b08111890 con 0x7f3b081083a0 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b08108cf0 0x7f3b0819db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b08108cf0 0x7f3b0819db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37556/0 (socket says 172.21.15.29:37556) 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 -- 172.21.15.29:0/1252980339 learned_addr learned my addr 172.21.15.29:0/1252980339 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b07fff640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b081083a0 0x7f3b08198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0c83a640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b08102a60 0x7f3b081982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 -- 172.21.15.29:0/1252980339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b081083a0 msgr2=0x7f3b08198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b081083a0 0x7f3b08198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 -- 172.21.15.29:0/1252980339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b08102a60 msgr2=0x7f3b081982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b08102a60 0x7f3b081982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 -- 172.21.15.29:0/1252980339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0819e230 con 0x7f3b08108cf0 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0d03b640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b08108cf0 0x7f3b0819db00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000cad0 tx=0x7f3b0000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b05ffb640 1 -- 172.21.15.29:0/1252980339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0000ae20 con 0x7f3b08108cf0 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b05ffb640 1 -- 172.21.15.29:0/1252980339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b00005020 con 0x7f3b08108cf0 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0819e520 con 0x7f3b08108cf0 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.678+0000 7f3b05ffb640 1 -- 172.21.15.29:0/1252980339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0000f6b0 con 0x7f3b08108cf0 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.679+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b081a4c80 con 0x7f3b08108cf0 2024-09-21T02:42:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.680+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b08102e70 con 0x7f3b08108cf0 2024-09-21T02:42:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.681+0000 7f3b05ffb640 1 -- 172.21.15.29:0/1252980339 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f3b000051d0 con 0x7f3b08108cf0 2024-09-21T02:42:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.682+0000 7f3b05ffb640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3af00b6590 0x7f3af00b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.683+0000 7f3b0c83a640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3af00b6590 0x7f3af00b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.683+0000 7f3b05ffb640 1 -- 172.21.15.29:0/1252980339 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3b000e7810 con 0x7f3b08108cf0 2024-09-21T02:42:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.683+0000 7f3b0c83a640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3af00b6590 0x7f3af00b8aa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3af800bbe0 tx=0x7f3af800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.686+0000 7f3b05ffb640 1 -- 172.21.15.29:0/1252980339 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3b00014030 con 0x7f3b08108cf0 2024-09-21T02:42:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:23 smithi029 ceph-mon[31006]: pgmap v487: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-09-21T02:42:23.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.915+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b081993f0 con 0x7f3b08108cf0 2024-09-21T02:42:23.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.915+0000 7f3b05ffb640 1 -- 172.21.15.29:0/1252980339 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11972 (secure 0 0 0) 0x7f3b000b2180 con 0x7f3b08108cf0 2024-09-21T02:42:23.917 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:42:23.917 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"btime":"2024-09-21T02:40:26:480768+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:40:26.304013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[3],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T02:42:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 30 2024-09-21T02:42:23.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3af00b6590 msgr2=0x7f3af00b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:23.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3af00b6590 0x7f3af00b8aa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3af800bbe0 tx=0x7f3af800b390 comp rx=0 tx=0).stop 2024-09-21T02:42:23.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b08108cf0 msgr2=0x7f3b0819db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:23.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b08108cf0 0x7f3b0819db00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000cad0 tx=0x7f3b0000cf90 comp rx=0 tx=0).stop 2024-09-21T02:42:23.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 shutdown_connections 2024-09-21T02:42:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3af00b6590 0x7f3af00b8aa0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b08108cf0 0x7f3b0819db00 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b081083a0 0x7f3b08198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 --2- 172.21.15.29:0/1252980339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b08102a60 0x7f3b081982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 >> 172.21.15.29:0/1252980339 conn(0x7f3b08079730 msgr2=0x7f3b081045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 shutdown_connections 2024-09-21T02:42:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:23.918+0000 7f3b0ea9e640 1 -- 172.21.15.29:0/1252980339 wait complete. 2024-09-21T02:42:24.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:23 smithi066 ceph-mon[33436]: pgmap v487: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-09-21T02:42:24.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 0.00 MB/sec execute 1 sec latency 25998.023 ms 2024-09-21T02:42:24.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:23 smithi190 ceph-mon[33871]: pgmap v487: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-09-21T02:42:24.788 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 50b9a31b --release --await' 2024-09-21T02:42:24.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 50b9a31b --release --await 2024-09-21T02:42:25.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:24 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1252980339' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:42:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:42:25.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30291 0.00 MB/sec execute 2 sec latency 26998.050 ms 2024-09-21T02:42:25.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:24 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1252980339' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:42:25.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:24 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1252980339' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:42:25.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.587+0000 7f276777a640 1 -- 172.21.15.29:0/171057912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760103b90 msgr2=0x7f2760103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.587+0000 7f276777a640 1 --2- 172.21.15.29:0/171057912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760103b90 0x7f2760103fa0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f275c009900 tx=0x7f275c02a410 comp rx=0 tx=0).stop 2024-09-21T02:42:25.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 -- 172.21.15.29:0/171057912 shutdown_connections 2024-09-21T02:42:25.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 --2- 172.21.15.29:0/171057912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2760109e20 0x7f2760110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 --2- 172.21.15.29:0/171057912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27601094d0 0x7f27601098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 --2- 172.21.15.29:0/171057912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760103b90 0x7f2760103fa0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 -- 172.21.15.29:0/171057912 >> 172.21.15.29:0/171057912 conn(0x7f27600ff600 msgr2=0x7f2760101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 -- 172.21.15.29:0/171057912 shutdown_connections 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 -- 172.21.15.29:0/171057912 wait complete. 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 Processor -- start 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.588+0000 7f276777a640 1 -- start start 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f276777a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2760103b90 0x7f276019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f276777a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27601094d0 0x7f276019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f276777a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760109e20 0x7f276019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f276777a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2760113990 con 0x7f2760109e20 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f276777a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2760113810 con 0x7f2760103b90 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f276777a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2760113b10 con 0x7f27601094d0 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760109e20 0x7f276019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760109e20 0x7f276019fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41070/0 (socket says 172.21.15.29:41070) 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 -- 172.21.15.29:0/4054235352 learned_addr learned my addr 172.21.15.29:0/4054235352 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2764d15640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27601094d0 0x7f276019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765516640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2760103b90 0x7f276019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 -- 172.21.15.29:0/4054235352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27601094d0 msgr2=0x7f276019a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27601094d0 0x7f276019a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 -- 172.21.15.29:0/4054235352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2760103b90 msgr2=0x7f276019a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2760103b90 0x7f276019a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 -- 172.21.15.29:0/4054235352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27601a0330 con 0x7f2760109e20 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765516640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2760103b90 0x7f276019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.589+0000 7f2765d17640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760109e20 0x7f276019fc00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f275800eaa0 tx=0x7f275800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.590+0000 7f27567fc640 1 -- 172.21.15.29:0/4054235352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275800cdb0 con 0x7f2760109e20 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.590+0000 7f27567fc640 1 -- 172.21.15.29:0/4054235352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2758005020 con 0x7f2760109e20 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.590+0000 7f276777a640 1 -- 172.21.15.29:0/4054235352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27601a0620 con 0x7f2760109e20 2024-09-21T02:42:25.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.590+0000 7f27567fc640 1 -- 172.21.15.29:0/4054235352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27580116d0 con 0x7f2760109e20 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.590+0000 7f276777a640 1 -- 172.21.15.29:0/4054235352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27601a6d80 con 0x7f2760109e20 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.593+0000 7f276dcee740 1 Processor -- start 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.593+0000 7f27567fc640 1 -- 172.21.15.29:0/4054235352 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f27580051d0 con 0x7f2760109e20 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.593+0000 7f276dcee740 1 -- start start 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f276dcee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 0x55b14db6c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f276dcee740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db6ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f276dcee740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db6cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f276dcee740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b14da57dc0 con 0x55b14d79d660 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f276dcee740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b14dae41f0 con 0x55b14d7a5840 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f276dcee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b14dad3620 con 0x55b14d7c6520 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765516640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db6ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765516640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db6ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41080/0 (socket says 172.21.15.29:41080) 2024-09-21T02:42:25.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765516640 1 -- 172.21.15.29:0/4094603429 learned_addr learned my addr 172.21.15.29:0/4094603429 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2764d15640 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db6cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2764d15640 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 0x55b14db6c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f27567fc640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f274c0b6510 0x7f274c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765516640 1 -- 172.21.15.29:0/4094603429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 msgr2=0x55b14db6c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765516640 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 0x55b14db6c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765516640 1 -- 172.21.15.29:0/4094603429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 msgr2=0x55b14db6cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765516640 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db6cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765516640 1 -- 172.21.15.29:0/4094603429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b14da9bfd0 con 0x55b14d79d660 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.594+0000 7f2765d17640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f274c0b6510 0x7f274c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.595+0000 7f27567fc640 1 -- 172.21.15.29:0/4054235352 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f27580e7870 con 0x7f2760109e20 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.595+0000 7f2765516640 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db6ca50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f275c000c00 tx=0x7f275c032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b20a198aa5542c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.595+0000 7f274b7fe640 1 -- 172.21.15.29:0/4094603429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c03f070 con 0x55b14d79d660 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.595+0000 7f274b7fe640 1 -- 172.21.15.29:0/4094603429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f275c004030 con 0x55b14d79d660 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.595+0000 7f274b7fe640 1 -- 172.21.15.29:0/4094603429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c035070 con 0x55b14d79d660 2024-09-21T02:42:25.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 -- 172.21.15.29:0/4094603429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 msgr2=0x55b14db6ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f2765d17640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f274c0b6510 0x7f274c0b8a20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f27580a9c50 tx=0x7f2758071c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db6ca50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f275c000c00 tx=0x7f275c032d10 comp rx=0 tx=0).stop 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 -- 172.21.15.29:0/4094603429 shutdown_connections 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db6cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db6ca50 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 --2- 172.21.15.29:0/4094603429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 0x55b14db6c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 -- 172.21.15.29:0/4094603429 >> 172.21.15.29:0/4094603429 conn(0x55b14d7ba910 msgr2=0x55b14db6a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 -- 172.21.15.29:0/4094603429 shutdown_connections 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 -- 172.21.15.29:0/4094603429 wait complete. 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 -- start start 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.596+0000 7f276dcee740 1 Processor -- start 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f276dcee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 0x55b14db7bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f276dcee740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7a5840 0x55b14db7c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f276dcee740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7c6520 0x55b14db881d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f276dcee740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b14da6e300 con 0x55b14d7a5840 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f276dcee740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b14dad7430 con 0x55b14d7c6520 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f276dcee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b14d934bb0 con 0x55b14d79d660 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f276dcee740 10 client.0 _ll_register_callbacks cb 0x55b14da88120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2765516640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7a5840 0x55b14db7c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2765516640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7a5840 0x55b14db7c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41086/0 (socket says 172.21.15.29:41086) 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2765516640 1 -- 172.21.15.29:0/446701447 learned_addr learned my addr 172.21.15.29:0/446701447 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2764d15640 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 0x55b14db7bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2764d15640 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7c6520 0x55b14db881d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2765516640 1 -- 172.21.15.29:0/446701447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 msgr2=0x55b14db7bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2765516640 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 0x55b14db7bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2765516640 1 -- 172.21.15.29:0/446701447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7c6520 msgr2=0x55b14db881d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2765516640 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7c6520 0x55b14db881d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.597+0000 7f2765516640 1 -- 172.21.15.29:0/446701447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b14da5d780 con 0x55b14d7a5840 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f2765516640 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7a5840 0x55b14db7c4f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f275c006010 tx=0x7f275c004490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ee76fc5dd3ffd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f2748ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f2748ff9640 1 -- 172.21.15.29:0/446701447 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c035030 con 0x55b14d7a5840 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f2748ff9640 1 -- 172.21.15.29:0/446701447 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f275c031a70 con 0x55b14d7a5840 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f276dcee740 1 -- 172.21.15.29:0/446701447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f275c004030 con 0x55b14d7a5840 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f2748ff9640 1 -- 172.21.15.29:0/446701447 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f275c055050 con 0x55b14d7a5840 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f276dcee740 10 client.16068 fetch_fsmap learned FSMap version 30 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f276dcee740 1 -- 172.21.15.29:0/446701447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b14d831b00 con 0x55b14d7a5840 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f2748ff9640 1 -- 172.21.15.29:0/446701447 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f275c004030 con 0x55b14d7a5840 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.598+0000 7f2748ff9640 1 -- 172.21.15.29:0/446701447 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f275c0047e0 con 0x55b14d7a5840 2024-09-21T02:42:25.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f276dcee740 10 client.16068 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f276dcee740 10 client.16068 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f276dcee740 20 client.16068 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f276dcee740 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b14d784ea0 0x55b14db8e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f276dcee740 4 client.16068 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f276dcee740 1 -- 172.21.15.29:0/446701447 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b14dae6140 con 0x55b14d784ea0 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f2764d15640 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b14d784ea0 0x55b14db8e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f2764d15640 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b14d784ea0 0x55b14db8e0b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e25f71c346d165b4 server_cookie=b1ce4397f23b22c in_seq=0 out_seq=0 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.599+0000 7f2748ff9640 10 client.16068 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f2748ff9640 1 -- 172.21.15.29:0/446701447 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b14dae6140 con 0x55b14d784ea0 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f2748ff9640 10 client.16068 handle_command_reply: tid=0 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 client.16068 shutdown 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 -- 172.21.15.29:0/446701447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7a5840 msgr2=0x55b14db7c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7a5840 0x55b14db7c4f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f275c006010 tx=0x7f275c004490 comp rx=0 tx=0).stop 2024-09-21T02:42:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 -- 172.21.15.29:0/446701447 shutdown_connections 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7c6520 0x55b14db881d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b14d784ea0 0x55b14db8e0b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7a5840 0x55b14db7c4f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 --2- 172.21.15.29:0/446701447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 0x55b14db7bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f2748ff9640 0 client.16068 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 -- 172.21.15.29:0/446701447 >> 172.21.15.29:0/446701447 conn(0x55b14d7ba910 msgr2=0x55b14db68930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 -- 172.21.15.29:0/446701447 shutdown_connections 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.600+0000 7f276dcee740 1 -- 172.21.15.29:0/446701447 wait complete. 2024-09-21T02:42:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.601+0000 7f276dcee740 20 client.16068 trim_cache size 0 max 16384 2024-09-21T02:42:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f276dcee740 1 Processor -- start 2024-09-21T02:42:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f276dcee740 1 -- start start 2024-09-21T02:42:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f276dcee740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db69040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f276dcee740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db791a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f276dcee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 0x55b14db796e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f276dcee740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27601ae820 con 0x55b14d79d660 2024-09-21T02:42:25.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f276dcee740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f27601b0390 con 0x55b14d7a5840 2024-09-21T02:42:25.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f276dcee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f27601aeef0 con 0x55b14d7c6520 2024-09-21T02:42:25.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f2764d15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db69040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f2764d15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db69040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41096/0 (socket says 172.21.15.29:41096) 2024-09-21T02:42:25.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f2764d15640 1 -- 172.21.15.29:0/91228075 learned_addr learned my addr 172.21.15.29:0/91228075 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f2765516640 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db791a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f2764d15640 1 -- 172.21.15.29:0/91228075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 msgr2=0x55b14db796e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f2764d15640 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 0x55b14db796e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f2764d15640 1 -- 172.21.15.29:0/91228075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 msgr2=0x55b14db791a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.618+0000 7f2764d15640 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db791a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.619+0000 7f2764d15640 1 -- 172.21.15.29:0/91228075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b14d8a8960 con 0x55b14d79d660 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.619+0000 7f2765516640 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db791a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.619+0000 7f2764d15640 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db69040 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2760108f10 tx=0x7f2750005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2baa2492b020f37 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.619+0000 7f274affd640 1 -- 172.21.15.29:0/91228075 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27500044d0 con 0x55b14d79d660 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.619+0000 7f274affd640 1 -- 172.21.15.29:0/91228075 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2750012d60 con 0x55b14d79d660 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.619+0000 7f274affd640 1 -- 172.21.15.29:0/91228075 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750004a00 con 0x55b14d79d660 2024-09-21T02:42:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 -- 172.21.15.29:0/91228075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 msgr2=0x55b14db69040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db69040 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2760108f10 tx=0x7f2750005f90 comp rx=0 tx=0).stop 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 -- 172.21.15.29:0/91228075 shutdown_connections 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d7c6520 0x55b14db796e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db791a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 --2- 172.21.15.29:0/91228075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d79d660 0x55b14db69040 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 -- 172.21.15.29:0/91228075 >> 172.21.15.29:0/91228075 conn(0x55b14d7ba910 msgr2=0x55b14db68930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 -- 172.21.15.29:0/91228075 shutdown_connections 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 -- 172.21.15.29:0/91228075 wait complete. 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 -- start start 2024-09-21T02:42:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.620+0000 7f276dcee740 1 Processor -- start 2024-09-21T02:42:25.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f276dcee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 0x55b14db84b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f276dcee740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db7f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f276dcee740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7c6520 0x55b14db7f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f276dcee740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27601afc90 con 0x55b14d7c6520 2024-09-21T02:42:25.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f276dcee740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f27601af5c0 con 0x55b14d7a5840 2024-09-21T02:42:25.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f276dcee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f27601ac800 con 0x55b14d79d660 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f276dcee740 10 client.0 _ll_register_callbacks cb 0x7f2750009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7c6520 0x55b14db7f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7c6520 0x55b14db7f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41104/0 (socket says 172.21.15.29:41104) 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 -- 172.21.15.29:0/2060323619 learned_addr learned my addr 172.21.15.29:0/2060323619 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 0x55b14db84b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2765516640 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db7f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 -- 172.21.15.29:0/2060323619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 msgr2=0x55b14db84b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 0x55b14db84b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 -- 172.21.15.29:0/2060323619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 msgr2=0x55b14db7f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db7f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 -- 172.21.15.29:0/2060323619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b14dbf8850 con 0x55b14d7c6520 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2764d15640 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7c6520 0x55b14db7f880 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x55b14db84880 tx=0x7f2750011780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690279731edea833 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.621+0000 7f2748ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:42:25.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f2748ff9640 1 -- 172.21.15.29:0/2060323619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275002c040 con 0x55b14d7c6520 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f2748ff9640 1 -- 172.21.15.29:0/2060323619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2750004930 con 0x55b14d7c6520 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 1 -- 172.21.15.29:0/2060323619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2750012d60 con 0x55b14d7c6520 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f2748ff9640 1 -- 172.21.15.29:0/2060323619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2750004b30 con 0x55b14d7c6520 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 10 client.16074 fetch_fsmap learned FSMap version 30 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 1 -- 172.21.15.29:0/2060323619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b14dbf9990 con 0x55b14d7c6520 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f2748ff9640 1 -- 172.21.15.29:0/2060323619 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2750012d60 con 0x55b14d7c6520 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f2748ff9640 1 -- 172.21.15.29:0/2060323619 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f2750032020 con 0x55b14d7c6520 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 10 client.16074 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 10 client.16074 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 20 client.16074 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b14d75a250 0x55b14dbfc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 4 client.16074 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "50b9a31b", "release": true, "await": true}] 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.622+0000 7f276dcee740 1 -- 172.21.15.29:0/2060323619 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "50b9a31b", "release": true, "await": true}) -- 0x55b14dae1bf0 con 0x55b14d75a250 2024-09-21T02:42:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.623+0000 7f2765516640 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b14d75a250 0x55b14dbfc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:25.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.623+0000 7f2765516640 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b14d75a250 0x55b14dbfc100 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f10a596df8c4ed2e server_cookie=c2b00a9f5c55c726 in_seq=0 out_seq=0 2024-09-21T02:42:25.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.623+0000 7f2748ff9640 10 client.16074 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f2748ff9640 1 -- 172.21.15.29:0/2060323619 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b14dae1bf0 con 0x55b14d75a250 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f2748ff9640 10 client.16074 handle_command_reply: tid=0 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 client.16074 shutdown 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 -- 172.21.15.29:0/2060323619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7c6520 msgr2=0x55b14db7f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7c6520 0x55b14db7f880 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x55b14db84880 tx=0x7f2750011780 comp rx=0 tx=0).stop 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 -- 172.21.15.29:0/2060323619 shutdown_connections 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b14d75a250 0x55b14dbfc100 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b14d7c6520 0x55b14db7f880 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b14d7a5840 0x55b14db7f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 --2- 172.21.15.29:0/2060323619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b14d79d660 0x55b14db84b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f276dcee740 1 -- 172.21.15.29:0/2060323619 >> 172.21.15.29:0/2060323619 conn(0x55b14d7ba910 msgr2=0x55b14db6d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.634+0000 7f2748ff9640 0 client.16074 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.635+0000 7f276dcee740 1 -- 172.21.15.29:0/2060323619 shutdown_connections 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.635+0000 7f276dcee740 1 -- 172.21.15.29:0/2060323619 wait complete. 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.635+0000 7f276dcee740 20 client.16074 trim_cache size 0 max 16384 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 30, 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 16, 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stdout: "50b9a31b": { 2024-09-21T02:42:25.637 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 16, 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 147.8, 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:42:25.638 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:42:25.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:42:25.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:42:25.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:42:25.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:42:25.639 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:42:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.635+0000 7f276777a640 1 -- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f274c0b6510 msgr2=0x7f274c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.635+0000 7f276777a640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f274c0b6510 0x7f274c0b8a20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f27580a9c50 tx=0x7f2758071c60 comp rx=0 tx=0).stop 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.635+0000 7f276777a640 1 -- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760109e20 msgr2=0x7f276019fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.635+0000 7f276777a640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760109e20 0x7f276019fc00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f275800eaa0 tx=0x7f275800ef60 comp rx=0 tx=0).stop 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.636+0000 7f276777a640 1 -- 172.21.15.29:0/4054235352 shutdown_connections 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.636+0000 7f276777a640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f274c0b6510 0x7f274c0b8a20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.636+0000 7f276777a640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760109e20 0x7f276019fc00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.636+0000 7f276777a640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27601094d0 0x7f276019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.636+0000 7f276777a640 1 --2- 172.21.15.29:0/4054235352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2760103b90 0x7f276019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.636+0000 7f276777a640 1 -- 172.21.15.29:0/4054235352 >> 172.21.15.29:0/4054235352 conn(0x7f27600ff600 msgr2=0x7f2760101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.636+0000 7f276777a640 1 -- 172.21.15.29:0/4054235352 shutdown_connections 2024-09-21T02:42:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:42:25.636+0000 7f276777a640 1 -- 172.21.15.29:0/4054235352 wait complete. 2024-09-21T02:42:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:25 smithi029 ceph-mon[31006]: pgmap v488: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.1 KiB/s wr, 17 op/s 2024-09-21T02:42:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:25 smithi066 ceph-mon[33436]: pgmap v488: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.1 KiB/s wr, 17 op/s 2024-09-21T02:42:26.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 30587 0.12 MB/sec execute 3 sec latency 27565.237 ms 2024-09-21T02:42:26.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:25 smithi190 ceph-mon[33871]: pgmap v488: 129 pgs: 129 active+clean; 26 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.1 KiB/s wr, 17 op/s 2024-09-21T02:42:26.803 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '50b9a31b', total seconds elapsed: 31.7 2024-09-21T02:42:27.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 31331 1.78 MB/sec execute 4 sec latency 13.772 ms 2024-09-21T02:42:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:27 smithi066 ceph-mon[33436]: pgmap v489: 129 pgs: 129 active+clean; 26 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T02:42:28.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 31969 2.41 MB/sec execute 5 sec latency 15.027 ms 2024-09-21T02:42:28.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:27 smithi190 ceph-mon[33871]: pgmap v489: 129 pgs: 129 active+clean; 26 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T02:42:28.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:27 smithi029 ceph-mon[31006]: pgmap v489: 129 pgs: 129 active+clean; 26 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T02:42:29.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 32349 2.28 MB/sec execute 6 sec latency 16.191 ms 2024-09-21T02:42:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:29 smithi066 ceph-mon[33436]: pgmap v490: 129 pgs: 129 active+clean; 26 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-09-21T02:42:30.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 32738 2.02 MB/sec execute 7 sec latency 18.642 ms 2024-09-21T02:42:30.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:29 smithi190 ceph-mon[33871]: pgmap v490: 129 pgs: 129 active+clean; 26 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-09-21T02:42:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:29 smithi029 ceph-mon[31006]: pgmap v490: 129 pgs: 129 active+clean; 26 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-09-21T02:42:31.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:30 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:42:31.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 33390 2.35 MB/sec execute 8 sec latency 15.345 ms 2024-09-21T02:42:31.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:30 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:42:31.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:30 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:42:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:31 smithi066 ceph-mon[33436]: pgmap v491: 129 pgs: 129 active+clean; 29 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 370 KiB/s wr, 21 op/s 2024-09-21T02:42:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:31 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:42:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:31 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:42:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:31 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:42:32.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 34019 2.15 MB/sec execute 9 sec latency 19.374 ms 2024-09-21T02:42:32.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:31 smithi190 ceph-mon[33871]: pgmap v491: 129 pgs: 129 active+clean; 29 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 370 KiB/s wr, 21 op/s 2024-09-21T02:42:32.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:31 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:42:32.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:31 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:42:32.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:31 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:42:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:31 smithi029 ceph-mon[31006]: pgmap v491: 129 pgs: 129 active+clean; 29 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 370 KiB/s wr, 21 op/s 2024-09-21T02:42:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:31 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:42:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:31 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:42:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:31 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:42:33.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 34714 2.47 MB/sec execute 10 sec latency 13.334 ms 2024-09-21T02:42:33.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:32 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:42:33.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:32 smithi066 ceph-mon[33436]: pgmap v492: 129 pgs: 129 active+clean; 30 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-09-21T02:42:33.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:32 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:42:33.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:32 smithi029 ceph-mon[31006]: pgmap v492: 129 pgs: 129 active+clean; 30 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-09-21T02:42:33.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:32 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:42:33.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:32 smithi190 ceph-mon[33871]: pgmap v492: 129 pgs: 129 active+clean; 30 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-09-21T02:42:34.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 35300 2.56 MB/sec execute 11 sec latency 15.856 ms 2024-09-21T02:42:35.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 35819 2.71 MB/sec execute 12 sec latency 14.138 ms 2024-09-21T02:42:35.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:35 smithi190 ceph-mon[33871]: pgmap v493: 129 pgs: 129 active+clean; 30 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 710 KiB/s wr, 67 op/s 2024-09-21T02:42:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:35 smithi029 ceph-mon[31006]: pgmap v493: 129 pgs: 129 active+clean; 30 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 710 KiB/s wr, 67 op/s 2024-09-21T02:42:35.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:35 smithi066 ceph-mon[33436]: pgmap v493: 129 pgs: 129 active+clean; 30 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 710 KiB/s wr, 67 op/s 2024-09-21T02:42:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 -- 172.21.15.29:0/1078883606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc074103ff0 msgr2=0x7fc07410a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 --2- 172.21.15.29:0/1078883606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc074103ff0 0x7fc07410a990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc06c009ca0 tx=0x7fc06c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 -- 172.21.15.29:0/1078883606 shutdown_connections 2024-09-21T02:42:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 --2- 172.21.15.29:0/1078883606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc074103ff0 0x7fc07410a990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 --2- 172.21.15.29:0/1078883606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc074103650 0x7fc074103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 --2- 172.21.15.29:0/1078883606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0741025d0 0x7fc074102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 -- 172.21.15.29:0/1078883606 >> 172.21.15.29:0/1078883606 conn(0x7fc0740fdf30 msgr2=0x7fc0741003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 -- 172.21.15.29:0/1078883606 shutdown_connections 2024-09-21T02:42:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.523+0000 7fc07af58640 1 -- 172.21.15.29:0/1078883606 wait complete. 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc07af58640 1 Processor -- start 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc07af58640 1 -- start start 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc07af58640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0741025d0 0x7fc07406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc07af58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc074103650 0x7fc07419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc07af58640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc074103ff0 0x7fc0741a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc07af58640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc07410e070 con 0x7fc074103650 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc07af58640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc07410def0 con 0x7fc074103ff0 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc07af58640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc07410e1f0 con 0x7fc0741025d0 2024-09-21T02:42:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc073fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc074103650 0x7fc07419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc078cf4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0741025d0 0x7fc07406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc073fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc074103650 0x7fc07419eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51690/0 (socket says 172.21.15.29:51690) 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc073fff640 1 -- 172.21.15.29:0/2505074181 learned_addr learned my addr 172.21.15.29:0/2505074181 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc0794f5640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc074103ff0 0x7fc0741a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc078cf4640 1 -- 172.21.15.29:0/2505074181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc074103ff0 msgr2=0x7fc0741a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc074103ff0 0x7fc0741a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc078cf4640 1 -- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc074103650 msgr2=0x7fc07419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc074103650 0x7fc07419eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc078cf4640 1 -- 172.21.15.29:0/2505074181 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0741a18b0 con 0x7fc0741025d0 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.524+0000 7fc073fff640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc074103650 0x7fc07419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.525+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0741025d0 0x7fc07406c1f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc06800e9f0 tx=0x7fc06800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.525+0000 7fc071ffb640 1 -- 172.21.15.29:0/2505074181 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06800ccc0 con 0x7fc0741025d0 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.525+0000 7fc071ffb640 1 -- 172.21.15.29:0/2505074181 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0680108c0 con 0x7fc0741025d0 2024-09-21T02:42:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.525+0000 7fc07af58640 1 -- 172.21.15.29:0/2505074181 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0741a1ba0 con 0x7fc0741025d0 2024-09-21T02:42:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.525+0000 7fc071ffb640 1 -- 172.21.15.29:0/2505074181 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0680187e0 con 0x7fc0741025d0 2024-09-21T02:42:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.525+0000 7fc07af58640 1 -- 172.21.15.29:0/2505074181 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0741a20f0 con 0x7fc0741025d0 2024-09-21T02:42:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.528+0000 7fc081ebd740 1 Processor -- start 2024-09-21T02:42:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.528+0000 7fc081ebd740 1 -- start start 2024-09-21T02:42:35.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37646f1b0 0x55b376816e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b3768174c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376818010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc081ebd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b3767896f0 con 0x55b37646f1b0 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc081ebd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b376778b20 con 0x55b3764462f0 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc081ebd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b37677c930 con 0x55b37644e4d0 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc071ffb640 1 -- 172.21.15.29:0/2505074181 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc068010430 con 0x7fc0741025d0 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc0794f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37646f1b0 0x55b376816e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc0794f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37646f1b0 0x55b376816e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51706/0 (socket says 172.21.15.29:51706) 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc0794f5640 1 -- 172.21.15.29:0/1575713971 learned_addr learned my addr 172.21.15.29:0/1575713971 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc073fff640 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376818010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc078cf4640 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b3768174c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc073fff640 1 -- 172.21.15.29:0/1575713971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 msgr2=0x55b3768174c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc073fff640 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b3768174c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc073fff640 1 -- 172.21.15.29:0/1575713971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37646f1b0 msgr2=0x55b376816e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc073fff640 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37646f1b0 0x55b376816e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc073fff640 1 -- 172.21.15.29:0/1575713971 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b37673e530 con 0x55b37644e4d0 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc0794f5640 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37646f1b0 0x55b376816e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc078cf4640 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b3768174c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.529+0000 7fc073fff640 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376818010 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc060011510 tx=0x7fc0600119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc13e279aaa70571 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.530+0000 7fc05effd640 1 -- 172.21.15.29:0/1575713971 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060005020 con 0x55b37644e4d0 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.530+0000 7fc05effd640 1 -- 172.21.15.29:0/1575713971 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0600047a0 con 0x55b37644e4d0 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.530+0000 7fc05effd640 1 -- 172.21.15.29:0/1575713971 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06000f8e0 con 0x55b37644e4d0 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.530+0000 7fc071ffb640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc0640b6510 0x7fc0640b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.530+0000 7fc0794f5640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc0640b6510 0x7fc0640b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.530+0000 7fc071ffb640 1 -- 172.21.15.29:0/2505074181 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc0680e7960 con 0x7fc0741025d0 2024-09-21T02:42:35.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 -- 172.21.15.29:0/1575713971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 msgr2=0x55b376818010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376818010 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc060011510 tx=0x7fc0600119d0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc0794f5640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc0640b6510 0x7fc0640b8a20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc06c035000 tx=0x7fc06c031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:35.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 -- 172.21.15.29:0/1575713971 shutdown_connections 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376818010 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b3768174c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 --2- 172.21.15.29:0/1575713971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37646f1b0 0x55b376816e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 -- 172.21.15.29:0/1575713971 >> 172.21.15.29:0/1575713971 conn(0x55b376463300 msgr2=0x55b3768128e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 -- 172.21.15.29:0/1575713971 shutdown_connections 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 -- 172.21.15.29:0/1575713971 wait complete. 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 -- start start 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.531+0000 7fc081ebd740 1 Processor -- start 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3764462f0 0x55b3768205f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376820cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b37646f1b0 0x55b3768217b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc081ebd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b3766b0ab0 con 0x55b3764462f0 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc081ebd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b37671e730 con 0x55b37646f1b0 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc081ebd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b376818b30 con 0x55b37644e4d0 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc081ebd740 10 client.0 _ll_register_callbacks cb 0x55b3767726b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc078cf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3764462f0 0x55b3768205f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc0794f5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b37646f1b0 0x55b3768217b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc078cf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3764462f0 0x55b3768205f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51716/0 (socket says 172.21.15.29:51716) 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc078cf4640 1 -- 172.21.15.29:0/2552535630 learned_addr learned my addr 172.21.15.29:0/2552535630 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc073fff640 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376820cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc078cf4640 1 -- 172.21.15.29:0/2552535630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 msgr2=0x55b376820cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376820cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc078cf4640 1 -- 172.21.15.29:0/2552535630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b37646f1b0 msgr2=0x55b3768217b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b37646f1b0 0x55b3768217b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc078cf4640 1 -- 172.21.15.29:0/2552535630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3764da790 con 0x55b3764462f0 2024-09-21T02:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.532+0000 7fc0794f5640 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b37646f1b0 0x55b3768217b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3764462f0 0x55b3768205f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc0680a9df0 tx=0x7fc0680a96c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae3add567c8434a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc053fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc053fff640 1 -- 172.21.15.29:0/2552535630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0680acf90 con 0x55b3764462f0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc053fff640 1 -- 172.21.15.29:0/2552535630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc0680ac4f0 con 0x55b3764462f0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc081ebd740 1 -- 172.21.15.29:0/2552535630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0600047a0 con 0x55b3764462f0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc053fff640 1 -- 172.21.15.29:0/2552535630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc068010720 con 0x55b3764462f0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc081ebd740 10 client.16086 fetch_fsmap learned FSMap version 31 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc081ebd740 1 -- 172.21.15.29:0/2552535630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b376553620 con 0x55b3764462f0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc053fff640 1 -- 172.21.15.29:0/2552535630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0600047a0 con 0x55b3764462f0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc053fff640 1 -- 172.21.15.29:0/2552535630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc0680ac6f0 con 0x55b3764462f0 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc081ebd740 10 client.16086 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc081ebd740 10 client.16086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.533+0000 7fc081ebd740 20 client.16086 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.534+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b37642dd80 0x55b37682dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.534+0000 7fc081ebd740 4 client.16086 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.534+0000 7fc081ebd740 1 -- 172.21.15.29:0/2552535630 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b37678b640 con 0x55b37642dd80 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.534+0000 7fc073fff640 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b37642dd80 0x55b37682dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.534+0000 7fc053fff640 10 client.16086 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.534+0000 7fc073fff640 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b37642dd80 0x55b37682dcf0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=936433ade16b5d07 server_cookie=fcb69b8e7e66b80c in_seq=0 out_seq=0 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc053fff640 1 -- 172.21.15.29:0/2552535630 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b37678b640 con 0x55b37642dd80 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc053fff640 10 client.16086 handle_command_reply: tid=0 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 client.16086 shutdown 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 -- 172.21.15.29:0/2552535630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3764462f0 msgr2=0x55b3768205f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3764462f0 0x55b3768205f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc0680a9df0 tx=0x7fc0680a96c0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 -- 172.21.15.29:0/2552535630 shutdown_connections 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b37646f1b0 0x55b3768217b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376820cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b37642dd80 0x55b37682dcf0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc053fff640 0 client.16086 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:35.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2552535630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3764462f0 0x55b3768205f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 -- 172.21.15.29:0/2552535630 >> 172.21.15.29:0/2552535630 conn(0x55b376463300 msgr2=0x55b376810ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:35.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 -- 172.21.15.29:0/2552535630 shutdown_connections 2024-09-21T02:42:35.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.535+0000 7fc081ebd740 1 -- 172.21.15.29:0/2552535630 wait complete. 2024-09-21T02:42:35.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.536+0000 7fc081ebd740 20 client.16086 trim_cache size 0 max 16384 2024-09-21T02:42:35.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc081ebd740 1 Processor -- start 2024-09-21T02:42:35.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc081ebd740 1 -- start start 2024-09-21T02:42:35.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37642dd80 0x55b376812220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376811650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37644e4d0 0x55b376811b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc081ebd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc0741ac540 con 0x55b37644e4d0 2024-09-21T02:42:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc081ebd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc0741ae080 con 0x55b3764462f0 2024-09-21T02:42:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc081ebd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc0741acc10 con 0x55b37642dd80 2024-09-21T02:42:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc0794f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37644e4d0 0x55b376811b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc078cf4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37642dd80 0x55b376812220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc078cf4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37642dd80 0x55b376812220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48604/0 (socket says 172.21.15.29:48604) 2024-09-21T02:42:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc073fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376811650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.554+0000 7fc078cf4640 1 -- 172.21.15.29:0/1688721636 learned_addr learned my addr 172.21.15.29:0/1688721636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:35.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc0794f5640 1 -- 172.21.15.29:0/1688721636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37642dd80 msgr2=0x55b376812220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc0794f5640 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37642dd80 0x55b376812220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc0794f5640 1 -- 172.21.15.29:0/1688721636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 msgr2=0x55b376811650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc0794f5640 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376811650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc0794f5640 1 -- 172.21.15.29:0/1688721636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b376828a50 con 0x55b37644e4d0 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc073fff640 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376811650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc078cf4640 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37642dd80 0x55b376812220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc0794f5640 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37644e4d0 0x55b376811b90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc06c031920 tx=0x7fc06c041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e6ebd9d37177a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc05e7fc640 1 -- 172.21.15.29:0/1688721636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06c003af0 con 0x55b37644e4d0 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc05e7fc640 1 -- 172.21.15.29:0/1688721636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc06c036050 con 0x55b37644e4d0 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.555+0000 7fc05e7fc640 1 -- 172.21.15.29:0/1688721636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06c0305c0 con 0x55b37644e4d0 2024-09-21T02:42:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 -- 172.21.15.29:0/1688721636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37644e4d0 msgr2=0x55b376811b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37644e4d0 0x55b376811b90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc06c031920 tx=0x7fc06c041000 comp rx=0 tx=0).stop 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 -- 172.21.15.29:0/1688721636 shutdown_connections 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37644e4d0 0x55b376811b90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376811650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 --2- 172.21.15.29:0/1688721636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37642dd80 0x55b376812220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 -- 172.21.15.29:0/1688721636 >> 172.21.15.29:0/1688721636 conn(0x55b376463300 msgr2=0x55b37680d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 -- 172.21.15.29:0/1688721636 shutdown_connections 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 -- 172.21.15.29:0/1688721636 wait complete. 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 -- start start 2024-09-21T02:42:35.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.556+0000 7fc081ebd740 1 Processor -- start 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37642dd80 0x55b376827d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376828400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc081ebd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376826ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc081ebd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc0741ad9b0 con 0x55b37642dd80 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc081ebd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc0741ad2e0 con 0x55b3764462f0 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc081ebd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc0741aa520 con 0x55b37644e4d0 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc081ebd740 10 client.0 _ll_register_callbacks cb 0x7fc0680eaf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc078cf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37642dd80 0x55b376827d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc078cf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37642dd80 0x55b376827d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51730/0 (socket says 172.21.15.29:51730) 2024-09-21T02:42:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc078cf4640 1 -- 172.21.15.29:0/2096913502 learned_addr learned my addr 172.21.15.29:0/2096913502 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc073fff640 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376828400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc0794f5640 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376826ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc078cf4640 1 -- 172.21.15.29:0/2096913502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 msgr2=0x55b376826ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376826ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc078cf4640 1 -- 172.21.15.29:0/2096913502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 msgr2=0x55b376828400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376828400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc078cf4640 1 -- 172.21.15.29:0/2096913502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b376821b20 con 0x55b37642dd80 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.557+0000 7fc073fff640 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376828400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc078cf4640 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37642dd80 0x55b376827d40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc0680a9da0 tx=0x7fc0680ad230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107ad1339c8e25f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc053fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc053fff640 1 -- 172.21.15.29:0/2096913502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc068098020 con 0x55b37642dd80 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc053fff640 1 -- 172.21.15.29:0/2096913502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc0680aeaa0 con 0x55b37642dd80 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc081ebd740 1 -- 172.21.15.29:0/2096913502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc06c036050 con 0x55b37642dd80 2024-09-21T02:42:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc053fff640 1 -- 172.21.15.29:0/2096913502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc0680ad330 con 0x55b37642dd80 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc081ebd740 10 client.16092 fetch_fsmap learned FSMap version 31 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc081ebd740 1 -- 172.21.15.29:0/2096913502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b376825f40 con 0x55b37642dd80 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc053fff640 1 -- 172.21.15.29:0/2096913502 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc06c036050 con 0x55b37642dd80 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc053fff640 1 -- 172.21.15.29:0/2096913502 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc0680b6040 con 0x55b37642dd80 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc081ebd740 10 client.16092 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc081ebd740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.558+0000 7fc081ebd740 20 client.16092 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.559+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b37646f1b0 0x55b37689aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.559+0000 7fc081ebd740 4 client.16092 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.559+0000 7fc081ebd740 1 -- 172.21.15.29:0/2096913502 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b376811440 con 0x55b37646f1b0 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.559+0000 7fc073fff640 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b37646f1b0 0x55b37689aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.559+0000 7fc053fff640 10 client.16092 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.559+0000 7fc073fff640 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b37646f1b0 0x55b37689aaa0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f52f755e28cd23b server_cookie=3be8f80388423c43 in_seq=0 out_seq=0 2024-09-21T02:42:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.593+0000 7fc053fff640 1 -- 172.21.15.29:0/2096913502 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b376811440 con 0x55b37646f1b0 2024-09-21T02:42:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.593+0000 7fc053fff640 10 client.16092 handle_command_reply: tid=0 2024-09-21T02:42:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.593+0000 7fc081ebd740 1 client.16092 shutdown 2024-09-21T02:42:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.593+0000 7fc081ebd740 1 -- 172.21.15.29:0/2096913502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37642dd80 msgr2=0x55b376827d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.593+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37642dd80 0x55b376827d40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc0680a9da0 tx=0x7fc0680ad230 comp rx=0 tx=0).stop 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 1 -- 172.21.15.29:0/2096913502 shutdown_connections 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b37644e4d0 0x55b376826ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3764462f0 0x55b376828400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b37646f1b0 0x55b37689aaa0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 1 --2- 172.21.15.29:0/2096913502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b37642dd80 0x55b376827d40 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc053fff640 0 client.16092 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 1 -- 172.21.15.29:0/2096913502 >> 172.21.15.29:0/2096913502 conn(0x55b376463300 msgr2=0x55b3768115f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 1 -- 172.21.15.29:0/2096913502 shutdown_connections 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 1 -- 172.21.15.29:0/2096913502 wait complete. 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.594+0000 7fc081ebd740 20 client.16092 trim_cache size 0 max 16384 2024-09-21T02:42:35.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:42:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:42:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:42:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:42:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 -- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc0640b6510 msgr2=0x7fc0640b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc0640b6510 0x7fc0640b8a20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc06c035000 tx=0x7fc06c031000 comp rx=0 tx=0).stop 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 -- 172.21.15.29:0/2505074181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0741025d0 msgr2=0x7fc07406c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0741025d0 0x7fc07406c1f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc06800e9f0 tx=0x7fc06800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 -- 172.21.15.29:0/2505074181 shutdown_connections 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc0640b6510 0x7fc0640b8a20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc074103ff0 0x7fc0741a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc074103650 0x7fc07419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 --2- 172.21.15.29:0/2505074181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0741025d0 0x7fc07406c1f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 -- 172.21.15.29:0/2505074181 >> 172.21.15.29:0/2505074181 conn(0x7fc0740fdf30 msgr2=0x7fc074106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 -- 172.21.15.29:0/2505074181 shutdown_connections 2024-09-21T02:42:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:35.595+0000 7fc07af58640 1 -- 172.21.15.29:0/2505074181 wait complete. 2024-09-21T02:42:36.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 36119 2.53 MB/sec execute 13 sec latency 33.289 ms 2024-09-21T02:42:37.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 36729 2.50 MB/sec execute 14 sec latency 15.545 ms 2024-09-21T02:42:37.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:37 smithi190 ceph-mon[33871]: pgmap v494: 129 pgs: 129 active+clean; 30 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-09-21T02:42:37.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:37 smithi029 ceph-mon[31006]: pgmap v494: 129 pgs: 129 active+clean; 30 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-09-21T02:42:37.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:37 smithi066 ceph-mon[33436]: pgmap v494: 129 pgs: 129 active+clean; 30 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-09-21T02:42:38.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 37376 2.55 MB/sec execute 15 sec latency 19.503 ms 2024-09-21T02:42:39.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 38001 2.54 MB/sec execute 16 sec latency 14.224 ms 2024-09-21T02:42:39.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:39 smithi190 ceph-mon[33871]: pgmap v495: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 1.3 MiB/s wr, 91 op/s 2024-09-21T02:42:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:39 smithi029 ceph-mon[31006]: pgmap v495: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 1.3 MiB/s wr, 91 op/s 2024-09-21T02:42:39.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:39 smithi066 ceph-mon[33436]: pgmap v495: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 1.3 MiB/s wr, 91 op/s 2024-09-21T02:42:40.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 38821 2.78 MB/sec execute 17 sec latency 16.129 ms 2024-09-21T02:42:41.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 39327 2.87 MB/sec execute 18 sec latency 44.920 ms 2024-09-21T02:42:41.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:41 smithi190 ceph-mon[33871]: pgmap v496: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-09-21T02:42:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:41 smithi029 ceph-mon[31006]: pgmap v496: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-09-21T02:42:41.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:41 smithi066 ceph-mon[33436]: pgmap v496: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-09-21T02:42:42.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 39634 2.73 MB/sec execute 19 sec latency 11.504 ms 2024-09-21T02:42:43.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 40240 2.70 MB/sec execute 20 sec latency 17.909 ms 2024-09-21T02:42:43.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:43 smithi190 ceph-mon[33871]: pgmap v497: 129 pgs: 129 active+clean; 27 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:42:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:43 smithi029 ceph-mon[31006]: pgmap v497: 129 pgs: 129 active+clean; 27 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:42:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:43 smithi066 ceph-mon[33436]: pgmap v497: 129 pgs: 129 active+clean; 27 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:42:44.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 40925 2.73 MB/sec execute 21 sec latency 16.501 ms 2024-09-21T02:42:45.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 41513 2.69 MB/sec execute 22 sec latency 15.209 ms 2024-09-21T02:42:45.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:45 smithi190 ceph-mon[33871]: pgmap v498: 129 pgs: 129 active+clean; 29 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-09-21T02:42:45.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:45 smithi029 ceph-mon[31006]: pgmap v498: 129 pgs: 129 active+clean; 29 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-09-21T02:42:45.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:45 smithi066 ceph-mon[33436]: pgmap v498: 129 pgs: 129 active+clean; 29 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-09-21T02:42:46.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 42363 2.89 MB/sec execute 23 sec latency 14.711 ms 2024-09-21T02:42:47.072 INFO:tasks.workunit.client.0.smithi029.stdout: 1 42847 2.95 MB/sec execute 24 sec latency 15.933 ms 2024-09-21T02:42:47.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:47 smithi190 ceph-mon[33871]: pgmap v499: 129 pgs: 129 active+clean; 29 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.6 MiB/s wr, 126 op/s 2024-09-21T02:42:47.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:47 smithi029 ceph-mon[31006]: pgmap v499: 129 pgs: 129 active+clean; 29 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.6 MiB/s wr, 126 op/s 2024-09-21T02:42:47.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:47 smithi066 ceph-mon[33436]: pgmap v499: 129 pgs: 129 active+clean; 29 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.6 MiB/s wr, 126 op/s 2024-09-21T02:42:48.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 43112 2.84 MB/sec execute 25 sec latency 14.947 ms 2024-09-21T02:42:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 -- 172.21.15.29:0/1983408740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc42c103fc0 msgr2=0x7fc42c10a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 --2- 172.21.15.29:0/1983408740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc42c103fc0 0x7fc42c10a960 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc424009c90 tx=0x7fc42402a860 comp rx=0 tx=0).stop 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 -- 172.21.15.29:0/1983408740 shutdown_connections 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 --2- 172.21.15.29:0/1983408740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc42c103fc0 0x7fc42c10a960 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 --2- 172.21.15.29:0/1983408740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc42c103620 0x7fc42c103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 --2- 172.21.15.29:0/1983408740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc42c1025a0 0x7fc42c102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 -- 172.21.15.29:0/1983408740 >> 172.21.15.29:0/1983408740 conn(0x7fc42c0fdee0 msgr2=0x7fc42c100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 -- 172.21.15.29:0/1983408740 shutdown_connections 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.702+0000 7fc43280a640 1 -- 172.21.15.29:0/1983408740 wait complete. 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc43280a640 1 Processor -- start 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc43280a640 1 -- start start 2024-09-21T02:42:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc43280a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc42c1025a0 0x7fc42c199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc43280a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc42c103620 0x7fc42c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc43280a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc42c103fc0 0x7fc42c19b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc43280a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c10e0b0 con 0x7fc42c103fc0 2024-09-21T02:42:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc43280a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c10df30 con 0x7fc42c1025a0 2024-09-21T02:42:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc43280a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c10e230 con 0x7fc42c103620 2024-09-21T02:42:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc42c1025a0 0x7fc42c199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.703+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc42c103620 0x7fc42c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc42c1025a0 0x7fc42c199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:45522/0 (socket says 172.21.15.29:45522) 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc42c103620 0x7fc42c19a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39250/0 (socket says 172.21.15.29:39250) 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42bfff640 1 -- 172.21.15.29:0/1499083295 learned_addr learned my addr 172.21.15.29:0/1499083295 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc430da7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc42c103fc0 0x7fc42c19b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42bfff640 1 -- 172.21.15.29:0/1499083295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc42c103620 msgr2=0x7fc42c19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42bfff640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc42c103620 0x7fc42c19a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42bfff640 1 -- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc42c103fc0 msgr2=0x7fc42c19b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42bfff640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc42c103fc0 0x7fc42c19b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42bfff640 1 -- 172.21.15.29:0/1499083295 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42c19b890 con 0x7fc42c1025a0 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc42bfff640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc42c1025a0 0x7fc42c199ec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00e9f0 tx=0x7fc41c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc4297fa640 1 -- 172.21.15.29:0/1499083295 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c00cd80 con 0x7fc42c1025a0 2024-09-21T02:42:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc4297fa640 1 -- 172.21.15.29:0/1499083295 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc41c0108c0 con 0x7fc42c1025a0 2024-09-21T02:42:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc4297fa640 1 -- 172.21.15.29:0/1499083295 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c0187c0 con 0x7fc42c1025a0 2024-09-21T02:42:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.704+0000 7fc43280a640 1 -- 172.21.15.29:0/1499083295 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc42c19bb80 con 0x7fc42c1025a0 2024-09-21T02:42:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.705+0000 7fc43280a640 1 -- 172.21.15.29:0/1499083295 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc42c1a2270 con 0x7fc42c1025a0 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.707+0000 7fc4297fa640 1 -- 172.21.15.29:0/1499083295 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc41c010430 con 0x7fc42c1025a0 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc439852740 1 Processor -- start 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc439852740 1 -- start start 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc439852740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dad0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc439852740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dad08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc439852740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dad1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc439852740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56315da3e6f0 con 0x56315d6fb2f0 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc439852740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56315da2db20 con 0x56315d7241b0 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc439852740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56315da31930 con 0x56315d7034d0 2024-09-21T02:42:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc430da7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dad0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc430da7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dad0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:45534/0 (socket says 172.21.15.29:45534) 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc430da7640 1 -- 172.21.15.29:0/3464659065 learned_addr learned my addr 172.21.15.29:0/3464659065 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dad1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc42bfff640 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dad08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.708+0000 7fc430da7640 1 -- 172.21.15.29:0/3464659065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 msgr2=0x56315dad1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc4297fa640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4140b6590 0x7fc4140b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc430da7640 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dad1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc430da7640 1 -- 172.21.15.29:0/3464659065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 msgr2=0x56315dad08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc430da7640 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dad08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc430da7640 1 -- 172.21.15.29:0/3464659065 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56315d9f3530 con 0x56315d7241b0 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc42b7fe640 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dad1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc42bfff640 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dad08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:42:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc4297fa640 1 -- 172.21.15.29:0/1499083295 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc41c0e7950 con 0x7fc42c1025a0 2024-09-21T02:42:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc430da7640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4140b6590 0x7fc4140b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc430da7640 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dad0260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc424009c90 tx=0x7fc4240093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c881cdf761ee8e34 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc4127fc640 1 -- 172.21.15.29:0/3464659065 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42404d070 con 0x56315d7241b0 2024-09-21T02:42:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc4127fc640 1 -- 172.21.15.29:0/3464659065 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc424036070 con 0x56315d7241b0 2024-09-21T02:42:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.709+0000 7fc4127fc640 1 -- 172.21.15.29:0/3464659065 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4240044a0 con 0x56315d7241b0 2024-09-21T02:42:48.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 -- 172.21.15.29:0/3464659065 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 msgr2=0x56315dad0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc430da7640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4140b6590 0x7fc4140b8aa0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc424031870 tx=0x7fc42404b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dad0260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc424009c90 tx=0x7fc4240093d0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 -- 172.21.15.29:0/3464659065 shutdown_connections 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dad1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dad08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 --2- 172.21.15.29:0/3464659065 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dad0260 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 -- 172.21.15.29:0/3464659065 >> 172.21.15.29:0/3464659065 conn(0x56315d718300 msgr2=0x56315dac78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 -- 172.21.15.29:0/3464659065 shutdown_connections 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.710+0000 7fc439852740 1 -- 172.21.15.29:0/3464659065 wait complete. 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 1 -- start start 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 1 Processor -- start 2024-09-21T02:42:48.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d6fb2f0 0x56315dad9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dada0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d7241b0 0x56315dadac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56315d965ab0 con 0x56315d7241b0 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56315d9d3730 con 0x56315d7034d0 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56315dad1f40 con 0x56315d6fb2f0 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc439852740 10 client.0 _ll_register_callbacks cb 0x56315da276b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dada0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc430da7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d7241b0 0x56315dadac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d6fb2f0 0x56315dad9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dada0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:45550/0 (socket says 172.21.15.29:45550) 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc42b7fe640 1 -- 172.21.15.29:0/3803596713 learned_addr learned my addr 172.21.15.29:0/3803596713 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.711+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d6fb2f0 0x56315dad9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39276/0 (socket says 172.21.15.29:39276) 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc42b7fe640 1 -- 172.21.15.29:0/3803596713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d6fb2f0 msgr2=0x56315dad9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc42b7fe640 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d6fb2f0 0x56315dad9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc42b7fe640 1 -- 172.21.15.29:0/3803596713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d7241b0 msgr2=0x56315dadac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc42b7fe640 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d7241b0 0x56315dadac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc42b7fe640 1 -- 172.21.15.29:0/3803596713 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56315d78f790 con 0x56315d7034d0 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc430da7640 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d7241b0 0x56315dadac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc42bfff640 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d6fb2f0 0x56315dad9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc42b7fe640 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dada0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc420012840 tx=0x7fc420012d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66ee0e0d9202aa3a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc40b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:42:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc40b7fe640 1 -- 172.21.15.29:0/3803596713 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420004050 con 0x56315d7034d0 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc40b7fe640 1 -- 172.21.15.29:0/3803596713 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc4200041f0 con 0x56315d7034d0 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.712+0000 7fc439852740 1 -- 172.21.15.29:0/3803596713 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc424036070 con 0x56315d7034d0 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc40b7fe640 1 -- 172.21.15.29:0/3803596713 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc420025050 con 0x56315d7034d0 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc439852740 10 client.25652 fetch_fsmap learned FSMap version 31 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc439852740 1 -- 172.21.15.29:0/3803596713 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56315d808620 con 0x56315d7034d0 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc40b7fe640 1 -- 172.21.15.29:0/3803596713 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc424036070 con 0x56315d7034d0 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc40b7fe640 1 -- 172.21.15.29:0/3803596713 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc4200038f0 con 0x56315d7034d0 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc439852740 10 client.25652 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc439852740 10 client.25652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc439852740 20 client.25652 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc439852740 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56315d6e2d80 0x56315dae7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc439852740 4 client.25652 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc439852740 1 -- 172.21.15.29:0/3803596713 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56315da40640 con 0x56315d6e2d80 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.713+0000 7fc42bfff640 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56315d6e2d80 0x56315dae7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.714+0000 7fc40b7fe640 10 client.25652 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.714+0000 7fc42bfff640 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56315d6e2d80 0x56315dae7210 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51a5082a0ce3347c server_cookie=779973ca189d9950 in_seq=0 out_seq=0 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc40b7fe640 1 -- 172.21.15.29:0/3803596713 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56315da40640 con 0x56315d6e2d80 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc40b7fe640 10 client.25652 handle_command_reply: tid=0 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 client.25652 shutdown 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 -- 172.21.15.29:0/3803596713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 msgr2=0x56315dada0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dada0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc420012840 tx=0x7fc420012d00 comp rx=0 tx=0).stop 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 -- 172.21.15.29:0/3803596713 shutdown_connections 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56315d6e2d80 0x56315dae7210 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d7241b0 0x56315dadac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dada0f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 --2- 172.21.15.29:0/3803596713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d6fb2f0 0x56315dad9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc40b7fe640 0 client.25652 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 -- 172.21.15.29:0/3803596713 >> 172.21.15.29:0/3803596713 conn(0x56315d718300 msgr2=0x56315dac5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 -- 172.21.15.29:0/3803596713 shutdown_connections 2024-09-21T02:42:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 1 -- 172.21.15.29:0/3803596713 wait complete. 2024-09-21T02:42:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.715+0000 7fc439852740 20 client.25652 trim_cache size 0 max 16384 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.733+0000 7fc439852740 1 Processor -- start 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.733+0000 7fc439852740 1 -- start start 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc439852740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dae6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc439852740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dac7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc439852740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7241b0 0x56315dad8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc439852740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c1aa2f0 con 0x56315d6fb2f0 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc439852740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c1abe30 con 0x56315d7034d0 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc439852740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c1aa9c0 con 0x56315d7241b0 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dae6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dae6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53556/0 (socket says 172.21.15.29:53556) 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 -- 172.21.15.29:0/3204592129 learned_addr learned my addr 172.21.15.29:0/3204592129 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dac7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc430da7640 1 --2- 172.21.15.29:0/3204592129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7241b0 0x56315dad8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 -- 172.21.15.29:0/3204592129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7241b0 msgr2=0x56315dad8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 --2- 172.21.15.29:0/3204592129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7241b0 0x56315dad8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 -- 172.21.15.29:0/3204592129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 msgr2=0x56315dac7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 --2- 172.21.15.29:0/3204592129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dac7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 -- 172.21.15.29:0/3204592129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56315dac7010 con 0x56315d6fb2f0 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc42bfff640 1 --2- 172.21.15.29:0/3204592129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dae6fd0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00e9c0 tx=0x7fc41c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4cc536db1c3dc62 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.734+0000 7fc411ffb640 1 -- 172.21.15.29:0/3204592129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c010720 con 0x56315d6fb2f0 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.735+0000 7fc411ffb640 1 -- 172.21.15.29:0/3204592129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc41c0ad320 con 0x56315d6fb2f0 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.735+0000 7fc411ffb640 1 -- 172.21.15.29:0/3204592129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c010d50 con 0x56315d6fb2f0 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 -- 172.21.15.29:0/3204592129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 msgr2=0x56315dae6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 --2- 172.21.15.29:0/3204592129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dae6fd0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00e9c0 tx=0x7fc41c000bc0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 -- 172.21.15.29:0/3204592129 shutdown_connections 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 --2- 172.21.15.29:0/3204592129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7241b0 0x56315dad8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 --2- 172.21.15.29:0/3204592129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7034d0 0x56315dac7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 --2- 172.21.15.29:0/3204592129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dae6fd0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 -- 172.21.15.29:0/3204592129 >> 172.21.15.29:0/3204592129 conn(0x56315d718300 msgr2=0x56315dad83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 -- 172.21.15.29:0/3204592129 shutdown_connections 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 -- 172.21.15.29:0/3204592129 wait complete. 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 -- start start 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 Processor -- start 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dadcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.736+0000 7fc439852740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dadd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc439852740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dade0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc439852740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c1ab760 con 0x56315d6fb2f0 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc439852740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c1ab090 con 0x56315d7241b0 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc439852740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c1a82d0 con 0x56315d7034d0 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc439852740 10 client.0 _ll_register_callbacks cb 0x7fc420009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dadcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dadd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc430da7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dade0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dadcf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53560/0 (socket says 172.21.15.29:53560) 2024-09-21T02:42:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42bfff640 1 -- 172.21.15.29:0/2392922124 learned_addr learned my addr 172.21.15.29:0/2392922124 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42b7fe640 1 -- 172.21.15.29:0/2392922124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 msgr2=0x56315dade0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42b7fe640 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dade0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42b7fe640 1 -- 172.21.15.29:0/2392922124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 msgr2=0x56315dadcf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42b7fe640 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dadcf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42b7fe640 1 -- 172.21.15.29:0/2392922124 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56315dade820 con 0x56315d7034d0 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc430da7640 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dade0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc42b7fe640 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dadd5c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc420001eb0 tx=0x7fc420003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=228b2680b0ed8960 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.737+0000 7fc40b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.738+0000 7fc40b7fe640 1 -- 172.21.15.29:0/2392922124 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42002d020 con 0x56315d7034d0 2024-09-21T02:42:48.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.738+0000 7fc40b7fe640 1 -- 172.21.15.29:0/2392922124 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc420018040 con 0x56315d7034d0 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.738+0000 7fc439852740 1 -- 172.21.15.29:0/2392922124 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc41c0ad320 con 0x56315d7034d0 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.738+0000 7fc40b7fe640 1 -- 172.21.15.29:0/2392922124 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc420038050 con 0x56315d7034d0 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.738+0000 7fc439852740 10 client.25732 fetch_fsmap learned FSMap version 31 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.738+0000 7fc439852740 1 -- 172.21.15.29:0/2392922124 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56315dae1b90 con 0x56315d7034d0 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.738+0000 7fc40b7fe640 1 -- 172.21.15.29:0/2392922124 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc41c0ad320 con 0x56315d7034d0 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.738+0000 7fc40b7fe640 1 -- 172.21.15.29:0/2392922124 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc42000fa30 con 0x56315d7034d0 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc439852740 10 client.25732 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc439852740 10 client.25732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc439852740 20 client.25732 populate_metadata read hostname 'smithi029' 2024-09-21T02:42:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc439852740 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56315d6b81c0 0x56315db540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:42:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc439852740 4 client.25732 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:42:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc439852740 1 -- 172.21.15.29:0/2392922124 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56315dac6440 con 0x56315d6b81c0 2024-09-21T02:42:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc42bfff640 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56315d6b81c0 0x56315db540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:42:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc40b7fe640 10 client.25732 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.739+0000 7fc42bfff640 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56315d6b81c0 0x56315db540e0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4020005ce9a7db1 server_cookie=c6ec416c65c8068 in_seq=0 out_seq=0 2024-09-21T02:42:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.781+0000 7fc40b7fe640 1 -- 172.21.15.29:0/2392922124 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56315dac6440 con 0x56315d6b81c0 2024-09-21T02:42:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.781+0000 7fc40b7fe640 10 client.25732 handle_command_reply: tid=0 2024-09-21T02:42:48.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.781+0000 7fc439852740 1 client.25732 shutdown 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.781+0000 7fc439852740 1 -- 172.21.15.29:0/2392922124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 msgr2=0x56315dadd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.781+0000 7fc439852740 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dadd5c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc420001eb0 tx=0x7fc420003820 comp rx=0 tx=0).stop 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 1 -- 172.21.15.29:0/2392922124 shutdown_connections 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56315d7241b0 0x56315dade0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56315d7034d0 0x56315dadd5c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56315d6b81c0 0x56315db540e0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 1 --2- 172.21.15.29:0/2392922124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315d6fb2f0 0x56315dadcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 1 -- 172.21.15.29:0/2392922124 >> 172.21.15.29:0/2392922124 conn(0x56315d718300 msgr2=0x56315dad8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc40b7fe640 0 client.25732 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:42:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 1 -- 172.21.15.29:0/2392922124 shutdown_connections 2024-09-21T02:42:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 1 -- 172.21.15.29:0/2392922124 wait complete. 2024-09-21T02:42:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc439852740 20 client.25732 trim_cache size 0 max 16384 2024-09-21T02:42:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:42:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:42:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:42:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:42:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc43280a640 1 -- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4140b6590 msgr2=0x7fc4140b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc43280a640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4140b6590 0x7fc4140b8aa0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc424031870 tx=0x7fc42404b000 comp rx=0 tx=0).stop 2024-09-21T02:42:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc43280a640 1 -- 172.21.15.29:0/1499083295 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc42c1025a0 msgr2=0x7fc42c199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:42:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.782+0000 7fc43280a640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc42c1025a0 0x7fc42c199ec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00e9f0 tx=0x7fc41c00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.783+0000 7fc43280a640 1 -- 172.21.15.29:0/1499083295 shutdown_connections 2024-09-21T02:42:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.783+0000 7fc43280a640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4140b6590 0x7fc4140b8aa0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.783+0000 7fc43280a640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc42c103fc0 0x7fc42c19b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.783+0000 7fc43280a640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc42c103620 0x7fc42c19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.783+0000 7fc43280a640 1 --2- 172.21.15.29:0/1499083295 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc42c1025a0 0x7fc42c199ec0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:42:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.783+0000 7fc43280a640 1 -- 172.21.15.29:0/1499083295 >> 172.21.15.29:0/1499083295 conn(0x7fc42c0fdee0 msgr2=0x7fc42c0ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:42:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.783+0000 7fc43280a640 1 -- 172.21.15.29:0/1499083295 shutdown_connections 2024-09-21T02:42:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:42:48.783+0000 7fc43280a640 1 -- 172.21.15.29:0/1499083295 wait complete. 2024-09-21T02:42:49.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 43706 2.81 MB/sec execute 26 sec latency 21.578 ms 2024-09-21T02:42:49.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:49 smithi190 ceph-mon[33871]: pgmap v500: 129 pgs: 129 active+clean; 30 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:42:49.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:49 smithi029 ceph-mon[31006]: pgmap v500: 129 pgs: 129 active+clean; 30 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:42:49.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:49 smithi066 ceph-mon[33436]: pgmap v500: 129 pgs: 129 active+clean; 30 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:42:50.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 44386 2.83 MB/sec execute 27 sec latency 16.914 ms 2024-09-21T02:42:51.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 45006 2.79 MB/sec execute 28 sec latency 18.373 ms 2024-09-21T02:42:51.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:51 smithi190 ceph-mon[33871]: pgmap v501: 129 pgs: 129 active+clean; 31 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-09-21T02:42:51.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:51 smithi029 ceph-mon[31006]: pgmap v501: 129 pgs: 129 active+clean; 31 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-09-21T02:42:51.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:51 smithi066 ceph-mon[33436]: pgmap v501: 129 pgs: 129 active+clean; 31 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-09-21T02:42:52.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 45890 2.95 MB/sec execute 29 sec latency 13.815 ms 2024-09-21T02:42:53.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 46382 3.00 MB/sec execute 30 sec latency 18.596 ms 2024-09-21T02:42:53.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:53 smithi190 ceph-mon[33871]: pgmap v502: 129 pgs: 129 active+clean; 30 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:42:53.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:53 smithi029 ceph-mon[31006]: pgmap v502: 129 pgs: 129 active+clean; 30 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:42:53.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:53 smithi066 ceph-mon[33436]: pgmap v502: 129 pgs: 129 active+clean; 30 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:42:54.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 46627 2.91 MB/sec execute 31 sec latency 14.176 ms 2024-09-21T02:42:55.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 47220 2.88 MB/sec execute 32 sec latency 17.180 ms 2024-09-21T02:42:55.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:55 smithi190 ceph-mon[33871]: pgmap v503: 129 pgs: 129 active+clean; 26 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-09-21T02:42:55.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:55 smithi029 ceph-mon[31006]: pgmap v503: 129 pgs: 129 active+clean; 26 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-09-21T02:42:55.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:55 smithi066 ceph-mon[33436]: pgmap v503: 129 pgs: 129 active+clean; 26 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-09-21T02:42:56.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 47860 2.89 MB/sec execute 33 sec latency 27.197 ms 2024-09-21T02:42:57.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 48536 2.86 MB/sec execute 34 sec latency 16.388 ms 2024-09-21T02:42:57.373 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:57 smithi190 ceph-mon[33871]: pgmap v504: 129 pgs: 129 active+clean; 28 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-09-21T02:42:57.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:57 smithi029 ceph-mon[31006]: pgmap v504: 129 pgs: 129 active+clean; 28 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-09-21T02:42:57.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:57 smithi066 ceph-mon[33436]: pgmap v504: 129 pgs: 129 active+clean; 28 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-09-21T02:42:58.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 49429 2.99 MB/sec execute 35 sec latency 12.792 ms 2024-09-21T02:42:59.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 49915 3.03 MB/sec execute 36 sec latency 17.573 ms 2024-09-21T02:42:59.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:42:59 smithi029 ceph-mon[31006]: pgmap v505: 129 pgs: 129 active+clean; 27 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-09-21T02:42:59.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:42:59 smithi066 ceph-mon[33436]: pgmap v505: 129 pgs: 129 active+clean; 27 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-09-21T02:42:59.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:42:59 smithi190 ceph-mon[33871]: pgmap v505: 129 pgs: 129 active+clean; 27 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-09-21T02:43:00.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 50158 2.95 MB/sec execute 37 sec latency 15.501 ms 2024-09-21T02:43:01.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 50592 2.90 MB/sec execute 38 sec latency 157.162 ms 2024-09-21T02:43:01.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:01 smithi029 ceph-mon[31006]: pgmap v506: 129 pgs: 129 active+clean; 28 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:43:01.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:01 smithi066 ceph-mon[33436]: pgmap v506: 129 pgs: 129 active+clean; 28 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:43:01.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:01 smithi190 ceph-mon[33871]: pgmap v506: 129 pgs: 129 active+clean; 28 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:43:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.889+0000 7f05ca3c3640 1 -- 172.21.15.29:0/9718196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 msgr2=0x7f05c4103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.889+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/9718196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 0x7f05c4103a30 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009900 tx=0x7f05b802a410 comp rx=0 tx=0).stop 2024-09-21T02:43:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 -- 172.21.15.29:0/9718196 shutdown_connections 2024-09-21T02:43:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/9718196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05c4103fc0 0x7f05c410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/9718196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 0x7f05c4103a30 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/9718196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05c41025a0 0x7f05c4102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 -- 172.21.15.29:0/9718196 >> 172.21.15.29:0/9718196 conn(0x7f05c40fdee0 msgr2=0x7f05c4100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 -- 172.21.15.29:0/9718196 shutdown_connections 2024-09-21T02:43:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 -- 172.21.15.29:0/9718196 wait complete. 2024-09-21T02:43:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 Processor -- start 2024-09-21T02:43:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.890+0000 7f05ca3c3640 1 -- start start 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05ca3c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05c41025a0 0x7f05c406b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05ca3c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 0x7f05c419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05ca3c3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05c4103fc0 0x7f05c41a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05ca3c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05c410e0b0 con 0x7f05c4103620 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05ca3c3640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f05c410df30 con 0x7f05c4103fc0 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05ca3c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05c410e230 con 0x7f05c41025a0 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 0x7f05c419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 0x7f05c419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40958/0 (socket says 172.21.15.29:40958) 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 -- 172.21.15.29:0/1759404546 learned_addr learned my addr 172.21.15.29:0/1759404546 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c3fff640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05c41025a0 0x7f05c406b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c8960640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05c4103fc0 0x7f05c41a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 -- 172.21.15.29:0/1759404546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05c41025a0 msgr2=0x7f05c406b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05c41025a0 0x7f05c406b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 -- 172.21.15.29:0/1759404546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05c4103fc0 msgr2=0x7f05c41a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05c4103fc0 0x7f05c41a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 -- 172.21.15.29:0/1759404546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c41a1950 con 0x7f05c4103620 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c8960640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05c4103fc0 0x7f05c41a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c3fff640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05c41025a0 0x7f05c406b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.891+0000 7f05c37fe640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 0x7f05c419ebe0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009230 tx=0x7f05b8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.892+0000 7f05c17fa640 1 -- 172.21.15.29:0/1759404546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b803f030 con 0x7f05c4103620 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.892+0000 7f05ca3c3640 1 -- 172.21.15.29:0/1759404546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c41a1be0 con 0x7f05c4103620 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.892+0000 7f05c17fa640 1 -- 172.21.15.29:0/1759404546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05b8004440 con 0x7f05c4103620 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.892+0000 7f05c17fa640 1 -- 172.21.15.29:0/1759404546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8035070 con 0x7f05c4103620 2024-09-21T02:43:01.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.892+0000 7f05ca3c3640 1 -- 172.21.15.29:0/1759404546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c41a2120 con 0x7f05c4103620 2024-09-21T02:43:01.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05c17fa640 1 -- 172.21.15.29:0/1759404546 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f05b8042030 con 0x7f05c4103620 2024-09-21T02:43:01.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05d140b740 1 Processor -- start 2024-09-21T02:43:01.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05d140b740 1 -- start start 2024-09-21T02:43:01.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05d140b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05d140b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d12f0 0x55f2725a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05d140b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725a3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05d140b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f2725146f0 con 0x55f2721d12f0 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05d140b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f272503b20 con 0x55f2721d94d0 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05d140b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f272507930 con 0x55f2721fa1b0 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d12f0 0x55f2725a24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05c8960640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05c8960640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39160/0 (socket says 172.21.15.29:39160) 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05c8960640 1 -- 172.21.15.29:0/1460140041 learned_addr learned my addr 172.21.15.29:0/1460140041 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d12f0 0x55f2725a24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40962/0 (socket says 172.21.15.29:40962) 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.895+0000 7f05c37fe640 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c8960640 1 -- 172.21.15.29:0/1460140041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 msgr2=0x55f2725a3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c8960640 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725a3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c8960640 1 -- 172.21.15.29:0/1460140041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d12f0 msgr2=0x55f2725a24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c8960640 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d12f0 0x55f2725a24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c8960640 1 -- 172.21.15.29:0/1460140041 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2724c9530 con 0x55f2721fa1b0 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c17fa640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05ac0b6510 0x7f05ac0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c3fff640 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d12f0 0x55f2725a24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c3fff640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05ac0b6510 0x7f05ac0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c8960640 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a1e50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009f70 tx=0x7f05bc00f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0413f0d162346dd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05c17fa640 1 -- 172.21.15.29:0/1759404546 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f05b8107bd0 con 0x7f05c4103620 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05aa7fc640 1 -- 172.21.15.29:0/1460140041 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc003bc0 con 0x55f2721fa1b0 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05aa7fc640 1 -- 172.21.15.29:0/1460140041 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05bc017070 con 0x55f2721fa1b0 2024-09-21T02:43:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.896+0000 7f05aa7fc640 1 -- 172.21.15.29:0/1460140041 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc0116a0 con 0x55f2721fa1b0 2024-09-21T02:43:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 -- 172.21.15.29:0/1460140041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 msgr2=0x55f2725a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a1e50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009f70 tx=0x7f05bc00f7f0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05c3fff640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05ac0b6510 0x7f05ac0b8a20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009fd0 tx=0x7f05b4015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 -- 172.21.15.29:0/1460140041 shutdown_connections 2024-09-21T02:43:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d12f0 0x55f2725a24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 --2- 172.21.15.29:0/1460140041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a1e50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 -- 172.21.15.29:0/1460140041 >> 172.21.15.29:0/1460140041 conn(0x55f2721ee300 msgr2=0x55f27259d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 -- 172.21.15.29:0/1460140041 shutdown_connections 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.897+0000 7f05d140b740 1 -- 172.21.15.29:0/1460140041 wait complete. 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.898+0000 7f05d140b740 1 -- start start 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.898+0000 7f05d140b740 1 Processor -- start 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.898+0000 7f05d140b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 0x55f2725ab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.898+0000 7f05d140b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725abd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.898+0000 7f05d140b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.898+0000 7f05d140b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f27243bab0 con 0x55f2721d94d0 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.898+0000 7f05d140b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f2724a9730 con 0x55f2721d12f0 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.898+0000 7f05d140b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2725a3b30 con 0x55f2721fa1b0 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05d140b740 10 client.0 _ll_register_callbacks cb 0x55f2724fd6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725abd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40966/0 (socket says 172.21.15.29:40966) 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 -- 172.21.15.29:0/3331175604 learned_addr learned my addr 172.21.15.29:0/3331175604 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c3fff640 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 0x55f2725ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c8960640 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 -- 172.21.15.29:0/3331175604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 msgr2=0x55f2725ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 -- 172.21.15.29:0/3331175604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 msgr2=0x55f2725ab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 0x55f2725ab690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 -- 172.21.15.29:0/3331175604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f272265790 con 0x55f2721d94d0 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05c37fe640 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725abd50 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f05b80cac50 tx=0x7f05b8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131c252ced291b43 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05a37fe640 1 -- 172.21.15.29:0/3331175604 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8049070 con 0x55f2721d94d0 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05a37fe640 1 -- 172.21.15.29:0/3331175604 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f05b80ccc40 con 0x55f2721d94d0 2024-09-21T02:43:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.899+0000 7f05d140b740 1 -- 172.21.15.29:0/3331175604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05bc017070 con 0x55f2721d94d0 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05a37fe640 1 -- 172.21.15.29:0/3331175604 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f05b80c9880 con 0x55f2721d94d0 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05d140b740 10 client.16122 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05d140b740 1 -- 172.21.15.29:0/3331175604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f2722de620 con 0x55f2721d94d0 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05a37fe640 1 -- 172.21.15.29:0/3331175604 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05bc017070 con 0x55f2721d94d0 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05a37fe640 1 -- 172.21.15.29:0/3331175604 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f05b80d2130 con 0x55f2721d94d0 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05d140b740 10 client.16122 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05d140b740 10 client.16122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05d140b740 20 client.16122 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05d140b740 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2721b8d80 0x55f2725b8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05d140b740 4 client.16122 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05d140b740 1 -- 172.21.15.29:0/3331175604 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f272516640 con 0x55f2721b8d80 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.900+0000 7f05c3fff640 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2721b8d80 0x55f2725b8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.901+0000 7f05a37fe640 10 client.16122 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.901+0000 7f05c3fff640 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2721b8d80 0x55f2725b8de0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9bd67ae5cd1b3c7 server_cookie=cb7995b008c1b614 in_seq=0 out_seq=0 2024-09-21T02:43:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.901+0000 7f05a37fe640 1 -- 172.21.15.29:0/3331175604 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f272516640 con 0x55f2721b8d80 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.901+0000 7f05a37fe640 10 client.16122 handle_command_reply: tid=0 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.901+0000 7f05d140b740 1 client.16122 shutdown 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 -- 172.21.15.29:0/3331175604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 msgr2=0x55f2725abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725abd50 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f05b80cac50 tx=0x7f05b8009260 comp rx=0 tx=0).stop 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 -- 172.21.15.29:0/3331175604 shutdown_connections 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2721b8d80 0x55f2725b8de0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725abd50 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 --2- 172.21.15.29:0/3331175604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 0x55f2725ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05a37fe640 0 client.16122 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 -- 172.21.15.29:0/3331175604 >> 172.21.15.29:0/3331175604 conn(0x55f2721ee300 msgr2=0x55f27259bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 -- 172.21.15.29:0/3331175604 shutdown_connections 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 1 -- 172.21.15.29:0/3331175604 wait complete. 2024-09-21T02:43:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.902+0000 7f05d140b740 20 client.16122 trim_cache size 0 max 16384 2024-09-21T02:43:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05d140b740 1 Processor -- start 2024-09-21T02:43:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05d140b740 1 -- start start 2024-09-21T02:43:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05d140b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 0x55f2725a2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05d140b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05d140b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05d140b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41aa320 con 0x55f2721d94d0 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05d140b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41abe60 con 0x55f2721d12f0 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05d140b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41aa9f0 con 0x55f2721fa1b0 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40978/0 (socket says 172.21.15.29:40978) 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05c37fe640 1 -- 172.21.15.29:0/2656428422 learned_addr learned my addr 172.21.15.29:0/2656428422 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.921+0000 7f05c8960640 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05c3fff640 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 0x55f2725a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05c37fe640 1 -- 172.21.15.29:0/2656428422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 msgr2=0x55f2725a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05c37fe640 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05c37fe640 1 -- 172.21.15.29:0/2656428422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 msgr2=0x55f2725a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05c37fe640 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 0x55f2725a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05c37fe640 1 -- 172.21.15.29:0/2656428422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2725ab420 con 0x55f2721d94d0 2024-09-21T02:43:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05c8960640 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05c37fe640 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725a8710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f05b8109040 tx=0x7f05b81053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e5f8a704b8a07e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05a9ffb640 1 -- 172.21.15.29:0/2656428422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b80b6070 con 0x55f2721d94d0 2024-09-21T02:43:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05a9ffb640 1 -- 172.21.15.29:0/2656428422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05b80ce0e0 con 0x55f2721d94d0 2024-09-21T02:43:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.922+0000 7f05a9ffb640 1 -- 172.21.15.29:0/2656428422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8049070 con 0x55f2721d94d0 2024-09-21T02:43:01.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 -- 172.21.15.29:0/2656428422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 msgr2=0x55f2725a8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725a8710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f05b8109040 tx=0x7f05b81053a0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 -- 172.21.15.29:0/2656428422 shutdown_connections 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721fa1b0 0x55f2725a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721d94d0 0x55f2725a8710 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 --2- 172.21.15.29:0/2656428422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d12f0 0x55f2725a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 -- 172.21.15.29:0/2656428422 >> 172.21.15.29:0/2656428422 conn(0x55f2721ee300 msgr2=0x55f2725a1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 -- 172.21.15.29:0/2656428422 shutdown_connections 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 -- 172.21.15.29:0/2656428422 wait complete. 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.923+0000 7f05d140b740 1 -- start start 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05d140b740 1 Processor -- start 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05d140b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721d12f0 0x55f2725b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05d140b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05d140b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721fa1b0 0x55f2725b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05d140b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41ab790 con 0x55f2721fa1b0 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05d140b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41ab0c0 con 0x55f2721d94d0 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05d140b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41a8300 con 0x55f2721d12f0 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05d140b740 10 client.0 _ll_register_callbacks cb 0x7f05b810b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05c8960640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721fa1b0 0x55f2725b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05c8960640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721fa1b0 0x55f2725b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40994/0 (socket says 172.21.15.29:40994) 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05c8960640 1 -- 172.21.15.29:0/771534750 learned_addr learned my addr 172.21.15.29:0/771534750 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05c3fff640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721d12f0 0x55f2725b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.924+0000 7f05c37fe640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05c8960640 1 -- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721d12f0 msgr2=0x55f2725b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05c8960640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721d12f0 0x55f2725b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05c8960640 1 -- 172.21.15.29:0/771534750 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 msgr2=0x55f2725af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05c8960640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05c8960640 1 -- 172.21.15.29:0/771534750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2725afe40 con 0x55f2721fa1b0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05c37fe640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05c3fff640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721d12f0 0x55f2725b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05c8960640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721fa1b0 0x55f2725b0540 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009f70 tx=0x7f05bc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d55eedb5412b3e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05a37fe640 1 -- 172.21.15.29:0/771534750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc004d30 con 0x55f2721fa1b0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05a37fe640 1 -- 172.21.15.29:0/771534750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f05bc005040 con 0x55f2721fa1b0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05d140b740 1 -- 172.21.15.29:0/771534750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05b80ce0e0 con 0x55f2721fa1b0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05a37fe640 1 -- 172.21.15.29:0/771534750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f05bc036050 con 0x55f2721fa1b0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05d140b740 10 client.16128 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.925+0000 7f05d140b740 1 -- 172.21.15.29:0/771534750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f272623ca0 con 0x55f2721fa1b0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05a37fe640 1 -- 172.21.15.29:0/771534750 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05b80ce0e0 con 0x55f2721fa1b0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05a37fe640 1 -- 172.21.15.29:0/771534750 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f05bc028780 con 0x55f2721fa1b0 2024-09-21T02:43:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05d140b740 10 client.16128 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05d140b740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05d140b740 20 client.16128 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05d140b740 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f27218e1c0 0x55f272626430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05d140b740 4 client.16128 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:43:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05d140b740 1 -- 172.21.15.29:0/771534750 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2725120f0 con 0x55f27218e1c0 2024-09-21T02:43:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05c3fff640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f27218e1c0 0x55f272626430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05a37fe640 10 client.16128 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.926+0000 7f05c3fff640 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f27218e1c0 0x55f272626430 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52e7881df1960e4f server_cookie=e8c7ad46f7672c6f in_seq=0 out_seq=0 2024-09-21T02:43:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05a37fe640 1 -- 172.21.15.29:0/771534750 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f2725120f0 con 0x55f27218e1c0 2024-09-21T02:43:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05a37fe640 10 client.16128 handle_command_reply: tid=0 2024-09-21T02:43:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 client.16128 shutdown 2024-09-21T02:43:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 -- 172.21.15.29:0/771534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721fa1b0 msgr2=0x55f2725b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721fa1b0 0x55f2725b0540 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009f70 tx=0x7f05bc0047b0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 -- 172.21.15.29:0/771534750 shutdown_connections 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f27218e1c0 0x55f272626430 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2721fa1b0 0x55f2725b0540 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2721d94d0 0x55f2725af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 --2- 172.21.15.29:0/771534750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2721d12f0 0x55f2725b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05d140b740 1 -- 172.21.15.29:0/771534750 >> 172.21.15.29:0/771534750 conn(0x55f2721ee300 msgr2=0x55f2725a8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.971+0000 7f05a37fe640 0 client.16128 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05d140b740 1 -- 172.21.15.29:0/771534750 shutdown_connections 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05d140b740 1 -- 172.21.15.29:0/771534750 wait complete. 2024-09-21T02:43:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05d140b740 20 client.16128 trim_cache size 0 max 16384 2024-09-21T02:43:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:43:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:43:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:43:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 -- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05ac0b6510 msgr2=0x7f05ac0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05ac0b6510 0x7f05ac0b8a20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009fd0 tx=0x7f05b4015040 comp rx=0 tx=0).stop 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 -- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 msgr2=0x7f05c419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 0x7f05c419ebe0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009230 tx=0x7f05b8004040 comp rx=0 tx=0).stop 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 -- 172.21.15.29:0/1759404546 shutdown_connections 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05ac0b6510 0x7f05ac0b8a20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05c4103fc0 0x7f05c41a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c4103620 0x7f05c419ebe0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 --2- 172.21.15.29:0/1759404546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05c41025a0 0x7f05c406b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.972+0000 7f05ca3c3640 1 -- 172.21.15.29:0/1759404546 >> 172.21.15.29:0/1759404546 conn(0x7f05c40fdee0 msgr2=0x7f05c4106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.973+0000 7f05ca3c3640 1 -- 172.21.15.29:0/1759404546 shutdown_connections 2024-09-21T02:43:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:01.973+0000 7f05ca3c3640 1 -- 172.21.15.29:0/1759404546 wait complete. 2024-09-21T02:43:02.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 51227 2.93 MB/sec execute 39 sec latency 22.269 ms 2024-09-21T02:43:03.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 51878 2.88 MB/sec execute 40 sec latency 22.681 ms 2024-09-21T02:43:03.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:03 smithi029 ceph-mon[31006]: pgmap v507: 129 pgs: 129 active+clean; 29 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.5 MiB/s wr, 116 op/s 2024-09-21T02:43:03.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:03 smithi066 ceph-mon[33436]: pgmap v507: 129 pgs: 129 active+clean; 29 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.5 MiB/s wr, 116 op/s 2024-09-21T02:43:03.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:03 smithi190 ceph-mon[33871]: pgmap v507: 129 pgs: 129 active+clean; 29 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.5 MiB/s wr, 116 op/s 2024-09-21T02:43:04.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 52662 2.97 MB/sec execute 41 sec latency 12.303 ms 2024-09-21T02:43:05.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 53270 3.02 MB/sec execute 42 sec latency 16.398 ms 2024-09-21T02:43:05.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:05 smithi029 ceph-mon[31006]: pgmap v508: 129 pgs: 129 active+clean; 30 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.5 MiB/s wr, 148 op/s 2024-09-21T02:43:05.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:05 smithi066 ceph-mon[33436]: pgmap v508: 129 pgs: 129 active+clean; 30 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.5 MiB/s wr, 148 op/s 2024-09-21T02:43:05.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:05 smithi190 ceph-mon[33871]: pgmap v508: 129 pgs: 129 active+clean; 30 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.5 MiB/s wr, 148 op/s 2024-09-21T02:43:06.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 53601 2.99 MB/sec execute 43 sec latency 116.923 ms 2024-09-21T02:43:07.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 53960 2.92 MB/sec execute 44 sec latency 16.903 ms 2024-09-21T02:43:07.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:07 smithi066 ceph-mon[33436]: pgmap v509: 129 pgs: 129 active+clean; 30 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-09-21T02:43:07.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:07 smithi190 ceph-mon[33871]: pgmap v509: 129 pgs: 129 active+clean; 30 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-09-21T02:43:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:07 smithi029 ceph-mon[31006]: pgmap v509: 129 pgs: 129 active+clean; 30 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-09-21T02:43:08.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 54653 2.97 MB/sec execute 45 sec latency 15.223 ms 2024-09-21T02:43:09.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 55291 2.92 MB/sec execute 46 sec latency 16.153 ms 2024-09-21T02:43:09.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:09 smithi066 ceph-mon[33436]: pgmap v510: 129 pgs: 129 active+clean; 30 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-09-21T02:43:09.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:09 smithi190 ceph-mon[33871]: pgmap v510: 129 pgs: 129 active+clean; 30 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-09-21T02:43:09.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:09 smithi029 ceph-mon[31006]: pgmap v510: 129 pgs: 129 active+clean; 30 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-09-21T02:43:10.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 55982 2.97 MB/sec execute 47 sec latency 15.356 ms 2024-09-21T02:43:11.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 56570 2.98 MB/sec execute 48 sec latency 21.443 ms 2024-09-21T02:43:11.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:11 smithi066 ceph-mon[33436]: pgmap v511: 129 pgs: 129 active+clean; 32 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T02:43:11.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:11 smithi190 ceph-mon[33871]: pgmap v511: 129 pgs: 129 active+clean; 32 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T02:43:11.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:11 smithi029 ceph-mon[31006]: pgmap v511: 129 pgs: 129 active+clean; 32 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T02:43:12.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 57100 3.00 MB/sec execute 49 sec latency 19.381 ms 2024-09-21T02:43:13.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 57445 2.95 MB/sec execute 50 sec latency 17.760 ms 2024-09-21T02:43:13.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:13 smithi066 ceph-mon[33436]: pgmap v512: 129 pgs: 129 active+clean; 32 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-09-21T02:43:13.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:13 smithi190 ceph-mon[33871]: pgmap v512: 129 pgs: 129 active+clean; 32 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-09-21T02:43:13.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:13 smithi029 ceph-mon[31006]: pgmap v512: 129 pgs: 129 active+clean; 32 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-09-21T02:43:14.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 58067 2.94 MB/sec execute 51 sec latency 16.362 ms 2024-09-21T02:43:15.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 58748 2.94 MB/sec execute 52 sec latency 20.820 ms 2024-09-21T02:43:15.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/3401231516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 msgr2=0x7f5a6810a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/3401231516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 0x7f5a6810a960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009c90 tx=0x7f5a6002a860 comp rx=0 tx=0).stop 2024-09-21T02:43:15.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/3401231516 shutdown_connections 2024-09-21T02:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/3401231516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 0x7f5a6810a960 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/3401231516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a68103620 0x7f5a68103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/3401231516 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a681025a0 0x7f5a68102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/3401231516 >> 172.21.15.29:0/3401231516 conn(0x7f5a680fdee0 msgr2=0x7f5a68100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/3401231516 shutdown_connections 2024-09-21T02:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.091+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/3401231516 wait complete. 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6f2bd640 1 Processor -- start 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6f2bd640 1 -- start start 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6f2bd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a681025a0 0x7f5a68197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6f2bd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a68103620 0x7f5a681982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6f2bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 0x7f5a68198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6f2bd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6810cfd0 con 0x7f5a68103fc0 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6f2bd640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6810ce50 con 0x7f5a68103620 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6f2bd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6810d150 con 0x7f5a681025a0 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 0x7f5a68198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6d059640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a681025a0 0x7f5a68197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6d059640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a681025a0 0x7f5a68197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38264/0 (socket says 172.21.15.29:38264) 2024-09-21T02:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 0x7f5a68198e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53480/0 (socket says 172.21.15.29:53480) 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6d059640 1 -- 172.21.15.29:0/371517848 learned_addr learned my addr 172.21.15.29:0/371517848 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.092+0000 7f5a6c858640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a68103620 0x7f5a681982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6d059640 1 -- 172.21.15.29:0/371517848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a68103620 msgr2=0x7f5a681982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6d059640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a68103620 0x7f5a681982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6d059640 1 -- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 msgr2=0x7f5a68198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6d059640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 0x7f5a68198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6d059640 1 -- 172.21.15.29:0/371517848 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a68199670 con 0x7f5a681025a0 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 0x7f5a68198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6d059640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a681025a0 0x7f5a68197c30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5a64010ad0 tx=0x7f5a64010f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a5e7fc640 1 -- 172.21.15.29:0/371517848 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6401a020 con 0x7f5a681025a0 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/371517848 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a68199960 con 0x7f5a681025a0 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a5e7fc640 1 -- 172.21.15.29:0/371517848 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a6400ec30 con 0x7f5a681025a0 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a5e7fc640 1 -- 172.21.15.29:0/371517848 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a64015640 con 0x7f5a681025a0 2024-09-21T02:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.093+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/371517848 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a681a0050 con 0x7f5a681025a0 2024-09-21T02:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a7622e740 1 Processor -- start 2024-09-21T02:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a7622e740 1 -- start start 2024-09-21T02:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a154a1b0 0x5614a18f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 0x5614a18f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a15294d0 0x5614a18f3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a7622e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5614a18646f0 con 0x5614a15294d0 2024-09-21T02:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a7622e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5614a1853b20 con 0x5614a15212f0 2024-09-21T02:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a7622e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5614a1857930 con 0x5614a154a1b0 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a154a1b0 0x5614a18f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a154a1b0 0x5614a18f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38274/0 (socket says 172.21.15.29:38274) 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a6c858640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a15294d0 0x5614a18f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.096+0000 7f5a6d85a640 1 -- 172.21.15.29:0/2298584185 learned_addr learned my addr 172.21.15.29:0/2298584185 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a5e7fc640 1 -- 172.21.15.29:0/371517848 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f5a6401d020 con 0x7f5a681025a0 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a6d85a640 1 -- 172.21.15.29:0/2298584185 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 msgr2=0x5614a18f24c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/2298584185 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 0x5614a18f24c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a6d85a640 1 -- 172.21.15.29:0/2298584185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a15294d0 msgr2=0x5614a18f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/2298584185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a15294d0 0x5614a18f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a6d85a640 1 -- 172.21.15.29:0/2298584185 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614a1819530 con 0x5614a154a1b0 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a6c858640 1 --2- 172.21.15.29:0/2298584185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a15294d0 0x5614a18f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/2298584185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a154a1b0 0x5614a18f1e50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009c60 tx=0x7f5a600093d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2f218a51c9a5370 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a537fe640 1 -- 172.21.15.29:0/2298584185 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60004d10 con 0x5614a154a1b0 2024-09-21T02:43:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a537fe640 1 -- 172.21.15.29:0/2298584185 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a60036030 con 0x5614a154a1b0 2024-09-21T02:43:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a537fe640 1 -- 172.21.15.29:0/2298584185 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60030d90 con 0x5614a154a1b0 2024-09-21T02:43:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.097+0000 7f5a5e7fc640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a540b6510 0x7f5a540b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a6d059640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a540b6510 0x7f5a540b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a5e7fc640 1 -- 172.21.15.29:0/371517848 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5a640e9b30 con 0x7f5a681025a0 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 -- 172.21.15.29:0/2298584185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a154a1b0 msgr2=0x5614a18f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a6d059640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a540b6510 0x7f5a540b8a20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5a640abf10 tx=0x7f5a640a41f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 --2- 172.21.15.29:0/2298584185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a154a1b0 0x5614a18f1e50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009c60 tx=0x7f5a600093d0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 -- 172.21.15.29:0/2298584185 shutdown_connections 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 --2- 172.21.15.29:0/2298584185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a15294d0 0x5614a18f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 --2- 172.21.15.29:0/2298584185 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 0x5614a18f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 --2- 172.21.15.29:0/2298584185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a154a1b0 0x5614a18f1e50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 -- 172.21.15.29:0/2298584185 >> 172.21.15.29:0/2298584185 conn(0x5614a153e300 msgr2=0x5614a18ed8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 -- 172.21.15.29:0/2298584185 shutdown_connections 2024-09-21T02:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.098+0000 7f5a7622e740 1 -- 172.21.15.29:0/2298584185 wait complete. 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 1 -- start start 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 1 Processor -- start 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18fb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18fc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5614a178bab0 con 0x5614a154a1b0 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5614a17f9730 con 0x5614a15294d0 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5614a18f3ad0 con 0x5614a15212f0 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.099+0000 7f5a7622e740 10 client.0 _ll_register_callbacks cb 0x5614a184d6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6c858640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18fc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6c858640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:45006/0 (socket says 172.21.15.29:45006) 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6c858640 1 -- 172.21.15.29:0/784374909 learned_addr learned my addr 172.21.15.29:0/784374909 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18fb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6d85a640 1 -- 172.21.15.29:0/784374909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 msgr2=0x5614a18fb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18fb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6d85a640 1 -- 172.21.15.29:0/784374909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 msgr2=0x5614a18fbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6d85a640 1 -- 172.21.15.29:0/784374909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614a15b5790 con 0x5614a154a1b0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6c858640 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18fc840 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5a600319a0 tx=0x7f5a600095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdd1d757c00660c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a50ff9640 1 -- 172.21.15.29:0/784374909 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60004b10 con 0x5614a154a1b0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a50ff9640 1 -- 172.21.15.29:0/784374909 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5a60004cb0 con 0x5614a154a1b0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.100+0000 7f5a7622e740 1 -- 172.21.15.29:0/784374909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a60036030 con 0x5614a154a1b0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a50ff9640 1 -- 172.21.15.29:0/784374909 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5a60031a80 con 0x5614a154a1b0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a7622e740 10 client.16140 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a7622e740 1 -- 172.21.15.29:0/784374909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5614a162e620 con 0x5614a154a1b0 2024-09-21T02:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a50ff9640 1 -- 172.21.15.29:0/784374909 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a60036030 con 0x5614a154a1b0 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a50ff9640 1 -- 172.21.15.29:0/784374909 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f5a60045a60 con 0x5614a154a1b0 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a7622e740 10 client.16140 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a7622e740 10 client.16140 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a7622e740 20 client.16140 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a7622e740 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614a1508d80 0x5614a1908cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a7622e740 4 client.16140 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a7622e740 1 -- 172.21.15.29:0/784374909 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614a1866640 con 0x5614a1508d80 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.101+0000 7f5a6c858640 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614a1508d80 0x5614a1908cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.102+0000 7f5a6c858640 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614a1508d80 0x5614a1908cb0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=618a6e00884fb9f1 server_cookie=bb44959e3a662899 in_seq=0 out_seq=0 2024-09-21T02:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.102+0000 7f5a50ff9640 10 client.16140 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a50ff9640 1 -- 172.21.15.29:0/784374909 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5614a1866640 con 0x5614a1508d80 2024-09-21T02:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a50ff9640 10 client.16140 handle_command_reply: tid=0 2024-09-21T02:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 client.16140 shutdown 2024-09-21T02:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 -- 172.21.15.29:0/784374909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 msgr2=0x5614a18fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18fc840 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5a600319a0 tx=0x7f5a600095a0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 -- 172.21.15.29:0/784374909 shutdown_connections 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614a1508d80 0x5614a1908cb0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18fc840 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 --2- 172.21.15.29:0/784374909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18fb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a7622e740 1 -- 172.21.15.29:0/784374909 >> 172.21.15.29:0/784374909 conn(0x5614a153e300 msgr2=0x5614a18ebea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.103+0000 7f5a50ff9640 0 client.16140 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.104+0000 7f5a7622e740 1 -- 172.21.15.29:0/784374909 shutdown_connections 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.104+0000 7f5a7622e740 1 -- 172.21.15.29:0/784374909 wait complete. 2024-09-21T02:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.104+0000 7f5a7622e740 20 client.16140 trim_cache size 0 max 16384 2024-09-21T02:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.120+0000 7f5a7622e740 1 Processor -- start 2024-09-21T02:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a7622e740 1 -- start start 2024-09-21T02:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18ee010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a7622e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a681aa270 con 0x5614a154a1b0 2024-09-21T02:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a7622e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5a681abdb0 con 0x5614a15294d0 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a7622e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a681aa940 con 0x5614a15212f0 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18ee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38292/0 (socket says 172.21.15.29:38292) 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 -- 172.21.15.29:0/3790415841 learned_addr learned my addr 172.21.15.29:0/3790415841 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6c858640 1 --2- 172.21.15.29:0/3790415841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 -- 172.21.15.29:0/3790415841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 msgr2=0x5614a18fa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/3790415841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fa880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 -- 172.21.15.29:0/3790415841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 msgr2=0x5614a18ee010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/3790415841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18ee010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.121+0000 7f5a6d85a640 1 -- 172.21.15.29:0/3790415841 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614a1508fb0 con 0x5614a15212f0 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.122+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/3790415841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18f2ce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a60045ed0 tx=0x7f5a60031910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf4525f7cc6e46c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:15.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.122+0000 7f5a52ffd640 1 -- 172.21.15.29:0/3790415841 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60063070 con 0x5614a15212f0 2024-09-21T02:43:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.122+0000 7f5a52ffd640 1 -- 172.21.15.29:0/3790415841 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a60036040 con 0x5614a15212f0 2024-09-21T02:43:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.122+0000 7f5a52ffd640 1 -- 172.21.15.29:0/3790415841 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60005100 con 0x5614a15212f0 2024-09-21T02:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 -- 172.21.15.29:0/3790415841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 msgr2=0x5614a18f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 --2- 172.21.15.29:0/3790415841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18f2ce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a60045ed0 tx=0x7f5a60031910 comp rx=0 tx=0).stop 2024-09-21T02:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 -- 172.21.15.29:0/3790415841 shutdown_connections 2024-09-21T02:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 --2- 172.21.15.29:0/3790415841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a18ee010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 --2- 172.21.15.29:0/3790415841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15294d0 0x5614a18fa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 --2- 172.21.15.29:0/3790415841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15212f0 0x5614a18f2ce0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 -- 172.21.15.29:0/3790415841 >> 172.21.15.29:0/3790415841 conn(0x5614a153e300 msgr2=0x5614a18ebea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 -- 172.21.15.29:0/3790415841 shutdown_connections 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 -- 172.21.15.29:0/3790415841 wait complete. 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 -- start start 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.123+0000 7f5a7622e740 1 Processor -- start 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 0x5614a19009f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15294d0 0x5614a19010b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a7622e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a1901bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a7622e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a681ab6e0 con 0x5614a154a1b0 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a7622e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5a681ab010 con 0x5614a15212f0 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a7622e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a681a8250 con 0x5614a15294d0 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a7622e740 10 client.0 _ll_register_callbacks cb 0x7f5a58009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a1901bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a1901bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53536/0 (socket says 172.21.15.29:53536) 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 -- 172.21.15.29:0/1031445617 learned_addr learned my addr 172.21.15.29:0/1031445617 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 0x5614a19009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6c858640 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15294d0 0x5614a19010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 -- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15294d0 msgr2=0x5614a19010b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15294d0 0x5614a19010b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 -- 172.21.15.29:0/1031445617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 msgr2=0x5614a19009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 0x5614a19009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6d85a640 1 -- 172.21.15.29:0/1031445617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614a1902310 con 0x5614a154a1b0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.124+0000 7f5a6c858640 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15294d0 0x5614a19010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a6d85a640 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a1901bb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5a600099a0 tx=0x7f5a600306c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd7d5b926c423a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a50ff9640 1 -- 172.21.15.29:0/1031445617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60063030 con 0x5614a154a1b0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a7622e740 1 -- 172.21.15.29:0/1031445617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a60036040 con 0x5614a154a1b0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a50ff9640 1 -- 172.21.15.29:0/1031445617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5a60004a90 con 0x5614a154a1b0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a50ff9640 1 -- 172.21.15.29:0/1031445617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5a60072050 con 0x5614a154a1b0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a7622e740 10 client.16146 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a7622e740 1 -- 172.21.15.29:0/1031445617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5614a19036a0 con 0x5614a154a1b0 2024-09-21T02:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a50ff9640 1 -- 172.21.15.29:0/1031445617 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a60036040 con 0x5614a154a1b0 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.125+0000 7f5a50ff9640 1 -- 172.21.15.29:0/1031445617 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f5a60069070 con 0x5614a154a1b0 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a7622e740 10 client.16146 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a7622e740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a7622e740 20 client.16146 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a7622e740 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614a14de1c0 0x5614a19758b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a7622e740 4 client.16146 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a7622e740 1 -- 172.21.15.29:0/1031445617 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614a18620f0 con 0x5614a14de1c0 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a6c858640 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614a14de1c0 0x5614a19758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a50ff9640 10 client.16146 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.126+0000 7f5a6c858640 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614a14de1c0 0x5614a19758b0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1342019d57e8656f server_cookie=c655ade2acc286e7 in_seq=0 out_seq=0 2024-09-21T02:43:15.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.164+0000 7f5a50ff9640 1 -- 172.21.15.29:0/1031445617 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614a18620f0 con 0x5614a14de1c0 2024-09-21T02:43:15.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.164+0000 7f5a50ff9640 10 client.16146 handle_command_reply: tid=0 2024-09-21T02:43:15.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 client.16146 shutdown 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 -- 172.21.15.29:0/1031445617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 msgr2=0x5614a1901bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a1901bb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5a600099a0 tx=0x7f5a600306c0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 -- 172.21.15.29:0/1031445617 shutdown_connections 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614a14de1c0 0x5614a19758b0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614a154a1b0 0x5614a1901bb0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a15294d0 0x5614a19010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 --2- 172.21.15.29:0/1031445617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614a15212f0 0x5614a19009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 -- 172.21.15.29:0/1031445617 >> 172.21.15.29:0/1031445617 conn(0x5614a153e300 msgr2=0x5614a18ebea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a50ff9640 0 client.16146 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 -- 172.21.15.29:0/1031445617 shutdown_connections 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 1 -- 172.21.15.29:0/1031445617 wait complete. 2024-09-21T02:43:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.165+0000 7f5a7622e740 20 client.16146 trim_cache size 0 max 16384 2024-09-21T02:43:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:43:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:43:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:43:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:43:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a540b6510 msgr2=0x7f5a540b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a540b6510 0x7f5a540b8a20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5a640abf10 tx=0x7f5a640a41f0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/371517848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a681025a0 msgr2=0x7f5a68197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a681025a0 0x7f5a68197c30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5a64010ad0 tx=0x7f5a64010f90 comp rx=0 tx=0).stop 2024-09-21T02:43:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/371517848 shutdown_connections 2024-09-21T02:43:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5a540b6510 0x7f5a540b8a20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a68103fc0 0x7f5a68198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a68103620 0x7f5a681982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 --2- 172.21.15.29:0/371517848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a681025a0 0x7f5a68197c30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/371517848 >> 172.21.15.29:0/371517848 conn(0x7f5a680fdee0 msgr2=0x7f5a68106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/371517848 shutdown_connections 2024-09-21T02:43:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:15.166+0000 7f5a6f2bd640 1 -- 172.21.15.29:0/371517848 wait complete. 2024-09-21T02:43:15.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:15 smithi066 ceph-mon[33436]: pgmap v513: 129 pgs: 129 active+clean; 33 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 1.4 MiB/s wr, 163 op/s 2024-09-21T02:43:15.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:15 smithi190 ceph-mon[33871]: pgmap v513: 129 pgs: 129 active+clean; 33 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 1.4 MiB/s wr, 163 op/s 2024-09-21T02:43:15.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:15 smithi029 ceph-mon[31006]: pgmap v513: 129 pgs: 129 active+clean; 33 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 1.4 MiB/s wr, 163 op/s 2024-09-21T02:43:16.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 59338 2.93 MB/sec execute 53 sec latency 38.687 ms 2024-09-21T02:43:17.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 60096 3.00 MB/sec execute 54 sec latency 15.186 ms 2024-09-21T02:43:17.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:17 smithi066 ceph-mon[33436]: pgmap v514: 129 pgs: 129 active+clean; 33 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 1.5 MiB/s wr, 118 op/s 2024-09-21T02:43:17.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:17 smithi190 ceph-mon[33871]: pgmap v514: 129 pgs: 129 active+clean; 33 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 1.5 MiB/s wr, 118 op/s 2024-09-21T02:43:17.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:17 smithi029 ceph-mon[31006]: pgmap v514: 129 pgs: 129 active+clean; 33 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 1.5 MiB/s wr, 118 op/s 2024-09-21T02:43:18.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 60610 3.02 MB/sec execute 55 sec latency 16.173 ms 2024-09-21T02:43:19.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 60921 2.98 MB/sec execute 56 sec latency 14.575 ms 2024-09-21T02:43:19.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:19 smithi066 ceph-mon[33436]: pgmap v515: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-09-21T02:43:19.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:19 smithi190 ceph-mon[33871]: pgmap v515: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-09-21T02:43:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:19 smithi029 ceph-mon[31006]: pgmap v515: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-09-21T02:43:20.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1 61507 2.96 MB/sec execute 57 sec latency 15.894 ms 2024-09-21T02:43:21.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 62176 2.97 MB/sec execute 58 sec latency 13.503 ms 2024-09-21T02:43:21.425 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:21 smithi190 ceph-mon[33871]: pgmap v516: 129 pgs: 129 active+clean; 29 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-09-21T02:43:21.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:21 smithi066 ceph-mon[33436]: pgmap v516: 129 pgs: 129 active+clean; 29 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-09-21T02:43:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:21 smithi029 ceph-mon[31006]: pgmap v516: 129 pgs: 129 active+clean; 29 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-09-21T02:43:22.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 62770 2.95 MB/sec execute 59 sec latency 17.935 ms 2024-09-21T02:43:23.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 63628 3.02 MB/sec execute 60 sec latency 17.283 ms 2024-09-21T02:43:23.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:23 smithi066 ceph-mon[33436]: pgmap v517: 129 pgs: 129 active+clean; 29 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:43:23.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:23 smithi190 ceph-mon[33871]: pgmap v517: 129 pgs: 129 active+clean; 29 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:43:23.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:23 smithi029 ceph-mon[31006]: pgmap v517: 129 pgs: 129 active+clean; 29 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:43:24.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 64127 3.04 MB/sec execute 61 sec latency 14.688 ms 2024-09-21T02:43:25.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 64411 3.00 MB/sec execute 62 sec latency 13.842 ms 2024-09-21T02:43:25.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:25 smithi066 ceph-mon[33436]: pgmap v518: 129 pgs: 129 active+clean; 31 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-09-21T02:43:25.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:25 smithi190 ceph-mon[33871]: pgmap v518: 129 pgs: 129 active+clean; 31 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-09-21T02:43:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:25 smithi029 ceph-mon[31006]: pgmap v518: 129 pgs: 129 active+clean; 31 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-09-21T02:43:26.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 65033 2.98 MB/sec execute 63 sec latency 19.682 ms 2024-09-21T02:43:27.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 65704 2.99 MB/sec execute 64 sec latency 14.551 ms 2024-09-21T02:43:27.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:27 smithi066 ceph-mon[33436]: pgmap v519: 129 pgs: 129 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T02:43:27.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:27 smithi190 ceph-mon[33871]: pgmap v519: 129 pgs: 129 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T02:43:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:27 smithi029 ceph-mon[31006]: pgmap v519: 129 pgs: 129 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T02:43:28.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 66319 2.97 MB/sec execute 65 sec latency 15.609 ms 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 -- 172.21.15.29:0/2195740807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc103630 msgr2=0x7f2ffc103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 --2- 172.21.15.29:0/2195740807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc103630 0x7f2ffc103a40 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009900 tx=0x7f2fec02a410 comp rx=0 tx=0).stop 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 -- 172.21.15.29:0/2195740807 shutdown_connections 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 --2- 172.21.15.29:0/2195740807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ffc103fd0 0x7f2ffc10a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 --2- 172.21.15.29:0/2195740807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc103630 0x7f2ffc103a40 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 --2- 172.21.15.29:0/2195740807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ffc1025b0 0x7f2ffc102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 -- 172.21.15.29:0/2195740807 >> 172.21.15.29:0/2195740807 conn(0x7f2ffc0fdf30 msgr2=0x7f2ffc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 -- 172.21.15.29:0/2195740807 shutdown_connections 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.274+0000 7f300420a640 1 -- 172.21.15.29:0/2195740807 wait complete. 2024-09-21T02:43:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f300420a640 1 Processor -- start 2024-09-21T02:43:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f300420a640 1 -- start start 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f300420a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc1025b0 0x7f2ffc06be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f300420a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ffc103630 0x7f2ffc19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f300420a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ffc103fd0 0x7f2ffc1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f300420a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc10e0c0 con 0x7f2ffc1025b0 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f300420a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc10df40 con 0x7f2ffc103630 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f300420a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc10e240 con 0x7f2ffc103fd0 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f3001fa6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc1025b0 0x7f2ffc06be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f3001fa6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc1025b0 0x7f2ffc06be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60854/0 (socket says 172.21.15.29:60854) 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f3001fa6640 1 -- 172.21.15.29:0/1376064617 learned_addr learned my addr 172.21.15.29:0/1376064617 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f30027a7640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ffc103fd0 0x7f2ffc1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f30017a5640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ffc103630 0x7f2ffc19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f3001fa6640 1 -- 172.21.15.29:0/1376064617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ffc103fd0 msgr2=0x7f2ffc1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f3001fa6640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ffc103fd0 0x7f2ffc1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f3001fa6640 1 -- 172.21.15.29:0/1376064617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ffc103630 msgr2=0x7f2ffc19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f3001fa6640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ffc103630 0x7f2ffc19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f3001fa6640 1 -- 172.21.15.29:0/1376064617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc1a1960 con 0x7f2ffc1025b0 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.275+0000 7f30017a5640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ffc103630 0x7f2ffc19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.276+0000 7f3001fa6640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc1025b0 0x7f2ffc06be50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800e9f0 tx=0x7f2ff800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.276+0000 7f2ff2ffd640 1 -- 172.21.15.29:0/1376064617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff800cd80 con 0x7f2ffc1025b0 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.276+0000 7f2ff2ffd640 1 -- 172.21.15.29:0/1376064617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ff80108c0 con 0x7f2ffc1025b0 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.276+0000 7f300420a640 1 -- 172.21.15.29:0/1376064617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ffc1a1c50 con 0x7f2ffc1025b0 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.276+0000 7f2ff2ffd640 1 -- 172.21.15.29:0/1376064617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80187c0 con 0x7f2ffc1025b0 2024-09-21T02:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.276+0000 7f300420a640 1 -- 172.21.15.29:0/1376064617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ffc1a21a0 con 0x7f2ffc1025b0 2024-09-21T02:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f300b252740 1 Processor -- start 2024-09-21T02:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f300b252740 1 -- start start 2024-09-21T02:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f2ff2ffd640 1 -- 172.21.15.29:0/1376064617 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f2ff8010430 con 0x7f2ffc1025b0 2024-09-21T02:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f300b252740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7a5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f300b252740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f300b252740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d84d0 0x561f6d7a6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f300b252740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561f6d7136f0 con 0x561f6d3d02f0 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f300b252740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561f6d702b20 con 0x561f6d3f91b0 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.279+0000 7f300b252740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561f6d706930 con 0x561f6d3d84d0 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30017a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d84d0 0x561f6d7a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7a5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:49426/0 (socket says 172.21.15.29:49426) 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 -- 172.21.15.29:0/3741316067 learned_addr learned my addr 172.21.15.29:0/3741316067 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f3001fa6640 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 -- 172.21.15.29:0/3741316067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d84d0 msgr2=0x561f6d7a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d84d0 0x561f6d7a6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 -- 172.21.15.29:0/3741316067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 msgr2=0x561f6d7a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 -- 172.21.15.29:0/3741316067 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6d6c8530 con 0x561f6d3f91b0 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30017a5640 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d84d0 0x561f6d7a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f30027a7640 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7a5260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009fd0 tx=0x7f2ff40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd2eef9b3431ab4a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f2ff2ffd640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2fe80b6510 0x7f2fe80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f2fe7fff640 1 -- 172.21.15.29:0/3741316067 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4004d20 con 0x561f6d3f91b0 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f3001fa6640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2fe80b6510 0x7f2fe80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f2fe7fff640 1 -- 172.21.15.29:0/3741316067 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ff4004050 con 0x561f6d3f91b0 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.280+0000 7f2fe7fff640 1 -- 172.21.15.29:0/3741316067 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4005260 con 0x561f6d3f91b0 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f2ff2ffd640 1 -- 172.21.15.29:0/1376064617 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2ff80e7aa0 con 0x7f2ffc1025b0 2024-09-21T02:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f300b252740 1 -- 172.21.15.29:0/3741316067 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 msgr2=0x561f6d7a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f3001fa6640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2fe80b6510 0x7f2fe80b8a20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8000c00 tx=0x7f2ff80eb580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f300b252740 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7a5260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009fd0 tx=0x7f2ff40118d0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f300b252740 1 -- 172.21.15.29:0/3741316067 shutdown_connections 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f300b252740 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d84d0 0x561f6d7a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f300b252740 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f300b252740 1 --2- 172.21.15.29:0/3741316067 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7a5260 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.281+0000 7f300b252740 1 -- 172.21.15.29:0/3741316067 >> 172.21.15.29:0/3741316067 conn(0x561f6d3ed300 msgr2=0x561f6d79c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.282+0000 7f300b252740 1 -- 172.21.15.29:0/3741316067 shutdown_connections 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.282+0000 7f300b252740 1 -- 172.21.15.29:0/3741316067 wait complete. 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.282+0000 7f300b252740 1 -- start start 2024-09-21T02:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.282+0000 7f300b252740 1 Processor -- start 2024-09-21T02:43:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f300b252740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f300b252740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d84d0 0x561f6d7af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f300b252740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 0x561f6d7afce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f300b252740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561f6d63aab0 con 0x561f6d3d02f0 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f300b252740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561f6d6a8730 con 0x561f6d3d84d0 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f300b252740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561f6d7a6f40 con 0x561f6d3f91b0 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f300b252740 10 client.0 _ll_register_callbacks cb 0x561f6d6fc6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60882/0 (socket says 172.21.15.29:60882) 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 -- 172.21.15.29:0/2468112340 learned_addr learned my addr 172.21.15.29:0/2468112340 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30017a5640 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d84d0 0x561f6d7af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 0x561f6d7afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 -- 172.21.15.29:0/2468112340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 msgr2=0x561f6d7afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 0x561f6d7afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 -- 172.21.15.29:0/2468112340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d84d0 msgr2=0x561f6d7af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d84d0 0x561f6d7af170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 -- 172.21.15.29:0/2468112340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6d464790 con 0x561f6d3d02f0 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30017a5640 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d84d0 0x561f6d7af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f30027a7640 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7aeab0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4011dd0 tx=0x7f2ff40047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc148932602854ce server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.283+0000 7f2fe57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f2fe57fa640 1 -- 172.21.15.29:0/2468112340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff400f6e0 con 0x561f6d3d02f0 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f2fe57fa640 1 -- 172.21.15.29:0/2468112340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2ff400f880 con 0x561f6d3d02f0 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 1 -- 172.21.15.29:0/2468112340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff4004050 con 0x561f6d3d02f0 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f2fe57fa640 1 -- 172.21.15.29:0/2468112340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ff4036050 con 0x561f6d3d02f0 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 10 client.16158 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 1 -- 172.21.15.29:0/2468112340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561f6d4dd620 con 0x561f6d3d02f0 2024-09-21T02:43:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f2fe57fa640 1 -- 172.21.15.29:0/2468112340 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ff4004050 con 0x561f6d3d02f0 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f2fe57fa640 1 -- 172.21.15.29:0/2468112340 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2ff4025070 con 0x561f6d3d02f0 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 10 client.16158 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 10 client.16158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 20 client.16158 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561f6d3b7d80 0x561f6d7bc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 4 client.16158 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.284+0000 7f300b252740 1 -- 172.21.15.29:0/2468112340 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f6d715640 con 0x561f6d3b7d80 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.285+0000 7f30017a5640 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561f6d3b7d80 0x561f6d7bc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.285+0000 7f2fe57fa640 10 client.16158 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.285+0000 7f30017a5640 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561f6d3b7d80 0x561f6d7bc010 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=879dd41ef72d849d server_cookie=65d14ff90a6f39a9 in_seq=0 out_seq=0 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f2fe57fa640 1 -- 172.21.15.29:0/2468112340 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561f6d715640 con 0x561f6d3b7d80 2024-09-21T02:43:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f2fe57fa640 10 client.16158 handle_command_reply: tid=0 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 client.16158 shutdown 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 -- 172.21.15.29:0/2468112340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 msgr2=0x561f6d7aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7aeab0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4011dd0 tx=0x7f2ff40047e0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 -- 172.21.15.29:0/2468112340 shutdown_connections 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 0x561f6d7afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d84d0 0x561f6d7af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561f6d3b7d80 0x561f6d7bc010 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 --2- 172.21.15.29:0/2468112340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d02f0 0x561f6d7aeab0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f2fe57fa640 0 client.16158 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.286+0000 7f300b252740 1 -- 172.21.15.29:0/2468112340 >> 172.21.15.29:0/2468112340 conn(0x561f6d3ed300 msgr2=0x561f6d79aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.287+0000 7f300b252740 1 -- 172.21.15.29:0/2468112340 shutdown_connections 2024-09-21T02:43:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.287+0000 7f300b252740 1 -- 172.21.15.29:0/2468112340 wait complete. 2024-09-21T02:43:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.287+0000 7f300b252740 20 client.16158 trim_cache size 0 max 16384 2024-09-21T02:43:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f300b252740 1 Processor -- start 2024-09-21T02:43:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f300b252740 1 -- start start 2024-09-21T02:43:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f300b252740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d02f0 0x561f6d7a5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f300b252740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7afce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f300b252740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7b0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f300b252740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1aa440 con 0x561f6d3d84d0 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f300b252740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1abf80 con 0x561f6d3f91b0 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f300b252740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1aab10 con 0x561f6d3d02f0 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f30017a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f30017a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60884/0 (socket says 172.21.15.29:60884) 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f30017a5640 1 -- 172.21.15.29:0/694032731 learned_addr learned my addr 172.21.15.29:0/694032731 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f30027a7640 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.303+0000 7f30027a7640 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d02f0 0x561f6d7a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f30017a5640 1 -- 172.21.15.29:0/694032731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d02f0 msgr2=0x561f6d7a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f30017a5640 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d02f0 0x561f6d7a5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f30017a5640 1 -- 172.21.15.29:0/694032731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 msgr2=0x561f6d7b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f30017a5640 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f30017a5640 1 -- 172.21.15.29:0/694032731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6d79b400 con 0x561f6d3d84d0 2024-09-21T02:43:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f30027a7640 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f30017a5640 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7afce0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0057c0 tx=0x7f2fec0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0fae62fcc5480ba server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f2fe77fe640 1 -- 172.21.15.29:0/694032731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec035040 con 0x561f6d3d84d0 2024-09-21T02:43:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f2fe77fe640 1 -- 172.21.15.29:0/694032731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2fec0044d0 con 0x561f6d3d84d0 2024-09-21T02:43:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.304+0000 7f2fe77fe640 1 -- 172.21.15.29:0/694032731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec03d450 con 0x561f6d3d84d0 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 -- 172.21.15.29:0/694032731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 msgr2=0x561f6d7afce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7afce0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0057c0 tx=0x7f2fec0058d0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 -- 172.21.15.29:0/694032731 shutdown_connections 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3f91b0 0x561f6d7b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7afce0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 --2- 172.21.15.29:0/694032731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3d02f0 0x561f6d7a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 -- 172.21.15.29:0/694032731 >> 172.21.15.29:0/694032731 conn(0x561f6d3ed300 msgr2=0x561f6d7a5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 -- 172.21.15.29:0/694032731 shutdown_connections 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 -- 172.21.15.29:0/694032731 wait complete. 2024-09-21T02:43:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.305+0000 7f300b252740 1 -- start start 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f300b252740 1 Processor -- start 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f300b252740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d02f0 0x561f6d7b6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f300b252740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f300b252740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 0x561f6d7b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f300b252740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1ab8b0 con 0x561f6d3d84d0 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f300b252740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1ab1e0 con 0x561f6d3d02f0 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f300b252740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1a8420 con 0x561f6d3f91b0 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f300b252740 10 client.0 _ll_register_callbacks cb 0x7f2ff80eb450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f30017a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f30017a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7b6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60888/0 (socket says 172.21.15.29:60888) 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f30017a5640 1 -- 172.21.15.29:0/3313840470 learned_addr learned my addr 172.21.15.29:0/3313840470 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f30027a7640 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 0x561f6d7b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.306+0000 7f30027a7640 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d02f0 0x561f6d7b6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f30017a5640 1 -- 172.21.15.29:0/3313840470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 msgr2=0x561f6d7b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f30017a5640 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 0x561f6d7b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f30017a5640 1 -- 172.21.15.29:0/3313840470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d02f0 msgr2=0x561f6d7b6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f30017a5640 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d02f0 0x561f6d7b6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f30017a5640 1 -- 172.21.15.29:0/3313840470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6d7b5b60 con 0x561f6d3d84d0 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f30017a5640 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7b6c00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2fec02ab40 tx=0x7f2fec02ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0d0eb5e04eb631 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f2fe57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f2fe57fa640 1 -- 172.21.15.29:0/3313840470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec035030 con 0x561f6d3d84d0 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f2fe57fa640 1 -- 172.21.15.29:0/3313840470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2fec003d00 con 0x561f6d3d84d0 2024-09-21T02:43:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f300b252740 1 -- 172.21.15.29:0/3313840470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fec0044d0 con 0x561f6d3d84d0 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f2fe57fa640 1 -- 172.21.15.29:0/3313840470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2fec060050 con 0x561f6d3d84d0 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f300b252740 10 client.16164 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.307+0000 7f300b252740 1 -- 172.21.15.29:0/3313840470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561f6d7b49f0 con 0x561f6d3d84d0 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f2fe57fa640 1 -- 172.21.15.29:0/3313840470 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2fec0044d0 con 0x561f6d3d84d0 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f2fe57fa640 1 -- 172.21.15.29:0/3313840470 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2fec03a040 con 0x561f6d3d84d0 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f300b252740 10 client.16164 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f300b252740 10 client.16164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f300b252740 20 client.16164 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f300b252740 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561f6d38d1c0 0x561f6d828fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f300b252740 4 client.16164 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:43:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f300b252740 1 -- 172.21.15.29:0/3313840470 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f6d7110f0 con 0x561f6d38d1c0 2024-09-21T02:43:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f30027a7640 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561f6d38d1c0 0x561f6d828fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f2fe57fa640 10 client.16164 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.308+0000 7f30027a7640 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561f6d38d1c0 0x561f6d828fa0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d09f10e9b3acb30 server_cookie=f9e15ce38db116b6 in_seq=0 out_seq=0 2024-09-21T02:43:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f2fe57fa640 1 -- 172.21.15.29:0/3313840470 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f6d7110f0 con 0x561f6d38d1c0 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f2fe57fa640 10 client.16164 handle_command_reply: tid=0 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 client.16164 shutdown 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 -- 172.21.15.29:0/3313840470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 msgr2=0x561f6d7b6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7b6c00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2fec02ab40 tx=0x7f2fec02ab70 comp rx=0 tx=0).stop 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 -- 172.21.15.29:0/3313840470 shutdown_connections 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f6d3f91b0 0x561f6d7b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561f6d38d1c0 0x561f6d828fa0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f2fe57fa640 0 client.16164 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6d3d84d0 0x561f6d7b6c00 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 --2- 172.21.15.29:0/3313840470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561f6d3d02f0 0x561f6d7b6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 -- 172.21.15.29:0/3313840470 >> 172.21.15.29:0/3313840470 conn(0x561f6d3ed300 msgr2=0x561f6d7acb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 -- 172.21.15.29:0/3313840470 shutdown_connections 2024-09-21T02:43:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 1 -- 172.21.15.29:0/3313840470 wait complete. 2024-09-21T02:43:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.356+0000 7f300b252740 20 client.16164 trim_cache size 0 max 16384 2024-09-21T02:43:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:43:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:43:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:43:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:43:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.357+0000 7f300420a640 1 -- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2fe80b6510 msgr2=0x7f2fe80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.357+0000 7f300420a640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2fe80b6510 0x7f2fe80b8a20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8000c00 tx=0x7f2ff80eb580 comp rx=0 tx=0).stop 2024-09-21T02:43:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.357+0000 7f300420a640 1 -- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc1025b0 msgr2=0x7f2ffc06be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.357+0000 7f300420a640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc1025b0 0x7f2ffc06be50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800e9f0 tx=0x7f2ff800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.357+0000 7f300420a640 1 -- 172.21.15.29:0/1376064617 shutdown_connections 2024-09-21T02:43:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.358+0000 7f300420a640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2fe80b6510 0x7f2fe80b8a20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.358+0000 7f300420a640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ffc103fd0 0x7f2ffc1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.358+0000 7f300420a640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ffc103630 0x7f2ffc19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.358+0000 7f300420a640 1 --2- 172.21.15.29:0/1376064617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc1025b0 0x7f2ffc06be50 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.358+0000 7f300420a640 1 -- 172.21.15.29:0/1376064617 >> 172.21.15.29:0/1376064617 conn(0x7f2ffc0fdf30 msgr2=0x7f2ffc106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.358+0000 7f300420a640 1 -- 172.21.15.29:0/1376064617 shutdown_connections 2024-09-21T02:43:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:28.358+0000 7f300420a640 1 -- 172.21.15.29:0/1376064617 wait complete. 2024-09-21T02:43:29.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 67167 3.03 MB/sec execute 66 sec latency 26.385 ms 2024-09-21T02:43:29.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:29 smithi066 ceph-mon[33436]: pgmap v520: 129 pgs: 129 active+clean; 31 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-09-21T02:43:29.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:29 smithi190 ceph-mon[33871]: pgmap v520: 129 pgs: 129 active+clean; 31 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-09-21T02:43:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:29 smithi029 ceph-mon[31006]: pgmap v520: 129 pgs: 129 active+clean; 31 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-09-21T02:43:30.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 67664 3.05 MB/sec execute 67 sec latency 15.725 ms 2024-09-21T02:43:31.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 67921 3.01 MB/sec execute 68 sec latency 16.356 ms 2024-09-21T02:43:31.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:31 smithi066 ceph-mon[33436]: pgmap v521: 129 pgs: 129 active+clean; 28 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-09-21T02:43:31.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:31 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:43:31.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:31 smithi190 ceph-mon[33871]: pgmap v521: 129 pgs: 129 active+clean; 28 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-09-21T02:43:31.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:31 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:43:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:31 smithi029 ceph-mon[31006]: pgmap v521: 129 pgs: 129 active+clean; 28 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-09-21T02:43:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:31 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:43:32.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 68515 3.00 MB/sec execute 69 sec latency 17.334 ms 2024-09-21T02:43:32.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:43:32.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:43:32.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:43:32.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:43:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:43:32.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:43:32.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:43:32.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:43:32.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:43:33.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 69194 3.00 MB/sec execute 70 sec latency 18.153 ms 2024-09-21T02:43:33.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:33 smithi066 ceph-mon[33436]: pgmap v522: 129 pgs: 129 active+clean; 29 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.6 MiB/s wr, 124 op/s 2024-09-21T02:43:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:33 smithi190 ceph-mon[33871]: pgmap v522: 129 pgs: 129 active+clean; 29 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.6 MiB/s wr, 124 op/s 2024-09-21T02:43:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:33 smithi029 ceph-mon[31006]: pgmap v522: 129 pgs: 129 active+clean; 29 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.6 MiB/s wr, 124 op/s 2024-09-21T02:43:34.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 69812 2.98 MB/sec execute 71 sec latency 17.634 ms 2024-09-21T02:43:35.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 70706 3.05 MB/sec execute 72 sec latency 13.775 ms 2024-09-21T02:43:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:35 smithi066 ceph-mon[33436]: pgmap v523: 129 pgs: 129 active+clean; 28 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:43:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:35 smithi190 ceph-mon[33871]: pgmap v523: 129 pgs: 129 active+clean; 28 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:43:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:35 smithi029 ceph-mon[31006]: pgmap v523: 129 pgs: 129 active+clean; 28 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:43:36.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 71181 3.06 MB/sec execute 73 sec latency 39.942 ms 2024-09-21T02:43:37.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 71431 3.03 MB/sec execute 74 sec latency 14.359 ms 2024-09-21T02:43:37.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:37 smithi066 ceph-mon[33436]: pgmap v524: 129 pgs: 129 active+clean; 28 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-09-21T02:43:37.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:37 smithi190 ceph-mon[33871]: pgmap v524: 129 pgs: 129 active+clean; 28 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-09-21T02:43:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:37 smithi029 ceph-mon[31006]: pgmap v524: 129 pgs: 129 active+clean; 28 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-09-21T02:43:38.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 72001 3.01 MB/sec execute 75 sec latency 16.850 ms 2024-09-21T02:43:39.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 72629 3.02 MB/sec execute 76 sec latency 23.009 ms 2024-09-21T02:43:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:39 smithi066 ceph-mon[33436]: pgmap v525: 129 pgs: 129 active+clean; 29 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:43:39.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:39 smithi190 ceph-mon[33871]: pgmap v525: 129 pgs: 129 active+clean; 29 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:43:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:39 smithi029 ceph-mon[31006]: pgmap v525: 129 pgs: 129 active+clean; 29 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:43:40.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 73333 3.00 MB/sec execute 77 sec latency 15.410 ms 2024-09-21T02:43:41.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 74003 3.03 MB/sec execute 78 sec latency 127.153 ms 2024-09-21T02:43:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 -- 172.21.15.29:0/3416886751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10641025a0 msgr2=0x7f1064102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 --2- 172.21.15.29:0/3416886751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10641025a0 0x7f1064102a10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1060009900 tx=0x7f106002a410 comp rx=0 tx=0).stop 2024-09-21T02:43:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 -- 172.21.15.29:0/3416886751 shutdown_connections 2024-09-21T02:43:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 --2- 172.21.15.29:0/3416886751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1064103fc0 0x7f106410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 --2- 172.21.15.29:0/3416886751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1064103620 0x7f1064103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 --2- 172.21.15.29:0/3416886751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10641025a0 0x7f1064102a10 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 -- 172.21.15.29:0/3416886751 >> 172.21.15.29:0/3416886751 conn(0x7f10640fdee0 msgr2=0x7f1064100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 -- 172.21.15.29:0/3416886751 shutdown_connections 2024-09-21T02:43:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.464+0000 7f106be7e640 1 -- 172.21.15.29:0/3416886751 wait complete. 2024-09-21T02:43:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106be7e640 1 Processor -- start 2024-09-21T02:43:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106be7e640 1 -- start start 2024-09-21T02:43:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106be7e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10641025a0 0x7f106406b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106be7e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1064103620 0x7f106419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106be7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1064103fc0 0x7f10641a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106be7e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f106410e0c0 con 0x7f1064103fc0 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106be7e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f106410df40 con 0x7f10641025a0 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106be7e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f106410e240 con 0x7f1064103620 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106a41b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1064103fc0 0x7f10641a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f1069c1a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10641025a0 0x7f106406b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f1069419640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1064103620 0x7f106419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106a41b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1064103fc0 0x7f10641a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40686/0 (socket says 172.21.15.29:40686) 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.465+0000 7f106a41b640 1 -- 172.21.15.29:0/2370946547 learned_addr learned my addr 172.21.15.29:0/2370946547 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f1069419640 1 -- 172.21.15.29:0/2370946547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10641025a0 msgr2=0x7f106406b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f1069419640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10641025a0 0x7f106406b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f1069419640 1 -- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1064103fc0 msgr2=0x7f10641a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f1069419640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1064103fc0 0x7f10641a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f1069419640 1 -- 172.21.15.29:0/2370946547 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10641a1930 con 0x7f1064103620 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f1069c1a640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10641025a0 0x7f106406b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f106a41b640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1064103fc0 0x7f10641a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f1069419640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1064103620 0x7f106419ebc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f105400e9f0 tx=0x7f105400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f105affd640 1 -- 172.21.15.29:0/2370946547 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105400ccc0 con 0x7f1064103620 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f105affd640 1 -- 172.21.15.29:0/2370946547 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f10540108c0 con 0x7f1064103620 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f106be7e640 1 -- 172.21.15.29:0/2370946547 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10641a1c20 con 0x7f1064103620 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f105affd640 1 -- 172.21.15.29:0/2370946547 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540187e0 con 0x7f1064103620 2024-09-21T02:43:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.466+0000 7f106be7e640 1 -- 172.21.15.29:0/2370946547 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10641a2170 con 0x7f1064103620 2024-09-21T02:43:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.469+0000 7f1072deb740 1 Processor -- start 2024-09-21T02:43:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.469+0000 7f1072deb740 1 -- start start 2024-09-21T02:43:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.469+0000 7f105affd640 1 -- 172.21.15.29:0/2370946547 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f1054018980 con 0x7f1064103620 2024-09-21T02:43:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1072deb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a290e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1072deb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a2914c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1072deb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec84d0 0x55a07a292010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1072deb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a07a2036f0 con 0x55a079ee91b0 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1072deb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a07a1f2b20 con 0x55a079ec02f0 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1072deb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a07a1f6930 con 0x55a079ec84d0 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a2914c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f106a41b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a290e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a2914c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:57844/0 (socket says 172.21.15.29:57844) 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 -- 172.21.15.29:0/1929087010 learned_addr learned my addr 172.21.15.29:0/1929087010 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f106a41b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a290e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40694/0 (socket says 172.21.15.29:40694) 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069419640 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec84d0 0x55a07a292010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 -- 172.21.15.29:0/1929087010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec84d0 msgr2=0x55a07a292010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec84d0 0x55a07a292010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 -- 172.21.15.29:0/1929087010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 msgr2=0x55a07a290e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a290e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 -- 172.21.15.29:0/1929087010 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a07a1b8530 con 0x55a079ec02f0 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f106a41b640 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a290e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.470+0000 7f1069c1a640 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a2914c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f106002aa00 tx=0x7f1060004470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=171be65af4f180a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.471+0000 7f105affd640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10500b6510 0x7f10500b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.471+0000 7f106a41b640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10500b6510 0x7f10500b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.471+0000 7f104ffff640 1 -- 172.21.15.29:0/1929087010 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060035030 con 0x55a079ec02f0 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.471+0000 7f105affd640 1 -- 172.21.15.29:0/2370946547 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f10540e79c0 con 0x7f1064103620 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.471+0000 7f104ffff640 1 -- 172.21.15.29:0/1929087010 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1060040070 con 0x55a079ec02f0 2024-09-21T02:43:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.471+0000 7f104ffff640 1 -- 172.21.15.29:0/1929087010 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060031c30 con 0x55a079ec02f0 2024-09-21T02:43:41.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 -- 172.21.15.29:0/1929087010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 msgr2=0x55a07a2914c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a2914c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f106002aa00 tx=0x7f1060004470 comp rx=0 tx=0).stop 2024-09-21T02:43:41.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f106a41b640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10500b6510 0x7f10500b8a20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f105c00fbb0 tx=0x7f105c0094e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 -- 172.21.15.29:0/1929087010 shutdown_connections 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec84d0 0x55a07a292010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a2914c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 --2- 172.21.15.29:0/1929087010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a290e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 -- 172.21.15.29:0/1929087010 >> 172.21.15.29:0/1929087010 conn(0x55a079edd300 msgr2=0x55a07a28c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 -- 172.21.15.29:0/1929087010 shutdown_connections 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 -- 172.21.15.29:0/1929087010 wait complete. 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 -- start start 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.472+0000 7f1072deb740 1 Processor -- start 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1072deb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec02f0 0x55a07a29a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1072deb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a29ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1072deb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1072deb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a07a12aab0 con 0x55a079ec02f0 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1072deb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a07a198730 con 0x55a079ec84d0 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1072deb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a07a292ad0 con 0x55a079ee91b0 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1072deb740 10 client.0 _ll_register_callbacks cb 0x55a07a1ec6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069c1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec02f0 0x55a07a29a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069c1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec02f0 0x55a07a29a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40702/0 (socket says 172.21.15.29:40702) 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069c1a640 1 -- 172.21.15.29:0/2624423976 learned_addr learned my addr 172.21.15.29:0/2624423976 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f106a41b640 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069419640 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a29ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069c1a640 1 -- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 msgr2=0x55a07a29b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069c1a640 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069c1a640 1 -- 172.21.15.29:0/2624423976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 msgr2=0x55a07a29ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069c1a640 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a29ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f1069c1a640 1 -- 172.21.15.29:0/2624423976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a079f54790 con 0x55a079ec02f0 2024-09-21T02:43:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.473+0000 7f106a41b640 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f1069c1a640 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec02f0 0x55a07a29a640 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1060031730 tx=0x7f1060005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2a39a8c9b8f514 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f104d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f104d7fa640 1 -- 172.21.15.29:0/2624423976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060052060 con 0x55a079ec02f0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f104d7fa640 1 -- 172.21.15.29:0/2624423976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f10600035f0 con 0x55a079ec02f0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f1072deb740 1 -- 172.21.15.29:0/2624423976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1060040070 con 0x55a079ec02f0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f104d7fa640 1 -- 172.21.15.29:0/2624423976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1060035070 con 0x55a079ec02f0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f1072deb740 10 client.16176 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f1072deb740 1 -- 172.21.15.29:0/2624423976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a079fcd620 con 0x55a079ec02f0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.474+0000 7f104d7fa640 1 -- 172.21.15.29:0/2624423976 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1060040070 con 0x55a079ec02f0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f104d7fa640 1 -- 172.21.15.29:0/2624423976 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f10600457e0 con 0x55a079ec02f0 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f1072deb740 10 client.16176 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f1072deb740 10 client.16176 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f1072deb740 20 client.16176 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f1072deb740 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a079ea7d80 0x55a07a2a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f1072deb740 4 client.16176 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f1072deb740 1 -- 172.21.15.29:0/2624423976 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a07a205640 con 0x55a079ea7d80 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f1069419640 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a079ea7d80 0x55a07a2a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f104d7fa640 10 client.16176 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.475+0000 7f1069419640 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a079ea7d80 0x55a07a2a78e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e3bfb2f6ee04ae5 server_cookie=c1d2e4cc720dde45 in_seq=0 out_seq=0 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f104d7fa640 1 -- 172.21.15.29:0/2624423976 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a07a205640 con 0x55a079ea7d80 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f104d7fa640 10 client.16176 handle_command_reply: tid=0 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 client.16176 shutdown 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 -- 172.21.15.29:0/2624423976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec02f0 msgr2=0x55a07a29a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec02f0 0x55a07a29a640 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1060031730 tx=0x7f1060005890 comp rx=0 tx=0).stop 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 -- 172.21.15.29:0/2624423976 shutdown_connections 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a29ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a079ea7d80 0x55a07a2a78e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 --2- 172.21.15.29:0/2624423976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec02f0 0x55a07a29a640 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f1072deb740 1 -- 172.21.15.29:0/2624423976 >> 172.21.15.29:0/2624423976 conn(0x55a079edd300 msgr2=0x55a07a28aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.476+0000 7f104d7fa640 0 client.16176 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.477+0000 7f1072deb740 1 -- 172.21.15.29:0/2624423976 shutdown_connections 2024-09-21T02:43:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.477+0000 7f1072deb740 1 -- 172.21.15.29:0/2624423976 wait complete. 2024-09-21T02:43:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.477+0000 7f1072deb740 20 client.16176 trim_cache size 0 max 16384 2024-09-21T02:43:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.494+0000 7f1072deb740 1 Processor -- start 2024-09-21T02:43:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.494+0000 7f1072deb740 1 -- start start 2024-09-21T02:43:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1072deb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec02f0 0x55a07a299840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1072deb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a2a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1072deb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a28b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1072deb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f10641ac490 con 0x55a079ee91b0 2024-09-21T02:43:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1072deb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f10641adfd0 con 0x55a079ec84d0 2024-09-21T02:43:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1072deb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f10641acb60 con 0x55a079ec02f0 2024-09-21T02:43:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f106a41b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a28b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec02f0 0x55a07a299840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec02f0 0x55a07a299840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:33382/0 (socket says 172.21.15.29:33382) 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 -- 172.21.15.29:0/3495610138 learned_addr learned my addr 172.21.15.29:0/3495610138 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f106a41b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a28b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40706/0 (socket says 172.21.15.29:40706) 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069419640 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a2a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 -- 172.21.15.29:0/3495610138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 msgr2=0x55a07a2a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a2a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 -- 172.21.15.29:0/3495610138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 msgr2=0x55a07a28b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a28b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 -- 172.21.15.29:0/3495610138 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a07a29bb70 con 0x55a079ec02f0 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069419640 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a2a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f106a41b640 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a28b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f1069c1a640 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec02f0 0x55a07a299840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f106002a930 tx=0x7f1060031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad93bf0f477ca692 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.495+0000 7f104f7fe640 1 -- 172.21.15.29:0/3495610138 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060062070 con 0x55a079ec02f0 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f104f7fe640 1 -- 172.21.15.29:0/3495610138 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1060040070 con 0x55a079ec02f0 2024-09-21T02:43:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f104f7fe640 1 -- 172.21.15.29:0/3495610138 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060052040 con 0x55a079ec02f0 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 -- 172.21.15.29:0/3495610138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec02f0 msgr2=0x55a07a299840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec02f0 0x55a07a299840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f106002a930 tx=0x7f1060031400 comp rx=0 tx=0).stop 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 -- 172.21.15.29:0/3495610138 shutdown_connections 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ee91b0 0x55a07a28b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec84d0 0x55a07a2a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 --2- 172.21.15.29:0/3495610138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ec02f0 0x55a07a299840 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 -- 172.21.15.29:0/3495610138 >> 172.21.15.29:0/3495610138 conn(0x55a079edd300 msgr2=0x55a07a2875c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 -- 172.21.15.29:0/3495610138 shutdown_connections 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.496+0000 7f1072deb740 1 -- 172.21.15.29:0/3495610138 wait complete. 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 1 -- start start 2024-09-21T02:43:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 1 Processor -- start 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a29d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec84d0 0x55a07a29dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f10641ad900 con 0x55a079ec84d0 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f10641ad230 con 0x55a079ec02f0 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f10641aa470 con 0x55a079ee91b0 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1072deb740 10 client.0 _ll_register_callbacks cb 0x7f105c00f3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.497+0000 7f1069419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec84d0 0x55a07a29dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec84d0 0x55a07a29dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40722/0 (socket says 172.21.15.29:40722) 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069419640 1 -- 172.21.15.29:0/486828404 learned_addr learned my addr 172.21.15.29:0/486828404 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069c1a640 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a29d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f106a41b640 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069419640 1 -- 172.21.15.29:0/486828404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 msgr2=0x55a07a29e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069419640 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069419640 1 -- 172.21.15.29:0/486828404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 msgr2=0x55a07a29d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069419640 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a29d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069419640 1 -- 172.21.15.29:0/486828404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a07a29ef40 con 0x55a079ec84d0 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069c1a640 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a29d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1069419640 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec84d0 0x55a07a29dd50 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f10540aaaf0 tx=0x7f10540ed160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c81daeab091f503 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f104d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f104d7fa640 1 -- 172.21.15.29:0/486828404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540ad650 con 0x55a079ec84d0 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f104d7fa640 1 -- 172.21.15.29:0/486828404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f10540acae0 con 0x55a079ec84d0 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1072deb740 1 -- 172.21.15.29:0/486828404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1060040070 con 0x55a079ec84d0 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f104d7fa640 1 -- 172.21.15.29:0/486828404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f10540acff0 con 0x55a079ec84d0 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1072deb740 10 client.16182 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.498+0000 7f1072deb740 1 -- 172.21.15.29:0/486828404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a07a2a2220 con 0x55a079ec84d0 2024-09-21T02:43:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f104d7fa640 1 -- 172.21.15.29:0/486828404 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1060040070 con 0x55a079ec84d0 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f104d7fa640 1 -- 172.21.15.29:0/486828404 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f10540a91d0 con 0x55a079ec84d0 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f1072deb740 10 client.16182 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f1072deb740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f1072deb740 20 client.16182 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f1072deb740 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a079e7d1c0 0x55a07a3186c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f1072deb740 4 client.16182 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f1072deb740 1 -- 172.21.15.29:0/486828404 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a07a2010f0 con 0x55a079e7d1c0 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.499+0000 7f1069c1a640 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a079e7d1c0 0x55a07a3186c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.500+0000 7f104d7fa640 10 client.16182 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.500+0000 7f1069c1a640 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a079e7d1c0 0x55a07a3186c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae241c16d0d2a377 server_cookie=c15badffce8d612e in_seq=0 out_seq=0 2024-09-21T02:43:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.543+0000 7f104d7fa640 1 -- 172.21.15.29:0/486828404 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a07a2010f0 con 0x55a079e7d1c0 2024-09-21T02:43:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.543+0000 7f104d7fa640 10 client.16182 handle_command_reply: tid=0 2024-09-21T02:43:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.543+0000 7f1072deb740 1 client.16182 shutdown 2024-09-21T02:43:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.543+0000 7f1072deb740 1 -- 172.21.15.29:0/486828404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec84d0 msgr2=0x55a07a29dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.543+0000 7f1072deb740 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec84d0 0x55a07a29dd50 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f10540aaaf0 tx=0x7f10540ed160 comp rx=0 tx=0).stop 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 1 -- 172.21.15.29:0/486828404 shutdown_connections 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a079ee91b0 0x55a07a29e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a079e7d1c0 0x55a07a3186c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a079ec84d0 0x55a07a29dd50 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 1 --2- 172.21.15.29:0/486828404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a079ec02f0 0x55a07a29d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f104d7fa640 0 client.16182 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 1 -- 172.21.15.29:0/486828404 >> 172.21.15.29:0/486828404 conn(0x55a079edd300 msgr2=0x55a07a28b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 1 -- 172.21.15.29:0/486828404 shutdown_connections 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 1 -- 172.21.15.29:0/486828404 wait complete. 2024-09-21T02:43:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f1072deb740 20 client.16182 trim_cache size 0 max 16384 2024-09-21T02:43:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:43:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:43:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:43:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f106be7e640 1 -- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10500b6510 msgr2=0x7f10500b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f106be7e640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10500b6510 0x7f10500b8a20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f105c00fbb0 tx=0x7f105c0094e0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f106be7e640 1 -- 172.21.15.29:0/2370946547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1064103620 msgr2=0x7f106419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.544+0000 7f106be7e640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1064103620 0x7f106419ebc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f105400e9f0 tx=0x7f105400eeb0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.545+0000 7f106be7e640 1 -- 172.21.15.29:0/2370946547 shutdown_connections 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.545+0000 7f106be7e640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f10500b6510 0x7f10500b8a20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.545+0000 7f106be7e640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1064103fc0 0x7f10641a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.545+0000 7f106be7e640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1064103620 0x7f106419ebc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.545+0000 7f106be7e640 1 --2- 172.21.15.29:0/2370946547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10641025a0 0x7f106406b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.545+0000 7f106be7e640 1 -- 172.21.15.29:0/2370946547 >> 172.21.15.29:0/2370946547 conn(0x7f10640fdee0 msgr2=0x7f1064106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.545+0000 7f106be7e640 1 -- 172.21.15.29:0/2370946547 shutdown_connections 2024-09-21T02:43:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:41.545+0000 7f106be7e640 1 -- 172.21.15.29:0/2370946547 wait complete. 2024-09-21T02:43:41.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:41 smithi066 ceph-mon[33436]: pgmap v526: 129 pgs: 129 active+clean; 29 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:43:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:41 smithi190 ceph-mon[33871]: pgmap v526: 129 pgs: 129 active+clean; 29 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:43:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:41 smithi029 ceph-mon[31006]: pgmap v526: 129 pgs: 129 active+clean; 29 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:43:42.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 74562 3.06 MB/sec execute 79 sec latency 45.547 ms 2024-09-21T02:43:43.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 74935 3.04 MB/sec execute 80 sec latency 15.250 ms 2024-09-21T02:43:43.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:43 smithi066 ceph-mon[33436]: pgmap v527: 129 pgs: 129 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-09-21T02:43:43.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:43 smithi190 ceph-mon[33871]: pgmap v527: 129 pgs: 129 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-09-21T02:43:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:43 smithi029 ceph-mon[31006]: pgmap v527: 129 pgs: 129 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-09-21T02:43:44.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 75377 3.01 MB/sec execute 81 sec latency 17.438 ms 2024-09-21T02:43:45.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 76052 3.03 MB/sec execute 82 sec latency 14.591 ms 2024-09-21T02:43:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:45 smithi029 ceph-mon[31006]: pgmap v528: 129 pgs: 129 active+clean; 31 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-09-21T02:43:45.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:45 smithi066 ceph-mon[33436]: pgmap v528: 129 pgs: 129 active+clean; 31 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-09-21T02:43:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:45 smithi190 ceph-mon[33871]: pgmap v528: 129 pgs: 129 active+clean; 31 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-09-21T02:43:46.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 76620 3.00 MB/sec execute 83 sec latency 123.652 ms 2024-09-21T02:43:47.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 77306 3.03 MB/sec execute 84 sec latency 17.263 ms 2024-09-21T02:43:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:47 smithi066 ceph-mon[33436]: pgmap v529: 129 pgs: 129 active+clean; 32 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-09-21T02:43:47.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:47 smithi190 ceph-mon[33871]: pgmap v529: 129 pgs: 129 active+clean; 32 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-09-21T02:43:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:47 smithi029 ceph-mon[31006]: pgmap v529: 129 pgs: 129 active+clean; 32 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 1.5 MiB/s wr, 123 op/s 2024-09-21T02:43:48.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 77900 3.03 MB/sec execute 85 sec latency 20.639 ms 2024-09-21T02:43:49.074 INFO:tasks.workunit.client.0.smithi029.stdout: 1 78427 3.05 MB/sec execute 86 sec latency 16.989 ms 2024-09-21T02:43:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:49 smithi066 ceph-mon[33436]: pgmap v530: 129 pgs: 129 active+clean; 32 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-09-21T02:43:49.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:49 smithi190 ceph-mon[33871]: pgmap v530: 129 pgs: 129 active+clean; 32 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-09-21T02:43:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:49 smithi029 ceph-mon[31006]: pgmap v530: 129 pgs: 129 active+clean; 32 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-09-21T02:43:50.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 78760 3.02 MB/sec execute 87 sec latency 26.282 ms 2024-09-21T02:43:51.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 79471 3.04 MB/sec execute 88 sec latency 13.943 ms 2024-09-21T02:43:51.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:51 smithi066 ceph-mon[33436]: pgmap v531: 129 pgs: 129 active+clean; 33 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:43:51.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:51 smithi190 ceph-mon[33871]: pgmap v531: 129 pgs: 129 active+clean; 33 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:43:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:51 smithi029 ceph-mon[31006]: pgmap v531: 129 pgs: 129 active+clean; 33 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:43:52.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 80086 3.01 MB/sec execute 89 sec latency 15.399 ms 2024-09-21T02:43:53.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 80693 3.00 MB/sec execute 90 sec latency 16.979 ms 2024-09-21T02:43:53.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:53 smithi066 ceph-mon[33436]: pgmap v532: 129 pgs: 129 active+clean; 34 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:43:53.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:53 smithi190 ceph-mon[33871]: pgmap v532: 129 pgs: 129 active+clean; 34 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:43:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:53 smithi029 ceph-mon[31006]: pgmap v532: 129 pgs: 129 active+clean; 34 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:43:54.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 81417 3.04 MB/sec execute 91 sec latency 15.297 ms 2024-09-21T02:43:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 -- 172.21.15.29:0/939638675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 msgr2=0x7fde88102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 --2- 172.21.15.29:0/939638675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 0x7fde88102a10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fde84009900 tx=0x7fde8402a410 comp rx=0 tx=0).stop 2024-09-21T02:43:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 -- 172.21.15.29:0/939638675 shutdown_connections 2024-09-21T02:43:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 --2- 172.21.15.29:0/939638675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde88103fc0 0x7fde8810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 --2- 172.21.15.29:0/939638675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fde88103620 0x7fde88103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 --2- 172.21.15.29:0/939638675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 0x7fde88102a10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 -- 172.21.15.29:0/939638675 >> 172.21.15.29:0/939638675 conn(0x7fde880fdee0 msgr2=0x7fde88100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 -- 172.21.15.29:0/939638675 shutdown_connections 2024-09-21T02:43:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.647+0000 7fde904ec640 1 -- 172.21.15.29:0/939638675 wait complete. 2024-09-21T02:43:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde904ec640 1 Processor -- start 2024-09-21T02:43:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde904ec640 1 -- start start 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde904ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 0x7fde8806b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde904ec640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde88103620 0x7fde881a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde904ec640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fde88103fc0 0x7fde881a3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde904ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fde8810e080 con 0x7fde881025a0 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde904ec640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fde8810df00 con 0x7fde88103fc0 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde904ec640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fde8810e200 con 0x7fde88103620 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde8e288640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 0x7fde8806b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde8e288640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 0x7fde8806b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35760/0 (socket says 172.21.15.29:35760) 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde8e288640 1 -- 172.21.15.29:0/2559402384 learned_addr learned my addr 172.21.15.29:0/2559402384 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde8da87640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde88103620 0x7fde881a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.648+0000 7fde8ea89640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fde88103fc0 0x7fde881a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde8e288640 1 -- 172.21.15.29:0/2559402384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde88103620 msgr2=0x7fde881a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde8e288640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde88103620 0x7fde881a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde8e288640 1 -- 172.21.15.29:0/2559402384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fde88103fc0 msgr2=0x7fde881a3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde8e288640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fde88103fc0 0x7fde881a3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde8e288640 1 -- 172.21.15.29:0/2559402384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde881a3b30 con 0x7fde881025a0 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde8da87640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde88103620 0x7fde881a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde8e288640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 0x7fde8806b390 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fde84009230 tx=0x7fde840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde7f7fe640 1 -- 172.21.15.29:0/2559402384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8403f030 con 0x7fde881025a0 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde904ec640 1 -- 172.21.15.29:0/2559402384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde881a3dc0 con 0x7fde881025a0 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde7f7fe640 1 -- 172.21.15.29:0/2559402384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde84032e20 con 0x7fde881025a0 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde7f7fe640 1 -- 172.21.15.29:0/2559402384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84035070 con 0x7fde881025a0 2024-09-21T02:43:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.649+0000 7fde904ec640 1 -- 172.21.15.29:0/2559402384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde881a4220 con 0x7fde881025a0 2024-09-21T02:43:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.652+0000 7fde97459740 1 Processor -- start 2024-09-21T02:43:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.652+0000 7fde97459740 1 -- start start 2024-09-21T02:43:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde97459740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde97459740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde7f7fe640 1 -- 172.21.15.29:0/2559402384 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fde84042030 con 0x7fde881025a0 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde97459740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4aa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde97459740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a3dd41b6f0 con 0x55a3dd1011b0 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde97459740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a3dd40ab20 con 0x55a3dd0d82f0 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde97459740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a3dd40e930 con 0x55a3dd0e04d0 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8ea89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8e288640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8da87640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8da87640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4aa010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49978/0 (socket says 172.21.15.29:49978) 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8ea89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35774/0 (socket says 172.21.15.29:35774) 2024-09-21T02:43:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8da87640 1 -- 172.21.15.29:0/2509380535 learned_addr learned my addr 172.21.15.29:0/2509380535 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8da87640 1 -- 172.21.15.29:0/2509380535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 msgr2=0x55a3dd4a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8da87640 1 --2- 172.21.15.29:0/2509380535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8da87640 1 -- 172.21.15.29:0/2509380535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 msgr2=0x55a3dd4a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8da87640 1 --2- 172.21.15.29:0/2509380535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.653+0000 7fde8da87640 1 -- 172.21.15.29:0/2509380535 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3dd3d0530 con 0x55a3dd0e04d0 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.654+0000 7fde8da87640 1 --2- 172.21.15.29:0/2509380535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4aa010 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fde78010580 tx=0x7fde78010a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=912215eb651438fd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.654+0000 7fde7f7fe640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fde740b6510 0x7fde740b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.654+0000 7fde7cbfe640 1 -- 172.21.15.29:0/2509380535 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde78011760 con 0x55a3dd0e04d0 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.654+0000 7fde8ea89640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fde740b6510 0x7fde740b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.654+0000 7fde7cbfe640 1 -- 172.21.15.29:0/2509380535 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde78011d90 con 0x55a3dd0e04d0 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.654+0000 7fde7f7fe640 1 -- 172.21.15.29:0/2559402384 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fde84107be0 con 0x7fde881025a0 2024-09-21T02:43:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.654+0000 7fde7cbfe640 1 -- 172.21.15.29:0/2509380535 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7800ed90 con 0x55a3dd0e04d0 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 -- 172.21.15.29:0/2509380535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 msgr2=0x55a3dd4aa010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 --2- 172.21.15.29:0/2509380535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4aa010 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fde78010580 tx=0x7fde78010a40 comp rx=0 tx=0).stop 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde8ea89640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fde740b6510 0x7fde740b8a20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fde80011550 tx=0x7fde800094e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 -- 172.21.15.29:0/2509380535 shutdown_connections 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 --2- 172.21.15.29:0/2509380535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4aa010 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 --2- 172.21.15.29:0/2509380535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 --2- 172.21.15.29:0/2509380535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 -- 172.21.15.29:0/2509380535 >> 172.21.15.29:0/2509380535 conn(0x55a3dd0f5300 msgr2=0x55a3dd4a48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 -- 172.21.15.29:0/2509380535 shutdown_connections 2024-09-21T02:43:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 -- 172.21.15.29:0/2509380535 wait complete. 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 -- start start 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.655+0000 7fde97459740 1 Processor -- start 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde97459740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde97459740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde97459740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde97459740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a3dd342ab0 con 0x55a3dd0e04d0 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde97459740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a3dd3b0730 con 0x55a3dd0d82f0 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde97459740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a3dd4aab30 con 0x55a3dd1011b0 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde97459740 10 client.0 _ll_register_callbacks cb 0x55a3dd4046b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35786/0 (socket says 172.21.15.29:35786) 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 -- 172.21.15.29:0/2940726752 learned_addr learned my addr 172.21.15.29:0/2940726752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8ea89640 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8e288640 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 -- 172.21.15.29:0/2940726752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 msgr2=0x55a3dd4b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 -- 172.21.15.29:0/2940726752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 msgr2=0x55a3dd4b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 -- 172.21.15.29:0/2940726752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3dd16c790 con 0x55a3dd0e04d0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8e288640 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde8da87640 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fde78000c00 tx=0x7fde7800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70afe32b14186743 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.656+0000 7fde71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde71ffb640 1 -- 172.21.15.29:0/2940726752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7802b060 con 0x55a3dd0e04d0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde71ffb640 1 -- 172.21.15.29:0/2940726752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fde7800ed20 con 0x55a3dd0e04d0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 1 -- 172.21.15.29:0/2940726752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde78011d90 con 0x55a3dd0e04d0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde71ffb640 1 -- 172.21.15.29:0/2940726752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde78032050 con 0x55a3dd0e04d0 2024-09-21T02:43:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 10 client.16194 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 1 -- 172.21.15.29:0/2940726752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a3dd1e5620 con 0x55a3dd0e04d0 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde71ffb640 1 -- 172.21.15.29:0/2940726752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde78011d90 con 0x55a3dd0e04d0 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde71ffb640 1 -- 172.21.15.29:0/2940726752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fde7801c740 con 0x55a3dd0e04d0 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 10 client.16194 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 20 client.16194 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3dd0bfd80 0x55a3dd4bfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 4 client.16194 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.657+0000 7fde97459740 1 -- 172.21.15.29:0/2940726752 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3dd41d640 con 0x55a3dd0bfd80 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.658+0000 7fde8e288640 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3dd0bfd80 0x55a3dd4bfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.658+0000 7fde8e288640 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3dd0bfd80 0x55a3dd4bfa50 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=157ce2c59e24f548 server_cookie=a6f3407dbc539537 in_seq=0 out_seq=0 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.658+0000 7fde71ffb640 10 client.16194 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde71ffb640 1 -- 172.21.15.29:0/2940726752 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a3dd41d640 con 0x55a3dd0bfd80 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde71ffb640 10 client.16194 handle_command_reply: tid=0 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 client.16194 shutdown 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 -- 172.21.15.29:0/2940726752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 msgr2=0x55a3dd4b2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fde78000c00 tx=0x7fde7800be70 comp rx=0 tx=0).stop 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 -- 172.21.15.29:0/2940726752 shutdown_connections 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3dd0bfd80 0x55a3dd4bfa50 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2c80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 --2- 172.21.15.29:0/2940726752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde71ffb640 0 client.16194 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 -- 172.21.15.29:0/2940726752 >> 172.21.15.29:0/2940726752 conn(0x55a3dd0f5300 msgr2=0x55a3dd4a2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 -- 172.21.15.29:0/2940726752 shutdown_connections 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 1 -- 172.21.15.29:0/2940726752 wait complete. 2024-09-21T02:43:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.659+0000 7fde97459740 20 client.16194 trim_cache size 0 max 16384 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.676+0000 7fde97459740 1 Processor -- start 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.676+0000 7fde97459740 1 -- start start 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.676+0000 7fde97459740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.676+0000 7fde97459740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.676+0000 7fde97459740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.676+0000 7fde97459740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fde881ae7f0 con 0x55a3dd0e04d0 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.676+0000 7fde97459740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fde881b0330 con 0x55a3dd0d82f0 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.676+0000 7fde97459740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fde881aeec0 con 0x55a3dd1011b0 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35796/0 (socket says 172.21.15.29:35796) 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 -- 172.21.15.29:0/1285126673 learned_addr learned my addr 172.21.15.29:0/1285126673 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8ea89640 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8e288640 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 -- 172.21.15.29:0/1285126673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 msgr2=0x55a3dd4a8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 -- 172.21.15.29:0/1285126673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 msgr2=0x55a3dd4b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 -- 172.21.15.29:0/1285126673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3dd4a3fd0 con 0x55a3dd0e04d0 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde8da87640 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2b00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fde7800bd20 tx=0x7fde7800bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab350b9da27e42d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde73fff640 1 -- 172.21.15.29:0/1285126673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7802b040 con 0x55a3dd0e04d0 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde73fff640 1 -- 172.21.15.29:0/1285126673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde7801c910 con 0x55a3dd0e04d0 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.677+0000 7fde73fff640 1 -- 172.21.15.29:0/1285126673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde78005280 con 0x55a3dd0e04d0 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 -- 172.21.15.29:0/1285126673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 msgr2=0x55a3dd4b2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2b00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fde7800bd20 tx=0x7fde7800bf70 comp rx=0 tx=0).stop 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 -- 172.21.15.29:0/1285126673 shutdown_connections 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd1011b0 0x55a3dd4a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b2b00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 --2- 172.21.15.29:0/1285126673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 -- 172.21.15.29:0/1285126673 >> 172.21.15.29:0/1285126673 conn(0x55a3dd0f5300 msgr2=0x55a3dd49f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 -- 172.21.15.29:0/1285126673 shutdown_connections 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.678+0000 7fde97459740 1 -- 172.21.15.29:0/1285126673 wait complete. 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 1 -- start start 2024-09-21T02:43:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 1 Processor -- start 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fde881afc60 con 0x55a3dd1011b0 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fde881af590 con 0x55a3dd0d82f0 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fde881ac7d0 con 0x55a3dd0e04d0 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde97459740 10 client.0 _ll_register_callbacks cb 0x7fde80009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8ea89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8da87640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8ea89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35808/0 (socket says 172.21.15.29:35808) 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8ea89640 1 -- 172.21.15.29:0/3321685062 learned_addr learned my addr 172.21.15.29:0/3321685062 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8e288640 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8da87640 1 -- 172.21.15.29:0/3321685062 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 msgr2=0x55a3dd4b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8da87640 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8da87640 1 -- 172.21.15.29:0/3321685062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 msgr2=0x55a3dd4b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8da87640 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8da87640 1 -- 172.21.15.29:0/3321685062 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3dd4b9010 con 0x55a3dd0e04d0 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.679+0000 7fde8ea89640 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde8da87640 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b7d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fde7800b4f0 tx=0x7fde7801cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a370361baa38377 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde71ffb640 1 -- 172.21.15.29:0/3321685062 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7802b070 con 0x55a3dd0e04d0 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde97459740 1 -- 172.21.15.29:0/3321685062 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde7801c910 con 0x55a3dd0e04d0 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde71ffb640 1 -- 172.21.15.29:0/3321685062 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fde7801c480 con 0x55a3dd0e04d0 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde71ffb640 1 -- 172.21.15.29:0/3321685062 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde78048050 con 0x55a3dd0e04d0 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde97459740 10 client.25822 fetch_fsmap learned FSMap version 31 2024-09-21T02:43:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde97459740 1 -- 172.21.15.29:0/3321685062 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a3dd4ba3f0 con 0x55a3dd0e04d0 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde71ffb640 1 -- 172.21.15.29:0/3321685062 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde7801c910 con 0x55a3dd0e04d0 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde71ffb640 1 -- 172.21.15.29:0/3321685062 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fde78011bd0 con 0x55a3dd0e04d0 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde97459740 10 client.25822 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde97459740 10 client.25822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde97459740 20 client.25822 populate_metadata read hostname 'smithi029' 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.680+0000 7fde97459740 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3dd0951c0 0x55a3dd52c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.681+0000 7fde97459740 4 client.25822 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.681+0000 7fde97459740 1 -- 172.21.15.29:0/3321685062 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a3dd4190f0 con 0x55a3dd0951c0 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.681+0000 7fde8e288640 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3dd0951c0 0x55a3dd52c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.681+0000 7fde71ffb640 10 client.25822 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.681+0000 7fde8e288640 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3dd0951c0 0x55a3dd52c4c0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=badf639cd48eef42 server_cookie=9e41315a9b1c432e in_seq=0 out_seq=0 2024-09-21T02:43:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.727+0000 7fde71ffb640 1 -- 172.21.15.29:0/3321685062 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a3dd4190f0 con 0x55a3dd0951c0 2024-09-21T02:43:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.727+0000 7fde71ffb640 10 client.25822 handle_command_reply: tid=0 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 client.25822 shutdown 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 -- 172.21.15.29:0/3321685062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 msgr2=0x55a3dd4b7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b7d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fde7800b4f0 tx=0x7fde7801cc10 comp rx=0 tx=0).stop 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 -- 172.21.15.29:0/3321685062 shutdown_connections 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3dd0951c0 0x55a3dd52c4c0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3dd1011b0 0x55a3dd4b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde71ffb640 0 client.25822 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3dd0e04d0 0x55a3dd4b7d90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 --2- 172.21.15.29:0/3321685062 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3dd0d82f0 0x55a3dd4b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 -- 172.21.15.29:0/3321685062 >> 172.21.15.29:0/3321685062 conn(0x55a3dd0f5300 msgr2=0x55a3dd4a8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 -- 172.21.15.29:0/3321685062 shutdown_connections 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 1 -- 172.21.15.29:0/3321685062 wait complete. 2024-09-21T02:43:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.728+0000 7fde97459740 20 client.25822 trim_cache size 0 max 16384 2024-09-21T02:43:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:43:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:43:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:43:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 -- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fde740b6510 msgr2=0x7fde740b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fde740b6510 0x7fde740b8a20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fde80011550 tx=0x7fde800094e0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 -- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 msgr2=0x7fde8806b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 0x7fde8806b390 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fde84009230 tx=0x7fde840042f0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 -- 172.21.15.29:0/2559402384 shutdown_connections 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fde740b6510 0x7fde740b8a20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fde88103fc0 0x7fde881a3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde88103620 0x7fde881a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 --2- 172.21.15.29:0/2559402384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde881025a0 0x7fde8806b390 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 -- 172.21.15.29:0/2559402384 >> 172.21.15.29:0/2559402384 conn(0x7fde880fdee0 msgr2=0x7fde88106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:43:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 -- 172.21.15.29:0/2559402384 shutdown_connections 2024-09-21T02:43:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:43:54.729+0000 7fde904ec640 1 -- 172.21.15.29:0/2559402384 wait complete. 2024-09-21T02:43:55.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 81940 3.06 MB/sec execute 92 sec latency 39.264 ms 2024-09-21T02:43:55.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:55 smithi066 ceph-mon[33436]: pgmap v533: 129 pgs: 129 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-09-21T02:43:55.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:55 smithi190 ceph-mon[33871]: pgmap v533: 129 pgs: 129 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-09-21T02:43:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:55 smithi029 ceph-mon[31006]: pgmap v533: 129 pgs: 129 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-09-21T02:43:56.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 82237 3.03 MB/sec execute 93 sec latency 17.807 ms 2024-09-21T02:43:57.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 82851 3.02 MB/sec execute 94 sec latency 14.425 ms 2024-09-21T02:43:57.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:57 smithi066 ceph-mon[33436]: pgmap v534: 129 pgs: 129 active+clean; 34 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.6 MiB/s wr, 119 op/s 2024-09-21T02:43:57.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:57 smithi190 ceph-mon[33871]: pgmap v534: 129 pgs: 129 active+clean; 34 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.6 MiB/s wr, 119 op/s 2024-09-21T02:43:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:57 smithi029 ceph-mon[31006]: pgmap v534: 129 pgs: 129 active+clean; 34 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 1.6 MiB/s wr, 119 op/s 2024-09-21T02:43:58.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 83510 3.02 MB/sec execute 95 sec latency 16.688 ms 2024-09-21T02:43:59.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 84138 3.01 MB/sec execute 96 sec latency 19.849 ms 2024-09-21T02:43:59.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:43:59 smithi066 ceph-mon[33436]: pgmap v535: 129 pgs: 129 active+clean; 31 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:43:59.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:43:59 smithi190 ceph-mon[33871]: pgmap v535: 129 pgs: 129 active+clean; 31 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:43:59.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:43:59 smithi029 ceph-mon[31006]: pgmap v535: 129 pgs: 129 active+clean; 31 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:44:00.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 84945 3.05 MB/sec execute 97 sec latency 15.977 ms 2024-09-21T02:44:01.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 85445 3.06 MB/sec execute 98 sec latency 13.381 ms 2024-09-21T02:44:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:01 smithi066 ceph-mon[33436]: pgmap v536: 129 pgs: 129 active+clean; 31 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T02:44:01.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:01 smithi190 ceph-mon[33871]: pgmap v536: 129 pgs: 129 active+clean; 31 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T02:44:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:01 smithi029 ceph-mon[31006]: pgmap v536: 129 pgs: 129 active+clean; 31 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T02:44:02.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 85702 3.04 MB/sec execute 99 sec latency 15.864 ms 2024-09-21T02:44:03.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 86285 3.03 MB/sec execute 100 sec latency 26.922 ms 2024-09-21T02:44:03.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:03 smithi066 ceph-mon[33436]: pgmap v537: 129 pgs: 129 active+clean; 32 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-09-21T02:44:03.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:03 smithi190 ceph-mon[33871]: pgmap v537: 129 pgs: 129 active+clean; 32 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-09-21T02:44:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:03 smithi029 ceph-mon[31006]: pgmap v537: 129 pgs: 129 active+clean; 32 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-09-21T02:44:04.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 86940 3.03 MB/sec execute 101 sec latency 20.259 ms 2024-09-21T02:44:05.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 87569 3.02 MB/sec execute 102 sec latency 18.995 ms 2024-09-21T02:44:05.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:05 smithi066 ceph-mon[33436]: pgmap v538: 129 pgs: 129 active+clean; 32 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-09-21T02:44:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:05 smithi190 ceph-mon[33871]: pgmap v538: 129 pgs: 129 active+clean; 32 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-09-21T02:44:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:05 smithi029 ceph-mon[31006]: pgmap v538: 129 pgs: 129 active+clean; 32 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-09-21T02:44:06.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 88466 3.06 MB/sec execute 103 sec latency 13.470 ms 2024-09-21T02:44:07.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 88955 3.07 MB/sec execute 104 sec latency 14.389 ms 2024-09-21T02:44:07.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:07 smithi066 ceph-mon[33436]: pgmap v539: 129 pgs: 129 active+clean; 32 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.834+0000 7f4136bed640 1 -- 172.21.15.29:0/1241300742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41301025a0 msgr2=0x7f4130102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.834+0000 7f4136bed640 1 --2- 172.21.15.29:0/1241300742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41301025a0 0x7f4130102a10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4124009900 tx=0x7f412402a410 comp rx=0 tx=0).stop 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.834+0000 7f4136bed640 1 -- 172.21.15.29:0/1241300742 shutdown_connections 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.834+0000 7f4136bed640 1 --2- 172.21.15.29:0/1241300742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4130103fc0 0x7f413010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.834+0000 7f4136bed640 1 --2- 172.21.15.29:0/1241300742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4130103620 0x7f4130103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.834+0000 7f4136bed640 1 --2- 172.21.15.29:0/1241300742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41301025a0 0x7f4130102a10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.835+0000 7f4136bed640 1 -- 172.21.15.29:0/1241300742 >> 172.21.15.29:0/1241300742 conn(0x7f41300fdee0 msgr2=0x7f4130100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.835+0000 7f4136bed640 1 -- 172.21.15.29:0/1241300742 shutdown_connections 2024-09-21T02:44:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.835+0000 7f4136bed640 1 -- 172.21.15.29:0/1241300742 wait complete. 2024-09-21T02:44:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.835+0000 7f4136bed640 1 Processor -- start 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.835+0000 7f4136bed640 1 -- start start 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.835+0000 7f4136bed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41301025a0 0x7f4130106190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4136bed640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4130103620 0x7f4130104a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4136bed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130103fc0 0x7f4130104f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4136bed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f413010e080 con 0x7f4130103fc0 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4136bed640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f413010df00 con 0x7f4130103620 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4136bed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f413010e200 con 0x7f41301025a0 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f413518a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130103fc0 0x7f4130104f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41301025a0 0x7f4130106190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41301025a0 0x7f4130106190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49428/0 (socket says 172.21.15.29:49428) 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f413518a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130103fc0 0x7f4130104f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33886/0 (socket says 172.21.15.29:33886) 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 -- 172.21.15.29:0/2838580842 learned_addr learned my addr 172.21.15.29:0/2838580842 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f412ffff640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4130103620 0x7f4130104a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 -- 172.21.15.29:0/2838580842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4130103620 msgr2=0x7f4130104a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4130103620 0x7f4130104a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 -- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130103fc0 msgr2=0x7f4130104f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130103fc0 0x7f4130104f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 -- 172.21.15.29:0/2838580842 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41301056c0 con 0x7f41301025a0 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f413518a640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130103fc0 0x7f4130104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.836+0000 7f4134989640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41301025a0 0x7f4130106190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4124009230 tx=0x7f4124004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.837+0000 7f412dffb640 1 -- 172.21.15.29:0/2838580842 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412403f030 con 0x7f41301025a0 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.837+0000 7f4136bed640 1 -- 172.21.15.29:0/2838580842 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41301a3ee0 con 0x7f41301025a0 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.837+0000 7f412dffb640 1 -- 172.21.15.29:0/2838580842 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4124004440 con 0x7f41301025a0 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.837+0000 7f412dffb640 1 -- 172.21.15.29:0/2838580842 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124035070 con 0x7f41301025a0 2024-09-21T02:44:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.837+0000 7f4136bed640 1 -- 172.21.15.29:0/2838580842 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41301a4290 con 0x7f41301025a0 2024-09-21T02:44:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413dc36740 1 Processor -- start 2024-09-21T02:44:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413dc36740 1 -- start start 2024-09-21T02:44:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f412dffb640 1 -- 172.21.15.29:0/2838580842 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4124042030 con 0x7f41301025a0 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413dc36740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb6ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413dc36740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 0x56148cb6d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413dc36740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb6e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413dc36740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56148cadf6f0 con 0x56148c7a44d0 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413dc36740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56148caceb20 con 0x56148c7c51b0 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413dc36740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56148cad2930 con 0x56148c79c2f0 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413518a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb6ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f412ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb6e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413518a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb6ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:36812/0 (socket says 172.21.15.29:36812) 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f413518a640 1 -- 172.21.15.29:0/3001647380 learned_addr learned my addr 172.21.15.29:0/3001647380 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.840+0000 7f4134989640 1 --2- 172.21.15.29:0/3001647380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 0x56148cb6d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f413518a640 1 -- 172.21.15.29:0/3001647380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 msgr2=0x56148cb6d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f413518a640 1 --2- 172.21.15.29:0/3001647380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 0x56148cb6d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f413518a640 1 -- 172.21.15.29:0/3001647380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 msgr2=0x56148cb6e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f413518a640 1 --2- 172.21.15.29:0/3001647380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb6e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f413518a640 1 -- 172.21.15.29:0/3001647380 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56148ca94530 con 0x56148c7c51b0 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f412dffb640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41200b6510 0x7f41200b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f413518a640 1 --2- 172.21.15.29:0/3001647380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb6ce50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4128011820 tx=0x7f4128011ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed0b94d17283fe58 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f4134989640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41200b6510 0x7f41200b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f412dffb640 1 -- 172.21.15.29:0/2838580842 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4124107ae0 con 0x7f41301025a0 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f411affd640 1 -- 172.21.15.29:0/3001647380 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128011330 con 0x56148c7c51b0 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f411affd640 1 -- 172.21.15.29:0/3001647380 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f412800fe30 con 0x56148c7c51b0 2024-09-21T02:44:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.841+0000 7f411affd640 1 -- 172.21.15.29:0/3001647380 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128010430 con 0x56148c7c51b0 2024-09-21T02:44:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 -- 172.21.15.29:0/3001647380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 msgr2=0x56148cb6ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 --2- 172.21.15.29:0/3001647380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb6ce50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4128011820 tx=0x7f4128011ce0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f4134989640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41200b6510 0x7f41200b8a20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f41240cac10 tx=0x7f412410b5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 -- 172.21.15.29:0/3001647380 shutdown_connections 2024-09-21T02:44:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 --2- 172.21.15.29:0/3001647380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb6e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 --2- 172.21.15.29:0/3001647380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 0x56148cb6d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 --2- 172.21.15.29:0/3001647380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb6ce50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 -- 172.21.15.29:0/3001647380 >> 172.21.15.29:0/3001647380 conn(0x56148c7b9300 msgr2=0x56148cb688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 -- 172.21.15.29:0/3001647380 shutdown_connections 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.842+0000 7f413dc36740 1 -- 172.21.15.29:0/3001647380 wait complete. 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 1 -- start start 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 1 Processor -- start 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 0x56148cb76610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb76cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb77840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56148ca06ab0 con 0x56148c7a44d0 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56148ca74730 con 0x56148c7c51b0 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56148cb6eb30 con 0x56148c79c2f0 2024-09-21T02:44:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413dc36740 10 client.0 _ll_register_callbacks cb 0x56148cac86b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f412ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb76cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f412ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb76cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33904/0 (socket says 172.21.15.29:33904) 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f412ffff640 1 -- 172.21.15.29:0/888975276 learned_addr learned my addr 172.21.15.29:0/888975276 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413518a640 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 0x56148cb76610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.843+0000 7f413518a640 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb77840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f412ffff640 1 -- 172.21.15.29:0/888975276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 msgr2=0x56148cb76610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f412ffff640 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 0x56148cb76610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f412ffff640 1 -- 172.21.15.29:0/888975276 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 msgr2=0x56148cb77840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f412ffff640 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb77840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f412ffff640 1 -- 172.21.15.29:0/888975276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56148c830790 con 0x56148c7a44d0 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f412ffff640 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb76cd0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f411c012740 tx=0x7f411c012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171ce371f6bd6e50 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f410ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f410ffff640 1 -- 172.21.15.29:0/888975276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c00f920 con 0x56148c7a44d0 2024-09-21T02:44:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f410ffff640 1 -- 172.21.15.29:0/888975276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f411c00fac0 con 0x56148c7a44d0 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f413dc36740 1 -- 172.21.15.29:0/888975276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f412800fe30 con 0x56148c7a44d0 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f410ffff640 1 -- 172.21.15.29:0/888975276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f411c00fcc0 con 0x56148c7a44d0 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f413dc36740 10 client.16212 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.844+0000 7f413dc36740 1 -- 172.21.15.29:0/888975276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56148c8a9620 con 0x56148c7a44d0 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f410ffff640 1 -- 172.21.15.29:0/888975276 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f412800fe30 con 0x56148c7a44d0 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f410ffff640 1 -- 172.21.15.29:0/888975276 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f411c01c420 con 0x56148c7a44d0 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f413dc36740 10 client.16212 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f413dc36740 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f413dc36740 20 client.16212 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f413dc36740 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56148c783d80 0x56148cb83e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f413dc36740 4 client.16212 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f413dc36740 1 -- 172.21.15.29:0/888975276 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56148cae1640 con 0x56148c783d80 2024-09-21T02:44:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f413518a640 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56148c783d80 0x56148cb83e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f410ffff640 10 client.16212 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.845+0000 7f413518a640 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56148c783d80 0x56148cb83e90 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aba24f32ab747957 server_cookie=248fd309aa91860c in_seq=0 out_seq=0 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f410ffff640 1 -- 172.21.15.29:0/888975276 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56148cae1640 con 0x56148c783d80 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f410ffff640 10 client.16212 handle_command_reply: tid=0 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 client.16212 shutdown 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 -- 172.21.15.29:0/888975276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 msgr2=0x56148cb76cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb76cd0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f411c012740 tx=0x7f411c012c00 comp rx=0 tx=0).stop 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 -- 172.21.15.29:0/888975276 shutdown_connections 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7c51b0 0x56148cb77840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56148c783d80 0x56148cb83e90 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb76cd0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 --2- 172.21.15.29:0/888975276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c79c2f0 0x56148cb76610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f410ffff640 0 client.16212 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 -- 172.21.15.29:0/888975276 >> 172.21.15.29:0/888975276 conn(0x56148c7b9300 msgr2=0x56148cb66ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 -- 172.21.15.29:0/888975276 shutdown_connections 2024-09-21T02:44:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.846+0000 7f413dc36740 1 -- 172.21.15.29:0/888975276 wait complete. 2024-09-21T02:44:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.847+0000 7f413dc36740 20 client.16212 trim_cache size 0 max 16384 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.862+0000 7f413dc36740 1 Processor -- start 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413dc36740 1 -- start start 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413dc36740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c79c2f0 0x56148cb68220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413dc36740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb75830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413dc36740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 0x56148cb74b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413dc36740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f41301ae720 con 0x56148c7a44d0 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413dc36740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f41301b0260 con 0x56148c79c2f0 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413dc36740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f41301aedf0 con 0x56148c7c51b0 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f412ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb75830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f412ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb75830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33906/0 (socket says 172.21.15.29:33906) 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f412ffff640 1 -- 172.21.15.29:0/2816997453 learned_addr learned my addr 172.21.15.29:0/2816997453 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413518a640 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 0x56148cb74b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f413518a640 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c79c2f0 0x56148cb68220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f412ffff640 1 -- 172.21.15.29:0/2816997453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 msgr2=0x56148cb74b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f412ffff640 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 0x56148cb74b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f412ffff640 1 -- 172.21.15.29:0/2816997453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c79c2f0 msgr2=0x56148cb68220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f412ffff640 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c79c2f0 0x56148cb68220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.863+0000 7f412ffff640 1 -- 172.21.15.29:0/2816997453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56148cb7eae0 con 0x56148c7a44d0 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.864+0000 7f412ffff640 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb75830 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f411c005bc0 tx=0x7f411c00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ffc10cafb98a28 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.864+0000 7f411a7fc640 1 -- 172.21.15.29:0/2816997453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c00f920 con 0x56148c7a44d0 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.864+0000 7f411a7fc640 1 -- 172.21.15.29:0/2816997453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f411c012e50 con 0x56148c7a44d0 2024-09-21T02:44:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.864+0000 7f411a7fc640 1 -- 172.21.15.29:0/2816997453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c005340 con 0x56148c7a44d0 2024-09-21T02:44:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 -- 172.21.15.29:0/2816997453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 msgr2=0x56148cb75830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb75830 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f411c005bc0 tx=0x7f411c00f850 comp rx=0 tx=0).stop 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 -- 172.21.15.29:0/2816997453 shutdown_connections 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 0x56148cb74b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c7a44d0 0x56148cb75830 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 --2- 172.21.15.29:0/2816997453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c79c2f0 0x56148cb68220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 -- 172.21.15.29:0/2816997453 >> 172.21.15.29:0/2816997453 conn(0x56148c7b9300 msgr2=0x56148cb6de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 -- 172.21.15.29:0/2816997453 shutdown_connections 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 -- 172.21.15.29:0/2816997453 wait complete. 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 -- start start 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.865+0000 7f413dc36740 1 Processor -- start 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413dc36740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c79c2f0 0x56148cb7de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413dc36740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7a44d0 0x56148cb7e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413dc36740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 0x56148cb7cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413dc36740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f41301afb90 con 0x56148c79c2f0 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413dc36740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f41301af4c0 con 0x56148c7a44d0 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413dc36740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f41301ac700 con 0x56148c7c51b0 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413dc36740 10 client.0 _ll_register_callbacks cb 0x7f411c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c79c2f0 0x56148cb7de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c79c2f0 0x56148cb7de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33916/0 (socket says 172.21.15.29:33916) 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 -- 172.21.15.29:0/3833246347 learned_addr learned my addr 172.21.15.29:0/3833246347 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 0x56148cb7cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f412ffff640 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7a44d0 0x56148cb7e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 -- 172.21.15.29:0/3833246347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 msgr2=0x56148cb7cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 0x56148cb7cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 -- 172.21.15.29:0/3833246347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7a44d0 msgr2=0x56148cb7e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7a44d0 0x56148cb7e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 -- 172.21.15.29:0/3833246347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56148cb7d3c0 con 0x56148c79c2f0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f413518a640 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c79c2f0 0x56148cb7de30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f412800f890 tx=0x7f411c009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30908a1800a7bbb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.866+0000 7f410ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f410ffff640 1 -- 172.21.15.29:0/3833246347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128024040 con 0x56148c79c2f0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f410ffff640 1 -- 172.21.15.29:0/3833246347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f412801a480 con 0x56148c79c2f0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 1 -- 172.21.15.29:0/3833246347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f411c012e50 con 0x56148c79c2f0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f410ffff640 1 -- 172.21.15.29:0/3833246347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f412801a680 con 0x56148c79c2f0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 10 client.16218 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 1 -- 172.21.15.29:0/3833246347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56148cb7c2c0 con 0x56148c79c2f0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f410ffff640 1 -- 172.21.15.29:0/3833246347 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f411c012e50 con 0x56148c79c2f0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f410ffff640 1 -- 172.21.15.29:0/3833246347 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f412802a040 con 0x56148c79c2f0 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 10 client.16218 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:44:07.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 20 client.16218 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:07.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56148c7591c0 0x56148cbf0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:07.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 4 client.16218 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:44:07.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.867+0000 7f413dc36740 1 -- 172.21.15.29:0/3833246347 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56148cadd0f0 con 0x56148c7591c0 2024-09-21T02:44:07.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.868+0000 7f412ffff640 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56148c7591c0 0x56148cbf0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:07.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.868+0000 7f410ffff640 10 client.16218 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:07.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.868+0000 7f412ffff640 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56148c7591c0 0x56148cbf0c90 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6b45ea2bc93dec8 server_cookie=47d6c72b2c01e3b7 in_seq=0 out_seq=0 2024-09-21T02:44:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:07 smithi190 ceph-mon[33871]: pgmap v539: 129 pgs: 129 active+clean; 32 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:44:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:07 smithi029 ceph-mon[31006]: pgmap v539: 129 pgs: 129 active+clean; 32 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:44:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.915+0000 7f410ffff640 1 -- 172.21.15.29:0/3833246347 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56148cadd0f0 con 0x56148c7591c0 2024-09-21T02:44:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.915+0000 7f410ffff640 10 client.16218 handle_command_reply: tid=0 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.915+0000 7f413dc36740 1 client.16218 shutdown 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 -- 172.21.15.29:0/3833246347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c79c2f0 msgr2=0x56148cb7de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c79c2f0 0x56148cb7de30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f412800f890 tx=0x7f411c009e70 comp rx=0 tx=0).stop 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 -- 172.21.15.29:0/3833246347 shutdown_connections 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56148c7c51b0 0x56148cb7cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56148c7a44d0 0x56148cb7e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56148c7591c0 0x56148cbf0c90 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 --2- 172.21.15.29:0/3833246347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56148c79c2f0 0x56148cb7de30 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f410ffff640 0 client.16218 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 -- 172.21.15.29:0/3833246347 >> 172.21.15.29:0/3833246347 conn(0x56148c7b9300 msgr2=0x56148cb75830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 -- 172.21.15.29:0/3833246347 shutdown_connections 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 1 -- 172.21.15.29:0/3833246347 wait complete. 2024-09-21T02:44:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.916+0000 7f413dc36740 20 client.16218 trim_cache size 0 max 16384 2024-09-21T02:44:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:44:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:44:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:44:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 -- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41200b6510 msgr2=0x7f41200b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41200b6510 0x7f41200b8a20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f41240cac10 tx=0x7f412410b5c0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 -- 172.21.15.29:0/2838580842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41301025a0 msgr2=0x7f4130106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41301025a0 0x7f4130106190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4124009230 tx=0x7f4124004040 comp rx=0 tx=0).stop 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 -- 172.21.15.29:0/2838580842 shutdown_connections 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41200b6510 0x7f41200b8a20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130103fc0 0x7f4130104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4130103620 0x7f4130104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 --2- 172.21.15.29:0/2838580842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41301025a0 0x7f4130106190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 -- 172.21.15.29:0/2838580842 >> 172.21.15.29:0/2838580842 conn(0x7f41300fdee0 msgr2=0x7f41300ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 -- 172.21.15.29:0/2838580842 shutdown_connections 2024-09-21T02:44:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:07.917+0000 7f4136bed640 1 -- 172.21.15.29:0/2838580842 wait complete. 2024-09-21T02:44:08.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 89191 3.04 MB/sec execute 105 sec latency 22.526 ms 2024-09-21T02:44:09.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 89763 3.03 MB/sec execute 106 sec latency 16.758 ms 2024-09-21T02:44:09.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:09 smithi066 ceph-mon[33436]: pgmap v540: 129 pgs: 129 active+clean; 32 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-09-21T02:44:09.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:09 smithi190 ceph-mon[33871]: pgmap v540: 129 pgs: 129 active+clean; 32 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-09-21T02:44:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:09 smithi029 ceph-mon[31006]: pgmap v540: 129 pgs: 129 active+clean; 32 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-09-21T02:44:10.061 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-09-21T02:44:10.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:44:10.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 90378 3.04 MB/sec execute 107 sec latency 17.319 ms 2024-09-21T02:44:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:44:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 -- 172.21.15.29:0/1228599770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 msgr2=0x7fc1481085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/1228599770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 0x7fc1481085c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009900 tx=0x7fc13c02a410 comp rx=0 tx=0).stop 2024-09-21T02:44:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 -- 172.21.15.29:0/1228599770 shutdown_connections 2024-09-21T02:44:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/1228599770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc148108b00 0x7fc14810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/1228599770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 0x7fc1481085c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/1228599770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc148102870 0x7fc148102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 -- 172.21.15.29:0/1228599770 >> 172.21.15.29:0/1228599770 conn(0x7fc1480fe2c0 msgr2=0x7fc148100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 -- 172.21.15.29:0/1228599770 shutdown_connections 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.887+0000 7fc14c9a0640 1 -- 172.21.15.29:0/1228599770 wait complete. 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc14c9a0640 1 Processor -- start 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc14c9a0640 1 -- start start 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc14c9a0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc148102870 0x7fc14819afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc14c9a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 0x7fc14819b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc14c9a0640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc148108b00 0x7fc14819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc14c9a0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc148112640 con 0x7fc1481081b0 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc14c9a0640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481124c0 con 0x7fc148108b00 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc14c9a0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481127c0 con 0x7fc148102870 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.888+0000 7fc146ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 0x7fc14819b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc1477fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc148102870 0x7fc14819afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc146ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 0x7fc14819b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33930/0 (socket says 172.21.15.29:33930) 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc146ffd640 1 -- 172.21.15.29:0/2191408562 learned_addr learned my addr 172.21.15.29:0/2191408562 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc147fff640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc148108b00 0x7fc14819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc146ffd640 1 -- 172.21.15.29:0/2191408562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc148102870 msgr2=0x7fc14819afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc146ffd640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc148102870 0x7fc14819afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc146ffd640 1 -- 172.21.15.29:0/2191408562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc148108b00 msgr2=0x7fc14819ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc146ffd640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc148108b00 0x7fc14819ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc146ffd640 1 -- 172.21.15.29:0/2191408562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1481a6290 con 0x7fc1481081b0 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc1477fe640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc148102870 0x7fc14819afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc146ffd640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 0x7fc14819b510 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc13c004770 tx=0x7fc13c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc144ff9640 1 -- 172.21.15.29:0/2191408562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c03f030 con 0x7fc1481081b0 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc144ff9640 1 -- 172.21.15.29:0/2191408562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc13c004440 con 0x7fc1481081b0 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1481a6520 con 0x7fc1481081b0 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.889+0000 7fc144ff9640 1 -- 172.21.15.29:0/2191408562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c035070 con 0x7fc1481081b0 2024-09-21T02:44:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.890+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1481a6a00 con 0x7fc1481081b0 2024-09-21T02:44:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.891+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc148102c80 con 0x7fc1481081b0 2024-09-21T02:44:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.892+0000 7fc144ff9640 1 -- 172.21.15.29:0/2191408562 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc13c042030 con 0x7fc1481081b0 2024-09-21T02:44:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.893+0000 7fc144ff9640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1300b6510 0x7fc1300b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.893+0000 7fc1477fe640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1300b6510 0x7fc1300b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.893+0000 7fc144ff9640 1 -- 172.21.15.29:0/2191408562 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc13c107980 con 0x7fc1481081b0 2024-09-21T02:44:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.894+0000 7fc1477fe640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1300b6510 0x7fc1300b8a20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc14806b9e0 tx=0x7fc13800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:10.898+0000 7fc144ff9640 1 -- 172.21.15.29:0/2191408562 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc13c0d2270 con 0x7fc1481081b0 2024-09-21T02:44:11.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 91034 3.02 MB/sec execute 108 sec latency 42.595 ms 2024-09-21T02:44:11.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.168+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc14802e640 con 0x7fc1481081b0 2024-09-21T02:44:11.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.169+0000 7fc144ff9640 1 -- 172.21.15.29:0/2191408562 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7fc13c0d71b0 con 0x7fc1481081b0 2024-09-21T02:44:11.171 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:44:11.171 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:44:11.172 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1300b6510 msgr2=0x7fc1300b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1300b6510 0x7fc1300b8a20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc14806b9e0 tx=0x7fc13800b390 comp rx=0 tx=0).stop 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 msgr2=0x7fc14819b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 0x7fc14819b510 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc13c004770 tx=0x7fc13c004040 comp rx=0 tx=0).stop 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 shutdown_connections 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1300b6510 0x7fc1300b8a20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc148108b00 0x7fc14819ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1481081b0 0x7fc14819b510 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 --2- 172.21.15.29:0/2191408562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc148102870 0x7fc14819afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.171+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 >> 172.21.15.29:0/2191408562 conn(0x7fc1480fe2c0 msgr2=0x7fc1481006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.172+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 shutdown_connections 2024-09-21T02:44:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:11.172+0000 7fc14c9a0640 1 -- 172.21.15.29:0/2191408562 wait complete. 2024-09-21T02:44:11.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:11 smithi066 ceph-mon[33436]: pgmap v541: 129 pgs: 129 active+clean; 35 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-09-21T02:44:11.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:11 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2191408562' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:11.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:11 smithi190 ceph-mon[33871]: pgmap v541: 129 pgs: 129 active+clean; 35 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-09-21T02:44:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:11 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2191408562' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:11 smithi029 ceph-mon[31006]: pgmap v541: 129 pgs: 129 active+clean; 35 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-09-21T02:44:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:11 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2191408562' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:12.051 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-09-21T02:44:12.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-09-21T02:44:12.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 91754 3.05 MB/sec execute 109 sec latency 14.173 ms 2024-09-21T02:44:12.355 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:44:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 -- 172.21.15.29:0/1233172045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c1094d0 msgr2=0x7ff82c1098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 --2- 172.21.15.29:0/1233172045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c1094d0 0x7ff82c1098e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009900 tx=0x7ff81c02a410 comp rx=0 tx=0).stop 2024-09-21T02:44:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 -- 172.21.15.29:0/1233172045 shutdown_connections 2024-09-21T02:44:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 --2- 172.21.15.29:0/1233172045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff82c109e20 0x7ff82c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 --2- 172.21.15.29:0/1233172045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c1094d0 0x7ff82c1098e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 --2- 172.21.15.29:0/1233172045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c1054f0 0x7ff82c105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 -- 172.21.15.29:0/1233172045 >> 172.21.15.29:0/1233172045 conn(0x7ff82c079870 msgr2=0x7ff82c079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 -- 172.21.15.29:0/1233172045 shutdown_connections 2024-09-21T02:44:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.911+0000 7ff8334e1640 1 -- 172.21.15.29:0/1233172045 wait complete. 2024-09-21T02:44:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.912+0000 7ff8334e1640 1 Processor -- start 2024-09-21T02:44:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.912+0000 7ff8334e1640 1 -- start start 2024-09-21T02:44:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.912+0000 7ff8334e1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff82c1054f0 0x7ff82c19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.912+0000 7ff8334e1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c1094d0 0x7ff82c19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.912+0000 7ff8334e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c109e20 0x7ff82c19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff8334e1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c113990 con 0x7ff82c109e20 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff8334e1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c113810 con 0x7ff82c1054f0 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff8334e1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c113b10 con 0x7ff82c1094d0 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c1094d0 0x7ff82c19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff831a7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c109e20 0x7ff82c19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c1094d0 0x7ff82c19a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49506/0 (socket says 172.21.15.29:49506) 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 -- 172.21.15.29:0/3497236596 learned_addr learned my addr 172.21.15.29:0/3497236596 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff83127d640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff82c1054f0 0x7ff82c19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 -- 172.21.15.29:0/3497236596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff82c1054f0 msgr2=0x7ff82c19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff82c1054f0 0x7ff82c19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 -- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c109e20 msgr2=0x7ff82c19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c109e20 0x7ff82c19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 -- 172.21.15.29:0/3497236596 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c1a0350 con 0x7ff82c1094d0 2024-09-21T02:44:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff831a7e640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c109e20 0x7ff82c19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.913+0000 7ff830a7c640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c1094d0 0x7ff82c19a980 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff81c005940 tx=0x7ff81c0040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.914+0000 7ff8227fc640 1 -- 172.21.15.29:0/3497236596 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c03f030 con 0x7ff82c1094d0 2024-09-21T02:44:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.914+0000 7ff8334e1640 1 -- 172.21.15.29:0/3497236596 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82c1a05e0 con 0x7ff82c1094d0 2024-09-21T02:44:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.914+0000 7ff8227fc640 1 -- 172.21.15.29:0/3497236596 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff81c003a80 con 0x7ff82c1094d0 2024-09-21T02:44:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.914+0000 7ff8227fc640 1 -- 172.21.15.29:0/3497236596 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c035070 con 0x7ff82c1094d0 2024-09-21T02:44:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.914+0000 7ff8334e1640 1 -- 172.21.15.29:0/3497236596 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82c1a6da0 con 0x7ff82c1094d0 2024-09-21T02:44:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff8227fc640 1 -- 172.21.15.29:0/3497236596 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff81c042030 con 0x7ff82c1094d0 2024-09-21T02:44:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff839a55740 1 Processor -- start 2024-09-21T02:44:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff839a55740 1 -- start start 2024-09-21T02:44:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff839a55740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a55600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff839a55740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a55b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff839a55740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a56080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff839a55740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f4e19cbb90 con 0x55f4e1685980 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff839a55740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f4e19bafc0 con 0x55f4e16ae840 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff839a55740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f4e19bedd0 con 0x55f4e168db60 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff83127d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a55b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff83127d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a55b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33954/0 (socket says 172.21.15.29:33954) 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.917+0000 7ff83127d640 1 -- 172.21.15.29:0/2981266465 learned_addr learned my addr 172.21.15.29:0/2981266465 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff830a7c640 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a56080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff831a7e640 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a55600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff83127d640 1 -- 172.21.15.29:0/2981266465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 msgr2=0x55f4e1a56080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff83127d640 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a56080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff83127d640 1 -- 172.21.15.29:0/2981266465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 msgr2=0x55f4e1a55600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff83127d640 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a55600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff83127d640 1 -- 172.21.15.29:0/2981266465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4e1942060 con 0x55f4e1685980 2024-09-21T02:44:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff831a7e640 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a55600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:44:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff8227fc640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8180b6510 0x7ff8180b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff831a7e640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8180b6510 0x7ff8180b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.918+0000 7ff8227fc640 1 -- 172.21.15.29:0/3497236596 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff81c107bb0 con 0x7ff82c1094d0 2024-09-21T02:44:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.919+0000 7ff83127d640 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a55b40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff82800b760 tx=0x7ff82800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76732c932d54899 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.919+0000 7ff8177fe640 1 -- 172.21.15.29:0/2981266465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828003d70 con 0x55f4e1685980 2024-09-21T02:44:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.919+0000 7ff831a7e640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8180b6510 0x7ff8180b8a20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff824005fd0 tx=0x7ff8240096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.919+0000 7ff8177fe640 1 -- 172.21.15.29:0/2981266465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff828015070 con 0x55f4e1685980 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.919+0000 7ff8177fe640 1 -- 172.21.15.29:0/2981266465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828004c60 con 0x55f4e1685980 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 -- 172.21.15.29:0/2981266465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 msgr2=0x55f4e1a55b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a55b40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff82800b760 tx=0x7ff82800bc20 comp rx=0 tx=0).stop 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 -- 172.21.15.29:0/2981266465 shutdown_connections 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a56080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a55b40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 --2- 172.21.15.29:0/2981266465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a55600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 -- 172.21.15.29:0/2981266465 >> 172.21.15.29:0/2981266465 conn(0x55f4e16a2990 msgr2=0x55f4e1a534f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 -- 172.21.15.29:0/2981266465 shutdown_connections 2024-09-21T02:44:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.920+0000 7ff839a55740 1 -- 172.21.15.29:0/2981266465 wait complete. 2024-09-21T02:44:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 1 -- start start 2024-09-21T02:44:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 1 Processor -- start 2024-09-21T02:44:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a65620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e168db60 0x55f4e1a65b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e16ae840 0x55f4e1a71470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f4e181c7a0 con 0x55f4e1685980 2024-09-21T02:44:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f4e19600a0 con 0x55f4e168db60 2024-09-21T02:44:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f4e1a68140 con 0x55f4e16ae840 2024-09-21T02:44:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff839a55740 10 client.0 _ll_register_callbacks cb 0x55f4e193d3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff83127d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a65620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff83127d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a65620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33956/0 (socket says 172.21.15.29:33956) 2024-09-21T02:44:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff83127d640 1 -- 172.21.15.29:0/3224431496 learned_addr learned my addr 172.21.15.29:0/3224431496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.921+0000 7ff831a7e640 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e16ae840 0x55f4e1a71470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff830a7c640 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e168db60 0x55f4e1a65b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff83127d640 1 -- 172.21.15.29:0/3224431496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e16ae840 msgr2=0x55f4e1a71470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff83127d640 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e16ae840 0x55f4e1a71470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff83127d640 1 -- 172.21.15.29:0/3224431496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e168db60 msgr2=0x55f4e1a65b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff83127d640 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e168db60 0x55f4e1a65b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff83127d640 1 -- 172.21.15.29:0/3224431496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4e194e880 con 0x55f4e1685980 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff83127d640 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a65620 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff828006010 tx=0x7ff828003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fd9233511f153f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff814ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff814ff9640 1 -- 172.21.15.29:0/3224431496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82802a060 con 0x55f4e1685980 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff814ff9640 1 -- 172.21.15.29:0/3224431496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff828011650 con 0x55f4e1685980 2024-09-21T02:44:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff839a55740 1 -- 172.21.15.29:0/3224431496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff828015070 con 0x55f4e1685980 2024-09-21T02:44:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff814ff9640 1 -- 172.21.15.29:0/3224431496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff828031050 con 0x55f4e1685980 2024-09-21T02:44:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.922+0000 7ff839a55740 10 client.16236 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff839a55740 1 -- 172.21.15.29:0/3224431496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f4e1728b90 con 0x55f4e1685980 2024-09-21T02:44:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff814ff9640 1 -- 172.21.15.29:0/3224431496 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff828015070 con 0x55f4e1685980 2024-09-21T02:44:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff814ff9640 1 -- 172.21.15.29:0/3224431496 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff828019830 con 0x55f4e1685980 2024-09-21T02:44:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff839a55740 10 client.16236 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff839a55740 10 client.16236 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:44:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff839a55740 20 client.16236 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff839a55740 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f4e166d1c0 0x55f4e1a77400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff839a55740 4 client.16236 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff839a55740 1 -- 172.21.15.29:0/3224431496 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4e19cdae0 con 0x55f4e166d1c0 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.923+0000 7ff830a7c640 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f4e166d1c0 0x55f4e1a77400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.924+0000 7ff830a7c640 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f4e166d1c0 0x55f4e1a77400 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d79be60399c7b529 server_cookie=aa543c94d75c7dfa in_seq=0 out_seq=0 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.924+0000 7ff814ff9640 10 client.16236 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff814ff9640 1 -- 172.21.15.29:0/3224431496 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f4e19cdae0 con 0x55f4e166d1c0 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff814ff9640 10 client.16236 handle_command_reply: tid=0 2024-09-21T02:44:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 client.16236 shutdown 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 -- 172.21.15.29:0/3224431496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 msgr2=0x55f4e1a65620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a65620 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff828006010 tx=0x7ff828003da0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 -- 172.21.15.29:0/3224431496 shutdown_connections 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e16ae840 0x55f4e1a71470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e168db60 0x55f4e1a65b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f4e166d1c0 0x55f4e1a77400 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 --2- 172.21.15.29:0/3224431496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a65620 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 -- 172.21.15.29:0/3224431496 >> 172.21.15.29:0/3224431496 conn(0x55f4e16a2990 msgr2=0x55f4e1a56ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff814ff9640 0 client.16236 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:12.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 -- 172.21.15.29:0/3224431496 shutdown_connections 2024-09-21T02:44:12.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 1 -- 172.21.15.29:0/3224431496 wait complete. 2024-09-21T02:44:12.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.925+0000 7ff839a55740 20 client.16236 trim_cache size 0 max 16384 2024-09-21T02:44:12.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.944+0000 7ff839a55740 1 Processor -- start 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff839a55740 1 -- start start 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff839a55740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a63ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff839a55740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a641e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff839a55740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a71370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff839a55740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1ae7a0 con 0x55f4e1685980 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff839a55740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1b02e0 con 0x55f4e16ae840 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff839a55740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1aee70 con 0x55f4e168db60 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a63ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a63ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33964/0 (socket says 172.21.15.29:33964) 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 -- 172.21.15.29:0/2403537686 learned_addr learned my addr 172.21.15.29:0/2403537686 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff830a7c640 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff831a7e640 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a71370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 -- 172.21.15.29:0/2403537686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 msgr2=0x55f4e1a641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a641e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 -- 172.21.15.29:0/2403537686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 msgr2=0x55f4e1a71370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a71370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 -- 172.21.15.29:0/2403537686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4e1a4e8b0 con 0x55f4e1685980 2024-09-21T02:44:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.945+0000 7ff83127d640 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a63ca0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff828019c30 tx=0x7ff828005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af87f91951fd62 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.946+0000 7ff816ffd640 1 -- 172.21.15.29:0/2403537686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82802a050 con 0x55f4e1685980 2024-09-21T02:44:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.946+0000 7ff816ffd640 1 -- 172.21.15.29:0/2403537686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff828015070 con 0x55f4e1685980 2024-09-21T02:44:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.946+0000 7ff816ffd640 1 -- 172.21.15.29:0/2403537686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8280198b0 con 0x55f4e1685980 2024-09-21T02:44:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 -- 172.21.15.29:0/2403537686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 msgr2=0x55f4e1a63ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a63ca0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff828019c30 tx=0x7ff828005f30 comp rx=0 tx=0).stop 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 -- 172.21.15.29:0/2403537686 shutdown_connections 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e16ae840 0x55f4e1a71370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a641e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 --2- 172.21.15.29:0/2403537686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e1685980 0x55f4e1a63ca0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 -- 172.21.15.29:0/2403537686 >> 172.21.15.29:0/2403537686 conn(0x55f4e16a2990 msgr2=0x55f4e1a60560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 -- 172.21.15.29:0/2403537686 shutdown_connections 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 -- 172.21.15.29:0/2403537686 wait complete. 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 -- start start 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.947+0000 7ff839a55740 1 Processor -- start 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff839a55740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e1685980 0x55f4e1a6a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff839a55740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a6a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff839a55740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e16ae840 0x55f4e1a6af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff839a55740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1afc10 con 0x55f4e16ae840 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff839a55740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1af540 con 0x55f4e1685980 2024-09-21T02:44:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff839a55740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1ac780 con 0x55f4e168db60 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff831a7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e16ae840 0x55f4e1a6af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff830a7c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a6a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff831a7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e16ae840 0x55f4e1a6af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33968/0 (socket says 172.21.15.29:33968) 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff831a7e640 1 -- 172.21.15.29:0/700672504 learned_addr learned my addr 172.21.15.29:0/700672504 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff839a55740 10 client.0 _ll_register_callbacks cb 0x7ff828004590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff83127d640 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e1685980 0x55f4e1a6a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff830a7c640 1 -- 172.21.15.29:0/700672504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e1685980 msgr2=0x55f4e1a6a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff830a7c640 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e1685980 0x55f4e1a6a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff830a7c640 1 -- 172.21.15.29:0/700672504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e16ae840 msgr2=0x55f4e1a6af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff830a7c640 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e16ae840 0x55f4e1a6af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.948+0000 7ff830a7c640 1 -- 172.21.15.29:0/700672504 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4e1ae1dc0 con 0x55f4e168db60 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff83127d640 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e1685980 0x55f4e1a6a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff831a7e640 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e16ae840 0x55f4e1a6af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff830a7c640 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a6a9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff81c1052d0 tx=0x7ff81c0c9a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d105eec27509075 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff814ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:44:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff814ff9640 1 -- 172.21.15.29:0/700672504 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c049040 con 0x55f4e168db60 2024-09-21T02:44:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff814ff9640 1 -- 172.21.15.29:0/700672504 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff81c0d6ef0 con 0x55f4e168db60 2024-09-21T02:44:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff839a55740 1 -- 172.21.15.29:0/700672504 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff828015070 con 0x55f4e168db60 2024-09-21T02:44:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff814ff9640 1 -- 172.21.15.29:0/700672504 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff81c0044e0 con 0x55f4e168db60 2024-09-21T02:44:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff839a55740 10 client.25864 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.949+0000 7ff839a55740 1 -- 172.21.15.29:0/700672504 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f4e1ae3000 con 0x55f4e168db60 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff814ff9640 1 -- 172.21.15.29:0/700672504 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff828015070 con 0x55f4e168db60 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff814ff9640 1 -- 172.21.15.29:0/700672504 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff81c0ccc70 con 0x55f4e168db60 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff839a55740 10 client.25864 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff839a55740 10 client.25864 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff839a55740 20 client.25864 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff839a55740 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f4e1642570 0x55f4e1ae57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff839a55740 4 client.25864 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff839a55740 1 -- 172.21.15.29:0/700672504 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x55f4e19c9590 con 0x55f4e1642570 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff83127d640 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f4e1642570 0x55f4e1ae57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff814ff9640 10 client.25864 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:12.950+0000 7ff83127d640 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f4e1642570 0x55f4e1ae57f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b87e509bf7e19d9 server_cookie=bdb0574891a665de in_seq=0 out_seq=0 2024-09-21T02:44:13.075 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92276 3.07 MB/sec execute 110 sec latency 103.354 ms 2024-09-21T02:44:13.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff814ff9640 1 -- 172.21.15.29:0/700672504 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f4e19c9590 con 0x55f4e1642570 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff814ff9640 10 client.25864 handle_command_reply: tid=0 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 client.25864 shutdown 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 -- 172.21.15.29:0/700672504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 msgr2=0x55f4e1a6a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a6a9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff81c1052d0 tx=0x7ff81c0c9a70 comp rx=0 tx=0).stop 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 -- 172.21.15.29:0/700672504 shutdown_connections 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f4e1642570 0x55f4e1ae57f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4e16ae840 0x55f4e1a6af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4e168db60 0x55f4e1a6a9d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 --2- 172.21.15.29:0/700672504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f4e1685980 0x55f4e1a6a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff814ff9640 0 client.25864 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 -- 172.21.15.29:0/700672504 >> 172.21.15.29:0/700672504 conn(0x55f4e16a2990 msgr2=0x55f4e1a60560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 -- 172.21.15.29:0/700672504 shutdown_connections 2024-09-21T02:44:13.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.090+0000 7ff839a55740 1 -- 172.21.15.29:0/700672504 wait complete. 2024-09-21T02:44:13.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff839a55740 20 client.25864 trim_cache size 0 max 16384 2024-09-21T02:44:13.094 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:44:13.094 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:44:13.094 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:44:13.094 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 18, 2024-09-21T02:44:13.094 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:44:13.094 INFO:teuthology.orchestra.run.smithi029.stdout: "e924aa9e": { 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 18, 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 130.0, 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:44:13.095 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:44:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 -- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8180b6510 msgr2=0x7ff8180b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8180b6510 0x7ff8180b8a20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff824005fd0 tx=0x7ff8240096a0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 -- 172.21.15.29:0/3497236596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c1094d0 msgr2=0x7ff82c19a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c1094d0 0x7ff82c19a980 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff81c005940 tx=0x7ff81c0040e0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 -- 172.21.15.29:0/3497236596 shutdown_connections 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8180b6510 0x7ff8180b8a20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff82c109e20 0x7ff82c19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c1094d0 0x7ff82c19a980 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 --2- 172.21.15.29:0/3497236596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff82c1054f0 0x7ff82c19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 -- 172.21.15.29:0/3497236596 >> 172.21.15.29:0/3497236596 conn(0x7ff82c079870 msgr2=0x7ff82c06bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 -- 172.21.15.29:0/3497236596 shutdown_connections 2024-09-21T02:44:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:13.091+0000 7ff8334e1640 1 -- 172.21.15.29:0/3497236596 wait complete. 2024-09-21T02:44:13.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:13 smithi066 ceph-mon[33436]: pgmap v542: 129 pgs: 129 active+clean; 32 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-09-21T02:44:13.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:13 smithi190 ceph-mon[33871]: pgmap v542: 129 pgs: 129 active+clean; 32 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-09-21T02:44:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:13 smithi029 ceph-mon[31006]: pgmap v542: 129 pgs: 129 active+clean; 32 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-09-21T02:44:14.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 3.04 MB/sec execute 111 sec latency 974.226 ms 2024-09-21T02:44:14.660 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e924aa9e', quiesce took 4.6 seconds. Will release after: 5.4 2024-09-21T02:44:15.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 3.01 MB/sec execute 112 sec latency 1974.341 ms 2024-09-21T02:44:15.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:15 smithi066 ceph-mon[33436]: pgmap v543: 129 pgs: 129 active+clean; 31 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-09-21T02:44:15.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:15 smithi190 ceph-mon[33871]: pgmap v543: 129 pgs: 129 active+clean; 31 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-09-21T02:44:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:15 smithi029 ceph-mon[31006]: pgmap v543: 129 pgs: 129 active+clean; 31 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-09-21T02:44:16.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 2.98 MB/sec execute 113 sec latency 2974.465 ms 2024-09-21T02:44:17.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 2.96 MB/sec execute 114 sec latency 3974.570 ms 2024-09-21T02:44:17.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:17 smithi066 ceph-mon[33436]: pgmap v544: 129 pgs: 129 active+clean; 31 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-09-21T02:44:17.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:17 smithi190 ceph-mon[33871]: pgmap v544: 129 pgs: 129 active+clean; 31 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-09-21T02:44:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:17 smithi029 ceph-mon[31006]: pgmap v544: 129 pgs: 129 active+clean; 31 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-09-21T02:44:18.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 2.93 MB/sec execute 115 sec latency 4974.680 ms 2024-09-21T02:44:19.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 2.91 MB/sec execute 116 sec latency 5974.803 ms 2024-09-21T02:44:19.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:19 smithi066 ceph-mon[33436]: pgmap v545: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-09-21T02:44:19.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:19 smithi190 ceph-mon[33871]: pgmap v545: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-09-21T02:44:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:19 smithi029 ceph-mon[31006]: pgmap v545: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-09-21T02:44:20.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:44:20.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 2.88 MB/sec execute 117 sec latency 6974.876 ms 2024-09-21T02:44:20.337 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:44:20.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 -- 172.21.15.29:0/4051235798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 msgr2=0x7f22401080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:20.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 --2- 172.21.15.29:0/4051235798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 0x7f22401080b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2238009ca0 tx=0x7f223802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:44:20.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 -- 172.21.15.29:0/4051235798 shutdown_connections 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 --2- 172.21.15.29:0/4051235798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 0x7f22401080b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 --2- 172.21.15.29:0/4051235798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2240102770 0x7f2240107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 --2- 172.21.15.29:0/4051235798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f224006b2f0 0x7f2240102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 -- 172.21.15.29:0/4051235798 >> 172.21.15.29:0/4051235798 conn(0x7f2240079e40 msgr2=0x7f224007a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 -- 172.21.15.29:0/4051235798 shutdown_connections 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.876+0000 7f22457fb640 1 -- 172.21.15.29:0/4051235798 wait complete. 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.877+0000 7f22457fb640 1 Processor -- start 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.877+0000 7f22457fb640 1 -- start start 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.877+0000 7f22457fb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f224006b2f0 0x7f224019b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f22457fb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2240102770 0x7f224019b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f22457fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 0x7f224019bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f22457fb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f22401128f0 con 0x7f2240102b80 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f22457fb640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2240112770 con 0x7f2240102770 2024-09-21T02:44:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f22457fb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2240112a70 con 0x7f224006b2f0 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 0x7f224019bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f224006b2f0 0x7f224019b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 0x7f224019bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49678/0 (socket says 172.21.15.29:49678) 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 -- 172.21.15.29:0/2462316117 learned_addr learned my addr 172.21.15.29:0/2462316117 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f224006b2f0 0x7f224019b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58520/0 (socket says 172.21.15.29:58520) 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223e7fc640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2240102770 0x7f224019b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 -- 172.21.15.29:0/2462316117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f224006b2f0 msgr2=0x7f224019b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f224006b2f0 0x7f224019b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 -- 172.21.15.29:0/2462316117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2240102770 msgr2=0x7f224019b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2240102770 0x7f224019b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 -- 172.21.15.29:0/2462316117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22401a63f0 con 0x7f2240102b80 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223e7fc640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2240102770 0x7f224019b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:44:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.878+0000 7f223f7fe640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 0x7f224019bbe0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2238005cc0 tx=0x7f2238032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.879+0000 7f2227fff640 1 -- 172.21.15.29:0/2462316117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2238032e00 con 0x7f2240102b80 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.879+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22401a6680 con 0x7f2240102b80 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.879+0000 7f2227fff640 1 -- 172.21.15.29:0/2462316117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22380044d0 con 0x7f2240102b80 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.879+0000 7f2227fff640 1 -- 172.21.15.29:0/2462316117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2238030b70 con 0x7f2240102b80 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.879+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22401a6c10 con 0x7f2240102b80 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.880+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2240102230 con 0x7f2240102b80 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.881+0000 7f2227fff640 1 -- 172.21.15.29:0/2462316117 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f2238030d10 con 0x7f2240102b80 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.882+0000 7f2227fff640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22280b6540 0x7f22280b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.882+0000 7f223effd640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22280b6540 0x7f22280b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.882+0000 7f2227fff640 1 -- 172.21.15.29:0/2462316117 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2238108b00 con 0x7f2240102b80 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.883+0000 7f223effd640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22280b6540 0x7f22280b8a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2240109820 tx=0x7f223000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:20.885+0000 7f2227fff640 1 -- 172.21.15.29:0/2462316117 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f22380cefb0 con 0x7f2240102b80 2024-09-21T02:44:21.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 -- 172.21.15.29:0/3980318884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc14103fc0 msgr2=0x7fdc1410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 --2- 172.21.15.29:0/3980318884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc14103fc0 0x7fdc1410a960 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009910 tx=0x7fdc0c02a3f0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 -- 172.21.15.29:0/3980318884 shutdown_connections 2024-09-21T02:44:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 --2- 172.21.15.29:0/3980318884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc14103fc0 0x7fdc1410a960 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 --2- 172.21.15.29:0/3980318884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc14103620 0x7fdc14103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 --2- 172.21.15.29:0/3980318884 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc141025a0 0x7fdc14102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 -- 172.21.15.29:0/3980318884 >> 172.21.15.29:0/3980318884 conn(0x7fdc140fdee0 msgr2=0x7fdc14100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 -- 172.21.15.29:0/3980318884 shutdown_connections 2024-09-21T02:44:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 -- 172.21.15.29:0/3980318884 wait complete. 2024-09-21T02:44:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.018+0000 7fdc1c430640 1 Processor -- start 2024-09-21T02:44:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1c430640 1 -- start start 2024-09-21T02:44:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1c430640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc141025a0 0x7fdc1406b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1c430640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc14103620 0x7fdc141a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1c430640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc14103fc0 0x7fdc141a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1c430640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410e090 con 0x7fdc141025a0 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1c430640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410df10 con 0x7fdc14103fc0 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1c430640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410e210 con 0x7fdc14103620 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc141025a0 0x7fdc1406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc141025a0 0x7fdc1406b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49694/0 (socket says 172.21.15.29:49694) 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 -- 172.21.15.29:0/1490589726 learned_addr learned my addr 172.21.15.29:0/1490589726 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc199cb640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc14103620 0x7fdc141a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc14103fc0 0x7fdc141a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 -- 172.21.15.29:0/1490589726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc14103620 msgr2=0x7fdc141a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc14103620 0x7fdc141a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 -- 172.21.15.29:0/1490589726 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc14103fc0 msgr2=0x7fdc141a34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc14103fc0 0x7fdc141a34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 -- 172.21.15.29:0/1490589726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc141a3af0 con 0x7fdc141025a0 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc14103fc0 0x7fdc141a34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.019+0000 7fdc1a1cc640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc141025a0 0x7fdc1406b390 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800c890 tx=0x7fdc0800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.020+0000 7fdc077fe640 1 -- 172.21.15.29:0/1490589726 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08004220 con 0x7fdc141025a0 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.020+0000 7fdc077fe640 1 -- 172.21.15.29:0/1490589726 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc08007e40 con 0x7fdc141025a0 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.020+0000 7fdc077fe640 1 -- 172.21.15.29:0/1490589726 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08004a00 con 0x7fdc141025a0 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.020+0000 7fdc1c430640 1 -- 172.21.15.29:0/1490589726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc141a3de0 con 0x7fdc141025a0 2024-09-21T02:44:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.020+0000 7fdc1c430640 1 -- 172.21.15.29:0/1490589726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc141a4250 con 0x7fdc141025a0 2024-09-21T02:44:21.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc23478740 1 Processor -- start 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc23478740 1 -- start start 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc23478740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2ca1b0 0x558aec671e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc23478740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec6724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc23478740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2a94d0 0x558aec673010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc23478740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558aec5e46f0 con 0x558aec2ca1b0 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc23478740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x558aec5d3b20 con 0x558aec2a12f0 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc23478740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558aec5d7930 con 0x558aec2a94d0 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc1a9cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2ca1b0 0x558aec671e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc077fe640 1 -- 172.21.15.29:0/1490589726 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fdc080079b0 con 0x7fdc141025a0 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc1a9cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2ca1b0 0x558aec671e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49698/0 (socket says 172.21.15.29:49698) 2024-09-21T02:44:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.023+0000 7fdc1a9cd640 1 -- 172.21.15.29:0/3010333403 learned_addr learned my addr 172.21.15.29:0/3010333403 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc199cb640 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2a94d0 0x558aec673010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc1a1cc640 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec6724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc1a9cd640 1 -- 172.21.15.29:0/3010333403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2a94d0 msgr2=0x558aec673010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2a94d0 0x558aec673010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc1a9cd640 1 -- 172.21.15.29:0/3010333403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 msgr2=0x558aec6724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec6724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc1a9cd640 1 -- 172.21.15.29:0/3010333403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aec599530 con 0x558aec2ca1b0 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2ca1b0 0x558aec671e50 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009830 tx=0x7fdc0c031c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2444a8d29e3d2cd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc04bfe640 1 -- 172.21.15.29:0/3010333403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c041020 con 0x558aec2ca1b0 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc04bfe640 1 -- 172.21.15.29:0/3010333403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc0c037040 con 0x558aec2ca1b0 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc04bfe640 1 -- 172.21.15.29:0/3010333403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c036070 con 0x558aec2ca1b0 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.024+0000 7fdc077fe640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdc000b6510 0x7fdc000b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc1a1cc640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdc000b6510 0x7fdc000b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc077fe640 1 -- 172.21.15.29:0/1490589726 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdc080e7320 con 0x7fdc141025a0 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 -- 172.21.15.29:0/3010333403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2ca1b0 msgr2=0x558aec671e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2ca1b0 0x558aec671e50 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009830 tx=0x7fdc0c031c60 comp rx=0 tx=0).stop 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc1a1cc640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdc000b6510 0x7fdc000b8a20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdc080aa440 tx=0x7fdc08000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 -- 172.21.15.29:0/3010333403 shutdown_connections 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2a94d0 0x558aec673010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec6724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 --2- 172.21.15.29:0/3010333403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2ca1b0 0x558aec671e50 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 -- 172.21.15.29:0/3010333403 >> 172.21.15.29:0/3010333403 conn(0x558aec2be300 msgr2=0x558aec66d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 -- 172.21.15.29:0/3010333403 shutdown_connections 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.025+0000 7fdc23478740 1 -- 172.21.15.29:0/3010333403 wait complete. 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 1 -- start start 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 1 Processor -- start 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec67b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec67bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec67c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558aec50bab0 con 0x558aec2a94d0 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x558aec579730 con 0x558aec2a12f0 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558aec673ad0 con 0x558aec2ca1b0 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc23478740 10 client.0 _ll_register_callbacks cb 0x558aec5cd6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc199cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec67bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc199cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec67bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49714/0 (socket says 172.21.15.29:49714) 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc199cb640 1 -- 172.21.15.29:0/2909097042 learned_addr learned my addr 172.21.15.29:0/2909097042 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec67b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.026+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec67c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc199cb640 1 -- 172.21.15.29:0/2909097042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 msgr2=0x558aec67c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc199cb640 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec67c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc199cb640 1 -- 172.21.15.29:0/2909097042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 msgr2=0x558aec67b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc199cb640 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec67b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc199cb640 1 -- 172.21.15.29:0/2909097042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aec335790 con 0x558aec2a94d0 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc199cb640 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec67bbf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdc10010b70 tx=0x7fdc1000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6eea66800261ae5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdbfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdbfdffb640 1 -- 172.21.15.29:0/2909097042 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1001e060 con 0x558aec2a94d0 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdbfdffb640 1 -- 172.21.15.29:0/2909097042 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdc1000e5c0 con 0x558aec2a94d0 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc23478740 1 -- 172.21.15.29:0/2909097042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc0c037040 con 0x558aec2a94d0 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdbfdffb640 1 -- 172.21.15.29:0/2909097042 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdc10025050 con 0x558aec2a94d0 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc23478740 10 client.16260 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdc23478740 1 -- 172.21.15.29:0/2909097042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558aec3ae620 con 0x558aec2a94d0 2024-09-21T02:44:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.027+0000 7fdbfdffb640 1 -- 172.21.15.29:0/2909097042 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc0c037040 con 0x558aec2a94d0 2024-09-21T02:44:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdbfdffb640 1 -- 172.21.15.29:0/2909097042 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fdc10018870 con 0x558aec2a94d0 2024-09-21T02:44:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdc23478740 10 client.16260 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdc23478740 10 client.16260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:44:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdc23478740 20 client.16260 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdc23478740 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558aec288d80 0x558aec688cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdc23478740 4 client.16260 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:44:21.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdc23478740 1 -- 172.21.15.29:0/2909097042 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558aec5e6640 con 0x558aec288d80 2024-09-21T02:44:21.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558aec288d80 0x558aec688cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558aec288d80 0x558aec688cc0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f5c8d01acea9377 server_cookie=dc812ffc58e98729 in_seq=0 out_seq=0 2024-09-21T02:44:21.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.028+0000 7fdbfdffb640 10 client.16260 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:21.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdbfdffb640 1 -- 172.21.15.29:0/2909097042 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558aec5e6640 con 0x558aec288d80 2024-09-21T02:44:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdbfdffb640 10 client.16260 handle_command_reply: tid=0 2024-09-21T02:44:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 client.16260 shutdown 2024-09-21T02:44:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 -- 172.21.15.29:0/2909097042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 msgr2=0x558aec67bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec67bbf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdc10010b70 tx=0x7fdc1000f7f0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 -- 172.21.15.29:0/2909097042 shutdown_connections 2024-09-21T02:44:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec67c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558aec288d80 0x558aec688cc0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec67bbf0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 --2- 172.21.15.29:0/2909097042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec67b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.029+0000 7fdc23478740 1 -- 172.21.15.29:0/2909097042 >> 172.21.15.29:0/2909097042 conn(0x558aec2be300 msgr2=0x558aec672ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:21.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.030+0000 7fdbfdffb640 0 client.16260 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:21.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.030+0000 7fdc23478740 1 -- 172.21.15.29:0/2909097042 shutdown_connections 2024-09-21T02:44:21.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.030+0000 7fdc23478740 1 -- 172.21.15.29:0/2909097042 wait complete. 2024-09-21T02:44:21.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.030+0000 7fdc23478740 20 client.16260 trim_cache size 0 max 16384 2024-09-21T02:44:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.044+0000 7fdc23478740 1 Processor -- start 2024-09-21T02:44:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.044+0000 7fdc23478740 1 -- start start 2024-09-21T02:44:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.044+0000 7fdc23478740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec67bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.044+0000 7fdc23478740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec671df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.044+0000 7fdc23478740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec672330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.044+0000 7fdc23478740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141ae700 con 0x558aec2a94d0 2024-09-21T02:44:21.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.044+0000 7fdc23478740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141b0240 con 0x558aec2a12f0 2024-09-21T02:44:21.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.044+0000 7fdc23478740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141aedd0 con 0x558aec2ca1b0 2024-09-21T02:44:21.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec671df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec671df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49716/0 (socket says 172.21.15.29:49716) 2024-09-21T02:44:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 -- 172.21.15.29:0/2541323940 learned_addr learned my addr 172.21.15.29:0/2541323940 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec67bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 -- 172.21.15.29:0/2541323940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 msgr2=0x558aec672330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec672330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec672330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 -- 172.21.15.29:0/2541323940 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 msgr2=0x558aec67bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec67bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 -- 172.21.15.29:0/2541323940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aec67a480 con 0x558aec2a94d0 2024-09-21T02:44:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdc199cb640 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec671df0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdc10010010 tx=0x7fdc1000fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e9836bddc2c87 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdbfffff640 1 -- 172.21.15.29:0/2541323940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1002e070 con 0x558aec2a94d0 2024-09-21T02:44:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdbfffff640 1 -- 172.21.15.29:0/2541323940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc10018e50 con 0x558aec2a94d0 2024-09-21T02:44:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.045+0000 7fdbfffff640 1 -- 172.21.15.29:0/2541323940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000eb50 con 0x558aec2a94d0 2024-09-21T02:44:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 -- 172.21.15.29:0/2541323940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 msgr2=0x558aec671df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec671df0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdc10010010 tx=0x7fdc1000fd70 comp rx=0 tx=0).stop 2024-09-21T02:44:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 -- 172.21.15.29:0/2541323940 shutdown_connections 2024-09-21T02:44:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec672330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a94d0 0x558aec671df0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 --2- 172.21.15.29:0/2541323940 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a12f0 0x558aec67bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 -- 172.21.15.29:0/2541323940 >> 172.21.15.29:0/2541323940 conn(0x558aec2be300 msgr2=0x558aec67b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:21.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 -- 172.21.15.29:0/2541323940 shutdown_connections 2024-09-21T02:44:21.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 -- 172.21.15.29:0/2541323940 wait complete. 2024-09-21T02:44:21.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.046+0000 7fdc23478740 1 -- start start 2024-09-21T02:44:21.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc23478740 1 Processor -- start 2024-09-21T02:44:21.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc23478740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a12f0 0x558aec680680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc23478740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a94d0 0x558aec680d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc23478740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec6818b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc23478740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141afb70 con 0x558aec2a12f0 2024-09-21T02:44:21.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc23478740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141af4a0 con 0x558aec2a94d0 2024-09-21T02:44:21.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc23478740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141ac6e0 con 0x558aec2ca1b0 2024-09-21T02:44:21.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc23478740 10 client.0 _ll_register_callbacks cb 0x7fdc1000aa50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:21.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a12f0 0x558aec680680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a12f0 0x558aec680680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49732/0 (socket says 172.21.15.29:49732) 2024-09-21T02:44:21.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 -- 172.21.15.29:0/325812078 learned_addr learned my addr 172.21.15.29:0/325812078 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:21.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc199cb640 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a94d0 0x558aec680d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 -- 172.21.15.29:0/325812078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 msgr2=0x558aec6818b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:21.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec6818b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 -- 172.21.15.29:0/325812078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a94d0 msgr2=0x558aec680d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:21.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a94d0 0x558aec680d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 -- 172.21.15.29:0/325812078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aec682010 con 0x558aec2a12f0 2024-09-21T02:44:21.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc199cb640 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a94d0 0x558aec680d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:44:21.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdc1a9cd640 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a12f0 0x558aec680680 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009230 tx=0x7fdc0c031e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38621162fb3f6cc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:21.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdbfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:21.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.047+0000 7fdbfdffb640 1 -- 172.21.15.29:0/325812078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c036040 con 0x558aec2a12f0 2024-09-21T02:44:21.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdbfdffb640 1 -- 172.21.15.29:0/325812078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdc0c056f90 con 0x558aec2a12f0 2024-09-21T02:44:21.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 1 -- 172.21.15.29:0/325812078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc10018e50 con 0x558aec2a12f0 2024-09-21T02:44:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdbfdffb640 1 -- 172.21.15.29:0/325812078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdc0c063050 con 0x558aec2a12f0 2024-09-21T02:44:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 10 client.16266 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 1 -- 172.21.15.29:0/325812078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558aec6833d0 con 0x558aec2a12f0 2024-09-21T02:44:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdbfdffb640 1 -- 172.21.15.29:0/325812078 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc10018e50 con 0x558aec2a12f0 2024-09-21T02:44:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdbfdffb640 1 -- 172.21.15.29:0/325812078 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fdc0c0053c0 con 0x558aec2a12f0 2024-09-21T02:44:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 10 client.16266 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 10 client.16266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:44:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 20 client.16266 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558aec25e1c0 0x558aec6f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 4 client.16266 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:44:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc23478740 1 -- 172.21.15.29:0/325812078 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558aec5e20f0 con 0x558aec25e1c0 2024-09-21T02:44:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.048+0000 7fdc199cb640 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558aec25e1c0 0x558aec6f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.049+0000 7fdbfdffb640 10 client.16266 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.049+0000 7fdc199cb640 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558aec25e1c0 0x558aec6f56d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=612b7224d1f957f7 server_cookie=fd7281b65b7db85d in_seq=0 out_seq=0 2024-09-21T02:44:21.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 2.86 MB/sec execute 118 sec latency 7975.067 ms 2024-09-21T02:44:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdbfdffb640 1 -- 172.21.15.29:0/325812078 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558aec5e20f0 con 0x558aec25e1c0 2024-09-21T02:44:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdbfdffb640 10 client.16266 handle_command_reply: tid=0 2024-09-21T02:44:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdc23478740 1 client.16266 shutdown 2024-09-21T02:44:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdc23478740 1 -- 172.21.15.29:0/325812078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a12f0 msgr2=0x558aec680680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdc23478740 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a12f0 0x558aec680680 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009230 tx=0x7fdc0c031e50 comp rx=0 tx=0).stop 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdc23478740 1 -- 172.21.15.29:0/325812078 shutdown_connections 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdc23478740 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aec2ca1b0 0x558aec6818b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdc23478740 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x558aec2a94d0 0x558aec680d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdc23478740 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558aec25e1c0 0x558aec6f56d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.085+0000 7fdc23478740 1 --2- 172.21.15.29:0/325812078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558aec2a12f0 0x558aec680680 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc23478740 1 -- 172.21.15.29:0/325812078 >> 172.21.15.29:0/325812078 conn(0x558aec2be300 msgr2=0x558aec679690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdbfdffb640 0 client.16266 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc23478740 1 -- 172.21.15.29:0/325812078 shutdown_connections 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc23478740 1 -- 172.21.15.29:0/325812078 wait complete. 2024-09-21T02:44:21.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc23478740 20 client.16266 trim_cache size 0 max 16384 2024-09-21T02:44:21.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:44:21.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:44:21.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:44:21.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:44:21.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 -- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdc000b6510 msgr2=0x7fdc000b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdc000b6510 0x7fdc000b8a20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdc080aa440 tx=0x7fdc08000bc0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 -- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc141025a0 msgr2=0x7fdc1406b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc141025a0 0x7fdc1406b390 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800c890 tx=0x7fdc0800cd50 comp rx=0 tx=0).stop 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 -- 172.21.15.29:0/1490589726 shutdown_connections 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdc000b6510 0x7fdc000b8a20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc14103fc0 0x7fdc141a34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc14103620 0x7fdc141a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 --2- 172.21.15.29:0/1490589726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc141025a0 0x7fdc1406b390 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 -- 172.21.15.29:0/1490589726 >> 172.21.15.29:0/1490589726 conn(0x7fdc140fdee0 msgr2=0x7fdc14106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 -- 172.21.15.29:0/1490589726 shutdown_connections 2024-09-21T02:44:21.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:21.086+0000 7fdc1c430640 1 -- 172.21.15.29:0/1490589726 wait complete. 2024-09-21T02:44:21.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.132+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22401090d0 con 0x7f2240102b80 2024-09-21T02:44:21.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.133+0000 7f2227fff640 1 -- 172.21.15.29:0/2462316117 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f22380d3390 con 0x7f2240102b80 2024-09-21T02:44:21.135 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:44:21.135 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:44:21.135 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22280b6540 msgr2=0x7f22280b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22280b6540 0x7f22280b8a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2240109820 tx=0x7f223000b390 comp rx=0 tx=0).stop 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 msgr2=0x7f224019bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 0x7f224019bbe0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2238005cc0 tx=0x7f2238032a10 comp rx=0 tx=0).stop 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 shutdown_connections 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22280b6540 0x7f22280b8a50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2240102b80 0x7f224019bbe0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2240102770 0x7f224019b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 --2- 172.21.15.29:0/2462316117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f224006b2f0 0x7f224019b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 >> 172.21.15.29:0/2462316117 conn(0x7f2240079e40 msgr2=0x7f2240100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 shutdown_connections 2024-09-21T02:44:21.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:21.136+0000 7f22457fb640 1 -- 172.21.15.29:0/2462316117 wait complete. 2024-09-21T02:44:21.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:21 smithi066 ceph-mon[33436]: pgmap v546: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-09-21T02:44:21.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:21 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2462316117' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:21.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:21 smithi190 ceph-mon[33871]: pgmap v546: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-09-21T02:44:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:21 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2462316117' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:21 smithi029 ceph-mon[31006]: pgmap v546: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-09-21T02:44:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:21 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2462316117' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:21.986 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id e924aa9e --release --await' 2024-09-21T02:44:21.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id e924aa9e --release --await 2024-09-21T02:44:22.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92281 2.83 MB/sec execute 119 sec latency 8975.127 ms 2024-09-21T02:44:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:44:22.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 -- 172.21.15.29:0/237987020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c1083c0 msgr2=0x7f280c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 --2- 172.21.15.29:0/237987020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c1083c0 0x7f280c1087d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009900 tx=0x7f27f402a410 comp rx=0 tx=0).stop 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 -- 172.21.15.29:0/237987020 shutdown_connections 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 --2- 172.21.15.29:0/237987020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f280c108d10 0x7f280c10f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 --2- 172.21.15.29:0/237987020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c1083c0 0x7f280c1087d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 --2- 172.21.15.29:0/237987020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f280c102a80 0x7f280c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 -- 172.21.15.29:0/237987020 >> 172.21.15.29:0/237987020 conn(0x7f280c079740 msgr2=0x7f280c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 -- 172.21.15.29:0/237987020 shutdown_connections 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.831+0000 7f2812732640 1 -- 172.21.15.29:0/237987020 wait complete. 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.832+0000 7f2812732640 1 Processor -- start 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.832+0000 7f2812732640 1 -- start start 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f2812732640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c102a80 0x7f280c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f2812732640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f280c1083c0 0x7f280c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f2812732640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f280c108d10 0x7f280c19fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f2812732640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f280c113950 con 0x7f280c102a80 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f2812732640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f280c1137d0 con 0x7f280c108d10 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f2812732640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f280c113ad0 con 0x7f280c1083c0 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c102a80 0x7f280c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c102a80 0x7f280c19a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49744/0 (socket says 172.21.15.29:49744) 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 -- 172.21.15.29:0/1841858696 learned_addr learned my addr 172.21.15.29:0/1841858696 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f2803fff640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f280c1083c0 0x7f280c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f2810ccf640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f280c108d10 0x7f280c19fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 -- 172.21.15.29:0/1841858696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f280c1083c0 msgr2=0x7f280c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f280c1083c0 0x7f280c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 -- 172.21.15.29:0/1841858696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f280c108d10 msgr2=0x7f280c19fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f280c108d10 0x7f280c19fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 -- 172.21.15.29:0/1841858696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c1a0310 con 0x7f280c102a80 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.833+0000 7f280bfff640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c102a80 0x7f280c19a3a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c970 tx=0x7f27fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.834+0000 7f2809ffb640 1 -- 172.21.15.29:0/1841858696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc007bb0 con 0x7f280c102a80 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.834+0000 7f2809ffb640 1 -- 172.21.15.29:0/1841858696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27fc007d50 con 0x7f280c102a80 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.834+0000 7f2809ffb640 1 -- 172.21.15.29:0/1841858696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc004e40 con 0x7f280c102a80 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.834+0000 7f2812732640 1 -- 172.21.15.29:0/1841858696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280c1a0600 con 0x7f280c102a80 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.834+0000 7f2812732640 1 -- 172.21.15.29:0/1841858696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f280c1a6d60 con 0x7f280c102a80 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.836+0000 7f2818ca6740 1 Processor -- start 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.836+0000 7f2818ca6740 1 -- start start 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.836+0000 7f2809ffb640 1 -- 172.21.15.29:0/1841858696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f27fc005020 con 0x7f280c102a80 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 0x55d71435c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d71435ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71435cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2818ca6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d714247dc0 con 0x55d713f8d660 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2818ca6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d7142d41f0 con 0x55d713f95840 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2818ca6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d7142c3620 con 0x55d713fb6520 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d71435ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2810ccf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 0x55d71435c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d71435ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49760/0 (socket says 172.21.15.29:49760) 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 -- 172.21.15.29:0/4048430685 learned_addr learned my addr 172.21.15.29:0/4048430685 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2803fff640 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71435cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 -- 172.21.15.29:0/4048430685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 msgr2=0x55d71435c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 0x55d71435c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 -- 172.21.15.29:0/4048430685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 msgr2=0x55d71435cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71435cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 -- 172.21.15.29:0/4048430685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d71428bfd0 con 0x55d713f8d660 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2803fff640 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71435cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2809ffb640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27f80b6590 0x7f27f80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f280bfff640 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d71435ca50 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0ef580 tx=0x7f27fc0efa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b73226e39fd5df server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.837+0000 7f2810ccf640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27f80b6590 0x7f27f80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.838+0000 7f2809ffb640 1 -- 172.21.15.29:0/1841858696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f27fc0e7360 con 0x7f280c102a80 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.838+0000 7f28027fc640 1 -- 172.21.15.29:0/4048430685 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc0a9860 con 0x55d713f8d660 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.838+0000 7f28027fc640 1 -- 172.21.15.29:0/4048430685 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27fc0acd20 con 0x55d713f8d660 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.838+0000 7f28027fc640 1 -- 172.21.15.29:0/4048430685 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc0ac980 con 0x55d713f8d660 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 -- 172.21.15.29:0/4048430685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 msgr2=0x55d71435ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2810ccf640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27f80b6590 0x7f27f80b8aa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2804005fd0 tx=0x7f2804009620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d71435ca50 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0ef580 tx=0x7f27fc0efa40 comp rx=0 tx=0).stop 2024-09-21T02:44:22.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 -- 172.21.15.29:0/4048430685 shutdown_connections 2024-09-21T02:44:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71435cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d71435ca50 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 --2- 172.21.15.29:0/4048430685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 0x55d71435c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 -- 172.21.15.29:0/4048430685 >> 172.21.15.29:0/4048430685 conn(0x55d713faa910 msgr2=0x55d71435a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 -- 172.21.15.29:0/4048430685 shutdown_connections 2024-09-21T02:44:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 -- 172.21.15.29:0/4048430685 wait complete. 2024-09-21T02:44:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 -- start start 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.839+0000 7f2818ca6740 1 Processor -- start 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f8d660 0x55d71436c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71436c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143781d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2818ca6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d71425e300 con 0x55d713fb6520 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2818ca6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d7142c7430 con 0x55d713f95840 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2818ca6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d714124bb0 con 0x55d713f8d660 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2818ca6740 10 client.0 _ll_register_callbacks cb 0x55d714278120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143781d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143781d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49776/0 (socket says 172.21.15.29:49776) 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 -- 172.21.15.29:0/73012122 learned_addr learned my addr 172.21.15.29:0/73012122 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f280bfff640 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f8d660 0x55d71436c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2803fff640 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71436c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 -- 172.21.15.29:0/73012122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f8d660 msgr2=0x55d71436c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f8d660 0x55d71436c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 -- 172.21.15.29:0/73012122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 msgr2=0x55d71436c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71436c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 -- 172.21.15.29:0/73012122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d71424d780 con 0x55d713fb6520 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2803fff640 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71436c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f2810ccf640 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143781d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2804012850 tx=0x7f2804004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e1a8ff8cb966b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f27eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.840+0000 7f27eb7fe640 1 -- 172.21.15.29:0/73012122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280400fcf0 con 0x55d713fb6520 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f27eb7fe640 1 -- 172.21.15.29:0/73012122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f280401f420 con 0x55d713fb6520 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 1 -- 172.21.15.29:0/73012122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27fc0acd20 con 0x55d713fb6520 2024-09-21T02:44:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f27eb7fe640 1 -- 172.21.15.29:0/73012122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2804030050 con 0x55d713fb6520 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 10 client.16278 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 1 -- 172.21.15.29:0/73012122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d714021b00 con 0x55d713fb6520 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f27eb7fe640 1 -- 172.21.15.29:0/73012122 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27fc0acd20 con 0x55d713fb6520 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f27eb7fe640 1 -- 172.21.15.29:0/73012122 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f280401d440 con 0x55d713fb6520 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 10 client.16278 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 10 client.16278 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 20 client.16278 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d713f74ea0 0x55d71437e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 4 client.16278 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.841+0000 7f2818ca6740 1 -- 172.21.15.29:0/73012122 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7142d6140 con 0x55d713f74ea0 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.842+0000 7f280bfff640 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d713f74ea0 0x55d71437e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.842+0000 7f280bfff640 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d713f74ea0 0x55d71437e320 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc38ed33a9d94d5b server_cookie=821fd99b699188d2 in_seq=0 out_seq=0 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.842+0000 7f27eb7fe640 10 client.16278 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f27eb7fe640 1 -- 172.21.15.29:0/73012122 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d7142d6140 con 0x55d713f74ea0 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f27eb7fe640 10 client.16278 handle_command_reply: tid=0 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 client.16278 shutdown 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 -- 172.21.15.29:0/73012122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 msgr2=0x55d7143781d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143781d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2804012850 tx=0x7f2804004a40 comp rx=0 tx=0).stop 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 -- 172.21.15.29:0/73012122 shutdown_connections 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d713f74ea0 0x55d71437e320 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143781d0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d71436c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f27eb7fe640 0 client.16278 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:22.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 --2- 172.21.15.29:0/73012122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f8d660 0x55d71436c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 -- 172.21.15.29:0/73012122 >> 172.21.15.29:0/73012122 conn(0x55d713faa910 msgr2=0x55d714358930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:22.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 -- 172.21.15.29:0/73012122 shutdown_connections 2024-09-21T02:44:22.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.843+0000 7f2818ca6740 1 -- 172.21.15.29:0/73012122 wait complete. 2024-09-21T02:44:22.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.844+0000 7f2818ca6740 20 client.16278 trim_cache size 0 max 16384 2024-09-21T02:44:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f2818ca6740 1 Processor -- start 2024-09-21T02:44:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f2818ca6740 1 -- start start 2024-09-21T02:44:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d7143780d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d714369210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 0x55d714369750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f2818ca6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f280c1ae770 con 0x55d713f8d660 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f2818ca6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f280c1b02b0 con 0x55d713f95840 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f2818ca6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f280c1aee40 con 0x55d713fb6520 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.861+0000 7f280bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d7143780d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f280bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d7143780d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49786/0 (socket says 172.21.15.29:49786) 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f280bfff640 1 -- 172.21.15.29:0/1871970263 learned_addr learned my addr 172.21.15.29:0/1871970263 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f2810ccf640 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 0x55d714369750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f2803fff640 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d714369210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f280bfff640 1 -- 172.21.15.29:0/1871970263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 msgr2=0x55d714369750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f280bfff640 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 0x55d714369750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f280bfff640 1 -- 172.21.15.29:0/1871970263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 msgr2=0x55d714369210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f280bfff640 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d714369210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f280bfff640 1 -- 172.21.15.29:0/1871970263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d714098960 con 0x55d713f8d660 2024-09-21T02:44:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f2803fff640 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d714369210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:44:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f280bfff640 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d7143780d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0eff40 tx=0x7f27fc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0087b6c99c4969 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f2801ffb640 1 -- 172.21.15.29:0/1871970263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc05c020 con 0x55d713f8d660 2024-09-21T02:44:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f2801ffb640 1 -- 172.21.15.29:0/1871970263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27fc0acd20 con 0x55d713f8d660 2024-09-21T02:44:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.862+0000 7f2801ffb640 1 -- 172.21.15.29:0/1871970263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc01d040 con 0x55d713f8d660 2024-09-21T02:44:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 -- 172.21.15.29:0/1871970263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 msgr2=0x55d7143780d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d7143780d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0eff40 tx=0x7f27fc00cf60 comp rx=0 tx=0).stop 2024-09-21T02:44:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 -- 172.21.15.29:0/1871970263 shutdown_connections 2024-09-21T02:44:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713fb6520 0x55d714369750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f95840 0x55d714369210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 --2- 172.21.15.29:0/1871970263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713f8d660 0x55d7143780d0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 -- 172.21.15.29:0/1871970263 >> 172.21.15.29:0/1871970263 conn(0x55d713faa910 msgr2=0x55d71435a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 -- 172.21.15.29:0/1871970263 shutdown_connections 2024-09-21T02:44:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 -- 172.21.15.29:0/1871970263 wait complete. 2024-09-21T02:44:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.863+0000 7f2818ca6740 1 -- start start 2024-09-21T02:44:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2818ca6740 1 Processor -- start 2024-09-21T02:44:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f8d660 0x55d71436f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f95840 0x55d71436fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2818ca6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143e8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2818ca6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f280c1afbe0 con 0x55d713fb6520 2024-09-21T02:44:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2818ca6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f280c1af510 con 0x55d713f8d660 2024-09-21T02:44:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2818ca6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f280c1ac750 con 0x55d713f95840 2024-09-21T02:44:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2818ca6740 10 client.0 _ll_register_callbacks cb 0x7f2804009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2810ccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143e8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f280bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f8d660 0x55d71436f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2810ccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143e8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49798/0 (socket says 172.21.15.29:49798) 2024-09-21T02:44:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2810ccf640 1 -- 172.21.15.29:0/2437740937 learned_addr learned my addr 172.21.15.29:0/2437740937 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.864+0000 7f2803fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f95840 0x55d71436fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2810ccf640 1 -- 172.21.15.29:0/2437740937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f95840 msgr2=0x55d71436fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2810ccf640 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f95840 0x55d71436fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2810ccf640 1 -- 172.21.15.29:0/2437740937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f8d660 msgr2=0x55d71436f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2810ccf640 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f8d660 0x55d71436f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2810ccf640 1 -- 172.21.15.29:0/2437740937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7143e9190 con 0x55d713fb6520 2024-09-21T02:44:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2810ccf640 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143e8c50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f280401d7e0 tx=0x7f280400fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2ed5b448e7efd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f27eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f27eb7fe640 1 -- 172.21.15.29:0/2437740937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280403a020 con 0x55d713fb6520 2024-09-21T02:44:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f27eb7fe640 1 -- 172.21.15.29:0/2437740937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f280401da50 con 0x55d713fb6520 2024-09-21T02:44:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2818ca6740 1 -- 172.21.15.29:0/2437740937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27fc0acd20 con 0x55d713fb6520 2024-09-21T02:44:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f27eb7fe640 1 -- 172.21.15.29:0/2437740937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f280400f730 con 0x55d713fb6520 2024-09-21T02:44:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2818ca6740 10 client.16284 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f2818ca6740 1 -- 172.21.15.29:0/2437740937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d7143ea200 con 0x55d713fb6520 2024-09-21T02:44:22.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.865+0000 7f27eb7fe640 1 -- 172.21.15.29:0/2437740937 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27fc0acd20 con 0x55d713fb6520 2024-09-21T02:44:22.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f27eb7fe640 1 -- 172.21.15.29:0/2437740937 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2804033070 con 0x55d713fb6520 2024-09-21T02:44:22.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f2818ca6740 10 client.16284 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:22.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f2818ca6740 10 client.16284 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:44:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f2818ca6740 20 client.16284 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f2818ca6740 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d713f4a250 0x55d7143eca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f2818ca6740 4 client.16284 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e924aa9e", "release": true, "await": true}] 2024-09-21T02:44:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f2818ca6740 1 -- 172.21.15.29:0/2437740937 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e924aa9e", "release": true, "await": true}) -- 0x55d7142d1bf0 con 0x55d713f4a250 2024-09-21T02:44:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f280bfff640 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d713f4a250 0x55d7143eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f280bfff640 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d713f4a250 0x55d7143eca40 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa16379d28e57eb1 server_cookie=95d611cddafff4c9 in_seq=0 out_seq=0 2024-09-21T02:44:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.866+0000 7f27eb7fe640 10 client.16284 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f27eb7fe640 1 -- 172.21.15.29:0/2437740937 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d7142d1bf0 con 0x55d713f4a250 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f27eb7fe640 10 client.16284 handle_command_reply: tid=0 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 client.16284 shutdown 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 -- 172.21.15.29:0/2437740937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 msgr2=0x55d7143e8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143e8c50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f280401d7e0 tx=0x7f280400fe20 comp rx=0 tx=0).stop 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 -- 172.21.15.29:0/2437740937 shutdown_connections 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d713f4a250 0x55d7143eca40 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d713fb6520 0x55d7143e8c50 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d713f95840 0x55d71436fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f27eb7fe640 0 client.16284 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 --2- 172.21.15.29:0/2437740937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d713f8d660 0x55d71436f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 -- 172.21.15.29:0/2437740937 >> 172.21.15.29:0/2437740937 conn(0x55d713faa910 msgr2=0x55d71435a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 -- 172.21.15.29:0/2437740937 shutdown_connections 2024-09-21T02:44:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 1 -- 172.21.15.29:0/2437740937 wait complete. 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.882+0000 7f2818ca6740 20 client.16284 trim_cache size 0 max 16384 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 20, 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "e924aa9e": { 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 20, 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:44:22.885 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 130.0, 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:22.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 -- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27f80b6590 msgr2=0x7f27f80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27f80b6590 0x7f27f80b8aa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2804005fd0 tx=0x7f2804009620 comp rx=0 tx=0).stop 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 -- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c102a80 msgr2=0x7f280c19a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c102a80 0x7f280c19a3a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c970 tx=0x7f27fc00ce30 comp rx=0 tx=0).stop 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 -- 172.21.15.29:0/1841858696 shutdown_connections 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27f80b6590 0x7f27f80b8aa0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f280c108d10 0x7f280c19fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f280c1083c0 0x7f280c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 --2- 172.21.15.29:0/1841858696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280c102a80 0x7f280c19a3a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 -- 172.21.15.29:0/1841858696 >> 172.21.15.29:0/1841858696 conn(0x7f280c079740 msgr2=0x7f280c104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 -- 172.21.15.29:0/1841858696 shutdown_connections 2024-09-21T02:44:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:22.883+0000 7f2812732640 1 -- 172.21.15.29:0/1841858696 wait complete. 2024-09-21T02:44:23.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92409 2.81 MB/sec execute 120 sec latency 9773.187 ms 2024-09-21T02:44:23.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:23 smithi029 ceph-mon[31006]: pgmap v547: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 600 KiB/s wr, 61 op/s 2024-09-21T02:44:23.814 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e924aa9e', total seconds elapsed: 13.8 2024-09-21T02:44:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:23 smithi190 ceph-mon[33871]: pgmap v547: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 600 KiB/s wr, 61 op/s 2024-09-21T02:44:24.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:23 smithi066 ceph-mon[33436]: pgmap v547: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 600 KiB/s wr, 61 op/s 2024-09-21T02:44:24.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92700 2.80 MB/sec execute 121 sec latency 14.499 ms 2024-09-21T02:44:25.076 INFO:tasks.workunit.client.0.smithi029.stdout: 1 93206 2.78 MB/sec execute 122 sec latency 15.420 ms 2024-09-21T02:44:25.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:25 smithi190 ceph-mon[33871]: pgmap v548: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 351 KiB/s wr, 52 op/s 2024-09-21T02:44:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:25 smithi029 ceph-mon[31006]: pgmap v548: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 351 KiB/s wr, 52 op/s 2024-09-21T02:44:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:25 smithi066 ceph-mon[33436]: pgmap v548: 129 pgs: 129 active+clean; 31 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 351 KiB/s wr, 52 op/s 2024-09-21T02:44:26.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 93847 2.80 MB/sec execute 123 sec latency 16.680 ms 2024-09-21T02:44:27.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 94493 2.78 MB/sec execute 124 sec latency 17.604 ms 2024-09-21T02:44:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:27 smithi190 ceph-mon[33871]: pgmap v549: 129 pgs: 129 active+clean; 32 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 452 KiB/s wr, 62 op/s 2024-09-21T02:44:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:27 smithi029 ceph-mon[31006]: pgmap v549: 129 pgs: 129 active+clean; 32 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 452 KiB/s wr, 62 op/s 2024-09-21T02:44:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:27 smithi066 ceph-mon[33436]: pgmap v549: 129 pgs: 129 active+clean; 32 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 452 KiB/s wr, 62 op/s 2024-09-21T02:44:28.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 95256 2.81 MB/sec execute 125 sec latency 33.472 ms 2024-09-21T02:44:29.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 95887 2.83 MB/sec execute 126 sec latency 15.835 ms 2024-09-21T02:44:29.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:29 smithi190 ceph-mon[33871]: pgmap v550: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 248 KiB/s wr, 49 op/s 2024-09-21T02:44:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:29 smithi029 ceph-mon[31006]: pgmap v550: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 248 KiB/s wr, 49 op/s 2024-09-21T02:44:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:29 smithi066 ceph-mon[33436]: pgmap v550: 129 pgs: 129 active+clean; 30 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 248 KiB/s wr, 49 op/s 2024-09-21T02:44:30.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 96215 2.82 MB/sec execute 127 sec latency 17.174 ms 2024-09-21T02:44:31.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 96641 2.80 MB/sec execute 128 sec latency 20.664 ms 2024-09-21T02:44:31.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:31 smithi190 ceph-mon[33871]: pgmap v551: 129 pgs: 129 active+clean; 33 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 689 KiB/s wr, 81 op/s 2024-09-21T02:44:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:31 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:44:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:31 smithi029 ceph-mon[31006]: pgmap v551: 129 pgs: 129 active+clean; 33 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 689 KiB/s wr, 81 op/s 2024-09-21T02:44:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:31 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:44:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:31 smithi066 ceph-mon[33436]: pgmap v551: 129 pgs: 129 active+clean; 33 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 689 KiB/s wr, 81 op/s 2024-09-21T02:44:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:31 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:44:32.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 97284 2.81 MB/sec execute 129 sec latency 18.229 ms 2024-09-21T02:44:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:44:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:44:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:44:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:44:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:44:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:44:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:44:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:44:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:44:33.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 97907 2.80 MB/sec execute 130 sec latency 16.471 ms 2024-09-21T02:44:33.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:33 smithi029 ceph-mon[31006]: pgmap v552: 129 pgs: 129 active+clean; 33 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 903 KiB/s wr, 82 op/s 2024-09-21T02:44:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:33 smithi190 ceph-mon[33871]: pgmap v552: 129 pgs: 129 active+clean; 33 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 903 KiB/s wr, 82 op/s 2024-09-21T02:44:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:33 smithi066 ceph-mon[33436]: pgmap v552: 129 pgs: 129 active+clean; 33 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 903 KiB/s wr, 82 op/s 2024-09-21T02:44:34.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 98577 2.82 MB/sec execute 131 sec latency 14.847 ms 2024-09-21T02:44:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 -- 172.21.15.29:0/3110638720 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 msgr2=0x7ff490103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3110638720 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 0x7ff490103a40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff480009860 tx=0x7ff48002a340 comp rx=0 tx=0).stop 2024-09-21T02:44:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 -- 172.21.15.29:0/3110638720 shutdown_connections 2024-09-21T02:44:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3110638720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff490103fd0 0x7ff49010a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3110638720 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 0x7ff490103a40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3110638720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4901025b0 0x7ff490102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 -- 172.21.15.29:0/3110638720 >> 172.21.15.29:0/3110638720 conn(0x7ff4900fdf30 msgr2=0x7ff4901003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 -- 172.21.15.29:0/3110638720 shutdown_connections 2024-09-21T02:44:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.198+0000 7ff496e4d640 1 -- 172.21.15.29:0/3110638720 wait complete. 2024-09-21T02:44:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff496e4d640 1 Processor -- start 2024-09-21T02:44:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff496e4d640 1 -- start start 2024-09-21T02:44:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff496e4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4901025b0 0x7ff490197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff496e4d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 0x7ff4901982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff496e4d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff490103fd0 0x7ff490198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff496e4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff49010cfe0 con 0x7ff4901025b0 2024-09-21T02:44:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff496e4d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff49010ce60 con 0x7ff490103630 2024-09-21T02:44:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff496e4d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff49010d160 con 0x7ff490103fd0 2024-09-21T02:44:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff494be9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4901025b0 0x7ff490197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff487fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 0x7ff4901982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff4953ea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff490103fd0 0x7ff490198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff487fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 0x7ff4901982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:48668/0 (socket says 172.21.15.29:48668) 2024-09-21T02:44:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff494be9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4901025b0 0x7ff490197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59092/0 (socket says 172.21.15.29:59092) 2024-09-21T02:44:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.199+0000 7ff487fff640 1 -- 172.21.15.29:0/3751085078 learned_addr learned my addr 172.21.15.29:0/3751085078 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff487fff640 1 -- 172.21.15.29:0/3751085078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff490103fd0 msgr2=0x7ff490198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff487fff640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff490103fd0 0x7ff490198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff487fff640 1 -- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4901025b0 msgr2=0x7ff490197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff487fff640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4901025b0 0x7ff490197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff487fff640 1 -- 172.21.15.29:0/3751085078 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4901996c0 con 0x7ff490103630 2024-09-21T02:44:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff494be9640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4901025b0 0x7ff490197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff487fff640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 0x7ff4901982f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff48002a930 tx=0x7ff48002ad70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff485ffb640 1 -- 172.21.15.29:0/3751085078 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480040030 con 0x7ff490103630 2024-09-21T02:44:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff485ffb640 1 -- 172.21.15.29:0/3751085078 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff480032bc0 con 0x7ff490103630 2024-09-21T02:44:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff496e4d640 1 -- 172.21.15.29:0/3751085078 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff490199950 con 0x7ff490103630 2024-09-21T02:44:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff485ffb640 1 -- 172.21.15.29:0/3751085078 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480037040 con 0x7ff490103630 2024-09-21T02:44:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.200+0000 7ff496e4d640 1 -- 172.21.15.29:0/3751085078 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4901a00a0 con 0x7ff490103630 2024-09-21T02:44:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff49ddc7740 1 Processor -- start 2024-09-21T02:44:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff49ddc7740 1 -- start start 2024-09-21T02:44:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a6e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a6e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476a14d0 0x55a347a6f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a3479dc6f0 con 0x55a3476992f0 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a3479cbb20 con 0x55a3476a14d0 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a3479cf930 con 0x55a3476c21b0 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff485ffb640 1 -- 172.21.15.29:0/3751085078 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff480043070 con 0x7ff490103630 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff494be9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a6e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.203+0000 7ff4953ea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a6e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff494be9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a6e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59104/0 (socket says 172.21.15.29:59104) 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff494be9640 1 -- 172.21.15.29:0/3096051874 learned_addr learned my addr 172.21.15.29:0/3096051874 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff487fff640 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476a14d0 0x55a347a6f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff4953ea640 1 -- 172.21.15.29:0/3096051874 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476a14d0 msgr2=0x55a347a6f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff4953ea640 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476a14d0 0x55a347a6f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff4953ea640 1 -- 172.21.15.29:0/3096051874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 msgr2=0x55a347a6e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff4953ea640 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a6e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff4953ea640 1 -- 172.21.15.29:0/3096051874 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a347991530 con 0x55a3476c21b0 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff494be9640 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a6e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff4953ea640 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a6e260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff48800bf80 tx=0x7ff48800b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5600c64b41b35324 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.204+0000 7ff476ffd640 1 -- 172.21.15.29:0/3096051874 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48801a030 con 0x55a3476c21b0 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.205+0000 7ff476ffd640 1 -- 172.21.15.29:0/3096051874 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff488009e70 con 0x55a3476c21b0 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.205+0000 7ff476ffd640 1 -- 172.21.15.29:0/3096051874 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488004470 con 0x55a3476c21b0 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.205+0000 7ff485ffb640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4780b6590 0x7ff4780b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.205+0000 7ff494be9640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4780b6590 0x7ff4780b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.205+0000 7ff485ffb640 1 -- 172.21.15.29:0/3751085078 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff4801072a0 con 0x7ff490103630 2024-09-21T02:44:34.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.205+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3096051874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 msgr2=0x55a347a6e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.205+0000 7ff494be9640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4780b6590 0x7ff4780b8aa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff48c0128a0 tx=0x7ff48c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.205+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a6e260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff48800bf80 tx=0x7ff48800b710 comp rx=0 tx=0).stop 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3096051874 shutdown_connections 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476a14d0 0x55a347a6f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a6e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3096051874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a6e260 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3096051874 >> 172.21.15.29:0/3096051874 conn(0x55a3476b6300 msgr2=0x55a347a658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3096051874 shutdown_connections 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3096051874 wait complete. 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 -- start start 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.206+0000 7ff49ddc7740 1 Processor -- start 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a77a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a780e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476c21b0 0x55a347a78c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a347903ab0 con 0x55a3476c21b0 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a347971730 con 0x55a3476992f0 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a347a6ff40 con 0x55a3476a14d0 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff49ddc7740 10 client.0 _ll_register_callbacks cb 0x55a3479c56b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff494be9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a77a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a780e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff4953ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476c21b0 0x55a347a78c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a780e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:46944/0 (socket says 172.21.15.29:46944) 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 -- 172.21.15.29:0/3091853968 learned_addr learned my addr 172.21.15.29:0/3091853968 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff494be9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a77a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:48688/0 (socket says 172.21.15.29:48688) 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff4953ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476c21b0 0x55a347a78c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59114/0 (socket says 172.21.15.29:59114) 2024-09-21T02:44:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 -- 172.21.15.29:0/3091853968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 msgr2=0x55a347a77a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a77a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 -- 172.21.15.29:0/3091853968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476c21b0 msgr2=0x55a347a78c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476c21b0 0x55a347a78c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 -- 172.21.15.29:0/3091853968 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a34772d790 con 0x55a3476a14d0 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff487fff640 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a780e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff480005cd0 tx=0x7ff4800c91a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61343306cf34bf54 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff46ffff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.207+0000 7ff46ffff640 1 -- 172.21.15.29:0/3091853968 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4800ccf10 con 0x55a3476a14d0 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff46ffff640 1 -- 172.21.15.29:0/3091853968 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff480005480 con 0x55a3476a14d0 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3091853968 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff488009e70 con 0x55a3476a14d0 2024-09-21T02:44:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff46ffff640 1 -- 172.21.15.29:0/3091853968 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff4800cc8b0 con 0x55a3476a14d0 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 10 client.25906 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3091853968 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a3477a6620 con 0x55a3476a14d0 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff46ffff640 1 -- 172.21.15.29:0/3091853968 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff488009e70 con 0x55a3476a14d0 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff46ffff640 1 -- 172.21.15.29:0/3091853968 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff48004d060 con 0x55a3476a14d0 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 10 client.25906 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 10 client.25906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 20 client.25906 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a347680d80 0x55a347a851e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 4 client.25906 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3091853968 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3479de640 con 0x55a347680d80 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.208+0000 7ff494be9640 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a347680d80 0x55a347a851e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.209+0000 7ff494be9640 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a347680d80 0x55a347a851e0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd819473366f2587 server_cookie=9467ec5213faa979 in_seq=0 out_seq=0 2024-09-21T02:44:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.209+0000 7ff46ffff640 10 client.25906 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff46ffff640 1 -- 172.21.15.29:0/3091853968 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a3479de640 con 0x55a347680d80 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff46ffff640 10 client.25906 handle_command_reply: tid=0 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 client.25906 shutdown 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3091853968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 msgr2=0x55a347a780e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a780e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff480005cd0 tx=0x7ff4800c91a0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3091853968 shutdown_connections 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a347680d80 0x55a347a851e0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476c21b0 0x55a347a78c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff46ffff640 0 client.25906 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a780e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3091853968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a77a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3091853968 >> 172.21.15.29:0/3091853968 conn(0x55a3476b6300 msgr2=0x55a347a63ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3091853968 shutdown_connections 2024-09-21T02:44:34.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3091853968 wait complete. 2024-09-21T02:44:34.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.210+0000 7ff49ddc7740 20 client.25906 trim_cache size 0 max 16384 2024-09-21T02:44:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff49ddc7740 1 Processor -- start 2024-09-21T02:44:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff49ddc7740 1 -- start start 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a6f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476a14d0 0x55a347a605c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a75f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901aa2a0 con 0x55a3476a14d0 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901abde0 con 0x55a3476992f0 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901aa970 con 0x55a3476c21b0 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff494be9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a6f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff4953ea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a75f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff487fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476a14d0 0x55a347a605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff494be9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a6f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:48690/0 (socket says 172.21.15.29:48690) 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff494be9640 1 -- 172.21.15.29:0/3901140352 learned_addr learned my addr 172.21.15.29:0/3901140352 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff494be9640 1 -- 172.21.15.29:0/3901140352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 msgr2=0x55a347a75f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff494be9640 1 --2- 172.21.15.29:0/3901140352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a75f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff494be9640 1 -- 172.21.15.29:0/3901140352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476a14d0 msgr2=0x55a347a605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff494be9640 1 --2- 172.21.15.29:0/3901140352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476a14d0 0x55a347a605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.229+0000 7ff494be9640 1 -- 172.21.15.29:0/3901140352 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a347a7fef0 con 0x55a3476992f0 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.230+0000 7ff494be9640 1 --2- 172.21.15.29:0/3901140352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a6f1c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff48c005fd0 tx=0x7ff48c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fa450af3ac98357 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.230+0000 7ff4767fc640 1 -- 172.21.15.29:0/3901140352 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c004020 con 0x55a3476992f0 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.230+0000 7ff4767fc640 1 -- 172.21.15.29:0/3901140352 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff48c01dcf0 con 0x55a3476992f0 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.230+0000 7ff4767fc640 1 -- 172.21.15.29:0/3901140352 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c0035f0 con 0x55a3476992f0 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3901140352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 msgr2=0x55a347a6f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3901140352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a6f1c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff48c005fd0 tx=0x7ff48c01b000 comp rx=0 tx=0).stop 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3901140352 shutdown_connections 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3901140352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476c21b0 0x55a347a75f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3901140352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476a14d0 0x55a347a605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3901140352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476992f0 0x55a347a6f1c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3901140352 >> 172.21.15.29:0/3901140352 conn(0x55a3476b6300 msgr2=0x55a347a76c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3901140352 shutdown_connections 2024-09-21T02:44:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3901140352 wait complete. 2024-09-21T02:44:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 -- start start 2024-09-21T02:44:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.231+0000 7ff49ddc7740 1 Processor -- start 2024-09-21T02:44:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a7f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a7e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff49ddc7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476c21b0 0x55a347a7e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901ab710 con 0x55a3476992f0 2024-09-21T02:44:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901ab040 con 0x55a3476c21b0 2024-09-21T02:44:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff49ddc7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901a8280 con 0x55a3476a14d0 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff49ddc7740 10 client.0 _ll_register_callbacks cb 0x7ff48010aa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff494be9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a7f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff494be9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a7f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59126/0 (socket says 172.21.15.29:59126) 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff494be9640 1 -- 172.21.15.29:0/3245535389 learned_addr learned my addr 172.21.15.29:0/3245535389 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff487fff640 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a7e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff4953ea640 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476c21b0 0x55a347a7e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff494be9640 1 -- 172.21.15.29:0/3245535389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 msgr2=0x55a347a7e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff494be9640 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a7e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff494be9640 1 -- 172.21.15.29:0/3245535389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476c21b0 msgr2=0x55a347a7e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff494be9640 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476c21b0 0x55a347a7e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.232+0000 7ff494be9640 1 -- 172.21.15.29:0/3245535389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a347a7f020 con 0x55a3476992f0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff494be9640 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a7f670 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009e60 tx=0x7ff48c01df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec94346c39ad6b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff46ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff46ffff640 1 -- 172.21.15.29:0/3245535389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c01d8e0 con 0x55a3476992f0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff46ffff640 1 -- 172.21.15.29:0/3245535389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff48c01da80 con 0x55a3476992f0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3245535389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff48c01dcf0 con 0x55a3476992f0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff46ffff640 1 -- 172.21.15.29:0/3245535389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff48c049050 con 0x55a3476992f0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 10 client.16302 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3245535389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a347aefba0 con 0x55a3476992f0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff46ffff640 1 -- 172.21.15.29:0/3245535389 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff48c01dcf0 con 0x55a3476992f0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff46ffff640 1 -- 172.21.15.29:0/3245535389 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff48c044030 con 0x55a3476992f0 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 10 client.16302 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 10 client.16302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:44:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 20 client.16302 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3476561c0 0x55a347af2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 4 client.16302 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:44:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.233+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3245535389 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a347a6f8a0 con 0x55a3476561c0 2024-09-21T02:44:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.234+0000 7ff487fff640 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3476561c0 0x55a347af2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.234+0000 7ff46ffff640 10 client.16302 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.234+0000 7ff487fff640 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3476561c0 0x55a347af2570 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3658c9baa4e57a74 server_cookie=de8ed3ba6ba51830 in_seq=0 out_seq=0 2024-09-21T02:44:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.274+0000 7ff46ffff640 1 -- 172.21.15.29:0/3245535389 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a347a6f8a0 con 0x55a3476561c0 2024-09-21T02:44:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.274+0000 7ff46ffff640 10 client.16302 handle_command_reply: tid=0 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.274+0000 7ff49ddc7740 1 client.16302 shutdown 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3245535389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 msgr2=0x55a347a7f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a7f670 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009e60 tx=0x7ff48c01df30 comp rx=0 tx=0).stop 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3245535389 shutdown_connections 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a3476c21b0 0x55a347a7e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3476a14d0 0x55a347a7e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a3476561c0 0x55a347af2570 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 --2- 172.21.15.29:0/3245535389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3476992f0 0x55a347a7f670 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff46ffff640 0 client.16302 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3245535389 >> 172.21.15.29:0/3245535389 conn(0x55a3476b6300 msgr2=0x55a347a76c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3245535389 shutdown_connections 2024-09-21T02:44:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 1 -- 172.21.15.29:0/3245535389 wait complete. 2024-09-21T02:44:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.275+0000 7ff49ddc7740 20 client.16302 trim_cache size 0 max 16384 2024-09-21T02:44:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:44:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:44:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:44:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 -- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4780b6590 msgr2=0x7ff4780b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4780b6590 0x7ff4780b8aa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff48c0128a0 tx=0x7ff48c009260 comp rx=0 tx=0).stop 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 -- 172.21.15.29:0/3751085078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 msgr2=0x7ff4901982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 0x7ff4901982f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff48002a930 tx=0x7ff48002ad70 comp rx=0 tx=0).stop 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 -- 172.21.15.29:0/3751085078 shutdown_connections 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4780b6590 0x7ff4780b8aa0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff490103fd0 0x7ff490198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff490103630 0x7ff4901982f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 --2- 172.21.15.29:0/3751085078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4901025b0 0x7ff490197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 -- 172.21.15.29:0/3751085078 >> 172.21.15.29:0/3751085078 conn(0x7ff4900fdf30 msgr2=0x7ff490106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 -- 172.21.15.29:0/3751085078 shutdown_connections 2024-09-21T02:44:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:34.276+0000 7ff496e4d640 1 -- 172.21.15.29:0/3751085078 wait complete. 2024-09-21T02:44:35.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 99186 2.82 MB/sec execute 132 sec latency 41.853 ms 2024-09-21T02:44:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:35 smithi190 ceph-mon[33871]: pgmap v553: 129 pgs: 129 active+clean; 32 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 971 KiB/s wr, 92 op/s 2024-09-21T02:44:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:35 smithi029 ceph-mon[31006]: pgmap v553: 129 pgs: 129 active+clean; 32 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 971 KiB/s wr, 92 op/s 2024-09-21T02:44:36.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:35 smithi066 ceph-mon[33436]: pgmap v553: 129 pgs: 129 active+clean; 32 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 971 KiB/s wr, 92 op/s 2024-09-21T02:44:36.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 99719 2.84 MB/sec execute 133 sec latency 15.671 ms 2024-09-21T02:44:37.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 100058 2.82 MB/sec execute 134 sec latency 15.904 ms 2024-09-21T02:44:37.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:37 smithi190 ceph-mon[33871]: pgmap v554: 129 pgs: 129 active+clean; 29 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-09-21T02:44:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:37 smithi029 ceph-mon[31006]: pgmap v554: 129 pgs: 129 active+clean; 29 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-09-21T02:44:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:37 smithi066 ceph-mon[33436]: pgmap v554: 129 pgs: 129 active+clean; 29 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-09-21T02:44:38.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 100746 2.83 MB/sec execute 135 sec latency 14.925 ms 2024-09-21T02:44:39.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 101343 2.81 MB/sec execute 136 sec latency 22.028 ms 2024-09-21T02:44:39.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:39 smithi190 ceph-mon[33871]: pgmap v555: 129 pgs: 129 active+clean; 31 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-09-21T02:44:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:39 smithi029 ceph-mon[31006]: pgmap v555: 129 pgs: 129 active+clean; 31 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-09-21T02:44:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:39 smithi066 ceph-mon[33436]: pgmap v555: 129 pgs: 129 active+clean; 31 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-09-21T02:44:40.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 101956 2.81 MB/sec execute 137 sec latency 15.549 ms 2024-09-21T02:44:41.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 102688 2.84 MB/sec execute 138 sec latency 16.702 ms 2024-09-21T02:44:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:41 smithi190 ceph-mon[33871]: pgmap v556: 129 pgs: 129 active+clean; 31 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:44:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:41 smithi029 ceph-mon[31006]: pgmap v556: 129 pgs: 129 active+clean; 31 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:44:42.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:41 smithi066 ceph-mon[33436]: pgmap v556: 129 pgs: 129 active+clean; 31 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:44:42.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 103201 2.85 MB/sec execute 139 sec latency 56.644 ms 2024-09-21T02:44:42.386 INFO:tasks.quiescer.fs.[cephfs]:Total duration (38.5) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:44:42.387 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-09-21T02:44:42.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:44:42.658 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:44:43.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 103491 2.83 MB/sec execute 140 sec latency 14.975 ms 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 -- 172.21.15.29:0/950719139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb8104860 msgr2=0x7faeb8104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 --2- 172.21.15.29:0/950719139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb8104860 0x7faeb8104c70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7faea8009900 tx=0x7faea802a410 comp rx=0 tx=0).stop 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 -- 172.21.15.29:0/950719139 shutdown_connections 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 --2- 172.21.15.29:0/950719139 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faeb810de10 0x7faeb81102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 --2- 172.21.15.29:0/950719139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb8105290 0x7faeb810d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 --2- 172.21.15.29:0/950719139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb8104860 0x7faeb8104c70 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 -- 172.21.15.29:0/950719139 >> 172.21.15.29:0/950719139 conn(0x7faeb8079730 msgr2=0x7faeb8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 -- 172.21.15.29:0/950719139 shutdown_connections 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.214+0000 7faebd9ad640 1 -- 172.21.15.29:0/950719139 wait complete. 2024-09-21T02:44:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.215+0000 7faebd9ad640 1 Processor -- start 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.215+0000 7faebd9ad640 1 -- start start 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faebd9ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb8104860 0x7faeb819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faebd9ad640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faeb8105290 0x7faeb819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faebd9ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb810de10 0x7faeb819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faebd9ad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faeb8113930 con 0x7faeb810de10 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faebd9ad640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faeb81137b0 con 0x7faeb8105290 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faebd9ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faeb8113ab0 con 0x7faeb8104860 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb810de10 0x7faeb819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb8104860 0x7faeb819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb8104860 0x7faeb819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:46992/0 (socket says 172.21.15.29:46992) 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb6ffd640 1 -- 172.21.15.29:0/63181207 learned_addr learned my addr 172.21.15.29:0/63181207 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeaffff640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faeb8105290 0x7faeb819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb77fe640 1 -- 172.21.15.29:0/63181207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb8104860 msgr2=0x7faeb819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb77fe640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb8104860 0x7faeb819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb77fe640 1 -- 172.21.15.29:0/63181207 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faeb8105290 msgr2=0x7faeb819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb77fe640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faeb8105290 0x7faeb819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb77fe640 1 -- 172.21.15.29:0/63181207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb81a03b0 con 0x7faeb810de10 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb6ffd640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb8104860 0x7faeb819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.216+0000 7faeb77fe640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb810de10 0x7faeb819fc80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faeb000da10 tx=0x7faeb000ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.217+0000 7faebc9ab640 1 -- 172.21.15.29:0/63181207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb000bd70 con 0x7faeb810de10 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.217+0000 7faebc9ab640 1 -- 172.21.15.29:0/63181207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faeb0003e00 con 0x7faeb810de10 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.217+0000 7faebc9ab640 1 -- 172.21.15.29:0/63181207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb00106c0 con 0x7faeb810de10 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.217+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb81a06a0 con 0x7faeb810de10 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.217+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb81a6e00 con 0x7faeb810de10 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.218+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb806b4e0 con 0x7faeb810de10 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.220+0000 7faebc9ab640 1 -- 172.21.15.29:0/63181207 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7faeb0003970 con 0x7faeb810de10 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.220+0000 7faebc9ab640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faea40b6590 0x7faea40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.221+0000 7faeb6ffd640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faea40b6590 0x7faea40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:43.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.221+0000 7faebc9ab640 1 -- 172.21.15.29:0/63181207 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faeb00e7980 con 0x7faeb810de10 2024-09-21T02:44:43.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.221+0000 7faeb6ffd640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faea40b6590 0x7faea40b8aa0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faea8032fd0 tx=0x7faea80095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:43.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.225+0000 7faebc9ab640 1 -- 172.21.15.29:0/63181207 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faeb00ade90 con 0x7faeb810de10 2024-09-21T02:44:43.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.512+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faeb8108fc0 con 0x7faeb810de10 2024-09-21T02:44:43.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.512+0000 7faebc9ab640 1 -- 172.21.15.29:0/63181207 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7faeb00b2270 con 0x7faeb810de10 2024-09-21T02:44:43.514 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:44:43.514 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:44:43.514 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:44:43.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faea40b6590 msgr2=0x7faea40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:43.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faea40b6590 0x7faea40b8aa0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faea8032fd0 tx=0x7faea80095c0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb810de10 msgr2=0x7faeb819fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb810de10 0x7faeb819fc80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faeb000da10 tx=0x7faeb000ded0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 shutdown_connections 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faea40b6590 0x7faea40b8aa0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb810de10 0x7faeb819fc80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faeb8105290 0x7faeb819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 --2- 172.21.15.29:0/63181207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb8104860 0x7faeb819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 >> 172.21.15.29:0/63181207 conn(0x7faeb8079730 msgr2=0x7faeb81009c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 shutdown_connections 2024-09-21T02:44:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:43.515+0000 7faebd9ad640 1 -- 172.21.15.29:0/63181207 wait complete. 2024-09-21T02:44:43.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:43 smithi190 ceph-mon[33871]: pgmap v557: 129 pgs: 129 active+clean; 32 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-09-21T02:44:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:43 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/63181207' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:43 smithi029 ceph-mon[31006]: pgmap v557: 129 pgs: 129 active+clean; 32 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-09-21T02:44:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:43 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/63181207' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:44.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:43 smithi066 ceph-mon[33436]: pgmap v557: 129 pgs: 129 active+clean; 32 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-09-21T02:44:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:43 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/63181207' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:44:44.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104084 2.82 MB/sec execute 141 sec latency 17.464 ms 2024-09-21T02:44:44.396 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0' 2024-09-21T02:44:44.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0 2024-09-21T02:44:44.686 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:44:45.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104724 2.83 MB/sec execute 142 sec latency 21.016 ms 2024-09-21T02:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 -- 172.21.15.29:0/1071311858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 msgr2=0x7fbe44102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/1071311858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 0x7fbe44102e90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009900 tx=0x7fbe4002a410 comp rx=0 tx=0).stop 2024-09-21T02:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 -- 172.21.15.29:0/1071311858 shutdown_connections 2024-09-21T02:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/1071311858 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe44108d10 0x7fbe4410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/1071311858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe441083c0 0x7fbe441087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/1071311858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 0x7fbe44102e90 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 -- 172.21.15.29:0/1071311858 >> 172.21.15.29:0/1071311858 conn(0x7fbe44079760 msgr2=0x7fbe44079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 -- 172.21.15.29:0/1071311858 shutdown_connections 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.369+0000 7fbe4bc84640 1 -- 172.21.15.29:0/1071311858 wait complete. 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.370+0000 7fbe4bc84640 1 Processor -- start 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.370+0000 7fbe4bc84640 1 -- start start 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4bc84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 0x7fbe44199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4bc84640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe441083c0 0x7fbe44199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4bc84640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe44108d10 0x7fbe4419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4bc84640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44112830 con 0x7fbe44102a80 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4bc84640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441126b0 con 0x7fbe44108d10 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4bc84640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441129b0 con 0x7fbe441083c0 2024-09-21T02:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe49a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 0x7fbe44199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4a221640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe44108d10 0x7fbe4419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4921f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe441083c0 0x7fbe44199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe49a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 0x7fbe44199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32832/0 (socket says 172.21.15.29:32832) 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4921f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe441083c0 0x7fbe44199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:36190/0 (socket says 172.21.15.29:36190) 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe49a20640 1 -- 172.21.15.29:0/266801314 learned_addr learned my addr 172.21.15.29:0/266801314 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4a221640 1 -- 172.21.15.29:0/266801314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe441083c0 msgr2=0x7fbe44199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4a221640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe441083c0 0x7fbe44199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4a221640 1 -- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 msgr2=0x7fbe44199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4a221640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 0x7fbe44199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.371+0000 7fbe4a221640 1 -- 172.21.15.29:0/266801314 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4419f1d0 con 0x7fbe44108d10 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.372+0000 7fbe4a221640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe44108d10 0x7fbe4419eaa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009fd0 tx=0x7fbe3c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.372+0000 7fbe3affd640 1 -- 172.21.15.29:0/266801314 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c00ce20 con 0x7fbe44108d10 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.372+0000 7fbe3affd640 1 -- 172.21.15.29:0/266801314 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe3c005020 con 0x7fbe44108d10 2024-09-21T02:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.372+0000 7fbe4bc84640 1 -- 172.21.15.29:0/266801314 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe4419f4c0 con 0x7fbe44108d10 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.372+0000 7fbe3affd640 1 -- 172.21.15.29:0/266801314 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c010690 con 0x7fbe44108d10 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.372+0000 7fbe4bc84640 1 -- 172.21.15.29:0/266801314 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe4406ae60 con 0x7fbe44108d10 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.375+0000 7fbe3affd640 1 -- 172.21.15.29:0/266801314 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fbe3c004a40 con 0x7fbe44108d10 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.375+0000 7fbe521f8740 1 Processor -- start 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe521f8740 1 -- start start 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7ed9850 0x55a6c827fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb0990 0x55a6c82803b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb8b70 0x55a6c82808f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe521f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a6c81f6bb0 con 0x55a6c7eb0990 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe521f8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a6c81e5fe0 con 0x55a6c7eb8b70 2024-09-21T02:44:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe521f8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a6c81e9df0 con 0x55a6c7ed9850 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe49a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb0990 0x55a6c82803b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe4921f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7ed9850 0x55a6c827fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe49a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb0990 0x55a6c82803b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32838/0 (socket says 172.21.15.29:32838) 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe49a20640 1 -- 172.21.15.29:0/4128351193 learned_addr learned my addr 172.21.15.29:0/4128351193 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.376+0000 7fbe4921f640 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb8b70 0x55a6c82808f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe49a20640 1 -- 172.21.15.29:0/4128351193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7ed9850 msgr2=0x55a6c827fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe49a20640 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7ed9850 0x55a6c827fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe49a20640 1 -- 172.21.15.29:0/4128351193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb8b70 msgr2=0x55a6c82808f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe49a20640 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb8b70 0x55a6c82808f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe49a20640 1 -- 172.21.15.29:0/4128351193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6c7f53ba0 con 0x55a6c7eb0990 2024-09-21T02:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe4921f640 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7ed9850 0x55a6c827fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:45.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe3affd640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe300b6510 0x7fbe300b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe49a20640 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb0990 0x55a6c82803b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009a20 tx=0x7fbe40031b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e2db5971d593f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe4921f640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe300b6510 0x7fbe300b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.377+0000 7fbe2ffff640 1 -- 172.21.15.29:0/4128351193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe40046070 con 0x55a6c7eb0990 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.378+0000 7fbe3affd640 1 -- 172.21.15.29:0/266801314 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbe3c0e7d60 con 0x7fbe44108d10 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.378+0000 7fbe2ffff640 1 -- 172.21.15.29:0/4128351193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe40004030 con 0x55a6c7eb0990 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.378+0000 7fbe2ffff640 1 -- 172.21.15.29:0/4128351193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe40035070 con 0x55a6c7eb0990 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 -- 172.21.15.29:0/4128351193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb0990 msgr2=0x55a6c82803b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb0990 0x55a6c82803b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009a20 tx=0x7fbe40031b00 comp rx=0 tx=0).stop 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe4921f640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe300b6510 0x7fbe300b8a20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbe34015fd0 tx=0x7fbe34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 -- 172.21.15.29:0/4128351193 shutdown_connections 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb8b70 0x55a6c82808f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb0990 0x55a6c82803b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 --2- 172.21.15.29:0/4128351193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7ed9850 0x55a6c827fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 -- 172.21.15.29:0/4128351193 >> 172.21.15.29:0/4128351193 conn(0x55a6c7ecdc40 msgr2=0x55a6c827dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 -- 172.21.15.29:0/4128351193 shutdown_connections 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.379+0000 7fbe521f8740 1 -- 172.21.15.29:0/4128351193 wait complete. 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 1 -- start start 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 1 Processor -- start 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c828b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7eb8b70 0x55a6c828bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7ed9850 0x55a6c8297980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a6c80477b0 con 0x55a6c7ed9850 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a6c8180e90 con 0x55a6c7eb0990 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a6c828e690 con 0x55a6c7eb8b70 2024-09-21T02:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe49a20640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c828b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe49a20640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c828b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:43692/0 (socket says 172.21.15.29:43692) 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe49a20640 1 -- 172.21.15.29:0/2692409647 learned_addr learned my addr 172.21.15.29:0/2692409647 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe4921f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7eb8b70 0x55a6c828bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe521f8740 10 client.0 _ll_register_callbacks cb 0x55a6c81e3710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.380+0000 7fbe4a221640 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7ed9850 0x55a6c8297980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe49a20640 1 -- 172.21.15.29:0/2692409647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7eb8b70 msgr2=0x55a6c828bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe49a20640 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7eb8b70 0x55a6c828bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe49a20640 1 -- 172.21.15.29:0/2692409647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7ed9850 msgr2=0x55a6c8297980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe49a20640 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7ed9850 0x55a6c8297980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe49a20640 1 -- 172.21.15.29:0/2692409647 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6c8297f10 con 0x55a6c7eb0990 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe49a20640 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c828b9f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbe40044040 tx=0x7fbe40031ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce754a7eb309a854 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/2692409647 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe40035070 con 0x55a6c7eb0990 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe521f8740 1 -- 172.21.15.29:0/2692409647 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe40004030 con 0x55a6c7eb0990 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.381+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/2692409647 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbe40031600 con 0x55a6c7eb0990 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/2692409647 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbe40058050 con 0x55a6c7eb0990 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe521f8740 10 client.25838 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe521f8740 1 -- 172.21.15.29:0/2692409647 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a6c829a840 con 0x55a6c7eb0990 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/2692409647 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe40004030 con 0x55a6c7eb0990 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/2692409647 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fbe40053030 con 0x55a6c7eb0990 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe521f8740 10 client.25838 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe521f8740 10 client.25838 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe521f8740 20 client.25838 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe521f8740 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a6c7e981d0 0x55a6c829de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe521f8740 4 client.25838 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.382+0000 7fbe521f8740 1 -- 172.21.15.29:0/2692409647 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6c81f8b00 con 0x55a6c7e981d0 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.383+0000 7fbe4921f640 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a6c7e981d0 0x55a6c829de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.383+0000 7fbe2d7fa640 10 client.25838 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.383+0000 7fbe4921f640 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a6c7e981d0 0x55a6c829de20 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9cd093801001425 server_cookie=6c2cbe702a641394 in_seq=0 out_seq=0 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/2692409647 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a6c81f8b00 con 0x55a6c7e981d0 2024-09-21T02:44:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe2d7fa640 10 client.25838 handle_command_reply: tid=0 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 client.25838 shutdown 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 -- 172.21.15.29:0/2692409647 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 msgr2=0x55a6c828b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c828b9f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbe40044040 tx=0x7fbe40031ea0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 -- 172.21.15.29:0/2692409647 shutdown_connections 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a6c7e981d0 0x55a6c829de20 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7ed9850 0x55a6c8297980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7eb8b70 0x55a6c828bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe2d7fa640 0 client.25838 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 --2- 172.21.15.29:0/2692409647 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c828b9f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 -- 172.21.15.29:0/2692409647 >> 172.21.15.29:0/2692409647 conn(0x55a6c7ecdc40 msgr2=0x55a6c8281860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 -- 172.21.15.29:0/2692409647 shutdown_connections 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.384+0000 7fbe521f8740 1 -- 172.21.15.29:0/2692409647 wait complete. 2024-09-21T02:44:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.385+0000 7fbe521f8740 20 client.25838 trim_cache size 0 max 16384 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe521f8740 1 Processor -- start 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe521f8740 1 -- start start 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c8285e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c82978f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c8284f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe521f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441ae740 con 0x55a6c7eb8b70 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe521f8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441b0280 con 0x55a6c7eb0990 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe521f8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441aee10 con 0x55a6c7e981d0 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe4a221640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c8284f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe4921f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c82978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe4a221640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c8284f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32856/0 (socket says 172.21.15.29:32856) 2024-09-21T02:44:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe4921f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c82978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:43698/0 (socket says 172.21.15.29:43698) 2024-09-21T02:44:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe4a221640 1 -- 172.21.15.29:0/1005818740 learned_addr learned my addr 172.21.15.29:0/1005818740 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.405+0000 7fbe49a20640 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c8285e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe4a221640 1 -- 172.21.15.29:0/1005818740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 msgr2=0x55a6c8285e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe4a221640 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c8285e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe4a221640 1 -- 172.21.15.29:0/1005818740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 msgr2=0x55a6c82978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe4a221640 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c82978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe4a221640 1 -- 172.21.15.29:0/1005818740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6c8289cf0 con 0x55a6c7eb8b70 2024-09-21T02:44:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe49a20640 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c8285e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:45.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe4a221640 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c8284f80 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0e54e0 tx=0x7fbe3c0a9b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f899ddba98f4588c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:45.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe2f7fe640 1 -- 172.21.15.29:0/1005818740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0ae270 con 0x55a6c7eb8b70 2024-09-21T02:44:45.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe2f7fe640 1 -- 172.21.15.29:0/1005818740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe3c0b2650 con 0x55a6c7eb8b70 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.406+0000 7fbe2f7fe640 1 -- 172.21.15.29:0/1005818740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0a9de0 con 0x55a6c7eb8b70 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 -- 172.21.15.29:0/1005818740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 msgr2=0x55a6c8284f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c8284f80 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0e54e0 tx=0x7fbe3c0a9b40 comp rx=0 tx=0).stop 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 -- 172.21.15.29:0/1005818740 shutdown_connections 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c8284f80 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c82978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 --2- 172.21.15.29:0/1005818740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c8285e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 -- 172.21.15.29:0/1005818740 >> 172.21.15.29:0/1005818740 conn(0x55a6c7ecdc40 msgr2=0x55a6c82869a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 -- 172.21.15.29:0/1005818740 shutdown_connections 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.407+0000 7fbe521f8740 1 -- 172.21.15.29:0/1005818740 wait complete. 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 1 -- start start 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 1 Processor -- start 2024-09-21T02:44:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c829d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c829dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c83016a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441afbb0 con 0x55a6c7eb8b70 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441af4e0 con 0x55a6c7eb0990 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441ac720 con 0x55a6c7e981d0 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.408+0000 7fbe521f8740 10 client.0 _ll_register_callbacks cb 0x7fbe40044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe49a20640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c829d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4a221640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c83016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe49a20640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c829d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:36236/0 (socket says 172.21.15.29:36236) 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe49a20640 1 -- 172.21.15.29:0/611245080 learned_addr learned my addr 172.21.15.29:0/611245080 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4921f640 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c829dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4a221640 1 -- 172.21.15.29:0/611245080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 msgr2=0x55a6c829d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4a221640 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c829d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4a221640 1 -- 172.21.15.29:0/611245080 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 msgr2=0x55a6c829dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4a221640 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c829dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4a221640 1 -- 172.21.15.29:0/611245080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6c8301d10 con 0x55a6c7eb8b70 2024-09-21T02:44:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe49a20640 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c829d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4921f640 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c829dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:44:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe4a221640 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c83016a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0e54e0 tx=0x7fbe3c0051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f9b69c56a45d2d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/611245080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0b7590 con 0x55a6c7eb8b70 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/611245080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbe3c0cfb80 con 0x55a6c7eb8b70 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.409+0000 7fbe521f8740 1 -- 172.21.15.29:0/611245080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe3c0b2650 con 0x55a6c7eb8b70 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/611245080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbe3c0ae270 con 0x55a6c7eb8b70 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe521f8740 10 client.16326 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe521f8740 1 -- 172.21.15.29:0/611245080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a6c83090c0 con 0x55a6c7eb8b70 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/611245080 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbe3c0b2650 con 0x55a6c7eb8b70 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/611245080 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fbe3c0acdd0 con 0x55a6c7eb8b70 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe521f8740 10 client.16326 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:45.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe521f8740 10 client.16326 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:44:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe521f8740 20 client.16326 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe521f8740 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a6c7ed9850 0x55a6c830b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe521f8740 4 client.16326 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}] 2024-09-21T02:44:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.410+0000 7fbe521f8740 1 -- 172.21.15.29:0/611245080 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}) -- 0x55a6c81f45b0 con 0x55a6c7ed9850 2024-09-21T02:44:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.411+0000 7fbe49a20640 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a6c7ed9850 0x55a6c830b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.411+0000 7fbe2d7fa640 10 client.16326 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.411+0000 7fbe49a20640 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a6c7ed9850 0x55a6c830b850 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457231e002895895 server_cookie=a778a5eba5ca318c in_seq=0 out_seq=0 2024-09-21T02:44:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.507+0000 7fbe2d7fa640 1 -- 172.21.15.29:0/611245080 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a6c81f45b0 con 0x55a6c7ed9850 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.507+0000 7fbe2d7fa640 10 client.16326 handle_command_reply: tid=0 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.507+0000 7fbe521f8740 1 client.16326 shutdown 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.507+0000 7fbe521f8740 1 -- 172.21.15.29:0/611245080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 msgr2=0x55a6c83016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.507+0000 7fbe521f8740 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c83016a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0e54e0 tx=0x7fbe3c0051d0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.507+0000 7fbe521f8740 1 -- 172.21.15.29:0/611245080 shutdown_connections 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.507+0000 7fbe521f8740 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a6c7ed9850 0x55a6c830b850 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe521f8740 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a6c7eb8b70 0x55a6c83016a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe521f8740 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a6c7eb0990 0x55a6c829dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe521f8740 1 --2- 172.21.15.29:0/611245080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a6c7e981d0 0x55a6c829d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe2d7fa640 0 client.16326 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe521f8740 1 -- 172.21.15.29:0/611245080 >> 172.21.15.29:0/611245080 conn(0x55a6c7ecdc40 msgr2=0x55a6c82869a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe521f8740 1 -- 172.21.15.29:0/611245080 shutdown_connections 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe521f8740 1 -- 172.21.15.29:0/611245080 wait complete. 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe521f8740 20 client.16326 trim_cache size 0 max 16384 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 22, 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout: "2075e1c": { 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 22, 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:44:45.511 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.2, 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:45.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:45.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:45.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:44:45.513 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:44:45.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe4bc84640 1 -- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe300b6510 msgr2=0x7fbe300b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe300b6510 0x7fbe300b8a20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbe34015fd0 tx=0x7fbe34005f90 comp rx=0 tx=0).stop 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe4bc84640 1 -- 172.21.15.29:0/266801314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe44108d10 msgr2=0x7fbe4419eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe44108d10 0x7fbe4419eaa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009fd0 tx=0x7fbe3c00ef10 comp rx=0 tx=0).stop 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.508+0000 7fbe4bc84640 1 -- 172.21.15.29:0/266801314 shutdown_connections 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.509+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbe300b6510 0x7fbe300b8a20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.509+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe44108d10 0x7fbe4419eaa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.509+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe441083c0 0x7fbe44199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.509+0000 7fbe4bc84640 1 --2- 172.21.15.29:0/266801314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe44102a80 0x7fbe44199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.509+0000 7fbe4bc84640 1 -- 172.21.15.29:0/266801314 >> 172.21.15.29:0/266801314 conn(0x7fbe44079760 msgr2=0x7fbe44101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.509+0000 7fbe4bc84640 1 -- 172.21.15.29:0/266801314 shutdown_connections 2024-09-21T02:44:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:44:45.509+0000 7fbe4bc84640 1 -- 172.21.15.29:0/266801314 wait complete. 2024-09-21T02:44:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:45 smithi029 ceph-mon[31006]: pgmap v558: 129 pgs: 129 active+clean; 32 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-09-21T02:44:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:45 smithi190 ceph-mon[33871]: pgmap v558: 129 pgs: 129 active+clean; 32 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-09-21T02:44:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:45 smithi066 ceph-mon[33436]: pgmap v558: 129 pgs: 129 active+clean; 32 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-09-21T02:44:46.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.81 MB/sec execute 143 sec latency 552.129 ms 2024-09-21T02:44:47.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.79 MB/sec execute 144 sec latency 1552.218 ms 2024-09-21T02:44:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/1038126410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 msgr2=0x7f4bd8102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/1038126410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 0x7f4bd8102a10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8009ca0 tx=0x7f4bc802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/1038126410 shutdown_connections 2024-09-21T02:44:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/1038126410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bd8103fc0 0x7f4bd810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/1038126410 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4bd8103620 0x7f4bd8103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/1038126410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 0x7f4bd8102a10 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/1038126410 >> 172.21.15.29:0/1038126410 conn(0x7f4bd80fdee0 msgr2=0x7f4bd8100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/1038126410 shutdown_connections 2024-09-21T02:44:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.388+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/1038126410 wait complete. 2024-09-21T02:44:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.389+0000 7f4bdd4d5640 1 Processor -- start 2024-09-21T02:44:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.389+0000 7f4bdd4d5640 1 -- start start 2024-09-21T02:44:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bdd4d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 0x7f4bd8199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bdd4d5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bd8103620 0x7f4bd819a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bdd4d5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4bd8103fc0 0x7f4bd819b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bdd4d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810e050 con 0x7f4bd81025a0 2024-09-21T02:44:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bdd4d5640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810ded0 con 0x7f4bd8103fc0 2024-09-21T02:44:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bdd4d5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810e1d0 con 0x7f4bd8103620 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 0x7f4bd8199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 0x7f4bd8199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32888/0 (socket says 172.21.15.29:32888) 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd6ffd640 1 -- 172.21.15.29:0/3326032055 learned_addr learned my addr 172.21.15.29:0/3326032055 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bd8103620 0x7f4bd819a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4bd8103fc0 0x7f4bd819b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd6ffd640 1 -- 172.21.15.29:0/3326032055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bd8103620 msgr2=0x7f4bd819a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bd8103620 0x7f4bd819a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd6ffd640 1 -- 172.21.15.29:0/3326032055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4bd8103fc0 msgr2=0x7f4bd819b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4bd8103fc0 0x7f4bd819b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd6ffd640 1 -- 172.21.15.29:0/3326032055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd819b8f0 con 0x7f4bd81025a0 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.390+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4bd8103fc0 0x7f4bd819b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.391+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 0x7f4bd8199eb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4bc802ade0 tx=0x7f4bc8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.391+0000 7f4bcffff640 1 -- 172.21.15.29:0/3326032055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8004040 con 0x7f4bd81025a0 2024-09-21T02:44:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.391+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/3326032055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd819bb80 con 0x7f4bd81025a0 2024-09-21T02:44:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.391+0000 7f4bcffff640 1 -- 172.21.15.29:0/3326032055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4bc8004670 con 0x7f4bd81025a0 2024-09-21T02:44:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.391+0000 7f4bcffff640 1 -- 172.21.15.29:0/3326032055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8030d50 con 0x7f4bd81025a0 2024-09-21T02:44:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.391+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/3326032055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd81a22d0 con 0x7f4bd81025a0 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4be4440740 1 Processor -- start 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4be4440740 1 -- start start 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4be4440740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1f01b0 0x55a2de59c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4be4440740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de59c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4be4440740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de59d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4be4440740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2de50a6f0 con 0x55a2de1cf4d0 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4be4440740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a2de4f9b20 con 0x55a2de1f01b0 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4be4440740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a2de4fd930 con 0x55a2de1c72f0 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4bcffff640 1 -- 172.21.15.29:0/3326032055 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4bc80305c0 con 0x7f4bd81025a0 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4bd67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de59d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4bd67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de59d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32898/0 (socket says 172.21.15.29:32898) 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4bd67fc640 1 -- 172.21.15.29:0/274915246 learned_addr learned my addr 172.21.15.29:0/274915246 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.394+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1f01b0 0x55a2de59c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de59c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bd67fc640 1 -- 172.21.15.29:0/274915246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 msgr2=0x55a2de59c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de59c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bd67fc640 1 -- 172.21.15.29:0/274915246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1f01b0 msgr2=0x55a2de59c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1f01b0 0x55a2de59c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bd67fc640 1 -- 172.21.15.29:0/274915246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2de4bf530 con 0x55a2de1cf4d0 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de59c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de59d420 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000bd10 tx=0x7f4bc000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8704233428d289 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bcd7fa640 1 -- 172.21.15.29:0/274915246 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0015070 con 0x55a2de1cf4d0 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bcd7fa640 1 -- 172.21.15.29:0/274915246 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4bc001a8f0 con 0x55a2de1cf4d0 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bcd7fa640 1 -- 172.21.15.29:0/274915246 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0011860 con 0x55a2de1cf4d0 2024-09-21T02:44:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.395+0000 7f4bcffff640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bc40b6510 0x7f4bc40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bc40b6510 0x7f4bc40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4bcffff640 1 -- 172.21.15.29:0/3326032055 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4bc8108a20 con 0x7f4bd81025a0 2024-09-21T02:44:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 -- 172.21.15.29:0/274915246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 msgr2=0x55a2de59d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de59d420 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000bd10 tx=0x7f4bc000f730 comp rx=0 tx=0).stop 2024-09-21T02:44:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bc40b6510 0x7f4bc40b8a20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009d90 tx=0x7f4bd0009700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 -- 172.21.15.29:0/274915246 shutdown_connections 2024-09-21T02:44:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de59d420 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de59c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 --2- 172.21.15.29:0/274915246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1f01b0 0x55a2de59c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 -- 172.21.15.29:0/274915246 >> 172.21.15.29:0/274915246 conn(0x55a2de1e4300 msgr2=0x55a2de5938e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 -- 172.21.15.29:0/274915246 shutdown_connections 2024-09-21T02:44:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.396+0000 7f4be4440740 1 -- 172.21.15.29:0/274915246 wait complete. 2024-09-21T02:44:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 1 -- start start 2024-09-21T02:44:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 1 Processor -- start 2024-09-21T02:44:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1c72f0 0x55a2de5a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1f01b0 0x55a2de5a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2de431ab0 con 0x55a2de1cf4d0 2024-09-21T02:44:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a2de49f730 con 0x55a2de1c72f0 2024-09-21T02:44:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a2de59df40 con 0x55a2de1f01b0 2024-09-21T02:44:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4be4440740 10 client.0 _ll_register_callbacks cb 0x55a2de4f36b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4bd67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4bd67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32904/0 (socket says 172.21.15.29:32904) 2024-09-21T02:44:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.397+0000 7f4bd67fc640 1 -- 172.21.15.29:0/3261912079 learned_addr learned my addr 172.21.15.29:0/3261912079 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1c72f0 0x55a2de5a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1f01b0 0x55a2de5a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bd67fc640 1 -- 172.21.15.29:0/3261912079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1f01b0 msgr2=0x55a2de5a6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1f01b0 0x55a2de5a6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bd67fc640 1 -- 172.21.15.29:0/3261912079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1c72f0 msgr2=0x55a2de5a5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1c72f0 0x55a2de5a5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bd67fc640 1 -- 172.21.15.29:0/3261912079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2de25b790 con 0x55a2de1cf4d0 2024-09-21T02:44:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a60d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000b860 tx=0x7f4bc001a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ea3defd5680c6e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bbe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/3261912079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0015070 con 0x55a2de1cf4d0 2024-09-21T02:44:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/3261912079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4bc001acb0 con 0x55a2de1cf4d0 2024-09-21T02:44:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4be4440740 1 -- 172.21.15.29:0/3261912079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc001a8f0 con 0x55a2de1cf4d0 2024-09-21T02:44:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/3261912079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4bc002e050 con 0x55a2de1cf4d0 2024-09-21T02:44:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4be4440740 10 client.16338 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.398+0000 7f4be4440740 1 -- 172.21.15.29:0/3261912079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a2de2d4620 con 0x55a2de1cf4d0 2024-09-21T02:44:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/3261912079 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4bc001a8f0 con 0x55a2de1cf4d0 2024-09-21T02:44:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/3261912079 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f4bc0004880 con 0x55a2de1cf4d0 2024-09-21T02:44:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4be4440740 10 client.16338 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4be4440740 10 client.16338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:44:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4be4440740 20 client.16338 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4be4440740 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2de1aed80 0x55a2de5b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4be4440740 4 client.16338 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:44:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4be4440740 1 -- 172.21.15.29:0/3261912079 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2de50c640 con 0x55a2de1aed80 2024-09-21T02:44:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.399+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2de1aed80 0x55a2de5b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.400+0000 7f4bbe7fc640 10 client.16338 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.400+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2de1aed80 0x55a2de5b2d50 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdc0f9e9b9ae1d89 server_cookie=3180df19c16b8198 in_seq=0 out_seq=0 2024-09-21T02:44:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.400+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/3261912079 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a2de50c640 con 0x55a2de1aed80 2024-09-21T02:44:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.400+0000 7f4bbe7fc640 10 client.16338 handle_command_reply: tid=0 2024-09-21T02:44:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.400+0000 7f4be4440740 1 client.16338 shutdown 2024-09-21T02:44:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 -- 172.21.15.29:0/3261912079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 msgr2=0x55a2de5a60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a60d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000b860 tx=0x7f4bc001a690 comp rx=0 tx=0).stop 2024-09-21T02:44:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 -- 172.21.15.29:0/3261912079 shutdown_connections 2024-09-21T02:44:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1f01b0 0x55a2de5a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2de1aed80 0x55a2de5b2d50 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a60d0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 --2- 172.21.15.29:0/3261912079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1c72f0 0x55a2de5a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4bbe7fc640 0 client.16338 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 -- 172.21.15.29:0/3261912079 >> 172.21.15.29:0/3261912079 conn(0x55a2de1e4300 msgr2=0x55a2de591ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 -- 172.21.15.29:0/3261912079 shutdown_connections 2024-09-21T02:44:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 1 -- 172.21.15.29:0/3261912079 wait complete. 2024-09-21T02:44:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.401+0000 7f4be4440740 20 client.16338 trim_cache size 0 max 16384 2024-09-21T02:44:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4be4440740 1 Processor -- start 2024-09-21T02:44:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4be4440740 1 -- start start 2024-09-21T02:44:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4be4440740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de58e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4be4440740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4be4440740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4be4440740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd81aa3e0 con 0x55a2de1f01b0 2024-09-21T02:44:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4be4440740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd81abf20 con 0x55a2de1cf4d0 2024-09-21T02:44:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4be4440740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd81aaab0 con 0x55a2de1c72f0 2024-09-21T02:44:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4bd77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4bd77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5a5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32908/0 (socket says 172.21.15.29:32908) 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4bd77fe640 1 -- 172.21.15.29:0/2438313983 learned_addr learned my addr 172.21.15.29:0/2438313983 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.416+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bd77fe640 1 -- 172.21.15.29:0/2438313983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 msgr2=0x55a2de58e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de58e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de58e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bd77fe640 1 -- 172.21.15.29:0/2438313983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 msgr2=0x55a2de5a5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bd77fe640 1 -- 172.21.15.29:0/2438313983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2de5a6490 con 0x55a2de1f01b0 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:44:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5a5f50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00214e0 tx=0x7f4bd0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bc6b02eb3a803f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bccff9640 1 -- 172.21.15.29:0/2438313983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd002e070 con 0x55a2de1f01b0 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bccff9640 1 -- 172.21.15.29:0/2438313983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4bd00298a0 con 0x55a2de1f01b0 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.417+0000 7f4bccff9640 1 -- 172.21.15.29:0/2438313983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0011a80 con 0x55a2de1f01b0 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 -- 172.21.15.29:0/2438313983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 msgr2=0x55a2de5a5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5a5f50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00214e0 tx=0x7f4bd0016000 comp rx=0 tx=0).stop 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 -- 172.21.15.29:0/2438313983 shutdown_connections 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5a5f50 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 --2- 172.21.15.29:0/2438313983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de58e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 -- 172.21.15.29:0/2438313983 >> 172.21.15.29:0/2438313983 conn(0x55a2de1e4300 msgr2=0x55a2de592650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 -- 172.21.15.29:0/2438313983 shutdown_connections 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 -- 172.21.15.29:0/2438313983 wait complete. 2024-09-21T02:44:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.418+0000 7f4be4440740 1 -- start start 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4be4440740 1 Processor -- start 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4be4440740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de5aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4be4440740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5ab390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4be4440740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4be4440740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd81ab850 con 0x55a2de1f01b0 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4be4440740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd81ab180 con 0x55a2de1cf4d0 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4be4440740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd81a83c0 con 0x55a2de1c72f0 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4be4440740 10 client.0 _ll_register_callbacks cb 0x7f4bd0009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5abeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32920/0 (socket says 172.21.15.29:32920) 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 -- 172.21.15.29:0/341224974 learned_addr learned my addr 172.21.15.29:0/341224974 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de5aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd67fc640 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 -- 172.21.15.29:0/341224974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 msgr2=0x55a2de5aacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de5aacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 -- 172.21.15.29:0/341224974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 msgr2=0x55a2de5ab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5ab390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 -- 172.21.15.29:0/341224974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2de5ac610 con 0x55a2de1f01b0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bd77fe640 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5abeb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4bd002c040 tx=0x7f4bd0029a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd52d18ebb540a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.419+0000 7f4bbe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/341224974 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd002e040 con 0x55a2de1f01b0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/341224974 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4bd0005050 con 0x55a2de1f01b0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 1 -- 172.21.15.29:0/341224974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bd00298a0 con 0x55a2de1f01b0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/341224974 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4bd0040050 con 0x55a2de1f01b0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 10 client.16344 fetch_fsmap learned FSMap version 31 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 1 -- 172.21.15.29:0/341224974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a2de5ad930 con 0x55a2de1f01b0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/341224974 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4bd00298a0 con 0x55a2de1f01b0 2024-09-21T02:44:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/341224974 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f4bd000f040 con 0x55a2de1f01b0 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 10 client.16344 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 10 client.16344 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 20 client.16344 populate_metadata read hostname 'smithi029' 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2de1841c0 0x55a2de61f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 4 client.16344 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.420+0000 7f4be4440740 1 -- 172.21.15.29:0/341224974 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a2de592440 con 0x55a2de1841c0 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.421+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2de1841c0 0x55a2de61f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.421+0000 7f4bbe7fc640 10 client.16344 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.421+0000 7f4bd6ffd640 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2de1841c0 0x55a2de61f910 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=723c96b706a4f352 server_cookie=dd7f24f016ac7b0 in_seq=0 out_seq=0 2024-09-21T02:44:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4bbe7fc640 1 -- 172.21.15.29:0/341224974 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a2de592440 con 0x55a2de1841c0 2024-09-21T02:44:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4bbe7fc640 10 client.16344 handle_command_reply: tid=0 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 client.16344 shutdown 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 -- 172.21.15.29:0/341224974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 msgr2=0x55a2de5abeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5abeb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4bd002c040 tx=0x7f4bd0029a50 comp rx=0 tx=0).stop 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 -- 172.21.15.29:0/341224974 shutdown_connections 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2de1841c0 0x55a2de61f910 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2de1f01b0 0x55a2de5abeb0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2de1cf4d0 0x55a2de5ab390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4bbe7fc640 0 client.16344 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 --2- 172.21.15.29:0/341224974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2de1c72f0 0x55a2de5aacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 -- 172.21.15.29:0/341224974 >> 172.21.15.29:0/341224974 conn(0x55a2de1e4300 msgr2=0x55a2de58e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 -- 172.21.15.29:0/341224974 shutdown_connections 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 1 -- 172.21.15.29:0/341224974 wait complete. 2024-09-21T02:44:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.455+0000 7f4be4440740 20 client.16344 trim_cache size 0 max 16384 2024-09-21T02:44:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:44:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:44:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:44:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bc40b6510 msgr2=0x7f4bc40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bc40b6510 0x7f4bc40b8a20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009d90 tx=0x7f4bd0009700 comp rx=0 tx=0).stop 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 msgr2=0x7f4bd8199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 0x7f4bd8199eb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4bc802ade0 tx=0x7f4bc8003ba0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/3326032055 shutdown_connections 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4bc40b6510 0x7f4bc40b8a20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4bd8103fc0 0x7f4bd819b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bd8103620 0x7f4bd819a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 --2- 172.21.15.29:0/3326032055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bd81025a0 0x7f4bd8199eb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/3326032055 >> 172.21.15.29:0/3326032055 conn(0x7f4bd80fdee0 msgr2=0x7f4bd806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/3326032055 shutdown_connections 2024-09-21T02:44:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:44:47.456+0000 7f4bdd4d5640 1 -- 172.21.15.29:0/3326032055 wait complete. 2024-09-21T02:44:47.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:47 smithi190 ceph-mon[33871]: pgmap v559: 129 pgs: 129 active+clean; 33 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-09-21T02:44:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:47 smithi029 ceph-mon[31006]: pgmap v559: 129 pgs: 129 active+clean; 33 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-09-21T02:44:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:47 smithi066 ceph-mon[33436]: pgmap v559: 129 pgs: 129 active+clean; 33 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-09-21T02:44:48.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.77 MB/sec execute 145 sec latency 2552.256 ms 2024-09-21T02:44:49.077 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.75 MB/sec execute 146 sec latency 3552.344 ms 2024-09-21T02:44:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:49 smithi190 ceph-mon[33871]: pgmap v560: 129 pgs: 129 active+clean; 33 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:44:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:49 smithi029 ceph-mon[31006]: pgmap v560: 129 pgs: 129 active+clean; 33 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:44:50.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:49 smithi066 ceph-mon[33436]: pgmap v560: 129 pgs: 129 active+clean; 33 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:44:50.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.73 MB/sec execute 147 sec latency 4552.413 ms 2024-09-21T02:44:51.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.71 MB/sec execute 148 sec latency 5552.501 ms 2024-09-21T02:44:51.664 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2075e1c', quiesce took 9.3 seconds. Will release after: 9.95 2024-09-21T02:44:51.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:51 smithi190 ceph-mon[33871]: pgmap v561: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-09-21T02:44:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:51 smithi029 ceph-mon[31006]: pgmap v561: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-09-21T02:44:52.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:51 smithi066 ceph-mon[33436]: pgmap v561: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-09-21T02:44:52.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.70 MB/sec execute 149 sec latency 6552.623 ms 2024-09-21T02:44:53.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.68 MB/sec execute 150 sec latency 7552.724 ms 2024-09-21T02:44:53.812 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:53 smithi066 ceph-mon[33436]: pgmap v562: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 887 KiB/s wr, 92 op/s 2024-09-21T02:44:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:53 smithi190 ceph-mon[33871]: pgmap v562: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 887 KiB/s wr, 92 op/s 2024-09-21T02:44:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:53 smithi029 ceph-mon[31006]: pgmap v562: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 887 KiB/s wr, 92 op/s 2024-09-21T02:44:54.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.66 MB/sec execute 151 sec latency 8552.843 ms 2024-09-21T02:44:55.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.64 MB/sec execute 152 sec latency 9552.945 ms 2024-09-21T02:44:55.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:55 smithi190 ceph-mon[33871]: pgmap v563: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 635 KiB/s wr, 87 op/s 2024-09-21T02:44:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:55 smithi029 ceph-mon[31006]: pgmap v563: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 635 KiB/s wr, 87 op/s 2024-09-21T02:44:56.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:55 smithi066 ceph-mon[33436]: pgmap v563: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 635 KiB/s wr, 87 op/s 2024-09-21T02:44:56.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.63 MB/sec execute 153 sec latency 10553.045 ms 2024-09-21T02:44:57.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.61 MB/sec execute 154 sec latency 11553.190 ms 2024-09-21T02:44:57.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:57 smithi190 ceph-mon[33871]: pgmap v564: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 474 KiB/s wr, 56 op/s 2024-09-21T02:44:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:57 smithi029 ceph-mon[31006]: pgmap v564: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 474 KiB/s wr, 56 op/s 2024-09-21T02:44:58.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:57 smithi066 ceph-mon[33436]: pgmap v564: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 474 KiB/s wr, 56 op/s 2024-09-21T02:44:58.078 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.59 MB/sec execute 155 sec latency 12553.292 ms 2024-09-21T02:44:59.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.58 MB/sec execute 156 sec latency 13553.401 ms 2024-09-21T02:44:59.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:44:59 smithi190 ceph-mon[33871]: pgmap v565: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 122 KiB/s wr, 45 op/s 2024-09-21T02:44:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:44:59 smithi029 ceph-mon[31006]: pgmap v565: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 122 KiB/s wr, 45 op/s 2024-09-21T02:45:00.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:44:59 smithi066 ceph-mon[33436]: pgmap v565: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 122 KiB/s wr, 45 op/s 2024-09-21T02:45:00.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.56 MB/sec execute 157 sec latency 14553.526 ms 2024-09-21T02:45:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 -- 172.21.15.29:0/450458654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590101370 msgr2=0x7ff590101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 --2- 172.21.15.29:0/450458654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590101370 0x7ff590101780 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009900 tx=0x7ff58c02a410 comp rx=0 tx=0).stop 2024-09-21T02:45:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 -- 172.21.15.29:0/450458654 shutdown_connections 2024-09-21T02:45:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 --2- 172.21.15.29:0/450458654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590109480 0x7ff59010b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 --2- 172.21.15.29:0/450458654 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff590101cc0 0x7ff590108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 --2- 172.21.15.29:0/450458654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590101370 0x7ff590101780 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 -- 172.21.15.29:0/450458654 >> 172.21.15.29:0/450458654 conn(0x7ff5900fd270 msgr2=0x7ff5900ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 -- 172.21.15.29:0/450458654 shutdown_connections 2024-09-21T02:45:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.569+0000 7ff5976ec640 1 -- 172.21.15.29:0/450458654 wait complete. 2024-09-21T02:45:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff5976ec640 1 Processor -- start 2024-09-21T02:45:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff5976ec640 1 -- start start 2024-09-21T02:45:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff5976ec640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff590101370 0x7ff590199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff5976ec640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590101cc0 0x7ff59019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff5976ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590109480 0x7ff59019b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff5976ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff59010f220 con 0x7ff590109480 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff5976ec640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff59010f0a0 con 0x7ff590101370 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff5976ec640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff59010f3a0 con 0x7ff590101cc0 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff595c89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590109480 0x7ff59019b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff595c89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590109480 0x7ff59019b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47946/0 (socket says 172.21.15.29:47946) 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff595c89640 1 -- 172.21.15.29:0/2366470591 learned_addr learned my addr 172.21.15.29:0/2366470591 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.570+0000 7ff595488640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff590101370 0x7ff590199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff594c87640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590101cc0 0x7ff59019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff595c89640 1 -- 172.21.15.29:0/2366470591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590101cc0 msgr2=0x7ff59019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff595c89640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590101cc0 0x7ff59019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff595c89640 1 -- 172.21.15.29:0/2366470591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff590101370 msgr2=0x7ff590199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff595c89640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff590101370 0x7ff590199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff595c89640 1 -- 172.21.15.29:0/2366470591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59019b8f0 con 0x7ff590109480 2024-09-21T02:45:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff594c87640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590101cc0 0x7ff59019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:45:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff595488640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff590101370 0x7ff590199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:45:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff595c89640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590109480 0x7ff59019b0d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff58800eb10 tx=0x7ff58800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff5867fc640 1 -- 172.21.15.29:0/2366470591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588004b20 con 0x7ff590109480 2024-09-21T02:45:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff5867fc640 1 -- 172.21.15.29:0/2366470591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff588004cc0 con 0x7ff590109480 2024-09-21T02:45:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff5976ec640 1 -- 172.21.15.29:0/2366470591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59019bbe0 con 0x7ff590109480 2024-09-21T02:45:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff5867fc640 1 -- 172.21.15.29:0/2366470591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588010690 con 0x7ff590109480 2024-09-21T02:45:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.571+0000 7ff5976ec640 1 -- 172.21.15.29:0/2366470591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5901a22d0 con 0x7ff590109480 2024-09-21T02:45:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.574+0000 7ff59e65b740 1 Processor -- start 2024-09-21T02:45:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.574+0000 7ff5867fc640 1 -- 172.21.15.29:0/2366470591 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff588005020 con 0x7ff590109480 2024-09-21T02:45:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff59e65b740 1 -- start start 2024-09-21T02:45:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 0x56342d0734c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342ccaa4d0 0x56342d074010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff59e65b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56342cfe56f0 con 0x56342cccb1b0 2024-09-21T02:45:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff59e65b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56342cfd4b20 con 0x56342ccaa4d0 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff59e65b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56342cfd8930 con 0x56342cca22f0 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff594c87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff595488640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 0x56342d0734c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff594c87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47958/0 (socket says 172.21.15.29:47958) 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff594c87640 1 -- 172.21.15.29:0/311480181 learned_addr learned my addr 172.21.15.29:0/311480181 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff594c87640 1 --2- 172.21.15.29:0/311480181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342ccaa4d0 0x56342d074010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff5867fc640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff57c0b6510 0x7ff57c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff594c87640 1 -- 172.21.15.29:0/311480181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 msgr2=0x56342d0734c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff594c87640 1 --2- 172.21.15.29:0/311480181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 0x56342d0734c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff594c87640 1 -- 172.21.15.29:0/311480181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342ccaa4d0 msgr2=0x56342d074010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.575+0000 7ff594c87640 1 --2- 172.21.15.29:0/311480181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342ccaa4d0 0x56342d074010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.576+0000 7ff594c87640 1 -- 172.21.15.29:0/311480181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56342cf9a530 con 0x56342cccb1b0 2024-09-21T02:45:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.576+0000 7ff595c89640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff57c0b6510 0x7ff57c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.576+0000 7ff594c87640 1 --2- 172.21.15.29:0/311480181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072e50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff580013910 tx=0x7ff58000b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc85251f208866d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.576+0000 7ff5867fc640 1 -- 172.21.15.29:0/2366470591 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff5880e7d40 con 0x7ff590109480 2024-09-21T02:45:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.576+0000 7ff57b7fe640 1 -- 172.21.15.29:0/311480181 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58000e8f0 con 0x56342cccb1b0 2024-09-21T02:45:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.576+0000 7ff57b7fe640 1 -- 172.21.15.29:0/311480181 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5800113e0 con 0x56342cccb1b0 2024-09-21T02:45:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.576+0000 7ff57b7fe640 1 -- 172.21.15.29:0/311480181 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580010630 con 0x56342cccb1b0 2024-09-21T02:45:00.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 -- 172.21.15.29:0/311480181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 msgr2=0x56342d072e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 --2- 172.21.15.29:0/311480181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072e50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff580013910 tx=0x7ff58000b4c0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff595c89640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff57c0b6510 0x7ff57c0b8a20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff5880e49c0 tx=0x7ff5880d1040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:00.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 -- 172.21.15.29:0/311480181 shutdown_connections 2024-09-21T02:45:00.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 --2- 172.21.15.29:0/311480181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342ccaa4d0 0x56342d074010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 --2- 172.21.15.29:0/311480181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 0x56342d0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 --2- 172.21.15.29:0/311480181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072e50 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 -- 172.21.15.29:0/311480181 >> 172.21.15.29:0/311480181 conn(0x56342ccbf300 msgr2=0x56342d06e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 -- 172.21.15.29:0/311480181 shutdown_connections 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 -- 172.21.15.29:0/311480181 wait complete. 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.577+0000 7ff59e65b740 1 -- start start 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff59e65b740 1 Processor -- start 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 0x56342d07c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342ccaa4d0 0x56342d07cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cccb1b0 0x56342d07d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff59e65b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56342cf0cab0 con 0x56342ccaa4d0 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff59e65b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56342cf7a730 con 0x56342cccb1b0 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff59e65b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56342d074ad0 con 0x56342cca22f0 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff59e65b740 10 client.0 _ll_register_callbacks cb 0x56342cfce6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff595488640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342ccaa4d0 0x56342d07cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff595488640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342ccaa4d0 0x56342d07cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47970/0 (socket says 172.21.15.29:47970) 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff595488640 1 -- 172.21.15.29:0/22895662 learned_addr learned my addr 172.21.15.29:0/22895662 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff594c87640 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 0x56342d07c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.578+0000 7ff594c87640 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cccb1b0 0x56342d07d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff595488640 1 -- 172.21.15.29:0/22895662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 msgr2=0x56342d07c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff595488640 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 0x56342d07c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff595488640 1 -- 172.21.15.29:0/22895662 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cccb1b0 msgr2=0x56342d07d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff595488640 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cccb1b0 0x56342d07d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff595488640 1 -- 172.21.15.29:0/22895662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56342cd36790 con 0x56342ccaa4d0 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff595488640 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342ccaa4d0 0x56342d07cca0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff58c000c00 tx=0x7ff58c004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf1f8d4864a6f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff578ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff578ff9640 1 -- 172.21.15.29:0/22895662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c035040 con 0x56342ccaa4d0 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff578ff9640 1 -- 172.21.15.29:0/22895662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff58c003ac0 con 0x56342ccaa4d0 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff59e65b740 1 -- 172.21.15.29:0/22895662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5800113e0 con 0x56342ccaa4d0 2024-09-21T02:45:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff578ff9640 1 -- 172.21.15.29:0/22895662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff58c003cc0 con 0x56342ccaa4d0 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff59e65b740 10 client.16356 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff59e65b740 1 -- 172.21.15.29:0/22895662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56342cdaf620 con 0x56342ccaa4d0 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.579+0000 7ff578ff9640 1 -- 172.21.15.29:0/22895662 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5800113e0 con 0x56342ccaa4d0 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.580+0000 7ff578ff9640 1 -- 172.21.15.29:0/22895662 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff58c03d4e0 con 0x56342ccaa4d0 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.580+0000 7ff59e65b740 10 client.16356 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.580+0000 7ff59e65b740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.580+0000 7ff59e65b740 20 client.16356 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.580+0000 7ff59e65b740 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56342cc89d80 0x56342d089ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.580+0000 7ff59e65b740 4 client.16356 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.580+0000 7ff59e65b740 1 -- 172.21.15.29:0/22895662 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56342cfe7640 con 0x56342cc89d80 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.580+0000 7ff594c87640 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56342cc89d80 0x56342d089ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.581+0000 7ff578ff9640 10 client.16356 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:00.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.581+0000 7ff594c87640 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56342cc89d80 0x56342d089ce0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a5a9ea71d620004 server_cookie=2c777a1b7e060f5c in_seq=0 out_seq=0 2024-09-21T02:45:00.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.581+0000 7ff578ff9640 1 -- 172.21.15.29:0/22895662 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56342cfe7640 con 0x56342cc89d80 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.581+0000 7ff578ff9640 10 client.16356 handle_command_reply: tid=0 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.581+0000 7ff59e65b740 1 client.16356 shutdown 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.581+0000 7ff59e65b740 1 -- 172.21.15.29:0/22895662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342ccaa4d0 msgr2=0x56342d07cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.581+0000 7ff59e65b740 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342ccaa4d0 0x56342d07cca0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff58c000c00 tx=0x7ff58c004510 comp rx=0 tx=0).stop 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 1 -- 172.21.15.29:0/22895662 shutdown_connections 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cccb1b0 0x56342d07d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56342cc89d80 0x56342d089ce0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342ccaa4d0 0x56342d07cca0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 1 --2- 172.21.15.29:0/22895662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342cca22f0 0x56342d07c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff578ff9640 0 client.16356 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 1 -- 172.21.15.29:0/22895662 >> 172.21.15.29:0/22895662 conn(0x56342ccbf300 msgr2=0x56342d06cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 1 -- 172.21.15.29:0/22895662 shutdown_connections 2024-09-21T02:45:00.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 1 -- 172.21.15.29:0/22895662 wait complete. 2024-09-21T02:45:00.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.582+0000 7ff59e65b740 20 client.16356 trim_cache size 0 max 16384 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.596+0000 7ff59e65b740 1 Processor -- start 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff59e65b740 1 -- start start 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 0x56342d07ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 0x56342d07b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff59e65b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff5901ac640 con 0x56342cccb1b0 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff59e65b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff5901ae180 con 0x56342cca22f0 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff59e65b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff5901acd10 con 0x56342ccaa4d0 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47986/0 (socket says 172.21.15.29:47986) 2024-09-21T02:45:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 -- 172.21.15.29:0/3331107777 learned_addr learned my addr 172.21.15.29:0/3331107777 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff595488640 1 --2- 172.21.15.29:0/3331107777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 0x56342d07b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 -- 172.21.15.29:0/3331107777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 msgr2=0x56342d07b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 --2- 172.21.15.29:0/3331107777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 0x56342d07b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 -- 172.21.15.29:0/3331107777 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 msgr2=0x56342d07ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 --2- 172.21.15.29:0/3331107777 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 0x56342d07ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 -- 172.21.15.29:0/3331107777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56342d084ab0 con 0x56342cccb1b0 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.597+0000 7ff594c87640 1 --2- 172.21.15.29:0/3331107777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072df0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff580005d00 tx=0x7ff58000b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31d1a938b05dda4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff57affd640 1 -- 172.21.15.29:0/3331107777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580027040 con 0x56342cccb1b0 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff57affd640 1 -- 172.21.15.29:0/3331107777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff580013250 con 0x56342cccb1b0 2024-09-21T02:45:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff57affd640 1 -- 172.21.15.29:0/3331107777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580011640 con 0x56342cccb1b0 2024-09-21T02:45:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff59e65b740 1 -- 172.21.15.29:0/3331107777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 msgr2=0x56342d072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff59e65b740 1 --2- 172.21.15.29:0/3331107777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072df0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff580005d00 tx=0x7ff58000b4c0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff59e65b740 1 -- 172.21.15.29:0/3331107777 shutdown_connections 2024-09-21T02:45:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff59e65b740 1 --2- 172.21.15.29:0/3331107777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d072df0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff59e65b740 1 --2- 172.21.15.29:0/3331107777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 0x56342d07b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff59e65b740 1 --2- 172.21.15.29:0/3331107777 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 0x56342d07ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff59e65b740 1 -- 172.21.15.29:0/3331107777 >> 172.21.15.29:0/3331107777 conn(0x56342ccbf300 msgr2=0x56342d073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.598+0000 7ff59e65b740 1 -- 172.21.15.29:0/3331107777 shutdown_connections 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 -- 172.21.15.29:0/3331107777 wait complete. 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 -- start start 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 Processor -- start 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 0x56342d083f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 0x56342d082c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d083150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff5901adab0 con 0x56342cccb1b0 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff5901ad3e0 con 0x56342cca22f0 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff5901aa620 con 0x56342ccaa4d0 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff59e65b740 10 client.0 _ll_register_callbacks cb 0x7ff58c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.599+0000 7ff594c87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d083150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d083150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47994/0 (socket says 172.21.15.29:47994) 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 -- 172.21.15.29:0/1596467905 learned_addr learned my addr 172.21.15.29:0/1596467905 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 0x56342d083f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff595488640 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 0x56342d082c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 -- 172.21.15.29:0/1596467905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 msgr2=0x56342d082c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 0x56342d082c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 -- 172.21.15.29:0/1596467905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 msgr2=0x56342d083f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 0x56342d083f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 -- 172.21.15.29:0/1596467905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56342d083690 con 0x56342cccb1b0 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff594c87640 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d083150 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff58000b7f0 tx=0x7ff5800135b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6824ad93df6e9bda server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff578ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff578ff9640 1 -- 172.21.15.29:0/1596467905 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58003e040 con 0x56342cccb1b0 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff59e65b740 1 -- 172.21.15.29:0/1596467905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff580013250 con 0x56342cccb1b0 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff578ff9640 1 -- 172.21.15.29:0/1596467905 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff58000e8c0 con 0x56342cccb1b0 2024-09-21T02:45:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff578ff9640 1 -- 172.21.15.29:0/1596467905 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff58000ee70 con 0x56342cccb1b0 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff59e65b740 10 client.16362 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.600+0000 7ff59e65b740 1 -- 172.21.15.29:0/1596467905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56342d082450 con 0x56342cccb1b0 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff578ff9640 1 -- 172.21.15.29:0/1596467905 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff580013250 con 0x56342cccb1b0 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff578ff9640 1 -- 172.21.15.29:0/1596467905 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff58002d040 con 0x56342cccb1b0 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff59e65b740 10 client.16362 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff59e65b740 10 client.16362 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff59e65b740 20 client.16362 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff59e65b740 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56342cc5f1c0 0x56342d0f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff59e65b740 4 client.16362 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff59e65b740 1 -- 172.21.15.29:0/1596467905 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56342cfe30f0 con 0x56342cc5f1c0 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.601+0000 7ff595488640 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56342cc5f1c0 0x56342d0f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.602+0000 7ff578ff9640 10 client.16362 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.602+0000 7ff595488640 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56342cc5f1c0 0x56342d0f6a30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=513265cac2a9a791 server_cookie=534a9297bc9383f9 in_seq=0 out_seq=0 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff578ff9640 1 -- 172.21.15.29:0/1596467905 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56342cfe30f0 con 0x56342cc5f1c0 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff578ff9640 10 client.16362 handle_command_reply: tid=0 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 client.16362 shutdown 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 -- 172.21.15.29:0/1596467905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 msgr2=0x56342d083150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d083150 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff58000b7f0 tx=0x7ff5800135b0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 -- 172.21.15.29:0/1596467905 shutdown_connections 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56342cc5f1c0 0x56342d0f6a30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56342cccb1b0 0x56342d083150 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56342ccaa4d0 0x56342d082c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 --2- 172.21.15.29:0/1596467905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56342cca22f0 0x56342d083f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff578ff9640 0 client.16362 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:00.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 -- 172.21.15.29:0/1596467905 >> 172.21.15.29:0/1596467905 conn(0x56342ccbf300 msgr2=0x56342d072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:00.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 -- 172.21.15.29:0/1596467905 shutdown_connections 2024-09-21T02:45:00.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 1 -- 172.21.15.29:0/1596467905 wait complete. 2024-09-21T02:45:00.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.623+0000 7ff59e65b740 20 client.16362 trim_cache size 0 max 16384 2024-09-21T02:45:00.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:45:00.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:45:00.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:45:00.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:45:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 -- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff57c0b6510 msgr2=0x7ff57c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff57c0b6510 0x7ff57c0b8a20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff5880e49c0 tx=0x7ff5880d1040 comp rx=0 tx=0).stop 2024-09-21T02:45:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 -- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590109480 msgr2=0x7ff59019b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590109480 0x7ff59019b0d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff58800eb10 tx=0x7ff58800c2d0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 -- 172.21.15.29:0/2366470591 shutdown_connections 2024-09-21T02:45:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff57c0b6510 0x7ff57c0b8a20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff590109480 0x7ff59019b0d0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590101cc0 0x7ff59019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 --2- 172.21.15.29:0/2366470591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff590101370 0x7ff590199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 -- 172.21.15.29:0/2366470591 >> 172.21.15.29:0/2366470591 conn(0x7ff5900fd270 msgr2=0x7ff590104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 -- 172.21.15.29:0/2366470591 shutdown_connections 2024-09-21T02:45:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:00.624+0000 7ff5976ec640 1 -- 172.21.15.29:0/2366470591 wait complete. 2024-09-21T02:45:01.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.54 MB/sec execute 158 sec latency 15553.630 ms 2024-09-21T02:45:01.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:45:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:01 smithi190 ceph-mon[33871]: pgmap v566: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 25 op/s 2024-09-21T02:45:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:01 smithi029 ceph-mon[31006]: pgmap v566: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 25 op/s 2024-09-21T02:45:01.888 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:45:02.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:01 smithi066 ceph-mon[33436]: pgmap v566: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 25 op/s 2024-09-21T02:45:02.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.53 MB/sec execute 159 sec latency 16553.729 ms 2024-09-21T02:45:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 -- 172.21.15.29:0/1484430889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f8102a90 msgr2=0x7f36f8102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1484430889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f8102a90 0x7f36f8102ea0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009900 tx=0x7f36e802a410 comp rx=0 tx=0).stop 2024-09-21T02:45:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 -- 172.21.15.29:0/1484430889 shutdown_connections 2024-09-21T02:45:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1484430889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36f8108d20 0x7f36f810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1484430889 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36f81083d0 0x7f36f81087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1484430889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f8102a90 0x7f36f8102ea0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 -- 172.21.15.29:0/1484430889 >> 172.21.15.29:0/1484430889 conn(0x7f36f8079750 msgr2=0x7f36f8079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 -- 172.21.15.29:0/1484430889 shutdown_connections 2024-09-21T02:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.446+0000 7f36fc96e640 1 -- 172.21.15.29:0/1484430889 wait complete. 2024-09-21T02:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.447+0000 7f36fc96e640 1 Processor -- start 2024-09-21T02:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.447+0000 7f36fc96e640 1 -- start start 2024-09-21T02:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.447+0000 7f36fc96e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36f8102a90 0x7f36f81992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.447+0000 7f36fc96e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f81083d0 0x7f36f8199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.447+0000 7f36fc96e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36f8108d20 0x7f36f819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.447+0000 7f36fc96e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8112840 con 0x7f36f81083d0 2024-09-21T02:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.447+0000 7f36fc96e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f36f81126c0 con 0x7f36f8108d20 2024-09-21T02:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.447+0000 7f36fc96e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f36f81129c0 con 0x7f36f8102a90 2024-09-21T02:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f81083d0 0x7f36f8199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f81083d0 0x7f36f8199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48012/0 (socket says 172.21.15.29:48012) 2024-09-21T02:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 -- 172.21.15.29:0/1916175262 learned_addr learned my addr 172.21.15.29:0/1916175262 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f659c640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36f8102a90 0x7f36f81992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f6d9d640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36f8108d20 0x7f36f819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 -- 172.21.15.29:0/1916175262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36f8102a90 msgr2=0x7f36f81992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36f8102a90 0x7f36f81992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 -- 172.21.15.29:0/1916175262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36f8108d20 msgr2=0x7f36f819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36f8108d20 0x7f36f819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 -- 172.21.15.29:0/1916175262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f819f240 con 0x7f36f81083d0 2024-09-21T02:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36f5d9b640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f81083d0 0x7f36f8199870 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00e9c0 tx=0x7f36ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36e77fe640 1 -- 172.21.15.29:0/1916175262 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec00cd80 con 0x7f36f81083d0 2024-09-21T02:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36e77fe640 1 -- 172.21.15.29:0/1916175262 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36ec0054b0 con 0x7f36f81083d0 2024-09-21T02:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36e77fe640 1 -- 172.21.15.29:0/1916175262 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec010430 con 0x7f36f81083d0 2024-09-21T02:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f819f530 con 0x7f36f81083d0 2024-09-21T02:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.448+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f81a5c90 con 0x7f36f81083d0 2024-09-21T02:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.450+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f8103d10 con 0x7f36f81083d0 2024-09-21T02:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.451+0000 7f36e77fe640 1 -- 172.21.15.29:0/1916175262 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f36ec004f20 con 0x7f36f81083d0 2024-09-21T02:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.452+0000 7f36e77fe640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f36e00b6590 0x7f36e00b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.452+0000 7f36f659c640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f36e00b6590 0x7f36e00b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.452+0000 7f36e77fe640 1 -- 172.21.15.29:0/1916175262 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f36ec0e7c00 con 0x7f36f81083d0 2024-09-21T02:45:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.452+0000 7f36f659c640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f36e00b6590 0x7f36e00b8aa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f36e80057c0 tx=0x7f36e80058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.456+0000 7f36e77fe640 1 -- 172.21.15.29:0/1916175262 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f36ec0ae0b0 con 0x7f36f81083d0 2024-09-21T02:45:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.683+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36f802e640 con 0x7f36f81083d0 2024-09-21T02:45:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.684+0000 7f36e77fe640 1 -- 172.21.15.29:0/1916175262 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f36ec0b2490 con 0x7f36f81083d0 2024-09-21T02:45:02.685 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:45:02.686 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:45:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f36e00b6590 msgr2=0x7f36e00b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f36e00b6590 0x7f36e00b8aa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f36e80057c0 tx=0x7f36e80058d0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f81083d0 msgr2=0x7f36f8199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f81083d0 0x7f36f8199870 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00e9c0 tx=0x7f36ec00ee80 comp rx=0 tx=0).stop 2024-09-21T02:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 shutdown_connections 2024-09-21T02:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f36e00b6590 0x7f36e00b8aa0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36f8108d20 0x7f36f819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f81083d0 0x7f36f8199870 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 --2- 172.21.15.29:0/1916175262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36f8102a90 0x7f36f81992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 >> 172.21.15.29:0/1916175262 conn(0x7f36f8079750 msgr2=0x7f36f8101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 shutdown_connections 2024-09-21T02:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:02.686+0000 7f36fc96e640 1 -- 172.21.15.29:0/1916175262 wait complete. 2024-09-21T02:45:03.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.51 MB/sec execute 160 sec latency 17553.837 ms 2024-09-21T02:45:03.500 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 2075e1c --release --await' 2024-09-21T02:45:03.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 2075e1c --release --await 2024-09-21T02:45:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:45:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:03 smithi190 ceph-mon[33871]: pgmap v567: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-21T02:45:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:03 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1916175262' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:03 smithi029 ceph-mon[31006]: pgmap v567: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-21T02:45:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:03 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1916175262' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:03 smithi066 ceph-mon[33436]: pgmap v567: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-21T02:45:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:03 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1916175262' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:04.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104908 2.50 MB/sec execute 161 sec latency 18553.896 ms 2024-09-21T02:45:04.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 -- 172.21.15.29:0/150596499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 msgr2=0x7f425c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 --2- 172.21.15.29:0/150596499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 0x7f425c102e70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4258009900 tx=0x7f425802a410 comp rx=0 tx=0).stop 2024-09-21T02:45:04.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 -- 172.21.15.29:0/150596499 shutdown_connections 2024-09-21T02:45:04.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 --2- 172.21.15.29:0/150596499 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f425c108cf0 0x7f425c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 --2- 172.21.15.29:0/150596499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f425c1083a0 0x7f425c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 --2- 172.21.15.29:0/150596499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 0x7f425c102e70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 -- 172.21.15.29:0/150596499 >> 172.21.15.29:0/150596499 conn(0x7f425c079730 msgr2=0x7f425c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 -- 172.21.15.29:0/150596499 shutdown_connections 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.285+0000 7f4263ab1640 1 -- 172.21.15.29:0/150596499 wait complete. 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f4263ab1640 1 Processor -- start 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f4263ab1640 1 -- start start 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f4263ab1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 0x7f425c19b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f4263ab1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f425c1083a0 0x7f425c19b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f4263ab1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f425c108cf0 0x7f425c19bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f4263ab1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f425c112810 con 0x7f425c102a60 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f4263ab1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f425c112690 con 0x7f425c108cf0 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f4263ab1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f425c112990 con 0x7f425c1083a0 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f426184d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 0x7f425c19b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f426184d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 0x7f425c19b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39154/0 (socket says 172.21.15.29:39154) 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.286+0000 7f426184d640 1 -- 172.21.15.29:0/3467557030 learned_addr learned my addr 172.21.15.29:0/3467557030 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426204e640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f425c108cf0 0x7f425c19bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426104c640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f425c1083a0 0x7f425c19b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426184d640 1 -- 172.21.15.29:0/3467557030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f425c1083a0 msgr2=0x7f425c19b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426184d640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f425c1083a0 0x7f425c19b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426184d640 1 -- 172.21.15.29:0/3467557030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f425c108cf0 msgr2=0x7f425c19bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426184d640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f425c108cf0 0x7f425c19bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426184d640 1 -- 172.21.15.29:0/3467557030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425c1a6450 con 0x7f425c102a60 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426104c640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f425c1083a0 0x7f425c19b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f426184d640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 0x7f425c19b140 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4258009230 tx=0x7f42580319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f4252ffd640 1 -- 172.21.15.29:0/3467557030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425803f030 con 0x7f425c102a60 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f4252ffd640 1 -- 172.21.15.29:0/3467557030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42580043d0 con 0x7f425c102a60 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f4263ab1640 1 -- 172.21.15.29:0/3467557030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f425c1a66e0 con 0x7f425c102a60 2024-09-21T02:45:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f4252ffd640 1 -- 172.21.15.29:0/3467557030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4258035070 con 0x7f425c102a60 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.287+0000 7f4263ab1640 1 -- 172.21.15.29:0/3467557030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f425c1a6c10 con 0x7f425c102a60 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.290+0000 7f426a025740 1 Processor -- start 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.290+0000 7f426a025740 1 -- start start 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.290+0000 7f426a025740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054446510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.290+0000 7f426a025740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 0x561054446a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.290+0000 7f426a025740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054446f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426a025740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561054331dc0 con 0x5610540a0520 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426a025740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5610543be1f0 con 0x56105407f840 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426a025740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5610543ad620 con 0x561054077660 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054446510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054446510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39158/0 (socket says 172.21.15.29:39158) 2024-09-21T02:45:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 -- 172.21.15.29:0/350036644 learned_addr learned my addr 172.21.15.29:0/350036644 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f4252ffd640 1 -- 172.21.15.29:0/3467557030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4258042030 con 0x7f425c102a60 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426104c640 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054446f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 -- 172.21.15.29:0/350036644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 msgr2=0x561054446a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 0x561054446a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 -- 172.21.15.29:0/350036644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 msgr2=0x561054446f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054446f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 -- 172.21.15.29:0/350036644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561054375fd0 con 0x5610540a0520 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426104c640 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054446f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f426204e640 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054446510 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4254009f90 tx=0x7f42540118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27073bfe7bcb3e66 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.291+0000 7f4247fff640 1 -- 172.21.15.29:0/350036644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254004d20 con 0x5610540a0520 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.292+0000 7f4247fff640 1 -- 172.21.15.29:0/350036644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4254004050 con 0x5610540a0520 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.292+0000 7f4247fff640 1 -- 172.21.15.29:0/350036644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254005240 con 0x5610540a0520 2024-09-21T02:45:04.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.292+0000 7f4252ffd640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f42480b6590 0x7f42480b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.292+0000 7f426184d640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f42480b6590 0x7f42480b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.292+0000 7f4252ffd640 1 -- 172.21.15.29:0/3467557030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f42581079e0 con 0x7f425c102a60 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 -- 172.21.15.29:0/350036644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 msgr2=0x561054446510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054446510 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4254009f90 tx=0x7f42540118d0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426184d640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f42480b6590 0x7f42480b8aa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f42580caa60 tx=0x7f42580c20a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 -- 172.21.15.29:0/350036644 shutdown_connections 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054446f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 0x561054446a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 --2- 172.21.15.29:0/350036644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054446510 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 -- 172.21.15.29:0/350036644 >> 172.21.15.29:0/350036644 conn(0x561054094910 msgr2=0x561054444400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 -- 172.21.15.29:0/350036644 shutdown_connections 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 -- 172.21.15.29:0/350036644 wait complete. 2024-09-21T02:45:04.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 -- start start 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.293+0000 7f426a025740 1 Processor -- start 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426a025740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561054077660 0x561054455f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426a025740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x561054456450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426a025740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054456990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426a025740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561054348300 con 0x5610540a0520 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426a025740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5610543b1430 con 0x561054077660 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426a025740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56105420ebb0 con 0x56105407f840 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426a025740 10 client.0 _ll_register_callbacks cb 0x561054362120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426204e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054456990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426204e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054456990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39174/0 (socket says 172.21.15.29:39174) 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426204e640 1 -- 172.21.15.29:0/3561553322 learned_addr learned my addr 172.21.15.29:0/3561553322 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:04.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426204e640 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561054077660 0x561054455f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.294+0000 7f426104c640 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x561054456450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f426204e640 1 -- 172.21.15.29:0/3561553322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 msgr2=0x561054456450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f426204e640 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x561054456450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f426204e640 1 -- 172.21.15.29:0/3561553322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 msgr2=0x561054456990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f426204e640 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054456990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f426204e640 1 -- 172.21.15.29:0/3561553322 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561054337780 con 0x561054077660 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f426204e640 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561054077660 0x561054455f10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4254005c40 tx=0x7f42540118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6df21ed02dc31125 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f42457fa640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f42457fa640 1 -- 172.21.15.29:0/3561553322 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254005040 con 0x561054077660 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f42457fa640 1 -- 172.21.15.29:0/3561553322 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f42540051e0 con 0x561054077660 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.295+0000 7f426a025740 1 -- 172.21.15.29:0/3561553322 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4254004050 con 0x561054077660 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f42457fa640 1 -- 172.21.15.29:0/3561553322 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f42540053e0 con 0x561054077660 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f426a025740 10 client.25889 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f426a025740 1 -- 172.21.15.29:0/3561553322 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56105410bb00 con 0x561054077660 2024-09-21T02:45:04.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f42457fa640 1 -- 172.21.15.29:0/3561553322 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4254004050 con 0x561054077660 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f42457fa640 1 -- 172.21.15.29:0/3561553322 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f4254025040 con 0x561054077660 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f426a025740 10 client.25889 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f426a025740 10 client.25889 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f426a025740 20 client.25889 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f426a025740 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56105405eea0 0x5610544680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f426a025740 4 client.25889 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.296+0000 7f426a025740 1 -- 172.21.15.29:0/3561553322 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610543c0140 con 0x56105405eea0 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.297+0000 7f426104c640 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56105405eea0 0x5610544680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.297+0000 7f426104c640 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56105405eea0 0x5610544680f0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e3cb9ce10a42aed server_cookie=3fbb2f933574ffcf in_seq=0 out_seq=0 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.297+0000 7f42457fa640 10 client.25889 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f42457fa640 1 -- 172.21.15.29:0/3561553322 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5610543c0140 con 0x56105405eea0 2024-09-21T02:45:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f42457fa640 10 client.25889 handle_command_reply: tid=0 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 client.25889 shutdown 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 -- 172.21.15.29:0/3561553322 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561054077660 msgr2=0x561054455f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561054077660 0x561054455f10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4254005c40 tx=0x7f42540118d0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 -- 172.21.15.29:0/3561553322 shutdown_connections 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56105405eea0 0x5610544680f0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054456990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x561054456450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 --2- 172.21.15.29:0/3561553322 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561054077660 0x561054455f10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f42457fa640 0 client.25889 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 -- 172.21.15.29:0/3561553322 >> 172.21.15.29:0/3561553322 conn(0x561054094910 msgr2=0x561054442930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 -- 172.21.15.29:0/3561553322 shutdown_connections 2024-09-21T02:45:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 1 -- 172.21.15.29:0/3561553322 wait complete. 2024-09-21T02:45:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.298+0000 7f426a025740 20 client.25889 trim_cache size 0 max 16384 2024-09-21T02:45:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426a025740 1 Processor -- start 2024-09-21T02:45:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426a025740 1 -- start start 2024-09-21T02:45:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426a025740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 0x561054447f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426a025740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054453100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426a025740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054453640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426a025740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f425c1ae7f0 con 0x5610540a0520 2024-09-21T02:45:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426a025740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f425c1b0330 con 0x56105407f840 2024-09-21T02:45:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426a025740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f425c1aeec0 con 0x561054077660 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426204e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054453640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426104c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054453100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426104c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054453100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:44888/0 (socket says 172.21.15.29:44888) 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426104c640 1 -- 172.21.15.29:0/2828959792 learned_addr learned my addr 172.21.15.29:0/2828959792 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426204e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054453640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39180/0 (socket says 172.21.15.29:39180) 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.322+0000 7f426204e640 1 --2- 172.21.15.29:0/2828959792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 0x561054447f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f426204e640 1 -- 172.21.15.29:0/2828959792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 msgr2=0x561054447f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f426204e640 1 --2- 172.21.15.29:0/2828959792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 0x561054447f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f426204e640 1 -- 172.21.15.29:0/2828959792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 msgr2=0x561054453100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f426204e640 1 --2- 172.21.15.29:0/2828959792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054453100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f426204e640 1 -- 172.21.15.29:0/2828959792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561054182960 con 0x5610540a0520 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f426204e640 1 --2- 172.21.15.29:0/2828959792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054453640 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f4254009bd0 tx=0x7f42540021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d06175461cba79 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f42477fe640 1 -- 172.21.15.29:0/2828959792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254005240 con 0x5610540a0520 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f42477fe640 1 -- 172.21.15.29:0/2828959792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f425402d070 con 0x5610540a0520 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.323+0000 7f42477fe640 1 -- 172.21.15.29:0/2828959792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254028e60 con 0x5610540a0520 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 -- 172.21.15.29:0/2828959792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 msgr2=0x561054453640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 --2- 172.21.15.29:0/2828959792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054453640 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f4254009bd0 tx=0x7f42540021f0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 -- 172.21.15.29:0/2828959792 shutdown_connections 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 --2- 172.21.15.29:0/2828959792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5610540a0520 0x561054453640 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 --2- 172.21.15.29:0/2828959792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56105407f840 0x561054453100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 --2- 172.21.15.29:0/2828959792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561054077660 0x561054447f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 -- 172.21.15.29:0/2828959792 >> 172.21.15.29:0/2828959792 conn(0x561054094910 msgr2=0x561054450c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 -- 172.21.15.29:0/2828959792 shutdown_connections 2024-09-21T02:45:04.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.324+0000 7f426a025740 1 -- 172.21.15.29:0/2828959792 wait complete. 2024-09-21T02:45:04.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 1 -- start start 2024-09-21T02:45:04.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 1 Processor -- start 2024-09-21T02:45:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561054077660 0x56105445a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x56105445ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610540a0520 0x5610544d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f425c1afc60 con 0x561054077660 2024-09-21T02:45:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f425c1af590 con 0x5610540a0520 2024-09-21T02:45:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f425c1ac7d0 con 0x56105407f840 2024-09-21T02:45:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.325+0000 7f426a025740 10 client.0 _ll_register_callbacks cb 0x7f42540045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561054077660 0x56105445a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561054077660 0x56105445a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39188/0 (socket says 172.21.15.29:39188) 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 -- 172.21.15.29:0/3016300125 learned_addr learned my addr 172.21.15.29:0/3016300125 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426104c640 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x56105445ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610540a0520 0x5610544d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 -- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 msgr2=0x56105445ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x56105445ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 -- 172.21.15.29:0/3016300125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610540a0520 msgr2=0x5610544d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610540a0520 0x5610544d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 -- 172.21.15.29:0/3016300125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610544d1e60 con 0x561054077660 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426104c640 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x56105445ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f426204e640 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561054077660 0x56105445a8c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4254004e70 tx=0x7f4254005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26abe3c2233cc70b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f42457fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.326+0000 7f42457fa640 1 -- 172.21.15.29:0/3016300125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254047040 con 0x561054077660 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 1 -- 172.21.15.29:0/3016300125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f425402d070 con 0x561054077660 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f42457fa640 1 -- 172.21.15.29:0/3016300125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4254003d20 con 0x561054077660 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f42457fa640 1 -- 172.21.15.29:0/3016300125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4254003f40 con 0x561054077660 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 10 client.16386 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 1 -- 172.21.15.29:0/3016300125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5610544d3120 con 0x561054077660 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f42457fa640 1 -- 172.21.15.29:0/3016300125 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f425402d070 con 0x561054077660 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f42457fa640 1 -- 172.21.15.29:0/3016300125 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f4254032450 con 0x561054077660 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 10 client.16386 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 10 client.16386 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 20 client.16386 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561054034250 0x5610544d54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 4 client.16386 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2075e1c", "release": true, "await": true}] 2024-09-21T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.327+0000 7f426a025740 1 -- 172.21.15.29:0/3016300125 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2075e1c", "release": true, "await": true}) -- 0x5610543bbbf0 con 0x561054034250 2024-09-21T02:45:04.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.328+0000 7f426104c640 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561054034250 0x5610544d54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:04.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.328+0000 7f42457fa640 10 client.16386 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:04.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.328+0000 7f426104c640 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561054034250 0x5610544d54f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd25a36a2f842f9e server_cookie=9b720331c699ed72 in_seq=0 out_seq=0 2024-09-21T02:45:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.341+0000 7f42457fa640 1 -- 172.21.15.29:0/3016300125 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5610543bbbf0 con 0x561054034250 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.341+0000 7f42457fa640 10 client.16386 handle_command_reply: tid=0 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 client.16386 shutdown 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 -- 172.21.15.29:0/3016300125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561054077660 msgr2=0x56105445a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561054077660 0x56105445a8c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4254004e70 tx=0x7f4254005d10 comp rx=0 tx=0).stop 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 -- 172.21.15.29:0/3016300125 shutdown_connections 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5610540a0520 0x5610544d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56105407f840 0x56105445ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561054034250 0x5610544d54f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 --2- 172.21.15.29:0/3016300125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561054077660 0x56105445a8c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f42457fa640 0 client.16386 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 -- 172.21.15.29:0/3016300125 >> 172.21.15.29:0/3016300125 conn(0x561054094910 msgr2=0x561054455e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 -- 172.21.15.29:0/3016300125 shutdown_connections 2024-09-21T02:45:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 1 -- 172.21.15.29:0/3016300125 wait complete. 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.342+0000 7f426a025740 20 client.16386 trim_cache size 0 max 16384 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 24, 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stdout: "2075e1c": { 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 24, 2024-09-21T02:45:04.345 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.2, 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:45:04.346 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:45:04.347 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:45:04.347 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:04.347 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:04.347 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:04.347 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:04.347 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:04.347 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:45:04.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 -- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f42480b6590 msgr2=0x7f42480b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f42480b6590 0x7f42480b8aa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f42580caa60 tx=0x7f42580c20a0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 -- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 msgr2=0x7f425c19b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 0x7f425c19b140 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4258009230 tx=0x7f42580319a0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 -- 172.21.15.29:0/3467557030 shutdown_connections 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f42480b6590 0x7f42480b8aa0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f425c108cf0 0x7f425c19bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f425c1083a0 0x7f425c19b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 --2- 172.21.15.29:0/3467557030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f425c102a60 0x7f425c19b140 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 -- 172.21.15.29:0/3467557030 >> 172.21.15.29:0/3467557030 conn(0x7f425c079730 msgr2=0x7f425c101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 -- 172.21.15.29:0/3467557030 shutdown_connections 2024-09-21T02:45:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:04.343+0000 7f4263ab1640 1 -- 172.21.15.29:0/3467557030 wait complete. 2024-09-21T02:45:05.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 105318 2.49 MB/sec execute 162 sec latency 18812.250 ms 2024-09-21T02:45:05.091 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2075e1c', total seconds elapsed: 22.7 2024-09-21T02:45:05.091 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-09-21T02:45:05.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:45:05.396 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:45:05.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:05 smithi029 ceph-mon[31006]: pgmap v568: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2024-09-21T02:45:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:05 smithi190 ceph-mon[33871]: pgmap v568: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2024-09-21T02:45:05.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.950+0000 7f5ff6eee640 1 -- 172.21.15.29:0/144839538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff0108d20 msgr2=0x7f5ff010f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:05.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.950+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/144839538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff0108d20 0x7f5ff010f550 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8009900 tx=0x7f5fe802a410 comp rx=0 tx=0).stop 2024-09-21T02:45:05.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.951+0000 7f5ff6eee640 1 -- 172.21.15.29:0/144839538 shutdown_connections 2024-09-21T02:45:05.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.951+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/144839538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff0108d20 0x7f5ff010f550 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:05.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.951+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/144839538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ff01083d0 0x7f5ff01087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.951+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/144839538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ff0102ae0 0x7f5ff0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.951+0000 7f5ff6eee640 1 -- 172.21.15.29:0/144839538 >> 172.21.15.29:0/144839538 conn(0x7f5ff0079790 msgr2=0x7f5ff0079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.951+0000 7f5ff6eee640 1 -- 172.21.15.29:0/144839538 shutdown_connections 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.951+0000 7f5ff6eee640 1 -- 172.21.15.29:0/144839538 wait complete. 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff6eee640 1 Processor -- start 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff6eee640 1 -- start start 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff6eee640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ff0102ae0 0x7f5ff019b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff6eee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff01083d0 0x7f5ff019b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff6eee640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ff0108d20 0x7f5ff019bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff6eee640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff0112890 con 0x7f5ff01083d0 2024-09-21T02:45:05.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff6eee640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff0112710 con 0x7f5ff0102ae0 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff6eee640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff0112a10 con 0x7f5ff0108d20 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5feffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff01083d0 0x7f5ff019b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5feffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff01083d0 0x7f5ff019b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39200/0 (socket says 172.21.15.29:39200) 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5feffff640 1 -- 172.21.15.29:0/4244763122 learned_addr learned my addr 172.21.15.29:0/4244763122 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff548b640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ff0108d20 0x7f5ff019bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.952+0000 7f5ff4c8a640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ff0102ae0 0x7f5ff019b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5feffff640 1 -- 172.21.15.29:0/4244763122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ff0108d20 msgr2=0x7f5ff019bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5feffff640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ff0108d20 0x7f5ff019bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5feffff640 1 -- 172.21.15.29:0/4244763122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ff0102ae0 msgr2=0x7f5ff019b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5feffff640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ff0102ae0 0x7f5ff019b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5feffff640 1 -- 172.21.15.29:0/4244763122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff01a6390 con 0x7f5ff01083d0 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5ff548b640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ff0108d20 0x7f5ff019bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5feffff640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff01083d0 0x7f5ff019b6a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800c8c0 tx=0x7f5fd800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:05.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5fedffb640 1 -- 172.21.15.29:0/4244763122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8007bb0 con 0x7f5ff01083d0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff01a6680 con 0x7f5ff01083d0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5fedffb640 1 -- 172.21.15.29:0/4244763122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fd8007d50 con 0x7f5ff01083d0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5fedffb640 1 -- 172.21.15.29:0/4244763122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8007bb0 con 0x7f5ff01083d0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.953+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ff01991d0 con 0x7f5ff01083d0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.955+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff0102ef0 con 0x7f5ff01083d0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.956+0000 7f5fedffb640 1 -- 172.21.15.29:0/4244763122 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f5fd800fa00 con 0x7f5ff01083d0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.957+0000 7f5fedffb640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5fdc0b6900 0x7f5fdc0b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.957+0000 7f5ff4c8a640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5fdc0b6900 0x7f5fdc0b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.957+0000 7f5fedffb640 1 -- 172.21.15.29:0/4244763122 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5fd80079b0 con 0x7f5ff01083d0 2024-09-21T02:45:05.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.958+0000 7f5ff4c8a640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5fdc0b6900 0x7f5fdc0b8e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5ff006c3a0 tx=0x7f5fe0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:05.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:05.961+0000 7f5fedffb640 1 -- 172.21.15.29:0/4244763122 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5fd801d060 con 0x7f5ff01083d0 2024-09-21T02:45:06.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:05 smithi066 ceph-mon[33436]: pgmap v568: 129 pgs: 129 active+clean; 31 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2024-09-21T02:45:06.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106191 2.52 MB/sec execute 163 sec latency 18.119 ms 2024-09-21T02:45:06.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.205+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ff006bc50 con 0x7f5ff01083d0 2024-09-21T02:45:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.206+0000 7f5fedffb640 1 -- 172.21.15.29:0/4244763122 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f5fd8014030 con 0x7f5ff01083d0 2024-09-21T02:45:06.207 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:45:06.207 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:45:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:45:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5fdc0b6900 msgr2=0x7f5fdc0b8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5fdc0b6900 0x7f5fdc0b8e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5ff006c3a0 tx=0x7f5fe0009500 comp rx=0 tx=0).stop 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff01083d0 msgr2=0x7f5ff019b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff01083d0 0x7f5ff019b6a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800c8c0 tx=0x7f5fd800cd80 comp rx=0 tx=0).stop 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 shutdown_connections 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5fdc0b6900 0x7f5fdc0b8e10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ff0108d20 0x7f5ff019bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ff01083d0 0x7f5ff019b6a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 --2- 172.21.15.29:0/4244763122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ff0102ae0 0x7f5ff019b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 >> 172.21.15.29:0/4244763122 conn(0x7f5ff0079790 msgr2=0x7f5ff01043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 shutdown_connections 2024-09-21T02:45:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:06.208+0000 7f5ff6eee640 1 -- 172.21.15.29:0/4244763122 wait complete. 2024-09-21T02:45:06.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:06 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4244763122' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:06.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:06 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4244763122' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:06.972 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0' 2024-09-21T02:45:06.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0 2024-09-21T02:45:07.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:06 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4244763122' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:07.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106668 2.53 MB/sec execute 164 sec latency 14.638 ms 2024-09-21T02:45:07.262 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:45:07.734 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:07 smithi029 ceph-mon[31006]: pgmap v569: 129 pgs: 129 active+clean; 32 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 121 KiB/s wr, 14 op/s 2024-09-21T02:45:07.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 -- 172.21.15.29:0/182094537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c102a80 msgr2=0x7f513c102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:07.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 --2- 172.21.15.29:0/182094537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c102a80 0x7f513c102e90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5138009900 tx=0x7f513802a410 comp rx=0 tx=0).stop 2024-09-21T02:45:07.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 -- 172.21.15.29:0/182094537 shutdown_connections 2024-09-21T02:45:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 --2- 172.21.15.29:0/182094537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f513c108d10 0x7f513c10f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 --2- 172.21.15.29:0/182094537 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f513c1083c0 0x7f513c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 --2- 172.21.15.29:0/182094537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c102a80 0x7f513c102e90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 -- 172.21.15.29:0/182094537 >> 172.21.15.29:0/182094537 conn(0x7f513c079760 msgr2=0x7f513c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 -- 172.21.15.29:0/182094537 shutdown_connections 2024-09-21T02:45:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.845+0000 7f5144454640 1 -- 172.21.15.29:0/182094537 wait complete. 2024-09-21T02:45:07.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.846+0000 7f5144454640 1 Processor -- start 2024-09-21T02:45:07.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.846+0000 7f5144454640 1 -- start start 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f5144454640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f513c102a80 0x7f513c19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f5144454640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c1083c0 0x7f513c19b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f5144454640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f513c108d10 0x7f513c19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f5144454640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f513c112830 con 0x7f513c1083c0 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f5144454640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1126b0 con 0x7f513c102a80 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f5144454640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1129b0 con 0x7f513c108d10 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c1083c0 0x7f513c19b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c1083c0 0x7f513c19b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39216/0 (socket says 172.21.15.29:39216) 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51429f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f513c108d10 0x7f513c19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 -- 172.21.15.29:0/3841053155 learned_addr learned my addr 172.21.15.29:0/3841053155 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51421f0640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f513c102a80 0x7f513c19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 -- 172.21.15.29:0/3841053155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f513c108d10 msgr2=0x7f513c19bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f513c108d10 0x7f513c19bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 -- 172.21.15.29:0/3841053155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f513c102a80 msgr2=0x7f513c19b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f513c102a80 0x7f513c19b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 -- 172.21.15.29:0/3841053155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c1a6490 con 0x7f513c1083c0 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51429f1640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f513c108d10 0x7f513c19bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.847+0000 7f51419ef640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c1083c0 0x7f513c19b740 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f512c005fd0 tx=0x7f512c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.848+0000 7f51337fe640 1 -- 172.21.15.29:0/3841053155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c017070 con 0x7f513c1083c0 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.848+0000 7f51337fe640 1 -- 172.21.15.29:0/3841053155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f512c009c30 con 0x7f513c1083c0 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.848+0000 7f5144454640 1 -- 172.21.15.29:0/3841053155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513c1a6780 con 0x7f513c1083c0 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.848+0000 7f51337fe640 1 -- 172.21.15.29:0/3841053155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c003c70 con 0x7f513c1083c0 2024-09-21T02:45:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.848+0000 7f5144454640 1 -- 172.21.15.29:0/3841053155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513c1a6cc0 con 0x7f513c1083c0 2024-09-21T02:45:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.850+0000 7f51337fe640 1 -- 172.21.15.29:0/3841053155 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f512c003e10 con 0x7f513c1083c0 2024-09-21T02:45:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.851+0000 7f514a9c8740 1 Processor -- start 2024-09-21T02:45:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.851+0000 7f514a9c8740 1 -- start start 2024-09-21T02:45:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.851+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d002e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.851+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d0033b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.851+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d0038f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f514a9c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563d6cf79bb0 con 0x563d6cc33990 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f514a9c8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x563d6cf68fe0 con 0x563d6cc5c850 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f514a9c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563d6cf6cdf0 con 0x563d6cc3bb70 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51429f1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d002e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51429f1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d002e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:44954/0 (socket says 172.21.15.29:44954) 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51419ef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d0038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51429f1640 1 -- 172.21.15.29:0/3246929092 learned_addr learned my addr 172.21.15.29:0/3246929092 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51421f0640 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d0033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51337fe640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51280b6540 0x7f51280b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51429f1640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51280b6540 0x7f51280b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51419ef640 1 -- 172.21.15.29:0/3246929092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 msgr2=0x563d6d002e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51419ef640 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d002e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51419ef640 1 -- 172.21.15.29:0/3246929092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 msgr2=0x563d6d0033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51419ef640 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d0033b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51419ef640 1 -- 172.21.15.29:0/3246929092 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d6ccd6ba0 con 0x563d6cc3bb70 2024-09-21T02:45:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.852+0000 7f51421f0640 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d0033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:45:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.853+0000 7f51337fe640 1 -- 172.21.15.29:0/3841053155 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f512c0e7220 con 0x7f513c1083c0 2024-09-21T02:45:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.853+0000 7f51419ef640 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d0038f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f512c0e4970 tx=0x7f512c0a18e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdc62f9245a487b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.853+0000 7f51429f1640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51280b6540 0x7f51280b8a50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5134009fd0 tx=0x7f5134004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.853+0000 7f5130d9a640 1 -- 172.21.15.29:0/3246929092 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c0ace00 con 0x563d6cc3bb70 2024-09-21T02:45:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.853+0000 7f5130d9a640 1 -- 172.21.15.29:0/3246929092 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f512c092e50 con 0x563d6cc3bb70 2024-09-21T02:45:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.853+0000 7f5130d9a640 1 -- 172.21.15.29:0/3246929092 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c0947c0 con 0x563d6cc3bb70 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 -- 172.21.15.29:0/3246929092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 msgr2=0x563d6d0038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d0038f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f512c0e4970 tx=0x7f512c0a18e0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 -- 172.21.15.29:0/3246929092 shutdown_connections 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d0038f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d0033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3246929092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d002e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 -- 172.21.15.29:0/3246929092 >> 172.21.15.29:0/3246929092 conn(0x563d6cc50c40 msgr2=0x563d6d000d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 -- 172.21.15.29:0/3246929092 shutdown_connections 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 -- 172.21.15.29:0/3246929092 wait complete. 2024-09-21T02:45:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.854+0000 7f514a9c8740 1 -- start start 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f514a9c8740 1 Processor -- start 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d00e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d00ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d01a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f514a9c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563d6cdca7b0 con 0x563d6cc33990 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f514a9c8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x563d6cf03e90 con 0x563d6cc5c850 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f514a9c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563d6d0115f0 con 0x563d6cc3bb70 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f51421f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d00e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f51421f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d00e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39232/0 (socket says 172.21.15.29:39232) 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f514a9c8740 10 client.0 _ll_register_callbacks cb 0x563d6cf66710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f51421f0640 1 -- 172.21.15.29:0/3040513496 learned_addr learned my addr 172.21.15.29:0/3040513496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f51419ef640 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d00ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.855+0000 7f51429f1640 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d01a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f51419ef640 1 -- 172.21.15.29:0/3040513496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 msgr2=0x563d6d01a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f51419ef640 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d01a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f51419ef640 1 -- 172.21.15.29:0/3040513496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 msgr2=0x563d6d00e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f51419ef640 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d00e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f51419ef640 1 -- 172.21.15.29:0/3040513496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d6d01ae50 con 0x563d6cc3bb70 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f51429f1640 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d01a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f51419ef640 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d00ef10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f512c0e49f0 tx=0x7f512c0eaa50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46fdaa50537e8e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f5125ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f5125ffb640 1 -- 172.21.15.29:0/3040513496 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c0e4e60 con 0x563d6cc3bb70 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f514a9c8740 1 -- 172.21.15.29:0/3040513496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f512c092e50 con 0x563d6cc3bb70 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f5125ffb640 1 -- 172.21.15.29:0/3040513496 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f512c0c2ef0 con 0x563d6cc3bb70 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f5125ffb640 1 -- 172.21.15.29:0/3040513496 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f512c0ac7a0 con 0x563d6cc3bb70 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f514a9c8740 10 client.26002 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.856+0000 7f514a9c8740 1 -- 172.21.15.29:0/3040513496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563d6d01d6f0 con 0x563d6cc3bb70 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f5125ffb640 1 -- 172.21.15.29:0/3040513496 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f512c092e50 con 0x563d6cc3bb70 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f5125ffb640 1 -- 172.21.15.29:0/3040513496 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f512c096bd0 con 0x563d6cc3bb70 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f514a9c8740 10 client.26002 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f514a9c8740 10 client.26002 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f514a9c8740 20 client.26002 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563d6cc1b1d0 0x563d6d020a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f514a9c8740 4 client.26002 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f514a9c8740 1 -- 172.21.15.29:0/3040513496 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d6cf7bb00 con 0x563d6cc1b1d0 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.857+0000 7f51421f0640 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563d6cc1b1d0 0x563d6d020a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.858+0000 7f5125ffb640 10 client.26002 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.858+0000 7f51421f0640 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563d6cc1b1d0 0x563d6d020a50 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=304e8844266ee481 server_cookie=2e7fbbaa4df531c8 in_seq=0 out_seq=0 2024-09-21T02:45:07.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.858+0000 7f5125ffb640 1 -- 172.21.15.29:0/3040513496 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563d6cf7bb00 con 0x563d6cc1b1d0 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.858+0000 7f5125ffb640 10 client.26002 handle_command_reply: tid=0 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.858+0000 7f514a9c8740 1 client.26002 shutdown 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 -- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 msgr2=0x563d6d00ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d00ef10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f512c0e49f0 tx=0x7f512c0eaa50 comp rx=0 tx=0).stop 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 -- 172.21.15.29:0/3040513496 shutdown_connections 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d01a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc3bb70 0x563d6d00ef10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563d6cc1b1d0 0x563d6d020a50 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 --2- 172.21.15.29:0/3040513496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d00e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f5125ffb640 0 client.26002 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 -- 172.21.15.29:0/3040513496 >> 172.21.15.29:0/3040513496 conn(0x563d6cc50c40 msgr2=0x563d6d003ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 -- 172.21.15.29:0/3040513496 shutdown_connections 2024-09-21T02:45:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 1 -- 172.21.15.29:0/3040513496 wait complete. 2024-09-21T02:45:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.859+0000 7f514a9c8740 20 client.26002 trim_cache size 0 max 16384 2024-09-21T02:45:07.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:07 smithi190 ceph-mon[33871]: pgmap v569: 129 pgs: 129 active+clean; 32 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 121 KiB/s wr, 14 op/s 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.878+0000 7f514a9c8740 1 Processor -- start 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.878+0000 7f514a9c8740 1 -- start start 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.878+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d01a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.878+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc3bb70 0x563d6d000dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.878+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc5c850 0x563d6d0106b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.878+0000 7f514a9c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1ae830 con 0x563d6cc33990 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.878+0000 7f514a9c8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1b0370 con 0x563d6cc3bb70 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.878+0000 7f514a9c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1aef00 con 0x563d6cc5c850 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d01a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d01a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39234/0 (socket says 172.21.15.29:39234) 2024-09-21T02:45:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 -- 172.21.15.29:0/2104608970 learned_addr learned my addr 172.21.15.29:0/2104608970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51429f1640 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc5c850 0x563d6d0106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51419ef640 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc3bb70 0x563d6d000dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 -- 172.21.15.29:0/2104608970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc5c850 msgr2=0x563d6d0106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc5c850 0x563d6d0106b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 -- 172.21.15.29:0/2104608970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc3bb70 msgr2=0x563d6d000dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc3bb70 0x563d6d000dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 -- 172.21.15.29:0/2104608970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d6d005160 con 0x563d6cc33990 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f51421f0640 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d01a850 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f513802a930 tx=0x7f51380058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e596fbefa0941f5e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f5127fff640 1 -- 172.21.15.29:0/2104608970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513804b040 con 0x563d6cc33990 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f5127fff640 1 -- 172.21.15.29:0/2104608970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5138031890 con 0x563d6cc33990 2024-09-21T02:45:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.879+0000 7f5127fff640 1 -- 172.21.15.29:0/2104608970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138035040 con 0x563d6cc33990 2024-09-21T02:45:07.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 -- 172.21.15.29:0/2104608970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 msgr2=0x563d6d01a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d01a850 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f513802a930 tx=0x7f51380058d0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 -- 172.21.15.29:0/2104608970 shutdown_connections 2024-09-21T02:45:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc5c850 0x563d6d0106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc3bb70 0x563d6d000dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 --2- 172.21.15.29:0/2104608970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc33990 0x563d6d01a850 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 -- 172.21.15.29:0/2104608970 >> 172.21.15.29:0/2104608970 conn(0x563d6cc50c40 msgr2=0x563d6d00d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 -- 172.21.15.29:0/2104608970 shutdown_connections 2024-09-21T02:45:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.880+0000 7f514a9c8740 1 -- 172.21.15.29:0/2104608970 wait complete. 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.881+0000 7f514a9c8740 1 -- start start 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.881+0000 7f514a9c8740 1 Processor -- start 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.881+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc33990 0x563d6d024390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.881+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc3bb70 0x563d6d0228c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.881+0000 7f514a9c8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d022f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.881+0000 7f514a9c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1afca0 con 0x563d6cc3bb70 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f514a9c8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1af5d0 con 0x563d6cc5c850 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f514a9c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1ac810 con 0x563d6cc33990 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51419ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc3bb70 0x563d6d0228c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51429f1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d022f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51419ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc3bb70 0x563d6d0228c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39250/0 (socket says 172.21.15.29:39250) 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51419ef640 1 -- 172.21.15.29:0/1202050712 learned_addr learned my addr 172.21.15.29:0/1202050712 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51421f0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc33990 0x563d6d024390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f514a9c8740 10 client.0 _ll_register_callbacks cb 0x7f5134009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51429f1640 1 -- 172.21.15.29:0/1202050712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc33990 msgr2=0x563d6d024390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51429f1640 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc33990 0x563d6d024390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51429f1640 1 -- 172.21.15.29:0/1202050712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc3bb70 msgr2=0x563d6d0228c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51429f1640 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc3bb70 0x563d6d0228c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51429f1640 1 -- 172.21.15.29:0/1202050712 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d6d0236d0 con 0x563d6cc5c850 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51419ef640 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc3bb70 0x563d6d0228c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f51429f1640 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d022f60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f513400fc20 tx=0x7f5134004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=685f953514fcf3a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f5125ffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.882+0000 7f5125ffb640 1 -- 172.21.15.29:0/1202050712 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513401a040 con 0x563d6cc5c850 2024-09-21T02:45:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.883+0000 7f5125ffb640 1 -- 172.21.15.29:0/1202050712 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f513401fa10 con 0x563d6cc5c850 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.883+0000 7f514a9c8740 1 -- 172.21.15.29:0/1202050712 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5138031890 con 0x563d6cc5c850 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.883+0000 7f5125ffb640 1 -- 172.21.15.29:0/1202050712 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5134039050 con 0x563d6cc5c850 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.883+0000 7f514a9c8740 10 client.25916 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.883+0000 7f514a9c8740 1 -- 172.21.15.29:0/1202050712 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563d6d08c270 con 0x563d6cc5c850 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.883+0000 7f5125ffb640 1 -- 172.21.15.29:0/1202050712 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5138031890 con 0x563d6cc5c850 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.883+0000 7f5125ffb640 1 -- 172.21.15.29:0/1202050712 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f513402bbb0 con 0x563d6cc5c850 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f514a9c8740 10 client.25916 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f514a9c8740 10 client.25916 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f514a9c8740 20 client.25916 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f514a9c8740 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563d6cbf0580 0x563d6d08eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f514a9c8740 4 client.25916 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}] 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f514a9c8740 1 -- 172.21.15.29:0/1202050712 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}) -- 0x563d6cf775b0 con 0x563d6cbf0580 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f51421f0640 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563d6cbf0580 0x563d6d08eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:07.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f51421f0640 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563d6cbf0580 0x563d6d08eb50 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5686cc02fd5de138 server_cookie=57185486a04aa250 in_seq=0 out_seq=0 2024-09-21T02:45:07.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:07.884+0000 7f5125ffb640 10 client.25916 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:07 smithi066 ceph-mon[33436]: pgmap v569: 129 pgs: 129 active+clean; 32 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 121 KiB/s wr, 14 op/s 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f5125ffb640 1 -- 172.21.15.29:0/1202050712 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563d6cf775b0 con 0x563d6cbf0580 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f5125ffb640 10 client.25916 handle_command_reply: tid=0 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 client.25916 shutdown 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 -- 172.21.15.29:0/1202050712 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 msgr2=0x563d6d022f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d022f60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f513400fc20 tx=0x7f5134004a40 comp rx=0 tx=0).stop 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 -- 172.21.15.29:0/1202050712 shutdown_connections 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563d6cc5c850 0x563d6d022f60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563d6cbf0580 0x563d6d08eb50 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563d6cc3bb70 0x563d6d0228c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 --2- 172.21.15.29:0/1202050712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563d6cc33990 0x563d6d024390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f5125ffb640 0 client.25916 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 -- 172.21.15.29:0/1202050712 >> 172.21.15.29:0/1202050712 conn(0x563d6cc50c40 msgr2=0x563d6d010410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 -- 172.21.15.29:0/1202050712 shutdown_connections 2024-09-21T02:45:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.018+0000 7f514a9c8740 1 -- 172.21.15.29:0/1202050712 wait complete. 2024-09-21T02:45:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.019+0000 7f514a9c8740 20 client.25916 trim_cache size 0 max 16384 2024-09-21T02:45:08.022 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:45:08.022 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:45:08.022 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:45:08.022 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 26, 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "18c9256d": { 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 26, 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.2, 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:45:08.023 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:08.024 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.019+0000 7f5144454640 1 -- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51280b6540 msgr2=0x7f51280b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.019+0000 7f5144454640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51280b6540 0x7f51280b8a50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5134009fd0 tx=0x7f5134004ab0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.019+0000 7f5144454640 1 -- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c1083c0 msgr2=0x7f513c19b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.019+0000 7f5144454640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c1083c0 0x7f513c19b740 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f512c005fd0 tx=0x7f512c00dde0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.020+0000 7f5144454640 1 -- 172.21.15.29:0/3841053155 shutdown_connections 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.020+0000 7f5144454640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f51280b6540 0x7f51280b8a50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.020+0000 7f5144454640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f513c108d10 0x7f513c19bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.020+0000 7f5144454640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c1083c0 0x7f513c19b740 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.020+0000 7f5144454640 1 --2- 172.21.15.29:0/3841053155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f513c102a80 0x7f513c19b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.020+0000 7f5144454640 1 -- 172.21.15.29:0/3841053155 >> 172.21.15.29:0/3841053155 conn(0x7f513c079760 msgr2=0x7f513c101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.020+0000 7f5144454640 1 -- 172.21.15.29:0/3841053155 shutdown_connections 2024-09-21T02:45:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:08.020+0000 7f5144454640 1 -- 172.21.15.29:0/3841053155 wait complete. 2024-09-21T02:45:08.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.52 MB/sec execute 165 sec latency 121.320 ms 2024-09-21T02:45:08.850 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '18c9256d', quiesce took 3.8 seconds. Will release after: 15.45 2024-09-21T02:45:09.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.50 MB/sec execute 166 sec latency 1060.367 ms 2024-09-21T02:45:09.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:09 smithi190 ceph-mon[33871]: pgmap v570: 129 pgs: 129 active+clean; 33 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 281 KiB/s wr, 35 op/s 2024-09-21T02:45:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:09 smithi029 ceph-mon[31006]: pgmap v570: 129 pgs: 129 active+clean; 33 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 281 KiB/s wr, 35 op/s 2024-09-21T02:45:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:09 smithi066 ceph-mon[33436]: pgmap v570: 129 pgs: 129 active+clean; 33 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 281 KiB/s wr, 35 op/s 2024-09-21T02:45:10.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.49 MB/sec execute 167 sec latency 2060.486 ms 2024-09-21T02:45:11.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.47 MB/sec execute 168 sec latency 3060.591 ms 2024-09-21T02:45:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:11 smithi190 ceph-mon[33871]: pgmap v571: 129 pgs: 129 active+clean; 34 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 474 KiB/s wr, 46 op/s 2024-09-21T02:45:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:11 smithi029 ceph-mon[31006]: pgmap v571: 129 pgs: 129 active+clean; 34 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 474 KiB/s wr, 46 op/s 2024-09-21T02:45:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:11 smithi066 ceph-mon[33436]: pgmap v571: 129 pgs: 129 active+clean; 34 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 474 KiB/s wr, 46 op/s 2024-09-21T02:45:12.079 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.46 MB/sec execute 169 sec latency 4060.658 ms 2024-09-21T02:45:13.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.44 MB/sec execute 170 sec latency 5060.776 ms 2024-09-21T02:45:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 -- 172.21.15.29:0/2994125981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 msgr2=0x7fc85410a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 --2- 172.21.15.29:0/2994125981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 0x7fc85410a810 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00b0a0 tx=0x7fc84c02a710 comp rx=0 tx=0).stop 2024-09-21T02:45:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 -- 172.21.15.29:0/2994125981 shutdown_connections 2024-09-21T02:45:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 --2- 172.21.15.29:0/2994125981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 0x7fc85410a810 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 --2- 172.21.15.29:0/2994125981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8541056e0 0x7fc854107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 --2- 172.21.15.29:0/2994125981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc854102ca0 0x7fc8541050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 -- 172.21.15.29:0/2994125981 >> 172.21.15.29:0/2994125981 conn(0x7fc8540fd690 msgr2=0x7fc8540ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 -- 172.21.15.29:0/2994125981 shutdown_connections 2024-09-21T02:45:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.737+0000 7fc858eb4640 1 -- 172.21.15.29:0/2994125981 wait complete. 2024-09-21T02:45:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.738+0000 7fc858eb4640 1 Processor -- start 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.738+0000 7fc858eb4640 1 -- start start 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.738+0000 7fc858eb4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc854102ca0 0x7fc854199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.738+0000 7fc858eb4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8541056e0 0x7fc85419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.738+0000 7fc858eb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 0x7fc85419b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc858eb4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc85410e0d0 con 0x7fc854108300 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc858eb4640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc85410df50 con 0x7fc8541056e0 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc858eb4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc85410e250 con 0x7fc854102ca0 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 0x7fc85419b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 0x7fc85419b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39270/0 (socket says 172.21.15.29:39270) 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 -- 172.21.15.29:0/594397017 learned_addr learned my addr 172.21.15.29:0/594397017 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc85259c640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc854102ca0 0x7fc854199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc851d9b640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8541056e0 0x7fc85419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 -- 172.21.15.29:0/594397017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc854102ca0 msgr2=0x7fc854199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc854102ca0 0x7fc854199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 -- 172.21.15.29:0/594397017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8541056e0 msgr2=0x7fc85419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8541056e0 0x7fc85419a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 -- 172.21.15.29:0/594397017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85419b8b0 con 0x7fc854108300 2024-09-21T02:45:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc852d9d640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 0x7fc85419b090 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c090 tx=0x7fc84c0097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc84b7fe640 1 -- 172.21.15.29:0/594397017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c040030 con 0x7fc854108300 2024-09-21T02:45:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc858eb4640 1 -- 172.21.15.29:0/594397017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc85419bb40 con 0x7fc854108300 2024-09-21T02:45:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc84b7fe640 1 -- 172.21.15.29:0/594397017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc84c031730 con 0x7fc854108300 2024-09-21T02:45:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc84b7fe640 1 -- 172.21.15.29:0/594397017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c036040 con 0x7fc854108300 2024-09-21T02:45:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.739+0000 7fc858eb4640 1 -- 172.21.15.29:0/594397017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8541a2290 con 0x7fc854108300 2024-09-21T02:45:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.742+0000 7fc85fe28740 1 Processor -- start 2024-09-21T02:45:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.742+0000 7fc84b7fe640 1 -- 172.21.15.29:0/594397017 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc84c043070 con 0x7fc854108300 2024-09-21T02:45:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85fe28740 1 -- start start 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 0x559f7588be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754bb2f0 0x559f7588c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754c34d0 0x559f7588d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85fe28740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559f757fe6f0 con 0x559f754bb2f0 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85fe28740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559f757edb20 con 0x559f754c34d0 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85fe28740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559f757f1930 con 0x559f754e41b0 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754bb2f0 0x559f7588c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754bb2f0 0x559f7588c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39274/0 (socket says 172.21.15.29:39274) 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 -- 172.21.15.29:0/1479091902 learned_addr learned my addr 172.21.15.29:0/1479091902 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc851d9b640 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 0x559f7588be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc851d9b640 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754c34d0 0x559f7588d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 -- 172.21.15.29:0/1479091902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 msgr2=0x559f7588be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 0x559f7588be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 -- 172.21.15.29:0/1479091902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754c34d0 msgr2=0x559f7588d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754c34d0 0x559f7588d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 -- 172.21.15.29:0/1479091902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f757b3530 con 0x559f754bb2f0 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc84b7fe640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8400b6510 0x7fc8400b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.743+0000 7fc85259c640 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754bb2f0 0x559f7588c4c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc844009d20 tx=0x7fc844011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734b73de701c1853 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.744+0000 7fc851d9b640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8400b6510 0x7fc8400b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.744+0000 7fc84b7fe640 1 -- 172.21.15.29:0/594397017 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc84c107f30 con 0x7fc854108300 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.744+0000 7fc848ff9640 1 -- 172.21.15.29:0/1479091902 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84401e070 con 0x559f754bb2f0 2024-09-21T02:45:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.744+0000 7fc848ff9640 1 -- 172.21.15.29:0/1479091902 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc844019ac0 con 0x559f754bb2f0 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.744+0000 7fc848ff9640 1 -- 172.21.15.29:0/1479091902 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84400e9b0 con 0x559f754bb2f0 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 -- 172.21.15.29:0/1479091902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754bb2f0 msgr2=0x559f7588c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc851d9b640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8400b6510 0x7fc8400b8a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009460 tx=0x7fc83c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754bb2f0 0x559f7588c4c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc844009d20 tx=0x7fc844011810 comp rx=0 tx=0).stop 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 -- 172.21.15.29:0/1479091902 shutdown_connections 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754c34d0 0x559f7588d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754bb2f0 0x559f7588c4c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1479091902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 0x559f7588be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 -- 172.21.15.29:0/1479091902 >> 172.21.15.29:0/1479091902 conn(0x559f754d8300 msgr2=0x559f758878e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 -- 172.21.15.29:0/1479091902 shutdown_connections 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 -- 172.21.15.29:0/1479091902 wait complete. 2024-09-21T02:45:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 -- start start 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.745+0000 7fc85fe28740 1 Processor -- start 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75895610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75895cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 0x559f75896840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc85fe28740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559f75725ab0 con 0x559f754c34d0 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc85fe28740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559f75793730 con 0x559f754bb2f0 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc85fe28740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559f7588db30 con 0x559f754e41b0 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc85fe28740 10 client.0 _ll_register_callbacks cb 0x559f757e76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75895cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75895cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39280/0 (socket says 172.21.15.29:39280) 2024-09-21T02:45:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 -- 172.21.15.29:0/1359663347 learned_addr learned my addr 172.21.15.29:0/1359663347 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc85259c640 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75895610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc852d9d640 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 0x559f75896840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 -- 172.21.15.29:0/1359663347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 msgr2=0x559f75896840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 0x559f75896840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 -- 172.21.15.29:0/1359663347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 msgr2=0x559f75895610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75895610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 -- 172.21.15.29:0/1359663347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f7554f790 con 0x559f754c34d0 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc851d9b640 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75895cd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc83c011c30 tx=0x7fc83c0093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd269e7ab0569d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.746+0000 7fc839ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc839ffb640 1 -- 172.21.15.29:0/1359663347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c009e60 con 0x559f754c34d0 2024-09-21T02:45:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc839ffb640 1 -- 172.21.15.29:0/1359663347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc83c00e930 con 0x559f754c34d0 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 1 -- 172.21.15.29:0/1359663347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc844019ac0 con 0x559f754c34d0 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc839ffb640 1 -- 172.21.15.29:0/1359663347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc83c00eb30 con 0x559f754c34d0 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 10 client.16419 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 1 -- 172.21.15.29:0/1359663347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559f755c8620 con 0x559f754c34d0 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc839ffb640 1 -- 172.21.15.29:0/1359663347 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc844019ac0 con 0x559f754c34d0 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc839ffb640 1 -- 172.21.15.29:0/1359663347 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc83c0038b0 con 0x559f754c34d0 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 10 client.16419 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 20 client.16419 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559f754a2d80 0x559f758a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 4 client.16419 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:45:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.747+0000 7fc85fe28740 1 -- 172.21.15.29:0/1359663347 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f75800640 con 0x559f754a2d80 2024-09-21T02:45:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.748+0000 7fc85259c640 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559f754a2d80 0x559f758a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.748+0000 7fc85259c640 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559f754a2d80 0x559f758a2ec0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c024cfb560f0fee6 server_cookie=3d4fc1e6094780ac in_seq=0 out_seq=0 2024-09-21T02:45:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.748+0000 7fc839ffb640 10 client.16419 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc839ffb640 1 -- 172.21.15.29:0/1359663347 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559f75800640 con 0x559f754a2d80 2024-09-21T02:45:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc839ffb640 10 client.16419 handle_command_reply: tid=0 2024-09-21T02:45:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 client.16419 shutdown 2024-09-21T02:45:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 -- 172.21.15.29:0/1359663347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 msgr2=0x559f75895cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75895cd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc83c011c30 tx=0x7fc83c0093b0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 -- 172.21.15.29:0/1359663347 shutdown_connections 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754e41b0 0x559f75896840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559f754a2d80 0x559f758a2ec0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75895cd0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 --2- 172.21.15.29:0/1359663347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75895610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc839ffb640 0 client.16419 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 -- 172.21.15.29:0/1359663347 >> 172.21.15.29:0/1359663347 conn(0x559f754d8300 msgr2=0x559f75885ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 -- 172.21.15.29:0/1359663347 shutdown_connections 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 1 -- 172.21.15.29:0/1359663347 wait complete. 2024-09-21T02:45:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.749+0000 7fc85fe28740 20 client.16419 trim_cache size 0 max 16384 2024-09-21T02:45:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85fe28740 1 Processor -- start 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85fe28740 1 -- start start 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75894830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754c34d0 0x559f758825c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754e41b0 0x559f75893b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85fe28740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc8541ac500 con 0x559f754e41b0 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85fe28740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc8541ae040 con 0x559f754bb2f0 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85fe28740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc8541acbd0 con 0x559f754c34d0 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc852d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754e41b0 0x559f75893b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc851d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754c34d0 0x559f758825c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc852d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754e41b0 0x559f75893b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39286/0 (socket says 172.21.15.29:39286) 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc852d9d640 1 -- 172.21.15.29:0/2260510943 learned_addr learned my addr 172.21.15.29:0/2260510943 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.767+0000 7fc85259c640 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75894830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc852d9d640 1 -- 172.21.15.29:0/2260510943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754c34d0 msgr2=0x559f758825c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc852d9d640 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754c34d0 0x559f758825c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc852d9d640 1 -- 172.21.15.29:0/2260510943 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 msgr2=0x559f75894830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc852d9d640 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75894830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc852d9d640 1 -- 172.21.15.29:0/2260510943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f75896bb0 con 0x559f754e41b0 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc851d9b640 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754c34d0 0x559f758825c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc85259c640 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75894830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc852d9d640 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754e41b0 0x559f75893b20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0caf50 tx=0x7fc84c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcc623333583163 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc83bfff640 1 -- 172.21.15.29:0/2260510943 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c004040 con 0x559f754e41b0 2024-09-21T02:45:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc83bfff640 1 -- 172.21.15.29:0/2260510943 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc84c0cd7f0 con 0x559f754e41b0 2024-09-21T02:45:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.768+0000 7fc83bfff640 1 -- 172.21.15.29:0/2260510943 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0c9f50 con 0x559f754e41b0 2024-09-21T02:45:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 -- 172.21.15.29:0/2260510943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754e41b0 msgr2=0x559f75893b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754e41b0 0x559f75893b20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0caf50 tx=0x7fc84c02a710 comp rx=0 tx=0).stop 2024-09-21T02:45:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 -- 172.21.15.29:0/2260510943 shutdown_connections 2024-09-21T02:45:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754e41b0 0x559f75893b20 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754c34d0 0x559f758825c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 --2- 172.21.15.29:0/2260510943 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754bb2f0 0x559f75894830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 -- 172.21.15.29:0/2260510943 >> 172.21.15.29:0/2260510943 conn(0x559f754d8300 msgr2=0x559f75896e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 -- 172.21.15.29:0/2260510943 shutdown_connections 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 -- 172.21.15.29:0/2260510943 wait complete. 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.769+0000 7fc85fe28740 1 -- start start 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85fe28740 1 Processor -- start 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754bb2f0 0x559f75908870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75908f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85fe28740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754e41b0 0x559f75909aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85fe28740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc8541ad970 con 0x559f754c34d0 2024-09-21T02:45:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85fe28740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc8541ad2a0 con 0x559f754e41b0 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85fe28740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc8541aa4e0 con 0x559f754bb2f0 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85fe28740 10 client.0 _ll_register_callbacks cb 0x7fc83c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc851d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75908f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc851d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75908f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39296/0 (socket says 172.21.15.29:39296) 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc851d9b640 1 -- 172.21.15.29:0/3983281882 learned_addr learned my addr 172.21.15.29:0/3983281882 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc852d9d640 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754e41b0 0x559f75909aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc85259c640 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754bb2f0 0x559f75908870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc851d9b640 1 -- 172.21.15.29:0/3983281882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754bb2f0 msgr2=0x559f75908870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc851d9b640 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754bb2f0 0x559f75908870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc851d9b640 1 -- 172.21.15.29:0/3983281882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754e41b0 msgr2=0x559f75909aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc851d9b640 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754e41b0 0x559f75909aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.770+0000 7fc851d9b640 1 -- 172.21.15.29:0/3983281882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f7590a200 con 0x559f754c34d0 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc851d9b640 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75908f30 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc83c003ee0 tx=0x7fc83c012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e054ff585746ff8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc839ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc839ffb640 1 -- 172.21.15.29:0/3983281882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c03a030 con 0x559f754c34d0 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc839ffb640 1 -- 172.21.15.29:0/3983281882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc83c00ecd0 con 0x559f754c34d0 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 1 -- 172.21.15.29:0/3983281882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc84c0cd7f0 con 0x559f754c34d0 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc839ffb640 1 -- 172.21.15.29:0/3983281882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc83c047050 con 0x559f754c34d0 2024-09-21T02:45:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 10 client.16425 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 1 -- 172.21.15.29:0/3983281882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559f759115d0 con 0x559f754c34d0 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc839ffb640 1 -- 172.21.15.29:0/3983281882 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc84c0cd7f0 con 0x559f754c34d0 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc839ffb640 1 -- 172.21.15.29:0/3983281882 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc83c003fa0 con 0x559f754c34d0 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 10 client.16425 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 20 client.16425 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559f754781c0 0x559f75913f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 4 client.16425 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.771+0000 7fc85fe28740 1 -- 172.21.15.29:0/3983281882 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f757fc0f0 con 0x559f754781c0 2024-09-21T02:45:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.772+0000 7fc85259c640 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559f754781c0 0x559f75913f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.772+0000 7fc85259c640 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559f754781c0 0x559f75913f90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2ce0a117eb99857 server_cookie=79575d0b47df8c89 in_seq=0 out_seq=0 2024-09-21T02:45:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.772+0000 7fc839ffb640 10 client.16425 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc839ffb640 1 -- 172.21.15.29:0/3983281882 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559f757fc0f0 con 0x559f754781c0 2024-09-21T02:45:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc839ffb640 10 client.16425 handle_command_reply: tid=0 2024-09-21T02:45:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 client.16425 shutdown 2024-09-21T02:45:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 -- 172.21.15.29:0/3983281882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 msgr2=0x559f75908f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75908f30 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc83c003ee0 tx=0x7fc83c012e70 comp rx=0 tx=0).stop 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 -- 172.21.15.29:0/3983281882 shutdown_connections 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559f754e41b0 0x559f75909aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559f754781c0 0x559f75913f90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559f754c34d0 0x559f75908f30 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 --2- 172.21.15.29:0/3983281882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f754bb2f0 0x559f75908870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc839ffb640 0 client.16425 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 -- 172.21.15.29:0/3983281882 >> 172.21.15.29:0/3983281882 conn(0x559f754d8300 msgr2=0x559f758825c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 -- 172.21.15.29:0/3983281882 shutdown_connections 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 1 -- 172.21.15.29:0/3983281882 wait complete. 2024-09-21T02:45:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.805+0000 7fc85fe28740 20 client.16425 trim_cache size 0 max 16384 2024-09-21T02:45:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:45:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:45:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:45:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 -- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8400b6510 msgr2=0x7fc8400b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8400b6510 0x7fc8400b8a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009460 tx=0x7fc83c005f90 comp rx=0 tx=0).stop 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 -- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 msgr2=0x7fc85419b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 0x7fc85419b090 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c090 tx=0x7fc84c0097f0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 -- 172.21.15.29:0/594397017 shutdown_connections 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8400b6510 0x7fc8400b8a20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854108300 0x7fc85419b090 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8541056e0 0x7fc85419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 --2- 172.21.15.29:0/594397017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc854102ca0 0x7fc854199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 -- 172.21.15.29:0/594397017 >> 172.21.15.29:0/594397017 conn(0x7fc8540fd690 msgr2=0x7fc854107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 -- 172.21.15.29:0/594397017 shutdown_connections 2024-09-21T02:45:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:13.806+0000 7fc858eb4640 1 -- 172.21.15.29:0/594397017 wait complete. 2024-09-21T02:45:13.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:13 smithi190 ceph-mon[33871]: pgmap v572: 129 pgs: 129 active+clean; 35 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 570 KiB/s wr, 46 op/s 2024-09-21T02:45:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:13 smithi029 ceph-mon[31006]: pgmap v572: 129 pgs: 129 active+clean; 35 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 570 KiB/s wr, 46 op/s 2024-09-21T02:45:14.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:13 smithi066 ceph-mon[33436]: pgmap v572: 129 pgs: 129 active+clean; 35 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 570 KiB/s wr, 46 op/s 2024-09-21T02:45:14.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.43 MB/sec execute 171 sec latency 6060.922 ms 2024-09-21T02:45:15.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.41 MB/sec execute 172 sec latency 7061.026 ms 2024-09-21T02:45:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:15 smithi190 ceph-mon[33871]: pgmap v573: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 572 KiB/s wr, 49 op/s 2024-09-21T02:45:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:15 smithi029 ceph-mon[31006]: pgmap v573: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 572 KiB/s wr, 49 op/s 2024-09-21T02:45:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:15 smithi066 ceph-mon[33436]: pgmap v573: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 572 KiB/s wr, 49 op/s 2024-09-21T02:45:16.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.40 MB/sec execute 173 sec latency 8061.084 ms 2024-09-21T02:45:17.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.39 MB/sec execute 174 sec latency 9061.209 ms 2024-09-21T02:45:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:17 smithi190 ceph-mon[33871]: pgmap v574: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 572 KiB/s wr, 48 op/s 2024-09-21T02:45:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:17 smithi029 ceph-mon[31006]: pgmap v574: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 572 KiB/s wr, 48 op/s 2024-09-21T02:45:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:17 smithi066 ceph-mon[33436]: pgmap v574: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 572 KiB/s wr, 48 op/s 2024-09-21T02:45:18.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.37 MB/sec execute 175 sec latency 10061.323 ms 2024-09-21T02:45:19.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.36 MB/sec execute 176 sec latency 11061.372 ms 2024-09-21T02:45:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:19 smithi190 ceph-mon[33871]: pgmap v575: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 459 KiB/s wr, 51 op/s 2024-09-21T02:45:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:19 smithi029 ceph-mon[31006]: pgmap v575: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 459 KiB/s wr, 51 op/s 2024-09-21T02:45:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:19 smithi066 ceph-mon[33436]: pgmap v575: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 459 KiB/s wr, 51 op/s 2024-09-21T02:45:20.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.35 MB/sec execute 177 sec latency 12061.484 ms 2024-09-21T02:45:21.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.33 MB/sec execute 178 sec latency 13061.599 ms 2024-09-21T02:45:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:21 smithi190 ceph-mon[33871]: pgmap v576: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 302 KiB/s wr, 35 op/s 2024-09-21T02:45:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:21 smithi029 ceph-mon[31006]: pgmap v576: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 302 KiB/s wr, 35 op/s 2024-09-21T02:45:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:21 smithi066 ceph-mon[33436]: pgmap v576: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 302 KiB/s wr, 35 op/s 2024-09-21T02:45:22.080 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.32 MB/sec execute 179 sec latency 14061.634 ms 2024-09-21T02:45:23.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.31 MB/sec execute 180 sec latency 15061.721 ms 2024-09-21T02:45:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:23 smithi190 ceph-mon[33871]: pgmap v577: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 107 KiB/s wr, 24 op/s 2024-09-21T02:45:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:23 smithi029 ceph-mon[31006]: pgmap v577: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 107 KiB/s wr, 24 op/s 2024-09-21T02:45:24.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:23 smithi066 ceph-mon[33436]: pgmap v577: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 107 KiB/s wr, 24 op/s 2024-09-21T02:45:24.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.29 MB/sec execute 181 sec latency 16061.847 ms 2024-09-21T02:45:24.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:45:24.575 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:45:25.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.28 MB/sec execute 182 sec latency 17061.881 ms 2024-09-21T02:45:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 -- 172.21.15.29:0/2652336315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa410e5c0 msgr2=0x7f9fa410e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2652336315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa410e5c0 0x7f9fa410e9d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009900 tx=0x7f9f9402a410 comp rx=0 tx=0).stop 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 -- 172.21.15.29:0/2652336315 shutdown_connections 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2652336315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa4114800 0x7f9fa411b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2652336315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9fa4113eb0 0x7f9fa41142c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2652336315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa410e5c0 0x7f9fa410e9d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 -- 172.21.15.29:0/2652336315 >> 172.21.15.29:0/2652336315 conn(0x7f9fa410a3c0 msgr2=0x7f9fa410c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 -- 172.21.15.29:0/2652336315 shutdown_connections 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.079+0000 7f9faaa62640 1 -- 172.21.15.29:0/2652336315 wait complete. 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9faaa62640 1 Processor -- start 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9faaa62640 1 -- start start 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9faaa62640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa410e5c0 0x7f9fa41a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9faaa62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa4113eb0 0x7f9fa41a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9faaa62640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9fa4114800 0x7f9fa41aa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9faaa62640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa411e380 con 0x7f9fa4113eb0 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9faaa62640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa411e200 con 0x7f9fa4114800 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9faaa62640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa411e500 con 0x7f9fa410e5c0 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9fa37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa4113eb0 0x7f9fa41a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9fa37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa4113eb0 0x7f9fa41a5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33518/0 (socket says 172.21.15.29:33518) 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.080+0000 7f9fa37fe640 1 -- 172.21.15.29:0/2220325487 learned_addr learned my addr 172.21.15.29:0/2220325487 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa8fff640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9fa4114800 0x7f9fa41aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa3fff640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa410e5c0 0x7f9fa41a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa37fe640 1 -- 172.21.15.29:0/2220325487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa410e5c0 msgr2=0x7f9fa41a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa37fe640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa410e5c0 0x7f9fa41a4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa37fe640 1 -- 172.21.15.29:0/2220325487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9fa4114800 msgr2=0x7f9fa41aa630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa37fe640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9fa4114800 0x7f9fa41aa630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa37fe640 1 -- 172.21.15.29:0/2220325487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa41aad60 con 0x7f9fa4113eb0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa8fff640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9fa4114800 0x7f9fa41aa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa37fe640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa4113eb0 0x7f9fa41a5390 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800d950 tx=0x7f9f9800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa17fa640 1 -- 172.21.15.29:0/2220325487 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98017070 con 0x7f9fa4113eb0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa17fa640 1 -- 172.21.15.29:0/2220325487 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f98009c30 con 0x7f9fa4113eb0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa41ab050 con 0x7f9fa4113eb0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9fa17fa640 1 -- 172.21.15.29:0/2220325487 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f980047a0 con 0x7f9fa4113eb0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.081+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa41b17b0 con 0x7f9fa4113eb0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.082+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa410e9d0 con 0x7f9fa4113eb0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.085+0000 7f9fa17fa640 1 -- 172.21.15.29:0/2220325487 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9f98004020 con 0x7f9fa4113eb0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.086+0000 7f9fa17fa640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f8c0b6510 0x7f9f8c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.086+0000 7f9fa3fff640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f8c0b6510 0x7f9f8c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.086+0000 7f9fa17fa640 1 -- 172.21.15.29:0/2220325487 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9f980e78f0 con 0x7f9fa4113eb0 2024-09-21T02:45:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.086+0000 7f9fa3fff640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f8c0b6510 0x7f9f8c0b8a20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9f94005f70 tx=0x7f9f94000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.089+0000 7f9fa17fa640 1 -- 172.21.15.29:0/2220325487 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9f98020020 con 0x7f9fa4113eb0 2024-09-21T02:45:25.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.315+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fa41a5f20 con 0x7f9fa4113eb0 2024-09-21T02:45:25.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.315+0000 7f9fa17fa640 1 -- 172.21.15.29:0/2220325487 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f9f980adda0 con 0x7f9fa4113eb0 2024-09-21T02:45:25.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:45:25.317 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:45:25.317 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:45:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f8c0b6510 msgr2=0x7f9f8c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f8c0b6510 0x7f9f8c0b8a20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9f94005f70 tx=0x7f9f94000bc0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa4113eb0 msgr2=0x7f9fa41a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa4113eb0 0x7f9fa41a5390 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800d950 tx=0x7f9f9800de10 comp rx=0 tx=0).stop 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 shutdown_connections 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f8c0b6510 0x7f9f8c0b8a20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9fa4114800 0x7f9fa41aa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fa4113eb0 0x7f9fa41a5390 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 --2- 172.21.15.29:0/2220325487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa410e5c0 0x7f9fa41a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 >> 172.21.15.29:0/2220325487 conn(0x7f9fa410a3c0 msgr2=0x7f9fa41135d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 shutdown_connections 2024-09-21T02:45:25.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:25.318+0000 7f9faaa62640 1 -- 172.21.15.29:0/2220325487 wait complete. 2024-09-21T02:45:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:25 smithi190 ceph-mon[33871]: pgmap v578: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-09-21T02:45:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:25 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2220325487' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:25 smithi029 ceph-mon[31006]: pgmap v578: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-09-21T02:45:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:25 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2220325487' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:25 smithi066 ceph-mon[33436]: pgmap v578: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-09-21T02:45:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:25 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2220325487' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:45:26.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106858 2.27 MB/sec execute 183 sec latency 18061.925 ms 2024-09-21T02:45:26.187 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 18c9256d --release --await' 2024-09-21T02:45:26.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 18c9256d --release --await 2024-09-21T02:45:26.450 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:45:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 -- 172.21.15.29:0/1509826851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368103fc0 msgr2=0x7f236810a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 --2- 172.21.15.29:0/1509826851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368103fc0 0x7f236810a960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2360009c90 tx=0x7f236002a860 comp rx=0 tx=0).stop 2024-09-21T02:45:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 -- 172.21.15.29:0/1509826851 shutdown_connections 2024-09-21T02:45:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 --2- 172.21.15.29:0/1509826851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368103fc0 0x7f236810a960 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 --2- 172.21.15.29:0/1509826851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2368103620 0x7f2368103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 --2- 172.21.15.29:0/1509826851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23681025a0 0x7f2368102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 -- 172.21.15.29:0/1509826851 >> 172.21.15.29:0/1509826851 conn(0x7f23680fdee0 msgr2=0x7f2368100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 -- 172.21.15.29:0/1509826851 shutdown_connections 2024-09-21T02:45:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.916+0000 7f236fb32640 1 -- 172.21.15.29:0/1509826851 wait complete. 2024-09-21T02:45:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236fb32640 1 Processor -- start 2024-09-21T02:45:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236fb32640 1 -- start start 2024-09-21T02:45:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236fb32640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23681025a0 0x7f236819c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236fb32640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2368103620 0x7f236819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236fb32640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2368103fc0 0x7f236819d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236fb32640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f236810e0b0 con 0x7f23681025a0 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236fb32640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f236810df30 con 0x7f2368103fc0 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236fb32640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f236810e230 con 0x7f2368103620 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23681025a0 0x7f236819c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23681025a0 0x7f236819c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33538/0 (socket says 172.21.15.29:33538) 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 -- 172.21.15.29:0/4121730165 learned_addr learned my addr 172.21.15.29:0/4121730165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d0cd640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2368103620 0x7f236819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236e0cf640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2368103fc0 0x7f236819d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 -- 172.21.15.29:0/4121730165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2368103620 msgr2=0x7f236819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2368103620 0x7f236819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 -- 172.21.15.29:0/4121730165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2368103fc0 msgr2=0x7f236819d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2368103fc0 0x7f236819d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 -- 172.21.15.29:0/4121730165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f236819da80 con 0x7f23681025a0 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d0cd640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2368103620 0x7f236819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.917+0000 7f236d8ce640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23681025a0 0x7f236819c0b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f235800e9f0 tx=0x7f235800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.918+0000 7f2366ffd640 1 -- 172.21.15.29:0/4121730165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235800cd80 con 0x7f23681025a0 2024-09-21T02:45:26.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.918+0000 7f2366ffd640 1 -- 172.21.15.29:0/4121730165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23580108c0 con 0x7f23681025a0 2024-09-21T02:45:26.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.918+0000 7f236fb32640 1 -- 172.21.15.29:0/4121730165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f236819dd70 con 0x7f23681025a0 2024-09-21T02:45:26.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.918+0000 7f2366ffd640 1 -- 172.21.15.29:0/4121730165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580187c0 con 0x7f23681025a0 2024-09-21T02:45:26.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.918+0000 7f236fb32640 1 -- 172.21.15.29:0/4121730165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23681a4460 con 0x7f23681025a0 2024-09-21T02:45:26.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.920+0000 7f2376a9e740 1 Processor -- start 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f2376a9e740 1 -- start start 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45231b0 0x55c1d48cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f2366ffd640 1 -- 172.21.15.29:0/4121730165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f2358010430 con 0x7f23681025a0 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 0x55c1d48cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45024d0 0x55c1d48cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f2376a9e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c1d483d6f0 con 0x55c1d45231b0 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f2376a9e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c1d482cb20 con 0x55c1d45024d0 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f2376a9e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c1d4830930 con 0x55c1d44fa2f0 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236e0cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45231b0 0x55c1d48cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236e0cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45231b0 0x55c1d48cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33550/0 (socket says 172.21.15.29:33550) 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236e0cf640 1 -- 172.21.15.29:0/3572937137 learned_addr learned my addr 172.21.15.29:0/3572937137 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236d8ce640 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 0x55c1d48cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236d0cd640 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45024d0 0x55c1d48cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236e0cf640 1 -- 172.21.15.29:0/3572937137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 msgr2=0x55c1d48cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236e0cf640 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 0x55c1d48cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236e0cf640 1 -- 172.21.15.29:0/3572937137 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45024d0 msgr2=0x55c1d48cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236e0cf640 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45024d0 0x55c1d48cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.921+0000 7f236e0cf640 1 -- 172.21.15.29:0/3572937137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1d47f2530 con 0x55c1d45231b0 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.922+0000 7f236e0cf640 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45231b0 0x55c1d48cae50 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2360009c90 tx=0x7f2360005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b53c915265bf9a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.922+0000 7f234ffff640 1 -- 172.21.15.29:0/3572937137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23600316b0 con 0x55c1d45231b0 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.922+0000 7f234ffff640 1 -- 172.21.15.29:0/3572937137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2360040070 con 0x55c1d45231b0 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.922+0000 7f234ffff640 1 -- 172.21.15.29:0/3572937137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360003e50 con 0x55c1d45231b0 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.922+0000 7f2366ffd640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23500b6510 0x7f23500b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.922+0000 7f236d8ce640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23500b6510 0x7f23500b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.922+0000 7f2366ffd640 1 -- 172.21.15.29:0/4121730165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f23580e7b50 con 0x7f23681025a0 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 -- 172.21.15.29:0/3572937137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45231b0 msgr2=0x55c1d48cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f236d8ce640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23500b6510 0x7f23500b8a20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f235800c6a0 tx=0x7f23580eb630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45231b0 0x55c1d48cae50 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2360009c90 tx=0x7f2360005dd0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 -- 172.21.15.29:0/3572937137 shutdown_connections 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45024d0 0x55c1d48cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 0x55c1d48cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 --2- 172.21.15.29:0/3572937137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45231b0 0x55c1d48cae50 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 -- 172.21.15.29:0/3572937137 >> 172.21.15.29:0/3572937137 conn(0x55c1d4517300 msgr2=0x55c1d48c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 -- 172.21.15.29:0/3572937137 shutdown_connections 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 -- 172.21.15.29:0/3572937137 wait complete. 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 -- start start 2024-09-21T02:45:26.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.923+0000 7f2376a9e740 1 Processor -- start 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48d4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45231b0 0x55c1d48d5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f2376a9e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c1d4764ab0 con 0x55c1d45024d0 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f2376a9e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c1d47d2730 con 0x55c1d44fa2f0 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f2376a9e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c1d48ccad0 con 0x55c1d45231b0 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f2376a9e740 10 client.0 _ll_register_callbacks cb 0x55c1d48266b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236d0cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48d4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236d0cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48d4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33566/0 (socket says 172.21.15.29:33566) 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236d0cd640 1 -- 172.21.15.29:0/539887577 learned_addr learned my addr 172.21.15.29:0/539887577 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236e0cf640 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45231b0 0x55c1d48d5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236e0cf640 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236d0cd640 1 -- 172.21.15.29:0/539887577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45231b0 msgr2=0x55c1d48d5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236d0cd640 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45231b0 0x55c1d48d5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236d0cd640 1 -- 172.21.15.29:0/539887577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d44fa2f0 msgr2=0x55c1d48d45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236d0cd640 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236d0cd640 1 -- 172.21.15.29:0/539887577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1d458e790 con 0x55c1d45024d0 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.924+0000 7f236e0cf640 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f236d0cd640 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48d4c90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f235c00bfc0 tx=0x7f235c011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f97edd4209f4175 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f234d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f234d7fa640 1 -- 172.21.15.29:0/539887577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c012710 con 0x55c1d45024d0 2024-09-21T02:45:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f234d7fa640 1 -- 172.21.15.29:0/539887577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f235c0128b0 con 0x55c1d45024d0 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f2376a9e740 1 -- 172.21.15.29:0/539887577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2360040070 con 0x55c1d45024d0 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f234d7fa640 1 -- 172.21.15.29:0/539887577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f235c012ab0 con 0x55c1d45024d0 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f2376a9e740 10 client.16443 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f2376a9e740 1 -- 172.21.15.29:0/539887577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c1d4607620 con 0x55c1d45024d0 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f234d7fa640 1 -- 172.21.15.29:0/539887577 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2360040070 con 0x55c1d45024d0 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.925+0000 7f234d7fa640 1 -- 172.21.15.29:0/539887577 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f235c01a3d0 con 0x55c1d45024d0 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f2376a9e740 10 client.16443 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f2376a9e740 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f2376a9e740 20 client.16443 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f2376a9e740 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c1d44e1d80 0x55c1d48e1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f2376a9e740 4 client.16443 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:45:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f2376a9e740 1 -- 172.21.15.29:0/539887577 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1d483f640 con 0x55c1d44e1d80 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f236e0cf640 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c1d44e1d80 0x55c1d48e1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f236e0cf640 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c1d44e1d80 0x55c1d48e1e40 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=836ee88d3f11b21e server_cookie=da35cdec9bf13dd4 in_seq=0 out_seq=0 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.926+0000 7f234d7fa640 10 client.16443 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f234d7fa640 1 -- 172.21.15.29:0/539887577 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c1d483f640 con 0x55c1d44e1d80 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f234d7fa640 10 client.16443 handle_command_reply: tid=0 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 client.16443 shutdown 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 -- 172.21.15.29:0/539887577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 msgr2=0x55c1d48d4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48d4c90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f235c00bfc0 tx=0x7f235c011870 comp rx=0 tx=0).stop 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 -- 172.21.15.29:0/539887577 shutdown_connections 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45231b0 0x55c1d48d5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c1d44e1d80 0x55c1d48e1e40 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48d4c90 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 --2- 172.21.15.29:0/539887577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f234d7fa640 0 client.16443 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 -- 172.21.15.29:0/539887577 >> 172.21.15.29:0/539887577 conn(0x55c1d4517300 msgr2=0x55c1d48c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 -- 172.21.15.29:0/539887577 shutdown_connections 2024-09-21T02:45:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.927+0000 7f2376a9e740 1 -- 172.21.15.29:0/539887577 wait complete. 2024-09-21T02:45:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.928+0000 7f2376a9e740 20 client.16443 trim_cache size 0 max 16384 2024-09-21T02:45:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f2376a9e740 1 Processor -- start 2024-09-21T02:45:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f2376a9e740 1 -- start start 2024-09-21T02:45:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45024d0 0x55c1d48cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 0x55c1d48cb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f2376a9e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f23681ae800 con 0x55c1d44fa2f0 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f2376a9e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f23681b0340 con 0x55c1d45231b0 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f2376a9e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f23681aeed0 con 0x55c1d45024d0 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236e0cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236e0cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33574/0 (socket says 172.21.15.29:33574) 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236e0cf640 1 -- 172.21.15.29:0/1535099410 learned_addr learned my addr 172.21.15.29:0/1535099410 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236d0cd640 1 --2- 172.21.15.29:0/1535099410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45024d0 0x55c1d48cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236e0cf640 1 -- 172.21.15.29:0/1535099410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45024d0 msgr2=0x55c1d48cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236e0cf640 1 --2- 172.21.15.29:0/1535099410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45024d0 0x55c1d48cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236e0cf640 1 -- 172.21.15.29:0/1535099410 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 msgr2=0x55c1d48cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236e0cf640 1 --2- 172.21.15.29:0/1535099410 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 0x55c1d48cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.943+0000 7f236e0cf640 1 -- 172.21.15.29:0/1535099410 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1d48dcaa0 con 0x55c1d44fa2f0 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.944+0000 7f236e0cf640 1 --2- 172.21.15.29:0/1535099410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d3230 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2360031040 tx=0x7f23600095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedb0a9541bf6745 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.944+0000 7f234f7fe640 1 -- 172.21.15.29:0/1535099410 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23600039c0 con 0x55c1d44fa2f0 2024-09-21T02:45:26.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.944+0000 7f234f7fe640 1 -- 172.21.15.29:0/1535099410 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2360040040 con 0x55c1d44fa2f0 2024-09-21T02:45:26.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.944+0000 7f234f7fe640 1 -- 172.21.15.29:0/1535099410 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360030560 con 0x55c1d44fa2f0 2024-09-21T02:45:26.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 -- 172.21.15.29:0/1535099410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d44fa2f0 msgr2=0x55c1d48d3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 --2- 172.21.15.29:0/1535099410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d3230 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2360031040 tx=0x7f23600095a0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 -- 172.21.15.29:0/1535099410 shutdown_connections 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 --2- 172.21.15.29:0/1535099410 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 0x55c1d48cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 --2- 172.21.15.29:0/1535099410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d45024d0 0x55c1d48cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 --2- 172.21.15.29:0/1535099410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d44fa2f0 0x55c1d48d3230 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 -- 172.21.15.29:0/1535099410 >> 172.21.15.29:0/1535099410 conn(0x55c1d4517300 msgr2=0x55c1d48d2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 -- 172.21.15.29:0/1535099410 shutdown_connections 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 -- 172.21.15.29:0/1535099410 wait complete. 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 -- start start 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.945+0000 7f2376a9e740 1 Processor -- start 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 0x55c1d48dc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48dad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f2376a9e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 0x55c1d48db2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f2376a9e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f23681afc70 con 0x55c1d45024d0 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f2376a9e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f23681af5a0 con 0x55c1d45231b0 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f2376a9e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f23681ac7e0 con 0x55c1d44fa2f0 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f2376a9e740 10 client.0 _ll_register_callbacks cb 0x7f235c00b4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48dad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48dad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33580/0 (socket says 172.21.15.29:33580) 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 -- 172.21.15.29:0/2206838910 learned_addr learned my addr 172.21.15.29:0/2206838910 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236e0cf640 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 0x55c1d48dc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 -- 172.21.15.29:0/2206838910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 msgr2=0x55c1d48dc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 0x55c1d48dc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 -- 172.21.15.29:0/2206838910 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 msgr2=0x55c1d48db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236e0cf640 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 0x55c1d48db2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 0x55c1d48db2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 -- 172.21.15.29:0/2206838910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1d48db940 con 0x55c1d45024d0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236e0cf640 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 0x55c1d48db2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f236d0cd640 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48dad90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f235c011e60 tx=0x7f235c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5441d9e48d8a4693 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.946+0000 7f234d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f234d7fa640 1 -- 172.21.15.29:0/2206838910 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c01a5d0 con 0x55c1d45024d0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f234d7fa640 1 -- 172.21.15.29:0/2206838910 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f235c01aab0 con 0x55c1d45024d0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 1 -- 172.21.15.29:0/2206838910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2360040040 con 0x55c1d45024d0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f234d7fa640 1 -- 172.21.15.29:0/2206838910 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f235c03a050 con 0x55c1d45024d0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 10 client.16449 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 1 -- 172.21.15.29:0/2206838910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c1d494c800 con 0x55c1d45024d0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f234d7fa640 1 -- 172.21.15.29:0/2206838910 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2360040040 con 0x55c1d45024d0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f234d7fa640 1 -- 172.21.15.29:0/2206838910 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f235c005440 con 0x55c1d45024d0 2024-09-21T02:45:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 10 client.16449 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 10 client.16449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 20 client.16449 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c1d44b71c0 0x55c1d494f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 4 client.16449 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:45:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.947+0000 7f2376a9e740 1 -- 172.21.15.29:0/2206838910 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c1d483b0f0 con 0x55c1d44b71c0 2024-09-21T02:45:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.948+0000 7f236e0cf640 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c1d44b71c0 0x55c1d494f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.948+0000 7f236e0cf640 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c1d44b71c0 0x55c1d494f0d0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb428a8bae258456 server_cookie=3ad9a42de1edbef6 in_seq=0 out_seq=0 2024-09-21T02:45:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.948+0000 7f234d7fa640 10 client.16449 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.964+0000 7f234d7fa640 1 -- 172.21.15.29:0/2206838910 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c1d483b0f0 con 0x55c1d44b71c0 2024-09-21T02:45:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.964+0000 7f234d7fa640 10 client.16449 handle_command_reply: tid=0 2024-09-21T02:45:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.964+0000 7f2376a9e740 1 client.16449 shutdown 2024-09-21T02:45:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.964+0000 7f2376a9e740 1 -- 172.21.15.29:0/2206838910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 msgr2=0x55c1d48dad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.964+0000 7f2376a9e740 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48dad90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f235c011e60 tx=0x7f235c011e90 comp rx=0 tx=0).stop 2024-09-21T02:45:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 1 -- 172.21.15.29:0/2206838910 shutdown_connections 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c1d45231b0 0x55c1d48db2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c1d44b71c0 0x55c1d494f0d0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c1d45024d0 0x55c1d48dad90 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 1 --2- 172.21.15.29:0/2206838910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c1d44fa2f0 0x55c1d48dc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 1 -- 172.21.15.29:0/2206838910 >> 172.21.15.29:0/2206838910 conn(0x55c1d4517300 msgr2=0x55c1d48cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f234d7fa640 0 client.16449 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 1 -- 172.21.15.29:0/2206838910 shutdown_connections 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 1 -- 172.21.15.29:0/2206838910 wait complete. 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f2376a9e740 20 client.16449 trim_cache size 0 max 16384 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:45:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:45:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:45:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:45:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 -- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23500b6510 msgr2=0x7f23500b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23500b6510 0x7f23500b8a20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f235800c6a0 tx=0x7f23580eb630 comp rx=0 tx=0).stop 2024-09-21T02:45:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 -- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23681025a0 msgr2=0x7f236819c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23681025a0 0x7f236819c0b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f235800e9f0 tx=0x7f235800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 -- 172.21.15.29:0/4121730165 shutdown_connections 2024-09-21T02:45:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f23500b6510 0x7f23500b8a20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2368103fc0 0x7f236819d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2368103620 0x7f236819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 --2- 172.21.15.29:0/4121730165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23681025a0 0x7f236819c0b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.965+0000 7f236fb32640 1 -- 172.21.15.29:0/4121730165 >> 172.21.15.29:0/4121730165 conn(0x7f23680fdee0 msgr2=0x7f23680ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.966+0000 7f236fb32640 1 -- 172.21.15.29:0/4121730165 shutdown_connections 2024-09-21T02:45:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:26.966+0000 7f236fb32640 1 -- 172.21.15.29:0/4121730165 wait complete. 2024-09-21T02:45:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.032+0000 7f6c9d61b640 1 -- 172.21.15.29:0/1881137773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c981084d0 msgr2=0x7f6c9810ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.032+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/1881137773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c981084d0 0x7f6c9810ed00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000adb0 tx=0x7f6c9002b7a0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 -- 172.21.15.29:0/1881137773 shutdown_connections 2024-09-21T02:45:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/1881137773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c981084d0 0x7f6c9810ed00 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/1881137773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c98107b80 0x7f6c98107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/1881137773 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c98102240 0x7f6c98102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 -- 172.21.15.29:0/1881137773 >> 172.21.15.29:0/1881137773 conn(0x7f6c98079f90 msgr2=0x7f6c9807a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 -- 172.21.15.29:0/1881137773 shutdown_connections 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 -- 172.21.15.29:0/1881137773 wait complete. 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 Processor -- start 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.033+0000 7f6c9d61b640 1 -- start start 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c9d61b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c98102240 0x7f6c9819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c9d61b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98107b80 0x7f6c9819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c9d61b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c981084d0 0x7f6c9819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c9d61b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6c981117c0 con 0x7f6c98107b80 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c9d61b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6c98111640 con 0x7f6c981084d0 2024-09-21T02:45:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c9d61b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6c98111940 con 0x7f6c98102240 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98107b80 0x7f6c9819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98107b80 0x7f6c9819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33596/0 (socket says 172.21.15.29:33596) 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 -- 172.21.15.29:0/3852792493 learned_addr learned my addr 172.21.15.29:0/3852792493 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c98102240 0x7f6c9819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c981084d0 0x7f6c9819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 -- 172.21.15.29:0/3852792493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c98102240 msgr2=0x7f6c9819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c98102240 0x7f6c9819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 -- 172.21.15.29:0/3852792493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c981084d0 msgr2=0x7f6c9819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c981084d0 0x7f6c9819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 -- 172.21.15.29:0/3852792493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c981a02f0 con 0x7f6c98107b80 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c981084d0 0x7f6c9819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.034+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98107b80 0x7f6c9819a920 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00d740 tx=0x7f6c8c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.035+0000 7f6c7ffff640 1 -- 172.21.15.29:0/3852792493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c017070 con 0x7f6c98107b80 2024-09-21T02:45:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.035+0000 7f6c9d61b640 1 -- 172.21.15.29:0/3852792493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c981a05e0 con 0x7f6c98107b80 2024-09-21T02:45:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.035+0000 7f6c7ffff640 1 -- 172.21.15.29:0/3852792493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c8c00de50 con 0x7f6c98107b80 2024-09-21T02:45:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.035+0000 7f6c7ffff640 1 -- 172.21.15.29:0/3852792493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c00f620 con 0x7f6c98107b80 2024-09-21T02:45:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.035+0000 7f6c9d61b640 1 -- 172.21.15.29:0/3852792493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c981a6d40 con 0x7f6c98107b80 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.037+0000 7f6ca3b8f740 1 Processor -- start 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.037+0000 7f6c7ffff640 1 -- 172.21.15.29:0/3852792493 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f6c8c01a070 con 0x7f6c98107b80 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.037+0000 7f6ca3b8f740 1 -- start start 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e4c520 0x56307a1f2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e23660 0x56307a1f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a1f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56307a0dddc0 con 0x563079e2b840 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56307a16a1f0 con 0x563079e23660 2024-09-21T02:45:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56307a159620 con 0x563079e4c520 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a1f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a1f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33608/0 (socket says 172.21.15.29:33608) 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c967fc640 1 -- 172.21.15.29:0/3344692155 learned_addr learned my addr 172.21.15.29:0/3344692155 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e23660 0x56307a1f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e4c520 0x56307a1f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c967fc640 1 -- 172.21.15.29:0/3344692155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e4c520 msgr2=0x56307a1f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e4c520 0x56307a1f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c967fc640 1 -- 172.21.15.29:0/3344692155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e23660 msgr2=0x56307a1f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e23660 0x56307a1f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.038+0000 7f6c967fc640 1 -- 172.21.15.29:0/3344692155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56307a121fd0 con 0x563079e2b840 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.039+0000 7f6c7ffff640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6c800b6510 0x7f6c800b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.039+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a1f2f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0eafd0 tx=0x7f6c8c0ef6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a89d5a1343dc2b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.039+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6c800b6510 0x7f6c800b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.039+0000 7f6c7ffff640 1 -- 172.21.15.29:0/3852792493 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6c8c0e7040 con 0x7f6c98107b80 2024-09-21T02:45:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.039+0000 7f6c7d7fa640 1 -- 172.21.15.29:0/3344692155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0cee60 con 0x563079e2b840 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.039+0000 7f6c7d7fa640 1 -- 172.21.15.29:0/3344692155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c8c0ac960 con 0x563079e2b840 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.039+0000 7f6c7d7fa640 1 -- 172.21.15.29:0/3344692155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0a94a0 con 0x563079e2b840 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3344692155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 msgr2=0x56307a1f2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6c800b6510 0x7f6c800b8a20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005e30 tx=0x7f6c9000a910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a1f2f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0eafd0 tx=0x7f6c8c0ef6f0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3344692155 shutdown_connections 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a1f2f90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e23660 0x56307a1f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3344692155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e4c520 0x56307a1f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3344692155 >> 172.21.15.29:0/3344692155 conn(0x563079e40910 msgr2=0x56307a1f0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3344692155 shutdown_connections 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.040+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3344692155 wait complete. 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 1 -- start start 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 1 Processor -- start 2024-09-21T02:45:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a1fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 0x56307a1fe080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e4c520 0x56307a1fe600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56307a0f4300 con 0x563079e4c520 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56307a15d430 con 0x563079e2b840 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563079fbabb0 con 0x563079e23660 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6ca3b8f740 10 client.0 _ll_register_callbacks cb 0x56307a10e120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6c977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e4c520 0x56307a1fe600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6c977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e4c520 0x56307a1fe600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33624/0 (socket says 172.21.15.29:33624) 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6c977fe640 1 -- 172.21.15.29:0/3958887202 learned_addr learned my addr 172.21.15.29:0/3958887202 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a1fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.041+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 0x56307a1fe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c977fe640 1 -- 172.21.15.29:0/3958887202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 msgr2=0x56307a1fdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a1fdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c977fe640 1 -- 172.21.15.29:0/3958887202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 msgr2=0x56307a1fe080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 0x56307a1fe080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c977fe640 1 -- 172.21.15.29:0/3958887202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56307a0e3780 con 0x563079e4c520 2024-09-21T02:45:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e4c520 0x56307a1fe600 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6c90032840 tx=0x7f6c9000a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03497e238f04571 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c7aff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3958887202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90004450 con 0x563079e4c520 2024-09-21T02:45:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3958887202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6c900045f0 con 0x563079e4c520 2024-09-21T02:45:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3958887202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c8c0ac960 con 0x563079e4c520 2024-09-21T02:45:27.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3958887202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c900047f0 con 0x563079e4c520 2024-09-21T02:45:27.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6ca3b8f740 10 client.16461 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3958887202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563079eb7b00 con 0x563079e4c520 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.042+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3958887202 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c8c0ac960 con 0x563079e4c520 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.043+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3958887202 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6c90050070 con 0x563079e4c520 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.043+0000 7f6ca3b8f740 10 client.16461 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.043+0000 7f6ca3b8f740 10 client.16461 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.043+0000 7f6ca3b8f740 20 client.16461 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.043+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563079e0aea0 0x56307a214300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.043+0000 7f6ca3b8f740 4 client.16461 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.043+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3958887202 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56307a16c140 con 0x563079e0aea0 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.043+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563079e0aea0 0x56307a214300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.044+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563079e0aea0 0x56307a214300 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b24d1041799afb84 server_cookie=c6dc6d9547ce488a in_seq=0 out_seq=0 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.044+0000 7f6c7aff5640 10 client.16461 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.044+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3958887202 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56307a16c140 con 0x563079e0aea0 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.044+0000 7f6c7aff5640 10 client.16461 handle_command_reply: tid=0 2024-09-21T02:45:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 client.16461 shutdown 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3958887202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e4c520 msgr2=0x56307a1fe600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e4c520 0x56307a1fe600 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6c90032840 tx=0x7f6c9000a6d0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3958887202 shutdown_connections 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563079e0aea0 0x56307a214300 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e4c520 0x56307a1fe600 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 0x56307a1fe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6c7aff5640 0 client.16461 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3958887202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a1fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3958887202 >> 172.21.15.29:0/3958887202 conn(0x563079e40910 msgr2=0x56307a1f56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3958887202 shutdown_connections 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3958887202 wait complete. 2024-09-21T02:45:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.045+0000 7f6ca3b8f740 20 client.16461 trim_cache size 0 max 16384 2024-09-21T02:45:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.062+0000 7f6ca3b8f740 1 Processor -- start 2024-09-21T02:45:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.062+0000 7f6ca3b8f740 1 -- start start 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e0aea0 0x56307a1ffb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a219aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a219fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6c981ae850 con 0x563079e2b840 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6c981b0390 con 0x563079e0aea0 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6c981aef20 con 0x563079e23660 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a219fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a219fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33640/0 (socket says 172.21.15.29:33640) 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 -- 172.21.15.29:0/1527544711 learned_addr learned my addr 172.21.15.29:0/1527544711 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c967fc640 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a219aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e0aea0 0x56307a1ffb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 -- 172.21.15.29:0/1527544711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 msgr2=0x56307a219aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a219aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 -- 172.21.15.29:0/1527544711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e0aea0 msgr2=0x56307a1ffb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e0aea0 0x56307a1ffb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 -- 172.21.15.29:0/1527544711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563079f2e960 con 0x563079e2b840 2024-09-21T02:45:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e0aea0 0x56307a1ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:45:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.063+0000 7f6c977fe640 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a219fe0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000ab70 tx=0x7f6c90005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfe0d994269a2a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6c7cff9640 1 -- 172.21.15.29:0/1527544711 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90004450 con 0x563079e2b840 2024-09-21T02:45:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6c7cff9640 1 -- 172.21.15.29:0/1527544711 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c90032400 con 0x563079e2b840 2024-09-21T02:45:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6c7cff9640 1 -- 172.21.15.29:0/1527544711 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90047830 con 0x563079e2b840 2024-09-21T02:45:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/1527544711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 msgr2=0x56307a219fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a219fe0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000ab70 tx=0x7f6c90005dc0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/1527544711 shutdown_connections 2024-09-21T02:45:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e2b840 0x56307a219fe0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a219aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/1527544711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e0aea0 0x56307a1ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.064+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/1527544711 >> 172.21.15.29:0/1527544711 conn(0x563079e40910 msgr2=0x56307a1f3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.065+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/1527544711 shutdown_connections 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.065+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/1527544711 wait complete. 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.065+0000 7f6ca3b8f740 1 -- start start 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.065+0000 7f6ca3b8f740 1 Processor -- start 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e0aea0 0x56307a2027c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a202d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6ca3b8f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 0x56307a277d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6c981afcc0 con 0x563079e0aea0 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6c981af5f0 con 0x563079e2b840 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6ca3b8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6c981ac830 con 0x563079e23660 2024-09-21T02:45:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6ca3b8f740 10 client.0 _ll_register_callbacks cb 0x7f6c9000aaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e0aea0 0x56307a2027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e0aea0 0x56307a2027c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33644/0 (socket says 172.21.15.29:33644) 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 -- 172.21.15.29:0/3404564509 learned_addr learned my addr 172.21.15.29:0/3404564509 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a202d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c977fe640 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 0x56307a277d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 -- 172.21.15.29:0/3404564509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 msgr2=0x56307a202d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a202d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 -- 172.21.15.29:0/3404564509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 msgr2=0x56307a277d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 0x56307a277d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 -- 172.21.15.29:0/3404564509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56307a2783c0 con 0x563079e0aea0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c96ffd640 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e0aea0 0x56307a2027c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6c88005ca0 tx=0x7f6c880119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ebb4d379aa90fe6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c7aff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3404564509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c88004020 con 0x563079e0aea0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3404564509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6c880041c0 con 0x563079e0aea0 2024-09-21T02:45:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.066+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3404564509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56307a167bf0 con 0x563079e0aea0 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3404564509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c880043c0 con 0x563079e0aea0 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6ca3b8f740 10 client.16467 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3404564509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56307a27f6b0 con 0x563079e0aea0 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3404564509 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56307a167bf0 con 0x563079e0aea0 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3404564509 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6c88004c60 con 0x563079e0aea0 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6ca3b8f740 10 client.16467 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6ca3b8f740 10 client.16467 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6ca3b8f740 20 client.16467 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563079e4c520 0x56307a281ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6ca3b8f740 4 client.16467 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "18c9256d", "release": true, "await": true}] 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3404564509 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "18c9256d", "release": true, "await": true}) -- 0x56307a165020 con 0x563079e4c520 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.067+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563079e4c520 0x56307a281ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.068+0000 7f6c967fc640 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563079e4c520 0x56307a281ef0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50b03ebd830e74a4 server_cookie=dab515e2e28eed72 in_seq=0 out_seq=0 2024-09-21T02:45:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.068+0000 7f6c7aff5640 10 client.16467 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6c7aff5640 1 -- 172.21.15.29:0/3404564509 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56307a165020 con 0x563079e4c520 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6c7aff5640 10 client.16467 handle_command_reply: tid=0 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 client.16467 shutdown 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3404564509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e0aea0 msgr2=0x56307a2027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e0aea0 0x56307a2027c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6c88005ca0 tx=0x7f6c880119a0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3404564509 shutdown_connections 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563079e2b840 0x56307a277d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563079e23660 0x56307a202d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563079e4c520 0x56307a281ef0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 --2- 172.21.15.29:0/3404564509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563079e0aea0 0x56307a2027c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3404564509 >> 172.21.15.29:0/3404564509 conn(0x563079e40910 msgr2=0x56307a1fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6c7aff5640 0 client.16467 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3404564509 shutdown_connections 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 1 -- 172.21.15.29:0/3404564509 wait complete. 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.077+0000 7f6ca3b8f740 20 client.16467 trim_cache size 0 max 16384 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 28, 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout: "18c9256d": { 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 28, 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:45:27.080 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.2, 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:27.081 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:27.082 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:45:27.082 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:45:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 -- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6c800b6510 msgr2=0x7f6c800b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6c800b6510 0x7f6c800b8a20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005e30 tx=0x7f6c9000a910 comp rx=0 tx=0).stop 2024-09-21T02:45:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 -- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98107b80 msgr2=0x7f6c9819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98107b80 0x7f6c9819a920 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00d740 tx=0x7f6c8c00dc00 comp rx=0 tx=0).stop 2024-09-21T02:45:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 -- 172.21.15.29:0/3852792493 shutdown_connections 2024-09-21T02:45:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6c800b6510 0x7f6c800b8a20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c981084d0 0x7f6c9819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98107b80 0x7f6c9819a920 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 --2- 172.21.15.29:0/3852792493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c98102240 0x7f6c9819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 -- 172.21.15.29:0/3852792493 >> 172.21.15.29:0/3852792493 conn(0x7f6c98079f90 msgr2=0x7f6c98105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 -- 172.21.15.29:0/3852792493 shutdown_connections 2024-09-21T02:45:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:45:27.078+0000 7f6c9d61b640 1 -- 172.21.15.29:0/3852792493 wait complete. 2024-09-21T02:45:27.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106860 2.26 MB/sec execute 184 sec latency 19061.579 ms 2024-09-21T02:45:27.835 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:27 smithi029 ceph-mon[31006]: pgmap v579: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-09-21T02:45:27.866 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '18c9256d', total seconds elapsed: 22.8 2024-09-21T02:45:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:27 smithi190 ceph-mon[33871]: pgmap v579: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-09-21T02:45:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:27 smithi066 ceph-mon[33436]: pgmap v579: 129 pgs: 129 active+clean; 35 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-09-21T02:45:28.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 107322 2.25 MB/sec execute 185 sec latency 16.732 ms 2024-09-21T02:45:29.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 107957 2.26 MB/sec execute 186 sec latency 16.988 ms 2024-09-21T02:45:29.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:29 smithi190 ceph-mon[33871]: pgmap v580: 129 pgs: 129 active+clean; 35 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 30 KiB/s wr, 23 op/s 2024-09-21T02:45:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:29 smithi029 ceph-mon[31006]: pgmap v580: 129 pgs: 129 active+clean; 35 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 30 KiB/s wr, 23 op/s 2024-09-21T02:45:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:29 smithi066 ceph-mon[33436]: pgmap v580: 129 pgs: 129 active+clean; 35 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 30 KiB/s wr, 23 op/s 2024-09-21T02:45:30.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 108593 2.25 MB/sec execute 187 sec latency 20.124 ms 2024-09-21T02:45:31.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 109372 2.28 MB/sec execute 188 sec latency 20.844 ms 2024-09-21T02:45:31.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:31 smithi190 ceph-mon[33871]: pgmap v581: 129 pgs: 129 active+clean; 35 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 112 KiB/s wr, 30 op/s 2024-09-21T02:45:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:31 smithi029 ceph-mon[31006]: pgmap v581: 129 pgs: 129 active+clean; 35 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 112 KiB/s wr, 30 op/s 2024-09-21T02:45:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:31 smithi066 ceph-mon[33436]: pgmap v581: 129 pgs: 129 active+clean; 35 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 112 KiB/s wr, 30 op/s 2024-09-21T02:45:32.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 109978 2.29 MB/sec execute 189 sec latency 18.480 ms 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:45:32.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:45:32.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:45:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:45:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:45:32.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:45:33.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:45:33.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:45:33.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:45:33.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 110371 2.29 MB/sec execute 190 sec latency 20.192 ms 2024-09-21T02:45:33.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:33 smithi190 ceph-mon[33871]: pgmap v582: 129 pgs: 129 active+clean; 34 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 290 KiB/s wr, 35 op/s 2024-09-21T02:45:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:33 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi066 to 176.0M 2024-09-21T02:45:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:33 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi190 to 176.0M 2024-09-21T02:45:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:33 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi029 to 177.0M 2024-09-21T02:45:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:33 smithi190 ceph-mon[33871]: Unable to set osd_memory_target on smithi066 to 184555212: error parsing value: Value '184555212' is below minimum 939524096 2024-09-21T02:45:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:33 smithi190 ceph-mon[33871]: Unable to set osd_memory_target on smithi190 to 184583884: error parsing value: Value '184583884' is below minimum 939524096 2024-09-21T02:45:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:33 smithi190 ceph-mon[33871]: Unable to set osd_memory_target on smithi029 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T02:45:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:33 smithi029 ceph-mon[31006]: pgmap v582: 129 pgs: 129 active+clean; 34 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 290 KiB/s wr, 35 op/s 2024-09-21T02:45:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:33 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi066 to 176.0M 2024-09-21T02:45:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:33 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi190 to 176.0M 2024-09-21T02:45:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:33 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi029 to 177.0M 2024-09-21T02:45:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:33 smithi029 ceph-mon[31006]: Unable to set osd_memory_target on smithi066 to 184555212: error parsing value: Value '184555212' is below minimum 939524096 2024-09-21T02:45:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:33 smithi029 ceph-mon[31006]: Unable to set osd_memory_target on smithi190 to 184583884: error parsing value: Value '184583884' is below minimum 939524096 2024-09-21T02:45:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:33 smithi029 ceph-mon[31006]: Unable to set osd_memory_target on smithi029 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T02:45:34.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:33 smithi066 ceph-mon[33436]: pgmap v582: 129 pgs: 129 active+clean; 34 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 290 KiB/s wr, 35 op/s 2024-09-21T02:45:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:33 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi066 to 176.0M 2024-09-21T02:45:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:33 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi190 to 176.0M 2024-09-21T02:45:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:33 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi029 to 177.0M 2024-09-21T02:45:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:33 smithi066 ceph-mon[33436]: Unable to set osd_memory_target on smithi066 to 184555212: error parsing value: Value '184555212' is below minimum 939524096 2024-09-21T02:45:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:33 smithi066 ceph-mon[33436]: Unable to set osd_memory_target on smithi190 to 184583884: error parsing value: Value '184583884' is below minimum 939524096 2024-09-21T02:45:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:33 smithi066 ceph-mon[33436]: Unable to set osd_memory_target on smithi029 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T02:45:34.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 110756 2.28 MB/sec execute 191 sec latency 18.133 ms 2024-09-21T02:45:35.081 INFO:tasks.workunit.client.0.smithi029.stdout: 1 111424 2.29 MB/sec execute 192 sec latency 15.158 ms 2024-09-21T02:45:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:35 smithi029 ceph-mon[31006]: pgmap v583: 129 pgs: 129 active+clean; 35 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 452 KiB/s wr, 49 op/s 2024-09-21T02:45:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:35 smithi066 ceph-mon[33436]: pgmap v583: 129 pgs: 129 active+clean; 35 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 452 KiB/s wr, 49 op/s 2024-09-21T02:45:36.082 INFO:tasks.workunit.client.0.smithi029.stdout: 1 112056 2.28 MB/sec execute 193 sec latency 16.135 ms 2024-09-21T02:45:36.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:35 smithi190 ceph-mon[33871]: pgmap v583: 129 pgs: 129 active+clean; 35 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 452 KiB/s wr, 49 op/s 2024-09-21T02:45:37.082 INFO:tasks.workunit.client.0.smithi029.stdout: 1 112620 2.28 MB/sec execute 194 sec latency 25.930 ms 2024-09-21T02:45:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:37 smithi029 ceph-mon[31006]: pgmap v584: 129 pgs: 129 active+clean; 37 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 891 KiB/s wr, 79 op/s 2024-09-21T02:45:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:37 smithi066 ceph-mon[33436]: pgmap v584: 129 pgs: 129 active+clean; 37 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 891 KiB/s wr, 79 op/s 2024-09-21T02:45:38.082 INFO:tasks.workunit.client.0.smithi029.stdout: 1 113338 2.30 MB/sec execute 195 sec latency 17.436 ms 2024-09-21T02:45:38.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:37 smithi190 ceph-mon[33871]: pgmap v584: 129 pgs: 129 active+clean; 37 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 891 KiB/s wr, 79 op/s 2024-09-21T02:45:39.082 INFO:tasks.workunit.client.0.smithi029.stdout: 1 113862 2.31 MB/sec execute 196 sec latency 15.039 ms 2024-09-21T02:45:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:39 smithi029 ceph-mon[31006]: pgmap v585: 129 pgs: 129 active+clean; 36 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 965 KiB/s wr, 85 op/s 2024-09-21T02:45:40.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:39 smithi066 ceph-mon[33436]: pgmap v585: 129 pgs: 129 active+clean; 36 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 965 KiB/s wr, 85 op/s 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 -- 172.21.15.29:0/2840070017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 msgr2=0x7f70d810a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 --2- 172.21.15.29:0/2840070017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 0x7f70d810a960 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009c90 tx=0x7f70d002a860 comp rx=0 tx=0).stop 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 -- 172.21.15.29:0/2840070017 shutdown_connections 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 --2- 172.21.15.29:0/2840070017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 0x7f70d810a960 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 --2- 172.21.15.29:0/2840070017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d8103620 0x7f70d8103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 --2- 172.21.15.29:0/2840070017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70d81025a0 0x7f70d8102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 -- 172.21.15.29:0/2840070017 >> 172.21.15.29:0/2840070017 conn(0x7f70d80fdee0 msgr2=0x7f70d8100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 -- 172.21.15.29:0/2840070017 shutdown_connections 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.068+0000 7f70de09b640 1 -- 172.21.15.29:0/2840070017 wait complete. 2024-09-21T02:45:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70de09b640 1 Processor -- start 2024-09-21T02:45:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70de09b640 1 -- start start 2024-09-21T02:45:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70de09b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d81025a0 0x7f70d8199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70de09b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70d8103620 0x7f70d819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70de09b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 0x7f70d819b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70de09b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70d810e0b0 con 0x7f70d8103fc0 2024-09-21T02:45:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70de09b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f70d810df30 con 0x7f70d8103620 2024-09-21T02:45:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70de09b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f70d810e230 con 0x7f70d81025a0 2024-09-21T02:45:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 0x7f70d819b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d81025a0 0x7f70d8199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70ceffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70d8103620 0x7f70d819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d81025a0 0x7f70d8199e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32880/0 (socket says 172.21.15.29:32880) 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 0x7f70d819b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52942/0 (socket says 172.21.15.29:52942) 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.069+0000 7f70d77fe640 1 -- 172.21.15.29:0/3689605749 learned_addr learned my addr 172.21.15.29:0/3689605749 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.070+0000 7f70d77fe640 1 -- 172.21.15.29:0/3689605749 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70d8103620 msgr2=0x7f70d819a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.070+0000 7f70d77fe640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70d8103620 0x7f70d819a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.070+0000 7f70d77fe640 1 -- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 msgr2=0x7f70d819b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.070+0000 7f70d77fe640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 0x7f70d819b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.070+0000 7f70d77fe640 1 -- 172.21.15.29:0/3689605749 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d819b860 con 0x7f70d81025a0 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.070+0000 7f70d77fe640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d81025a0 0x7f70d8199e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f70c800e9f0 tx=0x7f70c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.070+0000 7f70d57fa640 1 -- 172.21.15.29:0/3689605749 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c800cd80 con 0x7f70d81025a0 2024-09-21T02:45:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.071+0000 7f70de09b640 1 -- 172.21.15.29:0/3689605749 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d819bb50 con 0x7f70d81025a0 2024-09-21T02:45:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.071+0000 7f70de09b640 1 -- 172.21.15.29:0/3689605749 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d8104af0 con 0x7f70d81025a0 2024-09-21T02:45:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.071+0000 7f70d57fa640 1 -- 172.21.15.29:0/3689605749 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70c80108c0 con 0x7f70d81025a0 2024-09-21T02:45:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.071+0000 7f70d57fa640 1 -- 172.21.15.29:0/3689605749 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c80187c0 con 0x7f70d81025a0 2024-09-21T02:45:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.075+0000 7f70e500c740 1 Processor -- start 2024-09-21T02:45:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.075+0000 7f70d57fa640 1 -- 172.21.15.29:0/3689605749 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f70c8010430 con 0x7f70d81025a0 2024-09-21T02:45:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.075+0000 7f70e500c740 1 -- start start 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70e500c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550cbb1b0 0x55f55106b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70e500c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f55106bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70e500c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550c9a4d0 0x55f55106c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70e500c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f550fd56f0 con 0x55f550c9a4d0 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70e500c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f550fc4b20 con 0x55f550c922f0 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70e500c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f550fc8930 con 0x55f550cbb1b0 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70ceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550c9a4d0 0x55f55106c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70ceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550c9a4d0 0x55f55106c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52956/0 (socket says 172.21.15.29:52956) 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70ceffd640 1 -- 172.21.15.29:0/3219541310 learned_addr learned my addr 172.21.15.29:0/3219541310 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70d7fff640 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550cbb1b0 0x55f55106b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70d77fe640 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f55106bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70ceffd640 1 -- 172.21.15.29:0/3219541310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550cbb1b0 msgr2=0x55f55106b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70ceffd640 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550cbb1b0 0x55f55106b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.076+0000 7f70ceffd640 1 -- 172.21.15.29:0/3219541310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 msgr2=0x55f55106bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.077+0000 7f70d57fa640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70c40b6590 0x7f70c40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.077+0000 7f70d57fa640 1 -- 172.21.15.29:0/3689605749 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f70c80e79d0 con 0x7f70d81025a0 2024-09-21T02:45:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.077+0000 7f70ceffd640 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f55106bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.077+0000 7f70d7fff640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70c40b6590 0x7f70c40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.077+0000 7f70ceffd640 1 -- 172.21.15.29:0/3219541310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f550f8a530 con 0x55f550c9a4d0 2024-09-21T02:45:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.078+0000 7f70d77fe640 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f55106bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:45:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.078+0000 7f70ceffd640 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550c9a4d0 0x55f55106c820 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009d20 tx=0x7f70c0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33c257756926435 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.078+0000 7f70d7fff640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70c40b6590 0x7f70c40b8aa0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009c60 tx=0x7f70d003f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.079+0000 7f70cdffb640 1 -- 172.21.15.29:0/3219541310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c001e070 con 0x55f550c9a4d0 2024-09-21T02:45:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.080+0000 7f70cdffb640 1 -- 172.21.15.29:0/3219541310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70c0019ac0 con 0x55f550c9a4d0 2024-09-21T02:45:40.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.080+0000 7f70cdffb640 1 -- 172.21.15.29:0/3219541310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c000ea00 con 0x55f550c9a4d0 2024-09-21T02:45:40.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.081+0000 7f70e500c740 1 -- 172.21.15.29:0/3219541310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550c9a4d0 msgr2=0x55f55106c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.081+0000 7f70e500c740 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550c9a4d0 0x55f55106c820 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009d20 tx=0x7f70c0011810 comp rx=0 tx=0).stop 2024-09-21T02:45:40.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 114194 2.30 MB/sec execute 197 sec latency 17.594 ms 2024-09-21T02:45:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.081+0000 7f70e500c740 1 -- 172.21.15.29:0/3219541310 shutdown_connections 2024-09-21T02:45:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.081+0000 7f70e500c740 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550c9a4d0 0x55f55106c820 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.081+0000 7f70e500c740 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f55106bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.081+0000 7f70e500c740 1 --2- 172.21.15.29:0/3219541310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550cbb1b0 0x55f55106b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.081+0000 7f70e500c740 1 -- 172.21.15.29:0/3219541310 >> 172.21.15.29:0/3219541310 conn(0x55f550caf300 msgr2=0x55f5510670f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.081+0000 7f70e500c740 1 -- 172.21.15.29:0/3219541310 shutdown_connections 2024-09-21T02:45:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.082+0000 7f70e500c740 1 -- 172.21.15.29:0/3219541310 wait complete. 2024-09-21T02:45:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.082+0000 7f70e500c740 1 -- start start 2024-09-21T02:45:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.082+0000 7f70e500c740 1 Processor -- start 2024-09-21T02:45:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70e500c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551074e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70e500c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f5510754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70e500c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551076060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70e500c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f550efcab0 con 0x55f550cbb1b0 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70e500c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f550f6a730 con 0x55f550c9a4d0 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70e500c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f55106d340 con 0x55f550c922f0 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70e500c740 10 client.0 _ll_register_callbacks cb 0x55f550fbe6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32904/0 (socket says 172.21.15.29:32904) 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551076060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70d77fe640 1 -- 172.21.15.29:0/1575577918 learned_addr learned my addr 172.21.15.29:0/1575577918 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70ceffd640 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f5510754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70ceffd640 1 -- 172.21.15.29:0/1575577918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 msgr2=0x55f551074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70ceffd640 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70ceffd640 1 -- 172.21.15.29:0/1575577918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 msgr2=0x55f551076060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.083+0000 7f70ceffd640 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551076060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70ceffd640 1 -- 172.21.15.29:0/1575577918 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f550d26790 con 0x55f550c9a4d0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70ceffd640 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f5510754f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f70c0005890 tx=0x7f70c0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dc3754f1f87f7aa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1575577918 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c001e050 con 0x55f550c9a4d0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70e500c740 1 -- 172.21.15.29:0/1575577918 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70c0019ac0 con 0x55f550c9a4d0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1575577918 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f70c0019630 con 0x55f550c9a4d0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1575577918 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f70c0033050 con 0x55f550c9a4d0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70e500c740 10 client.25970 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.084+0000 7f70e500c740 1 -- 172.21.15.29:0/1575577918 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f550d9f620 con 0x55f550c9a4d0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1575577918 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70c0019ac0 con 0x55f550c9a4d0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1575577918 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f70c0003680 con 0x55f550c9a4d0 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70e500c740 10 client.25970 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70e500c740 10 client.25970 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70e500c740 20 client.25970 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70e500c740 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f550c79d80 0x55f551082320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70e500c740 4 client.25970 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:45:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70e500c740 1 -- 172.21.15.29:0/1575577918 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f550fd7640 con 0x55f550c79d80 2024-09-21T02:45:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.085+0000 7f70d77fe640 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f550c79d80 0x55f551082320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.086+0000 7f70b6ffd640 10 client.25970 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.086+0000 7f70d77fe640 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f550c79d80 0x55f551082320 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a800b02cea72bb5 server_cookie=5c51184faee35c6b in_seq=0 out_seq=0 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1575577918 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f550fd7640 con 0x55f550c79d80 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70b6ffd640 10 client.25970 handle_command_reply: tid=0 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 client.25970 shutdown 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 -- 172.21.15.29:0/1575577918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 msgr2=0x55f5510754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f5510754f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f70c0005890 tx=0x7f70c0004270 comp rx=0 tx=0).stop 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 -- 172.21.15.29:0/1575577918 shutdown_connections 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f550c79d80 0x55f551082320 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70b6ffd640 0 client.25970 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f5510754f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 --2- 172.21.15.29:0/1575577918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 -- 172.21.15.29:0/1575577918 >> 172.21.15.29:0/1575577918 conn(0x55f550caf300 msgr2=0x55f55105cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 -- 172.21.15.29:0/1575577918 shutdown_connections 2024-09-21T02:45:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 1 -- 172.21.15.29:0/1575577918 wait complete. 2024-09-21T02:45:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.087+0000 7f70e500c740 20 client.25970 trim_cache size 0 max 16384 2024-09-21T02:45:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.102+0000 7f70e500c740 1 Processor -- start 2024-09-21T02:45:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.102+0000 7f70e500c740 1 -- start start 2024-09-21T02:45:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70e500c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551075540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70e500c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f55107cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70e500c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551067530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70e500c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800fb50 con 0x55f550cbb1b0 2024-09-21T02:45:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70e500c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f70b8011690 con 0x55f550c9a4d0 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70e500c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f70b8010220 con 0x55f550c922f0 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551067530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551075540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551067530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52984/0 (socket says 172.21.15.29:52984) 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551075540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32906/0 (socket says 172.21.15.29:32906) 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70ceffd640 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f55107cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 -- 172.21.15.29:0/229681270 learned_addr learned my addr 172.21.15.29:0/229681270 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 -- 172.21.15.29:0/229681270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 msgr2=0x55f551075540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551075540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 -- 172.21.15.29:0/229681270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 msgr2=0x55f55107cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f55107cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 -- 172.21.15.29:0/229681270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f551059d40 con 0x55f550cbb1b0 2024-09-21T02:45:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d77fe640 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551075540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70ceffd640 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f55107cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.103+0000 7f70d7fff640 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551067530 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009c90 tx=0x7f70d003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb5e33cbfdcfa45 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.104+0000 7f70cd7fa640 1 -- 172.21.15.29:0/229681270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0044070 con 0x55f550cbb1b0 2024-09-21T02:45:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.104+0000 7f70cd7fa640 1 -- 172.21.15.29:0/229681270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70d0036040 con 0x55f550cbb1b0 2024-09-21T02:45:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.104+0000 7f70cd7fa640 1 -- 172.21.15.29:0/229681270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d003ca50 con 0x55f550cbb1b0 2024-09-21T02:45:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.104+0000 7f70e500c740 1 -- 172.21.15.29:0/229681270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 msgr2=0x55f551067530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.104+0000 7f70e500c740 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551067530 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009c90 tx=0x7f70d003e000 comp rx=0 tx=0).stop 2024-09-21T02:45:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.104+0000 7f70e500c740 1 -- 172.21.15.29:0/229681270 shutdown_connections 2024-09-21T02:45:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.104+0000 7f70e500c740 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f551067530 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c9a4d0 0x55f55107cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 --2- 172.21.15.29:0/229681270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c922f0 0x55f551075540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 -- 172.21.15.29:0/229681270 >> 172.21.15.29:0/229681270 conn(0x55f550caf300 msgr2=0x55f551074e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 -- 172.21.15.29:0/229681270 shutdown_connections 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 -- 172.21.15.29:0/229681270 wait complete. 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 -- start start 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 Processor -- start 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f551079f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c9a4d0 0x55f55107a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f55107b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70b8010fc0 con 0x55f550cbb1b0 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f70b80108f0 con 0x55f550c922f0 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800db30 con 0x55f550c9a4d0 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.105+0000 7f70e500c740 10 client.0 _ll_register_callbacks cb 0x7f70c001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f55107b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70ceffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c9a4d0 0x55f55107a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f55107b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52986/0 (socket says 172.21.15.29:52986) 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70d7fff640 1 -- 172.21.15.29:0/2716371608 learned_addr learned my addr 172.21.15.29:0/2716371608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70ceffd640 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c9a4d0 0x55f55107a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32908/0 (socket says 172.21.15.29:32908) 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70d77fe640 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f551079f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70ceffd640 1 -- 172.21.15.29:0/2716371608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 msgr2=0x55f551079f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70ceffd640 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f551079f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70ceffd640 1 -- 172.21.15.29:0/2716371608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 msgr2=0x55f55107b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70ceffd640 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f55107b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70ceffd640 1 -- 172.21.15.29:0/2716371608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f55107b8b0 con 0x55f550c9a4d0 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70d7fff640 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f55107b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70ceffd640 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c9a4d0 0x55f55107a5e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f70c0011d10 tx=0x7f70c0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bd942ce79fdbbe1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2716371608 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c003b070 con 0x55f550c9a4d0 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70e500c740 1 -- 172.21.15.29:0/2716371608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70d0036040 con 0x55f550c9a4d0 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.106+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2716371608 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f70c0019430 con 0x55f550c9a4d0 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2716371608 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f70c0047050 con 0x55f550c9a4d0 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70e500c740 10 client.26080 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70e500c740 1 -- 172.21.15.29:0/2716371608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f55107cba0 con 0x55f550c9a4d0 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2716371608 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70d0036040 con 0x55f550c9a4d0 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2716371608 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f70c0025450 con 0x55f550c9a4d0 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70e500c740 10 client.26080 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70e500c740 10 client.26080 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70e500c740 20 client.26080 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70e500c740 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f550c4f1c0 0x55f5510eee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70e500c740 4 client.26080 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:45:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70e500c740 1 -- 172.21.15.29:0/2716371608 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f550fd30f0 con 0x55f550c4f1c0 2024-09-21T02:45:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.107+0000 7f70d77fe640 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f550c4f1c0 0x55f5510eee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.108+0000 7f70b6ffd640 10 client.26080 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.108+0000 7f70d77fe640 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f550c4f1c0 0x55f5510eee20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52d75426224f208e server_cookie=257400592fb64e71 in_seq=0 out_seq=0 2024-09-21T02:45:40.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:39 smithi190 ceph-mon[33871]: pgmap v585: 129 pgs: 129 active+clean; 36 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 965 KiB/s wr, 85 op/s 2024-09-21T02:45:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.146+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2716371608 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f550fd30f0 con 0x55f550c4f1c0 2024-09-21T02:45:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.146+0000 7f70b6ffd640 10 client.26080 handle_command_reply: tid=0 2024-09-21T02:45:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.146+0000 7f70e500c740 1 client.26080 shutdown 2024-09-21T02:45:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.146+0000 7f70e500c740 1 -- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c9a4d0 msgr2=0x55f55107a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.146+0000 7f70e500c740 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c9a4d0 0x55f55107a5e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f70c0011d10 tx=0x7f70c0009c00 comp rx=0 tx=0).stop 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 1 -- 172.21.15.29:0/2716371608 shutdown_connections 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f550c4f1c0 0x55f5510eee20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f550cbb1b0 0x55f55107b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f550c9a4d0 0x55f55107a5e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 1 --2- 172.21.15.29:0/2716371608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f550c922f0 0x55f551079f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70b6ffd640 0 client.26080 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 1 -- 172.21.15.29:0/2716371608 >> 172.21.15.29:0/2716371608 conn(0x55f550caf300 msgr2=0x55f55106f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 1 -- 172.21.15.29:0/2716371608 shutdown_connections 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 1 -- 172.21.15.29:0/2716371608 wait complete. 2024-09-21T02:45:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70e500c740 20 client.26080 trim_cache size 0 max 16384 2024-09-21T02:45:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:45:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:45:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:45:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70de09b640 1 -- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70c40b6590 msgr2=0x7f70c40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70de09b640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70c40b6590 0x7f70c40b8aa0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009c60 tx=0x7f70d003f040 comp rx=0 tx=0).stop 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70de09b640 1 -- 172.21.15.29:0/3689605749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d81025a0 msgr2=0x7f70d8199e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.147+0000 7f70de09b640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d81025a0 0x7f70d8199e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f70c800e9f0 tx=0x7f70c800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.148+0000 7f70de09b640 1 -- 172.21.15.29:0/3689605749 shutdown_connections 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.148+0000 7f70de09b640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70c40b6590 0x7f70c40b8aa0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.148+0000 7f70de09b640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8103fc0 0x7f70d819b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.148+0000 7f70de09b640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70d8103620 0x7f70d819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.148+0000 7f70de09b640 1 --2- 172.21.15.29:0/3689605749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d81025a0 0x7f70d8199e90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:40.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.148+0000 7f70de09b640 1 -- 172.21.15.29:0/3689605749 >> 172.21.15.29:0/3689605749 conn(0x7f70d80fdee0 msgr2=0x7f70d80ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:40.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.148+0000 7f70de09b640 1 -- 172.21.15.29:0/3689605749 shutdown_connections 2024-09-21T02:45:40.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:40.148+0000 7f70de09b640 1 -- 172.21.15.29:0/3689605749 wait complete. 2024-09-21T02:45:41.082 INFO:tasks.workunit.client.0.smithi029.stdout: 1 114795 2.30 MB/sec execute 198 sec latency 17.779 ms 2024-09-21T02:45:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:41 smithi029 ceph-mon[31006]: pgmap v586: 129 pgs: 129 active+clean; 37 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-09-21T02:45:42.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:41 smithi066 ceph-mon[33436]: pgmap v586: 129 pgs: 129 active+clean; 37 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-09-21T02:45:42.082 INFO:tasks.workunit.client.0.smithi029.stdout: 1 115453 2.31 MB/sec execute 199 sec latency 16.343 ms 2024-09-21T02:45:42.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:41 smithi190 ceph-mon[33871]: pgmap v586: 129 pgs: 129 active+clean; 37 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-09-21T02:45:43.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 116092 2.31 MB/sec execute 200 sec latency 22.145 ms 2024-09-21T02:45:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:43 smithi066 ceph-mon[33436]: pgmap v587: 129 pgs: 129 active+clean; 38 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.4 MiB/s wr, 111 op/s 2024-09-21T02:45:44.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 116857 2.33 MB/sec execute 201 sec latency 17.164 ms 2024-09-21T02:45:44.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:43 smithi190 ceph-mon[33871]: pgmap v587: 129 pgs: 129 active+clean; 38 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.4 MiB/s wr, 111 op/s 2024-09-21T02:45:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:43 smithi029 ceph-mon[31006]: pgmap v587: 129 pgs: 129 active+clean; 38 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.4 MiB/s wr, 111 op/s 2024-09-21T02:45:45.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 117366 2.34 MB/sec execute 202 sec latency 14.189 ms 2024-09-21T02:45:46.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:45 smithi066 ceph-mon[33436]: pgmap v588: 129 pgs: 129 active+clean; 34 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-09-21T02:45:46.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 117633 2.33 MB/sec execute 203 sec latency 15.556 ms 2024-09-21T02:45:46.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:45 smithi190 ceph-mon[33871]: pgmap v588: 129 pgs: 129 active+clean; 34 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-09-21T02:45:46.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:45 smithi029 ceph-mon[31006]: pgmap v588: 129 pgs: 129 active+clean; 34 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-09-21T02:45:47.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 118244 2.33 MB/sec execute 204 sec latency 17.504 ms 2024-09-21T02:45:48.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:47 smithi066 ceph-mon[33436]: pgmap v589: 129 pgs: 129 active+clean; 35 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-09-21T02:45:48.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 118823 2.33 MB/sec execute 205 sec latency 72.098 ms 2024-09-21T02:45:48.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:47 smithi190 ceph-mon[33871]: pgmap v589: 129 pgs: 129 active+clean; 35 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-09-21T02:45:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:47 smithi029 ceph-mon[31006]: pgmap v589: 129 pgs: 129 active+clean; 35 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-09-21T02:45:49.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 119480 2.33 MB/sec execute 206 sec latency 13.788 ms 2024-09-21T02:45:50.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:49 smithi066 ceph-mon[33436]: pgmap v590: 129 pgs: 129 active+clean; 35 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-09-21T02:45:50.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 120370 2.35 MB/sec execute 207 sec latency 14.563 ms 2024-09-21T02:45:50.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:49 smithi190 ceph-mon[33871]: pgmap v590: 129 pgs: 129 active+clean; 35 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-09-21T02:45:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:49 smithi029 ceph-mon[31006]: pgmap v590: 129 pgs: 129 active+clean; 35 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-09-21T02:45:51.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 120858 2.36 MB/sec execute 208 sec latency 15.338 ms 2024-09-21T02:45:51.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:51 smithi190 ceph-mon[33871]: pgmap v591: 129 pgs: 129 active+clean; 36 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:45:52.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:51 smithi066 ceph-mon[33436]: pgmap v591: 129 pgs: 129 active+clean; 36 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:45:52.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 121105 2.35 MB/sec execute 209 sec latency 13.118 ms 2024-09-21T02:45:52.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:51 smithi029 ceph-mon[31006]: pgmap v591: 129 pgs: 129 active+clean; 36 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-09-21T02:45:53.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 121628 2.35 MB/sec execute 210 sec latency 38.836 ms 2024-09-21T02:45:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 -- 172.21.15.29:0/3048943517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad41050f0 msgr2=0x7f2ad410ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/3048943517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad41050f0 0x7f2ad410ba90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009ca0 tx=0x7f2acc02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 -- 172.21.15.29:0/3048943517 shutdown_connections 2024-09-21T02:45:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/3048943517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad41050f0 0x7f2ad410ba90 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/3048943517 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ad4104750 0x7f2ad4104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/3048943517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41036d0 0x7f2ad4103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 -- 172.21.15.29:0/3048943517 >> 172.21.15.29:0/3048943517 conn(0x7f2ad40ff030 msgr2=0x7f2ad41014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 -- 172.21.15.29:0/3048943517 shutdown_connections 2024-09-21T02:45:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 -- 172.21.15.29:0/3048943517 wait complete. 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 Processor -- start 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.256+0000 7f2adc7c3640 1 -- start start 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2adc7c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41036d0 0x7f2ad4076e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2adc7c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad4104750 0x7f2ad4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2adc7c3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ad41050f0 0x7f2ad4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2adc7c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410f180 con 0x7f2ad4104750 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2adc7c3640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410f000 con 0x7f2ad41050f0 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2adc7c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410f300 con 0x7f2ad41036d0 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad4104750 0x7f2ad4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad4104750 0x7f2ad4075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35782/0 (socket says 172.21.15.29:35782) 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/2685271109 learned_addr learned my addr 172.21.15.29:0/2685271109 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ada55f640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41036d0 0x7f2ad4076e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2adad60640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ad41050f0 0x7f2ad4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/2685271109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41036d0 msgr2=0x7f2ad4076e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41036d0 0x7f2ad4076e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/2685271109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ad41050f0 msgr2=0x7f2ad4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ad41050f0 0x7f2ad4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/2685271109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad4076380 con 0x7f2ad4104750 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.257+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad4104750 0x7f2ad4075710 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400d950 tx=0x7f2ac400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.258+0000 7f2acb7fe640 1 -- 172.21.15.29:0/2685271109 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4017070 con 0x7f2ad4104750 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.258+0000 7f2acb7fe640 1 -- 172.21.15.29:0/2685271109 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ac4009c30 con 0x7f2ad4104750 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.258+0000 7f2adc7c3640 1 -- 172.21.15.29:0/2685271109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad41a3eb0 con 0x7f2ad4104750 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.258+0000 7f2acb7fe640 1 -- 172.21.15.29:0/2685271109 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac400f620 con 0x7f2ad4104750 2024-09-21T02:45:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.258+0000 7f2adc7c3640 1 -- 172.21.15.29:0/2685271109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad41a42c0 con 0x7f2ad4104750 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.261+0000 7f2ae3809740 1 Processor -- start 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.261+0000 7f2ae3809740 1 -- start start 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.261+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e0ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.261+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e0f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.261+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e10010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.261+0000 7f2ae3809740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563480d816f0 con 0x563480a464d0 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.261+0000 7f2ae3809740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x563480d70b20 con 0x563480a671b0 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.261+0000 7f2ae3809740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563480d74930 con 0x563480a3e2f0 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2ada55f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e0f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2ada55f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e0f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:60690/0 (socket says 172.21.15.29:60690) 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2adad60640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e0ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2ada55f640 1 -- 172.21.15.29:0/1088573494 learned_addr learned my addr 172.21.15.29:0/1088573494 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2acb7fe640 1 -- 172.21.15.29:0/2685271109 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f2ac4010460 con 0x7f2ad4104750 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e10010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2adad60640 1 -- 172.21.15.29:0/1088573494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 msgr2=0x563480e0f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2adad60640 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e0f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2adad60640 1 -- 172.21.15.29:0/1088573494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 msgr2=0x563480e10010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2adad60640 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e10010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2adad60640 1 -- 172.21.15.29:0/1088573494 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563480d36530 con 0x563480a671b0 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2ada55f640 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e0f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2adad60640 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e0ee50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2acc006010 tx=0x7f2acc009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4cfa0c32382ffd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.262+0000 7f2ac8ff9640 1 -- 172.21.15.29:0/1088573494 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc031a80 con 0x563480a671b0 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.263+0000 7f2ac8ff9640 1 -- 172.21.15.29:0/1088573494 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2acc036070 con 0x563480a671b0 2024-09-21T02:45:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.263+0000 7f2ac8ff9640 1 -- 172.21.15.29:0/1088573494 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc004430 con 0x563480a671b0 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.263+0000 7f2acb7fe640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2ac00b6510 0x7f2ac00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.263+0000 7f2ada55f640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2ac00b6510 0x7f2ac00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.263+0000 7f2acb7fe640 1 -- 172.21.15.29:0/2685271109 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2ac40e7210 con 0x7f2ad4104750 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 -- 172.21.15.29:0/1088573494 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 msgr2=0x563480e0ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ada55f640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2ac00b6510 0x7f2ac00b8a20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0012560 tx=0x7f2ad0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e0ee50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2acc006010 tx=0x7f2acc009880 comp rx=0 tx=0).stop 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 -- 172.21.15.29:0/1088573494 shutdown_connections 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e10010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e0f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 --2- 172.21.15.29:0/1088573494 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e0ee50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 -- 172.21.15.29:0/1088573494 >> 172.21.15.29:0/1088573494 conn(0x563480a5b300 msgr2=0x563480e0a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 -- 172.21.15.29:0/1088573494 shutdown_connections 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 -- 172.21.15.29:0/1088573494 wait complete. 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 -- start start 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.264+0000 7f2ae3809740 1 Processor -- start 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e18450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e18b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e19680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ae3809740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563480ca8ab0 con 0x563480a464d0 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ae3809740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x563480d16730 con 0x563480a671b0 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ae3809740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563480e10950 con 0x563480a3e2f0 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ae3809740 10 client.0 _ll_register_callbacks cb 0x563480d6a6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ad9d5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e18b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ad9d5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e18b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35804/0 (socket says 172.21.15.29:35804) 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/2862087270 learned_addr learned my addr 172.21.15.29:0/2862087270 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ada55f640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e18450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2adad60640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e19680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ada55f640 1 -- 172.21.15.29:0/2862087270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 msgr2=0x563480e19680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ada55f640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e19680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ada55f640 1 -- 172.21.15.29:0/2862087270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 msgr2=0x563480e18b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ada55f640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e18b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ada55f640 1 -- 172.21.15.29:0/2862087270 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563480ad2790 con 0x563480a3e2f0 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e18b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2adad60640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e19680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2ada55f640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e18450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000fa90 tx=0x7f2ad0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a76c9bc0f923333 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.265+0000 7f2abdffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2abdffb640 1 -- 172.21.15.29:0/2862087270 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0004020 con 0x563480a3e2f0 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2abdffb640 1 -- 172.21.15.29:0/2862087270 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2ad0018030 con 0x563480a3e2f0 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 1 -- 172.21.15.29:0/2862087270 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2acc036070 con 0x563480a3e2f0 2024-09-21T02:45:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2abdffb640 1 -- 172.21.15.29:0/2862087270 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ad0032050 con 0x563480a3e2f0 2024-09-21T02:45:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 10 client.26092 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 1 -- 172.21.15.29:0/2862087270 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563480b4b620 con 0x563480a3e2f0 2024-09-21T02:45:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2abdffb640 1 -- 172.21.15.29:0/2862087270 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2acc036070 con 0x563480a3e2f0 2024-09-21T02:45:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2abdffb640 1 -- 172.21.15.29:0/2862087270 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2ad00047a0 con 0x563480a3e2f0 2024-09-21T02:45:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 10 client.26092 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 10 client.26092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 20 client.26092 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563480a25d80 0x563480e25c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 4 client.26092 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.266+0000 7f2ae3809740 1 -- 172.21.15.29:0/2862087270 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563480d83640 con 0x563480a25d80 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.267+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563480a25d80 0x563480e25c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.267+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563480a25d80 0x563480e25c40 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90bccb01f2771496 server_cookie=3e56affc611dd7bb in_seq=0 out_seq=0 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.267+0000 7f2abdffb640 10 client.26092 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2abdffb640 1 -- 172.21.15.29:0/2862087270 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563480d83640 con 0x563480a25d80 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2abdffb640 10 client.26092 handle_command_reply: tid=0 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 client.26092 shutdown 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 -- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 msgr2=0x563480e18450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e18450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000fa90 tx=0x7f2ad0004730 comp rx=0 tx=0).stop 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 -- 172.21.15.29:0/2862087270 shutdown_connections 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a671b0 0x563480e19680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563480a25d80 0x563480e25c40 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e18b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 --2- 172.21.15.29:0/2862087270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a3e2f0 0x563480e18450 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2abdffb640 0 client.26092 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 -- 172.21.15.29:0/2862087270 >> 172.21.15.29:0/2862087270 conn(0x563480a5b300 msgr2=0x563480e08ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 -- 172.21.15.29:0/2862087270 shutdown_connections 2024-09-21T02:45:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 1 -- 172.21.15.29:0/2862087270 wait complete. 2024-09-21T02:45:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.268+0000 7f2ae3809740 20 client.26092 trim_cache size 0 max 16384 2024-09-21T02:45:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.289+0000 7f2ae3809740 1 Processor -- start 2024-09-21T02:45:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ae3809740 1 -- start start 2024-09-21T02:45:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 0x563480e0fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e169e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a671b0 0x563480e16f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ae3809740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41ae6e0 con 0x563480a464d0 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ae3809740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41b0220 con 0x563480a3e2f0 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ae3809740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41aedb0 con 0x563480a671b0 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ad9d5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e169e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ad9d5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e169e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35818/0 (socket says 172.21.15.29:35818) 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/472829841 learned_addr learned my addr 172.21.15.29:0/472829841 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2adad60640 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a671b0 0x563480e16f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.290+0000 7f2ada55f640 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 0x563480e0fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/472829841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a671b0 msgr2=0x563480e16f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a671b0 0x563480e16f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/472829841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 msgr2=0x563480e0fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 0x563480e0fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/472829841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563480e16660 con 0x563480a464d0 2024-09-21T02:45:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e169e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40e4d00 tx=0x7f2ac40e7510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543357f01cd89857 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2abffff640 1 -- 172.21.15.29:0/472829841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4020040 con 0x563480a464d0 2024-09-21T02:45:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2abffff640 1 -- 172.21.15.29:0/472829841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ac40e4e50 con 0x563480a464d0 2024-09-21T02:45:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.291+0000 7f2abffff640 1 -- 172.21.15.29:0/472829841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4010750 con 0x563480a464d0 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.293+0000 7f2ae3809740 1 -- 172.21.15.29:0/472829841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 msgr2=0x563480e169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.293+0000 7f2ae3809740 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e169e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40e4d00 tx=0x7f2ac40e7510 comp rx=0 tx=0).stop 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.293+0000 7f2ae3809740 1 -- 172.21.15.29:0/472829841 shutdown_connections 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.293+0000 7f2ae3809740 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a671b0 0x563480e16f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.293+0000 7f2ae3809740 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a464d0 0x563480e169e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.293+0000 7f2ae3809740 1 --2- 172.21.15.29:0/472829841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 0x563480e0fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.293+0000 7f2ae3809740 1 -- 172.21.15.29:0/472829841 >> 172.21.15.29:0/472829841 conn(0x563480a5b300 msgr2=0x563480e17650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.294+0000 7f2ae3809740 1 -- 172.21.15.29:0/472829841 shutdown_connections 2024-09-21T02:45:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.294+0000 7f2ae3809740 1 -- 172.21.15.29:0/472829841 wait complete. 2024-09-21T02:45:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.294+0000 7f2ae3809740 1 -- start start 2024-09-21T02:45:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.294+0000 7f2ae3809740 1 Processor -- start 2024-09-21T02:45:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 0x563480e200c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a464d0 0x563480e1ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ae3809740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a671b0 0x563480e1f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ae3809740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41afb50 con 0x563480a671b0 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ae3809740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41af480 con 0x563480a3e2f0 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ae3809740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41ac6c0 con 0x563480a464d0 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ae3809740 10 client.0 _ll_register_callbacks cb 0x7f2ad00099f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ad9d5e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a464d0 0x563480e1ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ad9d5e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a464d0 0x563480e1ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:60726/0 (socket says 172.21.15.29:60726) 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/3809113771 learned_addr learned my addr 172.21.15.29:0/3809113771 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ada55f640 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 0x563480e200c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2adad60640 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a671b0 0x563480e1f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/3809113771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 msgr2=0x563480e200c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 0x563480e200c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/3809113771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a671b0 msgr2=0x563480e1f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a671b0 0x563480e1f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.295+0000 7f2ad9d5e640 1 -- 172.21.15.29:0/3809113771 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563480e1f900 con 0x563480a464d0 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2adad60640 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a671b0 0x563480e1f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2ad9d5e640 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a464d0 0x563480e1ed80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40a9000 tx=0x7f2ac40e7600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b1c869f6f22e611 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2abdffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2abdffb640 1 -- 172.21.15.29:0/3809113771 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4020070 con 0x563480a464d0 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2abdffb640 1 -- 172.21.15.29:0/3809113771 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2ac40a8c20 con 0x563480a464d0 2024-09-21T02:45:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2ae3809740 1 -- 172.21.15.29:0/3809113771 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac40e4e50 con 0x563480a464d0 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2abdffb640 1 -- 172.21.15.29:0/3809113771 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ac4009de0 con 0x563480a464d0 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2ae3809740 10 client.26098 fetch_fsmap learned FSMap version 31 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2ae3809740 1 -- 172.21.15.29:0/3809113771 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563480e90640 con 0x563480a464d0 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.296+0000 7f2abdffb640 1 -- 172.21.15.29:0/3809113771 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ac40e4e50 con 0x563480a464d0 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2abdffb640 1 -- 172.21.15.29:0/3809113771 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2ac40b2c90 con 0x563480a464d0 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2ae3809740 10 client.26098 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2ae3809740 10 client.26098 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2ae3809740 20 client.26098 populate_metadata read hostname 'smithi029' 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2ae3809740 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5634809fb1c0 0x563480e92ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2ae3809740 4 client.26098 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2ae3809740 1 -- 172.21.15.29:0/3809113771 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563480d7f0f0 con 0x5634809fb1c0 2024-09-21T02:45:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2ada55f640 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5634809fb1c0 0x563480e92ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:45:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.297+0000 7f2abdffb640 10 client.26098 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.298+0000 7f2ada55f640 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5634809fb1c0 0x563480e92ef0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8452cfeb2057ae34 server_cookie=630a77fa7c6d8bbf in_seq=0 out_seq=0 2024-09-21T02:45:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.344+0000 7f2abdffb640 1 -- 172.21.15.29:0/3809113771 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563480d7f0f0 con 0x5634809fb1c0 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.344+0000 7f2abdffb640 10 client.26098 handle_command_reply: tid=0 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.344+0000 7f2ae3809740 1 client.26098 shutdown 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.344+0000 7f2ae3809740 1 -- 172.21.15.29:0/3809113771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a464d0 msgr2=0x563480e1ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.344+0000 7f2ae3809740 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a464d0 0x563480e1ed80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40a9000 tx=0x7f2ac40e7600 comp rx=0 tx=0).stop 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 1 -- 172.21.15.29:0/3809113771 shutdown_connections 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5634809fb1c0 0x563480e92ef0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563480a671b0 0x563480e1f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563480a464d0 0x563480e1ed80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 1 --2- 172.21.15.29:0/3809113771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563480a3e2f0 0x563480e200c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 1 -- 172.21.15.29:0/3809113771 >> 172.21.15.29:0/3809113771 conn(0x563480a5b300 msgr2=0x563480e12f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2abdffb640 0 client.26098 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:45:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 1 -- 172.21.15.29:0/3809113771 shutdown_connections 2024-09-21T02:45:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 1 -- 172.21.15.29:0/3809113771 wait complete. 2024-09-21T02:45:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2ae3809740 20 client.26098 trim_cache size 0 max 16384 2024-09-21T02:45:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:45:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:45:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:45:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:45:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2adc7c3640 1 -- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2ac00b6510 msgr2=0x7f2ac00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2ac00b6510 0x7f2ac00b8a20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0012560 tx=0x7f2ad0005f90 comp rx=0 tx=0).stop 2024-09-21T02:45:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2adc7c3640 1 -- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad4104750 msgr2=0x7f2ad4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:45:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad4104750 0x7f2ad4075710 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400d950 tx=0x7f2ac400de10 comp rx=0 tx=0).stop 2024-09-21T02:45:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2adc7c3640 1 -- 172.21.15.29:0/2685271109 shutdown_connections 2024-09-21T02:45:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.345+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2ac00b6510 0x7f2ac00b8a20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.346+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ad41050f0 0x7f2ad4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.346+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad4104750 0x7f2ad4075710 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.346+0000 7f2adc7c3640 1 --2- 172.21.15.29:0/2685271109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41036d0 0x7f2ad4076e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:45:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.346+0000 7f2adc7c3640 1 -- 172.21.15.29:0/2685271109 >> 172.21.15.29:0/2685271109 conn(0x7f2ad40ff030 msgr2=0x7f2ad4107a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:45:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.346+0000 7f2adc7c3640 1 -- 172.21.15.29:0/2685271109 shutdown_connections 2024-09-21T02:45:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:45:53.346+0000 7f2adc7c3640 1 -- 172.21.15.29:0/2685271109 wait complete. 2024-09-21T02:45:54.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:53 smithi066 ceph-mon[33436]: pgmap v592: 129 pgs: 129 active+clean; 35 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-09-21T02:45:54.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 122252 2.36 MB/sec execute 211 sec latency 21.906 ms 2024-09-21T02:45:54.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:53 smithi190 ceph-mon[33871]: pgmap v592: 129 pgs: 129 active+clean; 35 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-09-21T02:45:54.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:53 smithi029 ceph-mon[31006]: pgmap v592: 129 pgs: 129 active+clean; 35 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-09-21T02:45:55.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 122955 2.35 MB/sec execute 212 sec latency 15.386 ms 2024-09-21T02:45:56.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:55 smithi066 ceph-mon[33436]: pgmap v593: 129 pgs: 129 active+clean; 37 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:45:56.083 INFO:tasks.workunit.client.0.smithi029.stdout: 1 123661 2.37 MB/sec execute 213 sec latency 17.637 ms 2024-09-21T02:45:56.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:55 smithi190 ceph-mon[33871]: pgmap v593: 129 pgs: 129 active+clean; 37 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:45:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:55 smithi029 ceph-mon[31006]: pgmap v593: 129 pgs: 129 active+clean; 37 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:45:57.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 124256 2.38 MB/sec execute 214 sec latency 15.716 ms 2024-09-21T02:45:58.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:57 smithi066 ceph-mon[33436]: pgmap v594: 129 pgs: 129 active+clean; 33 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-09-21T02:45:58.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 124576 2.38 MB/sec execute 215 sec latency 14.101 ms 2024-09-21T02:45:58.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:57 smithi190 ceph-mon[33871]: pgmap v594: 129 pgs: 129 active+clean; 33 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-09-21T02:45:58.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:57 smithi029 ceph-mon[31006]: pgmap v594: 129 pgs: 129 active+clean; 33 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-09-21T02:45:59.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 125038 2.37 MB/sec execute 216 sec latency 17.244 ms 2024-09-21T02:46:00.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:45:59 smithi066 ceph-mon[33436]: pgmap v595: 129 pgs: 129 active+clean; 34 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-09-21T02:46:00.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 125682 2.38 MB/sec execute 217 sec latency 16.565 ms 2024-09-21T02:46:00.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:45:59 smithi190 ceph-mon[33871]: pgmap v595: 129 pgs: 129 active+clean; 34 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-09-21T02:46:00.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:45:59 smithi029 ceph-mon[31006]: pgmap v595: 129 pgs: 129 active+clean; 34 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-09-21T02:46:01.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 126328 2.37 MB/sec execute 218 sec latency 20.613 ms 2024-09-21T02:46:02.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:01 smithi066 ceph-mon[33436]: pgmap v596: 129 pgs: 129 active+clean; 35 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-09-21T02:46:02.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 126952 2.38 MB/sec execute 219 sec latency 50.600 ms 2024-09-21T02:46:02.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:01 smithi190 ceph-mon[33871]: pgmap v596: 129 pgs: 129 active+clean; 35 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-09-21T02:46:02.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:01 smithi029 ceph-mon[31006]: pgmap v596: 129 pgs: 129 active+clean; 35 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-09-21T02:46:03.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 127548 2.39 MB/sec execute 220 sec latency 55.873 ms 2024-09-21T02:46:03.934 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:03 smithi029 ceph-mon[31006]: pgmap v597: 129 pgs: 129 active+clean; 34 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:46:04.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:03 smithi066 ceph-mon[33436]: pgmap v597: 129 pgs: 129 active+clean; 34 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:46:04.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 128068 2.40 MB/sec execute 221 sec latency 14.886 ms 2024-09-21T02:46:04.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:03 smithi190 ceph-mon[33871]: pgmap v597: 129 pgs: 129 active+clean; 34 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:46:05.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 128387 2.39 MB/sec execute 222 sec latency 18.517 ms 2024-09-21T02:46:06.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:05 smithi066 ceph-mon[33436]: pgmap v598: 129 pgs: 129 active+clean; 35 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T02:46:06.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 129008 2.39 MB/sec execute 223 sec latency 16.797 ms 2024-09-21T02:46:06.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:05 smithi190 ceph-mon[33871]: pgmap v598: 129 pgs: 129 active+clean; 35 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T02:46:06.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:05 smithi029 ceph-mon[31006]: pgmap v598: 129 pgs: 129 active+clean; 35 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T02:46:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3987497683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103620 msgr2=0x7fbbc8103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3987497683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103620 0x7fbbc8103a30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009be0 tx=0x7fbbbc02a6c0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3987497683 shutdown_connections 2024-09-21T02:46:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3987497683 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc8103fc0 0x7fbbc810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3987497683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103620 0x7fbbc8103a30 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3987497683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc81025a0 0x7fbbc8102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3987497683 >> 172.21.15.29:0/3987497683 conn(0x7fbbc80fdee0 msgr2=0x7fbbc8100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3987497683 shutdown_connections 2024-09-21T02:46:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.453+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3987497683 wait complete. 2024-09-21T02:46:06.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcfd5f640 1 Processor -- start 2024-09-21T02:46:06.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcfd5f640 1 -- start start 2024-09-21T02:46:06.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcfd5f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc81025a0 0x7fbbc8199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcfd5f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc8103620 0x7fbbc819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcfd5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103fc0 0x7fbbc819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcfd5f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc810e0b0 con 0x7fbbc8103fc0 2024-09-21T02:46:06.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcfd5f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc810df30 con 0x7fbbc81025a0 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcfd5f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc810e230 con 0x7fbbc8103620 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbce2fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103fc0 0x7fbbc819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbce2fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103fc0 0x7fbbc819b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56862/0 (socket says 172.21.15.29:56862) 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbce2fc640 1 -- 172.21.15.29:0/3109340727 learned_addr learned my addr 172.21.15.29:0/3109340727 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc81025a0 0x7fbbc8199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.454+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc8103620 0x7fbbc819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbce2fc640 1 -- 172.21.15.29:0/3109340727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc8103620 msgr2=0x7fbbc819a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbce2fc640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc8103620 0x7fbbc819a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbce2fc640 1 -- 172.21.15.29:0/3109340727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc81025a0 msgr2=0x7fbbc8199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbce2fc640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc81025a0 0x7fbbc8199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbce2fc640 1 -- 172.21.15.29:0/3109340727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc819b920 con 0x7fbbc8103fc0 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbce2fc640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103fc0 0x7fbbc819b100 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000ea80 tx=0x7fbbc000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbbaffd640 1 -- 172.21.15.29:0/3109340727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc000cea0 con 0x7fbbc8103fc0 2024-09-21T02:46:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbbaffd640 1 -- 172.21.15.29:0/3109340727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbbc0004c80 con 0x7fbbc8103fc0 2024-09-21T02:46:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3109340727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc819bc10 con 0x7fbbc8103fc0 2024-09-21T02:46:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbbaffd640 1 -- 172.21.15.29:0/3109340727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0010430 con 0x7fbbc8103fc0 2024-09-21T02:46:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.455+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3109340727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc81a2300 con 0x7fbbc8103fc0 2024-09-21T02:46:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.458+0000 7fbbbaffd640 1 -- 172.21.15.29:0/3109340727 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fbbc0005000 con 0x7fbbc8103fc0 2024-09-21T02:46:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.458+0000 7fbbd6cc8740 1 Processor -- start 2024-09-21T02:46:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.458+0000 7fbbd6cc8740 1 -- start start 2024-09-21T02:46:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x558542987260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x5585429878d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x558542988420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5585428f56f0 con 0x5585425db1b0 2024-09-21T02:46:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5585428e4b20 con 0x5585425ba4d0 2024-09-21T02:46:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5585428e8930 con 0x5585425b22f0 2024-09-21T02:46:06.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcd2fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x558542987260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x5585429878d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcd2fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x558542987260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56868/0 (socket says 172.21.15.29:56868) 2024-09-21T02:46:06.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x5585429878d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:36358/0 (socket says 172.21.15.29:36358) 2024-09-21T02:46:06.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcd2fa640 1 -- 172.21.15.29:0/3641186832 learned_addr learned my addr 172.21.15.29:0/3641186832 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x558542988420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcdafb640 1 -- 172.21.15.29:0/3641186832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 msgr2=0x558542988420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x558542988420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcdafb640 1 -- 172.21.15.29:0/3641186832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 msgr2=0x558542987260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x558542987260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcdafb640 1 -- 172.21.15.29:0/3641186832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585428aa530 con 0x5585425b22f0 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x558542987260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.459+0000 7fbbbaffd640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6590 0x7fbbb40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.460+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6590 0x7fbbb40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.460+0000 7fbbbaffd640 1 -- 172.21.15.29:0/3109340727 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbbc00e79d0 con 0x7fbbc8103fc0 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.460+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x5585429878d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009c00 tx=0x7fbbc4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7b70fb107f6909 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.460+0000 7fbbb3fff640 1 -- 172.21.15.29:0/3641186832 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4004230 con 0x5585425b22f0 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.460+0000 7fbbb3fff640 1 -- 172.21.15.29:0/3641186832 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbbc40043d0 con 0x5585425b22f0 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.460+0000 7fbbb3fff640 1 -- 172.21.15.29:0/3641186832 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4004c00 con 0x5585425b22f0 2024-09-21T02:46:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/3641186832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 msgr2=0x5585429878d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6590 0x7fbbb40b8aa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009420 tx=0x7fbbbc005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x5585429878d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009c00 tx=0x7fbbc4011810 comp rx=0 tx=0).stop 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/3641186832 shutdown_connections 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x558542988420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x5585429878d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/3641186832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x558542987260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/3641186832 >> 172.21.15.29:0/3641186832 conn(0x5585425cf300 msgr2=0x55854297e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/3641186832 shutdown_connections 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/3641186832 wait complete. 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 -- start start 2024-09-21T02:46:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.461+0000 7fbbd6cc8740 1 Processor -- start 2024-09-21T02:46:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542990a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x5585429910c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 0x558542991c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55854281cab0 con 0x5585425b22f0 2024-09-21T02:46:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55854288a730 con 0x5585425ba4d0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558542988f40 con 0x5585425db1b0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbd6cc8740 10 client.0 _ll_register_callbacks cb 0x5585428de6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542990a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542990a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56872/0 (socket says 172.21.15.29:56872) 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcdafb640 1 -- 172.21.15.29:0/2788310938 learned_addr learned my addr 172.21.15.29:0/2788310938 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbce2fc640 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 0x558542991c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x5585429910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcdafb640 1 -- 172.21.15.29:0/2788310938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 msgr2=0x558542991c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 0x558542991c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcdafb640 1 -- 172.21.15.29:0/2788310938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 msgr2=0x5585429910c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x5585429910c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.462+0000 7fbbcdafb640 1 -- 172.21.15.29:0/2788310938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558542646790 con 0x5585425b22f0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542990a00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40096e0 tx=0x7fbbc4019b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117f290720c0c49f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbb17fa640 1 -- 172.21.15.29:0/2788310938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4028020 con 0x5585425b22f0 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbb17fa640 1 -- 172.21.15.29:0/2788310938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbbc4024620 con 0x5585425b22f0 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/2788310938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbc40043d0 con 0x5585425b22f0 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbb17fa640 1 -- 172.21.15.29:0/2788310938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbbc4033050 con 0x5585425b22f0 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbd6cc8740 10 client.16515 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/2788310938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5585426bf620 con 0x5585425b22f0 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.463+0000 7fbbb17fa640 1 -- 172.21.15.29:0/2788310938 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbbc40043d0 con 0x5585425b22f0 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbb17fa640 1 -- 172.21.15.29:0/2788310938 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fbbc4018070 con 0x5585425b22f0 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbd6cc8740 10 client.16515 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbd6cc8740 10 client.16515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbd6cc8740 20 client.16515 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558542599d80 0x55854299ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbd6cc8740 4 client.16515 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/2788310938 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585428f7640 con 0x558542599d80 2024-09-21T02:46:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558542599d80 0x55854299ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.464+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558542599d80 0x55854299ded0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddbbb977a82f789f server_cookie=2d2eaa2770901790 in_seq=0 out_seq=0 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.465+0000 7fbbb17fa640 10 client.16515 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbb17fa640 1 -- 172.21.15.29:0/2788310938 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5585428f7640 con 0x558542599d80 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbb17fa640 10 client.16515 handle_command_reply: tid=0 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 client.16515 shutdown 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/2788310938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 msgr2=0x558542990a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542990a00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40096e0 tx=0x7fbbc4019b70 comp rx=0 tx=0).stop 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/2788310938 shutdown_connections 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 0x558542991c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x5585429910c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x558542599d80 0x55854299ded0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/2788310938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542990a00 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbb17fa640 0 client.16515 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/2788310938 >> 172.21.15.29:0/2788310938 conn(0x5585425cf300 msgr2=0x55854297cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/2788310938 shutdown_connections 2024-09-21T02:46:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/2788310938 wait complete. 2024-09-21T02:46:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.466+0000 7fbbd6cc8740 20 client.16515 trim_cache size 0 max 16384 2024-09-21T02:46:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbd6cc8740 1 Processor -- start 2024-09-21T02:46:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbd6cc8740 1 -- start start 2024-09-21T02:46:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x55854298fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x55854297d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x55854297db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc81aa420 con 0x5585425db1b0 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc81abf60 con 0x5585425ba4d0 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc81aaaf0 con 0x5585425b22f0 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbce2fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x55854297db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x55854298fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x55854298fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:36386/0 (socket says 172.21.15.29:36386) 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbcdafb640 1 -- 172.21.15.29:0/806953000 learned_addr learned my addr 172.21.15.29:0/806953000 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbce2fc640 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x55854297db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56878/0 (socket says 172.21.15.29:56878) 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.482+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x55854297d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbcdafb640 1 -- 172.21.15.29:0/806953000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 msgr2=0x55854297d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x55854297d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbcdafb640 1 -- 172.21.15.29:0/806953000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 msgr2=0x55854297db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x55854297db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbcdafb640 1 -- 172.21.15.29:0/806953000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558542991fa0 con 0x5585425b22f0 2024-09-21T02:46:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbce2fc640 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x55854297db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x55854298fc00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40046d0 tx=0x7fbbc40197e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c04a31ad0ce138bd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbb37fe640 1 -- 172.21.15.29:0/806953000 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc403c070 con 0x5585425b22f0 2024-09-21T02:46:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbb37fe640 1 -- 172.21.15.29:0/806953000 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbbc401f040 con 0x5585425b22f0 2024-09-21T02:46:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.483+0000 7fbbb37fe640 1 -- 172.21.15.29:0/806953000 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4028070 con 0x5585425b22f0 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/806953000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 msgr2=0x55854298fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x55854298fc00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40046d0 tx=0x7fbbc40197e0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/806953000 shutdown_connections 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425db1b0 0x55854297db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x55854297d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/806953000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425b22f0 0x55854298fc00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/806953000 >> 172.21.15.29:0/806953000 conn(0x5585425cf300 msgr2=0x5585429880a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/806953000 shutdown_connections 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/806953000 wait complete. 2024-09-21T02:46:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 -- start start 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.484+0000 7fbbd6cc8740 1 Processor -- start 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542998410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x558542997110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbd6cc8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 0x558542997820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc81ab890 con 0x5585425b22f0 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc81ab1c0 con 0x5585425ba4d0 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbd6cc8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc81a8400 con 0x5585425db1b0 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbd6cc8740 10 client.0 _ll_register_callbacks cb 0x7fbbc0005ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542998410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbce2fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 0x558542997820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcdafb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542998410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56894/0 (socket says 172.21.15.29:56894) 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcdafb640 1 -- 172.21.15.29:0/302974744 learned_addr learned my addr 172.21.15.29:0/302974744 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x558542997110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcdafb640 1 -- 172.21.15.29:0/302974744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 msgr2=0x558542997820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 0x558542997820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcdafb640 1 -- 172.21.15.29:0/302974744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 msgr2=0x558542997110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x558542997110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.485+0000 7fbbcdafb640 1 -- 172.21.15.29:0/302974744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558542995c90 con 0x5585425b22f0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbcdafb640 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542998410 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4018650 tx=0x7fbbc40092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48eee0ed52543bc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbb17fa640 1 -- 172.21.15.29:0/302974744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc403c070 con 0x5585425b22f0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/302974744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbc401f040 con 0x5585425b22f0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbb17fa640 1 -- 172.21.15.29:0/302974744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fbbc4018920 con 0x5585425b22f0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbb17fa640 1 -- 172.21.15.29:0/302974744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbbc4047050 con 0x5585425b22f0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbd6cc8740 10 client.16521 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/302974744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558542a08a80 con 0x5585425b22f0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbb17fa640 1 -- 172.21.15.29:0/302974744 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbbc401f040 con 0x5585425b22f0 2024-09-21T02:46:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.486+0000 7fbbb17fa640 1 -- 172.21.15.29:0/302974744 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fbbc4018040 con 0x5585425b22f0 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbd6cc8740 10 client.16521 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbd6cc8740 10 client.16521 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbd6cc8740 20 client.16521 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55854256f1c0 0x558542a0b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbd6cc8740 4 client.16521 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/302974744 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5585428f30f0 con 0x55854256f1c0 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55854256f1c0 0x558542a0b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbb17fa640 10 client.16521 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.487+0000 7fbbcd2fa640 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55854256f1c0 0x558542a0b420 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b1f52cd5e6e7bf8 server_cookie=dbf8c7a6e4179d66 in_seq=0 out_seq=0 2024-09-21T02:46:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.541+0000 7fbbb17fa640 1 -- 172.21.15.29:0/302974744 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5585428f30f0 con 0x55854256f1c0 2024-09-21T02:46:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.541+0000 7fbbb17fa640 10 client.16521 handle_command_reply: tid=0 2024-09-21T02:46:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.541+0000 7fbbd6cc8740 1 client.16521 shutdown 2024-09-21T02:46:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.541+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/302974744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 msgr2=0x558542998410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.541+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542998410 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4018650 tx=0x7fbbc40092e0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/302974744 shutdown_connections 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585425db1b0 0x558542997820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5585425ba4d0 0x558542997110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55854256f1c0 0x558542a0b420 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 1 --2- 172.21.15.29:0/302974744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585425b22f0 0x558542998410 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbb17fa640 0 client.16521 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/302974744 >> 172.21.15.29:0/302974744 conn(0x5585425cf300 msgr2=0x55854297d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/302974744 shutdown_connections 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 1 -- 172.21.15.29:0/302974744 wait complete. 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbd6cc8740 20 client.16521 trim_cache size 0 max 16384 2024-09-21T02:46:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:46:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:46:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:46:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:46:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6590 msgr2=0x7fbbb40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6590 0x7fbbb40b8aa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009420 tx=0x7fbbbc005bd0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103fc0 msgr2=0x7fbbc819b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103fc0 0x7fbbc819b100 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000ea80 tx=0x7fbbc000ef40 comp rx=0 tx=0).stop 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3109340727 shutdown_connections 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fbbb40b6590 0x7fbbb40b8aa0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8103fc0 0x7fbbc819b100 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbc8103620 0x7fbbc819a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 --2- 172.21.15.29:0/3109340727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbc81025a0 0x7fbbc8199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.542+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3109340727 >> 172.21.15.29:0/3109340727 conn(0x7fbbc80fdee0 msgr2=0x7fbbc80ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.543+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3109340727 shutdown_connections 2024-09-21T02:46:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:06.543+0000 7fbbcfd5f640 1 -- 172.21.15.29:0/3109340727 wait complete. 2024-09-21T02:46:07.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 129636 2.39 MB/sec execute 224 sec latency 27.568 ms 2024-09-21T02:46:08.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:07 smithi066 ceph-mon[33436]: pgmap v599: 129 pgs: 129 active+clean; 35 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:46:08.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 130230 2.39 MB/sec execute 225 sec latency 21.399 ms 2024-09-21T02:46:08.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:07 smithi190 ceph-mon[33871]: pgmap v599: 129 pgs: 129 active+clean; 35 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:46:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:07 smithi029 ceph-mon[31006]: pgmap v599: 129 pgs: 129 active+clean; 35 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:46:09.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 131043 2.41 MB/sec execute 226 sec latency 17.181 ms 2024-09-21T02:46:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:09 smithi066 ceph-mon[33436]: pgmap v600: 129 pgs: 129 active+clean; 36 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-09-21T02:46:10.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 131545 2.42 MB/sec execute 227 sec latency 16.858 ms 2024-09-21T02:46:10.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:09 smithi190 ceph-mon[33871]: pgmap v600: 129 pgs: 129 active+clean; 36 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-09-21T02:46:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:09 smithi029 ceph-mon[31006]: pgmap v600: 129 pgs: 129 active+clean; 36 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-09-21T02:46:11.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 131843 2.41 MB/sec execute 228 sec latency 12.986 ms 2024-09-21T02:46:12.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:11 smithi066 ceph-mon[33436]: pgmap v601: 129 pgs: 129 active+clean; 37 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:46:12.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 132446 2.41 MB/sec execute 229 sec latency 17.380 ms 2024-09-21T02:46:12.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:11 smithi190 ceph-mon[33871]: pgmap v601: 129 pgs: 129 active+clean; 37 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:46:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:11 smithi029 ceph-mon[31006]: pgmap v601: 129 pgs: 129 active+clean; 37 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:46:13.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 133105 2.41 MB/sec execute 230 sec latency 21.682 ms 2024-09-21T02:46:14.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:13 smithi066 ceph-mon[33436]: pgmap v602: 129 pgs: 129 active+clean; 36 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-09-21T02:46:14.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 133711 2.41 MB/sec execute 231 sec latency 14.443 ms 2024-09-21T02:46:14.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:13 smithi190 ceph-mon[33871]: pgmap v602: 129 pgs: 129 active+clean; 36 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-09-21T02:46:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:13 smithi029 ceph-mon[31006]: pgmap v602: 129 pgs: 129 active+clean; 36 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-09-21T02:46:15.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 134572 2.43 MB/sec execute 232 sec latency 11.712 ms 2024-09-21T02:46:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:15 smithi066 ceph-mon[33436]: pgmap v603: 129 pgs: 129 active+clean; 37 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-09-21T02:46:16.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 135063 2.44 MB/sec execute 233 sec latency 15.993 ms 2024-09-21T02:46:16.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:15 smithi190 ceph-mon[33871]: pgmap v603: 129 pgs: 129 active+clean; 37 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-09-21T02:46:16.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:15 smithi029 ceph-mon[31006]: pgmap v603: 129 pgs: 129 active+clean; 37 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-09-21T02:46:17.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 135304 2.43 MB/sec execute 234 sec latency 16.371 ms 2024-09-21T02:46:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:17 smithi066 ceph-mon[33436]: pgmap v604: 129 pgs: 129 active+clean; 38 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-09-21T02:46:18.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 135880 2.43 MB/sec execute 235 sec latency 27.393 ms 2024-09-21T02:46:18.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:17 smithi190 ceph-mon[33871]: pgmap v604: 129 pgs: 129 active+clean; 38 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-09-21T02:46:18.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:17 smithi029 ceph-mon[31006]: pgmap v604: 129 pgs: 129 active+clean; 38 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-09-21T02:46:19.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 136513 2.43 MB/sec execute 236 sec latency 14.374 ms 2024-09-21T02:46:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.650+0000 7f8e15532640 1 -- 172.21.15.29:0/1096850514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 msgr2=0x7f8e10108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.650+0000 7f8e15532640 1 --2- 172.21.15.29:0/1096850514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 0x7f8e10108f40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009900 tx=0x7f8e0402a410 comp rx=0 tx=0).stop 2024-09-21T02:46:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 -- 172.21.15.29:0/1096850514 shutdown_connections 2024-09-21T02:46:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 --2- 172.21.15.29:0/1096850514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e10109480 0x7f8e1010b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 --2- 172.21.15.29:0/1096850514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 0x7f8e10108f40 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 --2- 172.21.15.29:0/1096850514 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e10101a90 0x7f8e10101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 -- 172.21.15.29:0/1096850514 >> 172.21.15.29:0/1096850514 conn(0x7f8e100fd6d0 msgr2=0x7f8e100ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 -- 172.21.15.29:0/1096850514 shutdown_connections 2024-09-21T02:46:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 -- 172.21.15.29:0/1096850514 wait complete. 2024-09-21T02:46:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 Processor -- start 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.651+0000 7f8e15532640 1 -- start start 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e15532640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e10101a90 0x7f8e10199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e15532640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 0x7f8e1019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e15532640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e10109480 0x7f8e1019b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e15532640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1010f1b0 con 0x7f8e10106b20 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e15532640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1010f030 con 0x7f8e10109480 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e15532640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1010f330 con 0x7f8e10101a90 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 0x7f8e1019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 0x7f8e1019a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36274/0 (socket says 172.21.15.29:36274) 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 -- 172.21.15.29:0/3118754636 learned_addr learned my addr 172.21.15.29:0/3118754636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0effd640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e10101a90 0x7f8e10199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e10109480 0x7f8e1019b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 -- 172.21.15.29:0/3118754636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e10101a90 msgr2=0x7f8e10199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e10101a90 0x7f8e10199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 -- 172.21.15.29:0/3118754636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e10109480 msgr2=0x7f8e1019b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e10109480 0x7f8e1019b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 -- 172.21.15.29:0/3118754636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1019b8d0 con 0x7f8e10106b20 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.652+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 0x7f8e1019a500 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009230 tx=0x7f8e04005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.653+0000 7f8df7fff640 1 -- 172.21.15.29:0/3118754636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0403f030 con 0x7f8e10106b20 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.653+0000 7f8df7fff640 1 -- 172.21.15.29:0/3118754636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e040043d0 con 0x7f8e10106b20 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.653+0000 7f8e15532640 1 -- 172.21.15.29:0/3118754636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1019bb60 con 0x7f8e10106b20 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.653+0000 7f8df7fff640 1 -- 172.21.15.29:0/3118754636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04035070 con 0x7f8e10106b20 2024-09-21T02:46:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.653+0000 7f8e15532640 1 -- 172.21.15.29:0/3118754636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e1019be70 con 0x7f8e10106b20 2024-09-21T02:46:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.655+0000 7f8e1c4a1740 1 Processor -- start 2024-09-21T02:46:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.655+0000 7f8e1c4a1740 1 -- start start 2024-09-21T02:46:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20002f0 0x55f2f23d14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f2f23436f0 con 0x55f2f20084d0 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f2f2332b20 con 0x55f2f20291b0 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2f2336930 con 0x55f2f20002f0 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8df7fff640 1 -- 172.21.15.29:0/3118754636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f8e04042030 con 0x7f8e10106b20 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20002f0 0x55f2f23d14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20002f0 0x55f2f23d14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:60936/0 (socket says 172.21.15.29:60936) 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 -- 172.21.15.29:0/1839459455 learned_addr learned my addr 172.21.15.29:0/1839459455 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 -- 172.21.15.29:0/1839459455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 msgr2=0x55f2f23d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 -- 172.21.15.29:0/1839459455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 msgr2=0x55f2f23d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23d2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 -- 172.21.15.29:0/1839459455 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2f22f8530 con 0x55f2f20002f0 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23d2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.656+0000 7f8e0effd640 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20002f0 0x55f2f23d14c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009cf0 tx=0x7f8e00011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b11d5805b68904f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.657+0000 7f8df57fa640 1 -- 172.21.15.29:0/1839459455 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00004230 con 0x55f2f20002f0 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.657+0000 7f8df7fff640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8df80b6510 0x7f8df80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.657+0000 7f8df57fa640 1 -- 172.21.15.29:0/1839459455 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e000043d0 con 0x55f2f20002f0 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.657+0000 7f8df57fa640 1 -- 172.21.15.29:0/1839459455 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00019750 con 0x55f2f20002f0 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.657+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8df80b6510 0x7f8df80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.657+0000 7f8df7fff640 1 -- 172.21.15.29:0/3118754636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e04107a50 con 0x7f8e10106b20 2024-09-21T02:46:19.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/1839459455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20002f0 msgr2=0x55f2f23d14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20002f0 0x55f2f23d14c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009cf0 tx=0x7f8e00011810 comp rx=0 tx=0).stop 2024-09-21T02:46:19.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8df80b6510 0x7f8df80b8a20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8e08005fd0 tx=0x7f8e08009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/1839459455 shutdown_connections 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23d2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20002f0 0x55f2f23d14c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/1839459455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/1839459455 >> 172.21.15.29:0/1839459455 conn(0x55f2f201d300 msgr2=0x55f2f23cc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/1839459455 shutdown_connections 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/1839459455 wait complete. 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 -- start start 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.658+0000 7f8e1c4a1740 1 Processor -- start 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23da580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23dac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23db7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f2f226aab0 con 0x55f2f20002f0 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f2f22d8730 con 0x55f2f20291b0 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2f23d2b30 con 0x55f2f20084d0 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e1c4a1740 10 client.0 _ll_register_callbacks cb 0x55f2f232c6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23da580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23dac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23dac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:60938/0 (socket says 172.21.15.29:60938) 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0e7fc640 1 -- 172.21.15.29:0/3540694986 learned_addr learned my addr 172.21.15.29:0/3540694986 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23db7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0effd640 1 -- 172.21.15.29:0/3540694986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 msgr2=0x55f2f23dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0effd640 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23dac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0effd640 1 -- 172.21.15.29:0/3540694986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 msgr2=0x55f2f23db7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0effd640 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23db7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0effd640 1 -- 172.21.15.29:0/3540694986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2f2094790 con 0x55f2f20002f0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23dac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8e0effd640 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23da580 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009cf0 tx=0x7f8e000038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce5c5c26ea124cc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.659+0000 7f8df2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8df2ff5640 1 -- 172.21.15.29:0/3540694986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00027070 con 0x55f2f20002f0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8df2ff5640 1 -- 172.21.15.29:0/3540694986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e00005240 con 0x55f2f20002f0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/3540694986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e000043d0 con 0x55f2f20002f0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8df2ff5640 1 -- 172.21.15.29:0/3540694986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8e00033050 con 0x55f2f20002f0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 10 client.16533 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/3540694986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f2f210d620 con 0x55f2f20002f0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8df2ff5640 1 -- 172.21.15.29:0/3540694986 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e000043d0 con 0x55f2f20002f0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8df2ff5640 1 -- 172.21.15.29:0/3540694986 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f8e00004c70 con 0x55f2f20002f0 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 10 client.16533 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:19.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 20 client.16533 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2f1fe7d80 0x55f2f23e7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 4 client.16533 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.660+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/3540694986 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2f2345640 con 0x55f2f1fe7d80 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.661+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2f1fe7d80 0x55f2f23e7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.661+0000 7f8df2ff5640 10 client.16533 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.661+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2f1fe7d80 0x55f2f23e7af0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfcbc73237d93e06 server_cookie=282ef404a4718351 in_seq=0 out_seq=0 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8df2ff5640 1 -- 172.21.15.29:0/3540694986 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f2f2345640 con 0x55f2f1fe7d80 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8df2ff5640 10 client.16533 handle_command_reply: tid=0 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 client.16533 shutdown 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/3540694986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 msgr2=0x55f2f23da580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23da580 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009cf0 tx=0x7f8e000038a0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/3540694986 shutdown_connections 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23db7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23dac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2f1fe7d80 0x55f2f23e7af0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/3540694986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23da580 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/3540694986 >> 172.21.15.29:0/3540694986 conn(0x55f2f201d300 msgr2=0x55f2f23caea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8df2ff5640 0 client.16533 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/3540694986 shutdown_connections 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.662+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/3540694986 wait complete. 2024-09-21T02:46:19.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.663+0000 7f8e1c4a1740 20 client.16533 trim_cache size 0 max 16384 2024-09-21T02:46:19.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e1c4a1740 1 Processor -- start 2024-09-21T02:46:19.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e1c4a1740 1 -- start start 2024-09-21T02:46:19.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23d1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101ac5a0 con 0x55f2f20002f0 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101ae0e0 con 0x55f2f20291b0 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101acc70 con 0x55f2f20084d0 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23d0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36296/0 (socket says 172.21.15.29:36296) 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e0effd640 1 -- 172.21.15.29:0/2388848255 learned_addr learned my addr 172.21.15.29:0/2388848255 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23d1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.682+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8e0effd640 1 -- 172.21.15.29:0/2388848255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 msgr2=0x55f2f23d1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8e0effd640 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23d1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8e0effd640 1 -- 172.21.15.29:0/2388848255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 msgr2=0x55f2f23d8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8e0effd640 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8e0effd640 1 -- 172.21.15.29:0/2388848255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2f23d9eb0 con 0x55f2f20002f0 2024-09-21T02:46:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8e0effd640 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23d0df0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8e00003fd0 tx=0x7f8e000096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65964a014585232 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8df4ff9640 1 -- 172.21.15.29:0/2388848255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00030070 con 0x55f2f20002f0 2024-09-21T02:46:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8df4ff9640 1 -- 172.21.15.29:0/2388848255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e000043d0 con 0x55f2f20002f0 2024-09-21T02:46:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.683+0000 7f8df4ff9640 1 -- 172.21.15.29:0/2388848255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00027040 con 0x55f2f20002f0 2024-09-21T02:46:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/2388848255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 msgr2=0x55f2f23d0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23d0df0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8e00003fd0 tx=0x7f8e000096b0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/2388848255 shutdown_connections 2024-09-21T02:46:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20291b0 0x55f2f23d8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20084d0 0x55f2f23d1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/2388848255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20002f0 0x55f2f23d0df0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/2388848255 >> 172.21.15.29:0/2388848255 conn(0x55f2f201d300 msgr2=0x55f2f23d97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/2388848255 shutdown_connections 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.684+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/2388848255 wait complete. 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 1 -- start start 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 1 Processor -- start 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20002f0 0x55f2f23dec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23df2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20291b0 0x55f2f23dfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101ada10 con 0x55f2f20084d0 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101ad340 con 0x55f2f20002f0 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e101aa580 con 0x55f2f20291b0 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e1c4a1740 10 client.0 _ll_register_callbacks cb 0x7f8e00009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23df2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20002f0 0x55f2f23dec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0effd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20002f0 0x55f2f23dec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:35134/0 (socket says 172.21.15.29:35134) 2024-09-21T02:46:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0effd640 1 -- 172.21.15.29:0/225658959 learned_addr learned my addr 172.21.15.29:0/225658959 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23df2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36304/0 (socket says 172.21.15.29:36304) 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20291b0 0x55f2f23dfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0e7fc640 1 -- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20291b0 msgr2=0x55f2f23dfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20291b0 0x55f2f23dfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0e7fc640 1 -- 172.21.15.29:0/225658959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20002f0 msgr2=0x55f2f23dec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20002f0 0x55f2f23dec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.685+0000 7f8e0e7fc640 1 -- 172.21.15.29:0/225658959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2f23e05a0 con 0x55f2f20084d0 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8e0effd640 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20002f0 0x55f2f23dec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8e0f7fe640 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20291b0 0x55f2f23dfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8e0e7fc640 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23df2d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8e040c9ee0 tx=0x7f8e040cd080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36038249e8b04218 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8df2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8df2ff5640 1 -- 172.21.15.29:0/225658959 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04049040 con 0x55f2f20084d0 2024-09-21T02:46:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8df2ff5640 1 -- 172.21.15.29:0/225658959 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e040d6ff0 con 0x55f2f20084d0 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/225658959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e000043d0 con 0x55f2f20084d0 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8df2ff5640 1 -- 172.21.15.29:0/225658959 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8e040c97b0 con 0x55f2f20084d0 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8e1c4a1740 10 client.16539 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/225658959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f2f23e1960 con 0x55f2f20084d0 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.686+0000 7f8df2ff5640 1 -- 172.21.15.29:0/225658959 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e000043d0 con 0x55f2f20084d0 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8df2ff5640 1 -- 172.21.15.29:0/225658959 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f8e040c9260 con 0x55f2f20084d0 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8e1c4a1740 10 client.16539 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8e1c4a1740 10 client.16539 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8e1c4a1740 20 client.16539 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2f1fbd1c0 0x55f2f24544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8e1c4a1740 4 client.16539 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/225658959 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2f23410f0 con 0x55f2f1fbd1c0 2024-09-21T02:46:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8e0effd640 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2f1fbd1c0 0x55f2f24544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:19.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8df2ff5640 10 client.16539 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:19.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.687+0000 7f8e0effd640 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2f1fbd1c0 0x55f2f24544c0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8d7501dfc3f3d0d server_cookie=7a0beb0e60bc2320 in_seq=0 out_seq=0 2024-09-21T02:46:19.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8df2ff5640 1 -- 172.21.15.29:0/225658959 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f2f23410f0 con 0x55f2f1fbd1c0 2024-09-21T02:46:19.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8df2ff5640 10 client.16539 handle_command_reply: tid=0 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 client.16539 shutdown 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/225658959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 msgr2=0x55f2f23df2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23df2d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8e040c9ee0 tx=0x7f8e040cd080 comp rx=0 tx=0).stop 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/225658959 shutdown_connections 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2f20291b0 0x55f2f23dfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2f1fbd1c0 0x55f2f24544c0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2f20084d0 0x55f2f23df2d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 --2- 172.21.15.29:0/225658959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2f20002f0 0x55f2f23dec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8df2ff5640 0 client.16539 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:19.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.734+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/225658959 >> 172.21.15.29:0/225658959 conn(0x55f2f201d300 msgr2=0x55f2f23d97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:19.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/225658959 shutdown_connections 2024-09-21T02:46:19.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e1c4a1740 1 -- 172.21.15.29:0/225658959 wait complete. 2024-09-21T02:46:19.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e1c4a1740 20 client.16539 trim_cache size 0 max 16384 2024-09-21T02:46:19.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:46:19.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:46:19.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:46:19.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:46:19.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 -- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8df80b6510 msgr2=0x7f8df80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8df80b6510 0x7f8df80b8a20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8e08005fd0 tx=0x7f8e08009500 comp rx=0 tx=0).stop 2024-09-21T02:46:19.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 -- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 msgr2=0x7f8e1019a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:19.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 0x7f8e1019a500 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009230 tx=0x7f8e04005e80 comp rx=0 tx=0).stop 2024-09-21T02:46:19.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 -- 172.21.15.29:0/3118754636 shutdown_connections 2024-09-21T02:46:19.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8df80b6510 0x7f8df80b8a20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e10109480 0x7f8e1019b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e10106b20 0x7f8e1019a500 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 --2- 172.21.15.29:0/3118754636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e10101a90 0x7f8e10199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:19.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 -- 172.21.15.29:0/3118754636 >> 172.21.15.29:0/3118754636 conn(0x7f8e100fd6d0 msgr2=0x7f8e10107a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:19.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 -- 172.21.15.29:0/3118754636 shutdown_connections 2024-09-21T02:46:19.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:19.735+0000 7f8e15532640 1 -- 172.21.15.29:0/3118754636 wait complete. 2024-09-21T02:46:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:19 smithi066 ceph-mon[33436]: pgmap v605: 129 pgs: 129 active+clean; 39 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:46:20.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 137157 2.43 MB/sec execute 237 sec latency 28.331 ms 2024-09-21T02:46:20.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:19 smithi190 ceph-mon[33871]: pgmap v605: 129 pgs: 129 active+clean; 39 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:46:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:19 smithi029 ceph-mon[31006]: pgmap v605: 129 pgs: 129 active+clean; 39 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:46:21.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 138006 2.44 MB/sec execute 238 sec latency 19.729 ms 2024-09-21T02:46:21.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:21 smithi190 ceph-mon[33871]: pgmap v606: 129 pgs: 129 active+clean; 35 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T02:46:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:21 smithi066 ceph-mon[33436]: pgmap v606: 129 pgs: 129 active+clean; 35 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T02:46:22.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 138506 2.45 MB/sec execute 239 sec latency 14.604 ms 2024-09-21T02:46:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:21 smithi029 ceph-mon[31006]: pgmap v606: 129 pgs: 129 active+clean; 35 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T02:46:23.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 138761 2.45 MB/sec execute 240 sec latency 71.991 ms 2024-09-21T02:46:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:23 smithi066 ceph-mon[33436]: pgmap v607: 129 pgs: 129 active+clean; 35 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-09-21T02:46:24.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 139173 2.44 MB/sec execute 241 sec latency 110.837 ms 2024-09-21T02:46:24.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:23 smithi190 ceph-mon[33871]: pgmap v607: 129 pgs: 129 active+clean; 35 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-09-21T02:46:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:23 smithi029 ceph-mon[31006]: pgmap v607: 129 pgs: 129 active+clean; 35 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-09-21T02:46:25.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 139836 2.45 MB/sec execute 242 sec latency 16.838 ms 2024-09-21T02:46:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:25 smithi066 ceph-mon[33436]: pgmap v608: 129 pgs: 129 active+clean; 36 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-09-21T02:46:26.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 140458 2.44 MB/sec execute 243 sec latency 18.844 ms 2024-09-21T02:46:26.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:25 smithi190 ceph-mon[33871]: pgmap v608: 129 pgs: 129 active+clean; 36 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-09-21T02:46:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:25 smithi029 ceph-mon[31006]: pgmap v608: 129 pgs: 129 active+clean; 36 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-09-21T02:46:27.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 141147 2.45 MB/sec execute 244 sec latency 21.040 ms 2024-09-21T02:46:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:27 smithi066 ceph-mon[33436]: pgmap v609: 129 pgs: 129 active+clean; 36 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:46:28.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 141867 2.47 MB/sec execute 245 sec latency 15.715 ms 2024-09-21T02:46:28.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:27 smithi190 ceph-mon[33871]: pgmap v609: 129 pgs: 129 active+clean; 36 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:46:28.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:27 smithi029 ceph-mon[31006]: pgmap v609: 129 pgs: 129 active+clean; 36 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:46:29.084 INFO:tasks.workunit.client.0.smithi029.stdout: 1 142286 2.47 MB/sec execute 246 sec latency 13.693 ms 2024-09-21T02:46:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:29 smithi066 ceph-mon[33436]: pgmap v610: 129 pgs: 129 active+clean; 36 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:46:30.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 142700 2.46 MB/sec execute 247 sec latency 28.932 ms 2024-09-21T02:46:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:29 smithi190 ceph-mon[33871]: pgmap v610: 129 pgs: 129 active+clean; 36 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:46:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:29 smithi029 ceph-mon[31006]: pgmap v610: 129 pgs: 129 active+clean; 36 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:46:31.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 143367 2.47 MB/sec execute 248 sec latency 14.608 ms 2024-09-21T02:46:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:31 smithi066 ceph-mon[33436]: pgmap v611: 129 pgs: 129 active+clean; 37 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-09-21T02:46:32.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 144007 2.46 MB/sec execute 249 sec latency 19.295 ms 2024-09-21T02:46:32.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:31 smithi190 ceph-mon[33871]: pgmap v611: 129 pgs: 129 active+clean; 37 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-09-21T02:46:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:31 smithi029 ceph-mon[31006]: pgmap v611: 129 pgs: 129 active+clean; 37 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-09-21T02:46:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.860+0000 7fb2a69ed640 1 -- 172.21.15.29:0/4125785987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a01025a0 msgr2=0x7fb2a0102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.860+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/4125785987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a01025a0 0x7fb2a0102a10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb290009900 tx=0x7fb29002a410 comp rx=0 tx=0).stop 2024-09-21T02:46:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.860+0000 7fb2a69ed640 1 -- 172.21.15.29:0/4125785987 shutdown_connections 2024-09-21T02:46:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.860+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/4125785987 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a0103fc0 0x7fb2a010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.860+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/4125785987 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2a0103620 0x7fb2a0103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.860+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/4125785987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a01025a0 0x7fb2a0102a10 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.860+0000 7fb2a69ed640 1 -- 172.21.15.29:0/4125785987 >> 172.21.15.29:0/4125785987 conn(0x7fb2a00fdee0 msgr2=0x7fb2a0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.860+0000 7fb2a69ed640 1 -- 172.21.15.29:0/4125785987 shutdown_connections 2024-09-21T02:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.861+0000 7fb2a69ed640 1 -- 172.21.15.29:0/4125785987 wait complete. 2024-09-21T02:46:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.861+0000 7fb2a69ed640 1 Processor -- start 2024-09-21T02:46:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.861+0000 7fb2a69ed640 1 -- start start 2024-09-21T02:46:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a69ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a01025a0 0x7fb2a0199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a69ed640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2a0103620 0x7fb2a019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a69ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a0103fc0 0x7fb2a019b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a69ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a010e080 con 0x7fb2a0103fc0 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a69ed640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a010df00 con 0x7fb2a0103620 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a69ed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a010e200 con 0x7fb2a01025a0 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a0103fc0 0x7fb2a019b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb29ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a01025a0 0x7fb2a0199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a0103fc0 0x7fb2a019b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36686/0 (socket says 172.21.15.29:36686) 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/1510174124 learned_addr learned my addr 172.21.15.29:0/1510174124 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.862+0000 7fb29f7fe640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2a0103620 0x7fb2a019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.863+0000 7fb29ffff640 1 -- 172.21.15.29:0/1510174124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2a0103620 msgr2=0x7fb2a019a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.863+0000 7fb29ffff640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2a0103620 0x7fb2a019a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.863+0000 7fb29ffff640 1 -- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a0103fc0 msgr2=0x7fb2a019b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.863+0000 7fb29ffff640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a0103fc0 0x7fb2a019b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.863+0000 7fb29ffff640 1 -- 172.21.15.29:0/1510174124 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a019b8a0 con 0x7fb2a01025a0 2024-09-21T02:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.864+0000 7fb29ffff640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a01025a0 0x7fb2a0199e60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb290005760 tx=0x7fb290005e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.864+0000 7fb29d7fa640 1 -- 172.21.15.29:0/1510174124 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb290035050 con 0x7fb2a01025a0 2024-09-21T02:46:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.864+0000 7fb2a69ed640 1 -- 172.21.15.29:0/1510174124 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a019bb30 con 0x7fb2a01025a0 2024-09-21T02:46:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.864+0000 7fb29d7fa640 1 -- 172.21.15.29:0/1510174124 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb290004350 con 0x7fb2a01025a0 2024-09-21T02:46:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.864+0000 7fb29d7fa640 1 -- 172.21.15.29:0/1510174124 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb290003a50 con 0x7fb2a01025a0 2024-09-21T02:46:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.864+0000 7fb2a69ed640 1 -- 172.21.15.29:0/1510174124 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a0104b10 con 0x7fb2a01025a0 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb2ada36740 1 Processor -- start 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb2ada36740 1 -- start start 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb29d7fa640 1 -- 172.21.15.29:0/1510174124 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb29003ac20 con 0x7fb2a01025a0 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b7ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b7d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb2ada36740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e1c2ae66f0 con 0x55e1c27a32f0 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb2ada36740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e1c2ad5b20 con 0x55e1c27cc1b0 2024-09-21T02:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.867+0000 7fb2ada36740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e1c2ad9930 con 0x55e1c27ab4d0 2024-09-21T02:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:40106/0 (socket says 172.21.15.29:40106) 2024-09-21T02:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/1107742791 learned_addr learned my addr 172.21.15.29:0/1107742791 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb29f7fe640 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b7d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb29ffff640 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b7ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/1107742791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 msgr2=0x55e1c2b7d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b7d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/1107742791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 msgr2=0x55e1c2b7ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b7ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/1107742791 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1c2a9b530 con 0x55e1c27cc1b0 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb29f7fe640 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b7d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.868+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7c660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb298011570 tx=0x7fb298011a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a66c2a7f750fbc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.869+0000 7fb2867fc640 1 -- 172.21.15.29:0/1107742791 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29801e070 con 0x55e1c27cc1b0 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.869+0000 7fb2867fc640 1 -- 172.21.15.29:0/1107742791 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb29800fad0 con 0x55e1c27cc1b0 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.869+0000 7fb2867fc640 1 -- 172.21.15.29:0/1107742791 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb298019c00 con 0x55e1c27cc1b0 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.869+0000 7fb29d7fa640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb2880b6590 0x7fb2880b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.869+0000 7fb29ffff640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb2880b6590 0x7fb2880b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.869+0000 7fb29d7fa640 1 -- 172.21.15.29:0/1510174124 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb29010b2d0 con 0x7fb2a01025a0 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 -- 172.21.15.29:0/1107742791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 msgr2=0x55e1c2b7c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7c660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb298011570 tx=0x7fb298011a30 comp rx=0 tx=0).stop 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb29ffff640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb2880b6590 0x7fb2880b8aa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb2900cd650 tx=0x7fb29010f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 -- 172.21.15.29:0/1107742791 shutdown_connections 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b7d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b7ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 --2- 172.21.15.29:0/1107742791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7c660 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 -- 172.21.15.29:0/1107742791 >> 172.21.15.29:0/1107742791 conn(0x55e1c27c0300 msgr2=0x55e1c2b6f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 -- 172.21.15.29:0/1107742791 shutdown_connections 2024-09-21T02:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 -- 172.21.15.29:0/1107742791 wait complete. 2024-09-21T02:46:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.870+0000 7fb2ada36740 1 -- start start 2024-09-21T02:46:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2ada36740 1 Processor -- start 2024-09-21T02:46:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b85e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b86500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b85e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b85e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36694/0 (socket says 172.21.15.29:36694) 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b87070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb29f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b86500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/703757294 learned_addr learned my addr 172.21.15.29:0/703757294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e1c2a0dab0 con 0x55e1c27a32f0 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e1c2a7b730 con 0x55e1c27cc1b0 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e1c2b7e340 con 0x55e1c27ab4d0 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.871+0000 7fb2ada36740 10 client.0 _ll_register_callbacks cb 0x55e1c2acf6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb29f7fe640 1 -- 172.21.15.29:0/703757294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 msgr2=0x55e1c2b87070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb29f7fe640 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b87070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb29f7fe640 1 -- 172.21.15.29:0/703757294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 msgr2=0x55e1c2b85e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb29f7fe640 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b85e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb29f7fe640 1 -- 172.21.15.29:0/703757294 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1c2837790 con 0x55e1c27ab4d0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb29f7fe640 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b86500 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb294012700 tx=0x7fb294012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c06a0284ddf1dda8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb27f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb27f7fe640 1 -- 172.21.15.29:0/703757294 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294004050 con 0x55e1c27ab4d0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.872+0000 7fb27f7fe640 1 -- 172.21.15.29:0/703757294 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb2940041f0 con 0x55e1c27ab4d0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.873+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb29800fad0 con 0x55e1c27ab4d0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.873+0000 7fb27f7fe640 1 -- 172.21.15.29:0/703757294 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb294025050 con 0x55e1c27ab4d0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.873+0000 7fb2ada36740 10 client.26146 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.873+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1c28b0620 con 0x55e1c27ab4d0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.873+0000 7fb27f7fe640 1 -- 172.21.15.29:0/703757294 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb29800fad0 con 0x55e1c27ab4d0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.873+0000 7fb27f7fe640 1 -- 172.21.15.29:0/703757294 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb294003880 con 0x55e1c27ab4d0 2024-09-21T02:46:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.873+0000 7fb2ada36740 10 client.26146 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.873+0000 7fb2ada36740 10 client.26146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.874+0000 7fb2ada36740 20 client.26146 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.874+0000 7fb2ada36740 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1c278ad80 0x55e1c2b93610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.874+0000 7fb2ada36740 4 client.26146 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.874+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1c2ae8640 con 0x55e1c278ad80 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.874+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1c278ad80 0x55e1c2b93610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.874+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1c278ad80 0x55e1c2b93610 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bacb993aae4648da server_cookie=825d420867c17f93 in_seq=0 out_seq=0 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.874+0000 7fb27f7fe640 10 client.26146 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.875+0000 7fb27f7fe640 1 -- 172.21.15.29:0/703757294 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e1c2ae8640 con 0x55e1c278ad80 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.875+0000 7fb27f7fe640 10 client.26146 handle_command_reply: tid=0 2024-09-21T02:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.875+0000 7fb2ada36740 1 client.26146 shutdown 2024-09-21T02:46:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 msgr2=0x55e1c2b86500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b86500 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb294012700 tx=0x7fb294012bc0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 shutdown_connections 2024-09-21T02:46:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b87070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b86500 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1c278ad80 0x55e1c2b93610 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb27f7fe640 0 client.26146 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 --2- 172.21.15.29:0/703757294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27a32f0 0x55e1c2b85e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 >> 172.21.15.29:0/703757294 conn(0x55e1c27c0300 msgr2=0x55e1c2b7cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 shutdown_connections 2024-09-21T02:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 1 -- 172.21.15.29:0/703757294 wait complete. 2024-09-21T02:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.876+0000 7fb2ada36740 20 client.26146 trim_cache size 0 max 16384 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.893+0000 7fb2ada36740 1 Processor -- start 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.893+0000 7fb2ada36740 1 -- start start 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.893+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27a32f0 0x55e1c2b81180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.893+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b6e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.893+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2ada36740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a01aa310 con 0x55e1c27ab4d0 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2ada36740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a01abe50 con 0x55e1c27cc1b0 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2ada36740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a01aa9e0 con 0x55e1c27a32f0 2024-09-21T02:46:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb29f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b6e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27a32f0 0x55e1c2b81180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27a32f0 0x55e1c2b81180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45968/0 (socket says 172.21.15.29:45968) 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb29f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b6e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36704/0 (socket says 172.21.15.29:36704) 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/2366415568 learned_addr learned my addr 172.21.15.29:0/2366415568 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/2366415568 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/2366415568 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 msgr2=0x55e1c2b7e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/2366415568 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/2366415568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 msgr2=0x55e1c2b6e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/2366415568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b6e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/2366415568 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1c2b857a0 con 0x55e1c27a32f0 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.894+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/2366415568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27a32f0 0x55e1c2b81180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb298011540 tx=0x7fb298011a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3efe98a47de274c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.895+0000 7fb285ffb640 1 -- 172.21.15.29:0/2366415568 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29801e040 con 0x55e1c27a32f0 2024-09-21T02:46:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.895+0000 7fb285ffb640 1 -- 172.21.15.29:0/2366415568 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb29800f640 con 0x55e1c27a32f0 2024-09-21T02:46:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.895+0000 7fb285ffb640 1 -- 172.21.15.29:0/2366415568 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2980199f0 con 0x55e1c27a32f0 2024-09-21T02:46:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 -- 172.21.15.29:0/2366415568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27a32f0 msgr2=0x55e1c2b81180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2366415568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27a32f0 0x55e1c2b81180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb298011540 tx=0x7fb298011a30 comp rx=0 tx=0).stop 2024-09-21T02:46:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 -- 172.21.15.29:0/2366415568 shutdown_connections 2024-09-21T02:46:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2366415568 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b7e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2366415568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b6e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2366415568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27a32f0 0x55e1c2b81180 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 -- 172.21.15.29:0/2366415568 >> 172.21.15.29:0/2366415568 conn(0x55e1c27c0300 msgr2=0x55e1c2b6dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 -- 172.21.15.29:0/2366415568 shutdown_connections 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 -- 172.21.15.29:0/2366415568 wait complete. 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 -- start start 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.896+0000 7fb2ada36740 1 Processor -- start 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27a32f0 0x55e1c2b8aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b8b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2ada36740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b8c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2ada36740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a01ab780 con 0x55e1c27ab4d0 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2ada36740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a01ab0b0 con 0x55e1c27a32f0 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2ada36740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a01a82f0 con 0x55e1c27cc1b0 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2ada36740 10 client.0 _ll_register_callbacks cb 0x7fb294009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27a32f0 0x55e1c2b8aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb29f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b8b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27a32f0 0x55e1c2b8aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:40128/0 (socket says 172.21.15.29:40128) 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/2946138685 learned_addr learned my addr 172.21.15.29:0/2946138685 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b8c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/2946138685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27cc1b0 msgr2=0x55e1c2b8c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b8c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/2946138685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 msgr2=0x55e1c2b8b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b8b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.897+0000 7fb2a4f8a640 1 -- 172.21.15.29:0/2946138685 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1c2b8c830 con 0x55e1c27a32f0 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.898+0000 7fb2a4f8a640 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27a32f0 0x55e1c2b8aea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb29802a040 tx=0x7fb298009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6dfe3bece4508ed server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.898+0000 7fb27f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.898+0000 7fb27f7fe640 1 -- 172.21.15.29:0/2946138685 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29801e040 con 0x55e1c27a32f0 2024-09-21T02:46:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.898+0000 7fb27f7fe640 1 -- 172.21.15.29:0/2946138685 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb298004b40 con 0x55e1c27a32f0 2024-09-21T02:46:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.898+0000 7fb2ada36740 1 -- 172.21.15.29:0/2946138685 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb29800f640 con 0x55e1c27a32f0 2024-09-21T02:46:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.898+0000 7fb27f7fe640 1 -- 172.21.15.29:0/2946138685 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb298004d40 con 0x55e1c27a32f0 2024-09-21T02:46:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.898+0000 7fb2ada36740 10 client.26042 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.898+0000 7fb2ada36740 1 -- 172.21.15.29:0/2946138685 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1c2b8db20 con 0x55e1c27a32f0 2024-09-21T02:46:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb27f7fe640 1 -- 172.21.15.29:0/2946138685 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb29800f640 con 0x55e1c27a32f0 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb27f7fe640 1 -- 172.21.15.29:0/2946138685 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb298005040 con 0x55e1c27a32f0 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb2ada36740 10 client.26042 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb2ada36740 10 client.26042 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb2ada36740 20 client.26042 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1c27601c0 0x55e1c2bfffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb2ada36740 4 client.26042 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb2ada36740 1 -- 172.21.15.29:0/2946138685 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e1c2ae40f0 con 0x55e1c27601c0 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.899+0000 7fb29f7fe640 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1c27601c0 0x55e1c2bfffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.900+0000 7fb27f7fe640 10 client.26042 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.900+0000 7fb29f7fe640 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1c27601c0 0x55e1c2bfffe0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68fe4130f4dc9f1b server_cookie=522638a4ce0a298f in_seq=0 out_seq=0 2024-09-21T02:46:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.942+0000 7fb27f7fe640 1 -- 172.21.15.29:0/2946138685 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e1c2ae40f0 con 0x55e1c27601c0 2024-09-21T02:46:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.942+0000 7fb27f7fe640 10 client.26042 handle_command_reply: tid=0 2024-09-21T02:46:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.942+0000 7fb2ada36740 1 client.26042 shutdown 2024-09-21T02:46:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 -- 172.21.15.29:0/2946138685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27a32f0 msgr2=0x55e1c2b8aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27a32f0 0x55e1c2b8aea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb29802a040 tx=0x7fb298009ea0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 -- 172.21.15.29:0/2946138685 shutdown_connections 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1c27cc1b0 0x55e1c2b8c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1c27601c0 0x55e1c2bfffe0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1c27ab4d0 0x55e1c2b8b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 --2- 172.21.15.29:0/2946138685 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1c27a32f0 0x55e1c2b8aea0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb27f7fe640 0 client.26042 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 -- 172.21.15.29:0/2946138685 >> 172.21.15.29:0/2946138685 conn(0x55e1c27c0300 msgr2=0x55e1c2b97dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 -- 172.21.15.29:0/2946138685 shutdown_connections 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 1 -- 172.21.15.29:0/2946138685 wait complete. 2024-09-21T02:46:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.943+0000 7fb2ada36740 20 client.26042 trim_cache size 0 max 16384 2024-09-21T02:46:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:46:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:46:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:46:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 -- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb2880b6590 msgr2=0x7fb2880b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb2880b6590 0x7fb2880b8aa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb2900cd650 tx=0x7fb29010f040 comp rx=0 tx=0).stop 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 -- 172.21.15.29:0/1510174124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a01025a0 msgr2=0x7fb2a0199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a01025a0 0x7fb2a0199e60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb290005760 tx=0x7fb290005e70 comp rx=0 tx=0).stop 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 -- 172.21.15.29:0/1510174124 shutdown_connections 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb2880b6590 0x7fb2880b8aa0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a0103fc0 0x7fb2a019b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2a0103620 0x7fb2a019a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 --2- 172.21.15.29:0/1510174124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a01025a0 0x7fb2a0199e60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 -- 172.21.15.29:0/1510174124 >> 172.21.15.29:0/1510174124 conn(0x7fb2a00fdee0 msgr2=0x7fb2a00ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 -- 172.21.15.29:0/1510174124 shutdown_connections 2024-09-21T02:46:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:32.944+0000 7fb2a69ed640 1 -- 172.21.15.29:0/1510174124 wait complete. 2024-09-21T02:46:33.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:32 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:46:33.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:32 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:46:33.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 144627 2.47 MB/sec execute 250 sec latency 23.130 ms 2024-09-21T02:46:33.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:32 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:46:34.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:33 smithi066 ceph-mon[33436]: pgmap v612: 129 pgs: 129 active+clean; 39 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-09-21T02:46:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:46:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:46:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:46:34.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 145266 2.47 MB/sec execute 251 sec latency 17.091 ms 2024-09-21T02:46:34.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:33 smithi190 ceph-mon[33871]: pgmap v612: 129 pgs: 129 active+clean; 39 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-09-21T02:46:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:46:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:46:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:46:34.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:33 smithi029 ceph-mon[31006]: pgmap v612: 129 pgs: 129 active+clean; 39 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.5 MiB/s wr, 129 op/s 2024-09-21T02:46:34.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:46:34.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:46:34.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:46:35.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 145783 2.48 MB/sec execute 252 sec latency 14.620 ms 2024-09-21T02:46:36.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:35 smithi066 ceph-mon[33436]: pgmap v613: 129 pgs: 129 active+clean; 35 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:46:36.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 146136 2.47 MB/sec execute 253 sec latency 17.266 ms 2024-09-21T02:46:36.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:35 smithi190 ceph-mon[33871]: pgmap v613: 129 pgs: 129 active+clean; 35 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:46:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:35 smithi029 ceph-mon[31006]: pgmap v613: 129 pgs: 129 active+clean; 35 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 1.5 MiB/s wr, 141 op/s 2024-09-21T02:46:37.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 146843 2.48 MB/sec execute 254 sec latency 13.892 ms 2024-09-21T02:46:38.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:37 smithi066 ceph-mon[33436]: pgmap v614: 129 pgs: 129 active+clean; 35 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 1.6 MiB/s wr, 108 op/s 2024-09-21T02:46:38.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 147467 2.48 MB/sec execute 255 sec latency 20.085 ms 2024-09-21T02:46:38.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:37 smithi190 ceph-mon[33871]: pgmap v614: 129 pgs: 129 active+clean; 35 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 1.6 MiB/s wr, 108 op/s 2024-09-21T02:46:38.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:37 smithi029 ceph-mon[31006]: pgmap v614: 129 pgs: 129 active+clean; 35 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 1.6 MiB/s wr, 108 op/s 2024-09-21T02:46:39.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 148071 2.48 MB/sec execute 256 sec latency 23.234 ms 2024-09-21T02:46:39.367 INFO:tasks.quiescer.fs.[cephfs]:Total duration (42.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:46:39.367 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.05 2024-09-21T02:46:39.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:46:39.656 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:46:39.950 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:39 smithi029 ceph-mon[31006]: pgmap v615: 129 pgs: 129 active+clean; 36 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:46:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:39 smithi066 ceph-mon[33436]: pgmap v615: 129 pgs: 129 active+clean; 36 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:46:40.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 148782 2.49 MB/sec execute 257 sec latency 67.899 ms 2024-09-21T02:46:40.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:39 smithi190 ceph-mon[33871]: pgmap v615: 129 pgs: 129 active+clean; 36 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:46:40.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.174+0000 7fc180394640 1 -- 172.21.15.29:0/771322408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc178108480 msgr2=0x7fc17810ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:40.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.174+0000 7fc180394640 1 --2- 172.21.15.29:0/771322408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc178108480 0x7fc17810ecb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc170009ca0 tx=0x7fc17002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.174+0000 7fc180394640 1 -- 172.21.15.29:0/771322408 shutdown_connections 2024-09-21T02:46:40.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.174+0000 7fc180394640 1 --2- 172.21.15.29:0/771322408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc178108480 0x7fc17810ecb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.174+0000 7fc180394640 1 --2- 172.21.15.29:0/771322408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc178107b30 0x7fc178107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.174+0000 7fc180394640 1 --2- 172.21.15.29:0/771322408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1781021f0 0x7fc178102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.174+0000 7fc180394640 1 -- 172.21.15.29:0/771322408 >> 172.21.15.29:0/771322408 conn(0x7fc178077c60 msgr2=0x7fc1780780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.175+0000 7fc180394640 1 -- 172.21.15.29:0/771322408 shutdown_connections 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.175+0000 7fc180394640 1 -- 172.21.15.29:0/771322408 wait complete. 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.175+0000 7fc180394640 1 Processor -- start 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.175+0000 7fc180394640 1 -- start start 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc180394640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1781021f0 0x7fc17819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc180394640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178107b30 0x7fc17819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc180394640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc178108480 0x7fc17819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc180394640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc1781139d0 con 0x7fc1781021f0 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e130640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1781021f0 0x7fc17819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc180394640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc178113850 con 0x7fc178108480 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc180394640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc178113b50 con 0x7fc178107b30 2024-09-21T02:46:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e130640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1781021f0 0x7fc17819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33902/0 (socket says 172.21.15.29:33902) 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17d92f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178107b30 0x7fc17819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e130640 1 -- 172.21.15.29:0/3844467514 learned_addr learned my addr 172.21.15.29:0/3844467514 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e931640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc178108480 0x7fc17819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e130640 1 -- 172.21.15.29:0/3844467514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178107b30 msgr2=0x7fc17819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e130640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178107b30 0x7fc17819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e130640 1 -- 172.21.15.29:0/3844467514 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc178108480 msgr2=0x7fc17819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e130640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc178108480 0x7fc17819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.176+0000 7fc17e130640 1 -- 172.21.15.29:0/3844467514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1781a02b0 con 0x7fc1781021f0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.177+0000 7fc17e130640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1781021f0 0x7fc17819a340 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc17400e9f0 tx=0x7fc17400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.177+0000 7fc16f7fe640 1 -- 172.21.15.29:0/3844467514 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17400cd80 con 0x7fc1781021f0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.177+0000 7fc16f7fe640 1 -- 172.21.15.29:0/3844467514 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1740054b0 con 0x7fc1781021f0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.177+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1781a05a0 con 0x7fc1781021f0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.177+0000 7fc16f7fe640 1 -- 172.21.15.29:0/3844467514 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174010430 con 0x7fc1781021f0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.177+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1781a6d00 con 0x7fc1781021f0 2024-09-21T02:46:40.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.178+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc178102600 con 0x7fc1781021f0 2024-09-21T02:46:40.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.180+0000 7fc16f7fe640 1 -- 172.21.15.29:0/3844467514 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc1740046c0 con 0x7fc1781021f0 2024-09-21T02:46:40.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.181+0000 7fc16f7fe640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1640b6590 0x7fc1640b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:40.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.181+0000 7fc17d92f640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1640b6590 0x7fc1640b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:40.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.181+0000 7fc16f7fe640 1 -- 172.21.15.29:0/3844467514 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc1740e7b40 con 0x7fc1781021f0 2024-09-21T02:46:40.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.182+0000 7fc17d92f640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1640b6590 0x7fc1640b8aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc178107830 tx=0x7fc168005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:40.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.185+0000 7fc16f7fe640 1 -- 172.21.15.29:0/3844467514 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc1740ae050 con 0x7fc1781021f0 2024-09-21T02:46:40.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.444+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc17819b470 con 0x7fc1781021f0 2024-09-21T02:46:40.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.445+0000 7fc16f7fe640 1 -- 172.21.15.29:0/3844467514 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7fc1740b2430 con 0x7fc1781021f0 2024-09-21T02:46:40.447 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:46:40.447 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:46:40.447 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:46:40.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1640b6590 msgr2=0x7fc1640b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:40.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1640b6590 0x7fc1640b8aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc178107830 tx=0x7fc168005f90 comp rx=0 tx=0).stop 2024-09-21T02:46:40.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1781021f0 msgr2=0x7fc17819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:40.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1781021f0 0x7fc17819a340 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc17400e9f0 tx=0x7fc17400edf0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 shutdown_connections 2024-09-21T02:46:40.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1640b6590 0x7fc1640b8aa0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc178108480 0x7fc17819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc178107b30 0x7fc17819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 --2- 172.21.15.29:0/3844467514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1781021f0 0x7fc17819a340 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:40.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 >> 172.21.15.29:0/3844467514 conn(0x7fc178077c60 msgr2=0x7fc178105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:40.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 shutdown_connections 2024-09-21T02:46:40.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:40.448+0000 7fc180394640 1 -- 172.21.15.29:0/3844467514 wait complete. 2024-09-21T02:46:41.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:40 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3844467514' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:46:41.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:40 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3844467514' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:46:41.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149287 2.50 MB/sec execute 258 sec latency 16.524 ms 2024-09-21T02:46:41.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:40 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3844467514' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:46:41.300 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0' 2024-09-21T02:46:41.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0 2024-09-21T02:46:41.592 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:46:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:41 smithi066 ceph-mon[33436]: pgmap v616: 129 pgs: 129 active+clean; 35 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:46:42.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:41 smithi029 ceph-mon[31006]: pgmap v616: 129 pgs: 129 active+clean; 35 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:46:42.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149537 2.49 MB/sec execute 259 sec latency 44.331 ms 2024-09-21T02:46:42.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:41 smithi190 ceph-mon[33871]: pgmap v616: 129 pgs: 129 active+clean; 35 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:46:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 -- 172.21.15.29:0/2730133525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 msgr2=0x7faa04104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 --2- 172.21.15.29:0/2730133525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 0x7faa04104c50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7faa00009900 tx=0x7faa0002a410 comp rx=0 tx=0).stop 2024-09-21T02:46:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 -- 172.21.15.29:0/2730133525 shutdown_connections 2024-09-21T02:46:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 --2- 172.21.15.29:0/2730133525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa0410ddf0 0x7faa04110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 --2- 172.21.15.29:0/2730133525 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa04105270 0x7faa0410d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 --2- 172.21.15.29:0/2730133525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 0x7faa04104c50 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 -- 172.21.15.29:0/2730133525 >> 172.21.15.29:0/2730133525 conn(0x7faa040796d0 msgr2=0x7faa04079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 -- 172.21.15.29:0/2730133525 shutdown_connections 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.158+0000 7faa0c229640 1 -- 172.21.15.29:0/2730133525 wait complete. 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0c229640 1 Processor -- start 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0c229640 1 -- start start 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0c229640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 0x7faa04102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0c229640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa04105270 0x7faa04102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0c229640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa0410ddf0 0x7faa041031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0c229640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa04113920 con 0x7faa04104840 2024-09-21T02:46:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0c229640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faa041137a0 con 0x7faa04105270 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0c229640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa04113aa0 con 0x7faa0410ddf0 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 0x7faa04102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa097c4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa04105270 0x7faa04102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa097c4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa04105270 0x7faa04102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:50324/0 (socket says 172.21.15.29:50324) 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 0x7faa04102750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33910/0 (socket says 172.21.15.29:33910) 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa097c4640 1 -- 172.21.15.29:0/1130158027 learned_addr learned my addr 172.21.15.29:0/1130158027 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.159+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa0410ddf0 0x7faa041031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa09fc5640 1 -- 172.21.15.29:0/1130158027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa0410ddf0 msgr2=0x7faa041031d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa0410ddf0 0x7faa041031d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa09fc5640 1 -- 172.21.15.29:0/1130158027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa04105270 msgr2=0x7faa04102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa04105270 0x7faa04102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa09fc5640 1 -- 172.21.15.29:0/1130158027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa041a86f0 con 0x7faa04104840 2024-09-21T02:46:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa097c4640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa04105270 0x7faa04102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 0x7faa04102750 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7faa00005940 tx=0x7faa00005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7fa9f6ffd640 1 -- 172.21.15.29:0/1130158027 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0003f070 con 0x7faa04104840 2024-09-21T02:46:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa0c229640 1 -- 172.21.15.29:0/1130158027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa041a8980 con 0x7faa04104840 2024-09-21T02:46:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7fa9f6ffd640 1 -- 172.21.15.29:0/1130158027 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa00003a80 con 0x7faa04104840 2024-09-21T02:46:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7fa9f6ffd640 1 -- 172.21.15.29:0/1130158027 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00035070 con 0x7faa04104840 2024-09-21T02:46:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.160+0000 7faa0c229640 1 -- 172.21.15.29:0/1130158027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa041a8eb0 con 0x7faa04104840 2024-09-21T02:46:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.163+0000 7fa9f6ffd640 1 -- 172.21.15.29:0/1130158027 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7faa000035f0 con 0x7faa04104840 2024-09-21T02:46:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.163+0000 7faa1279d740 1 Processor -- start 2024-09-21T02:46:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa1279d740 1 -- start start 2024-09-21T02:46:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa1279d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75996850 0x555c75d3ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa1279d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d3d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa1279d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d3d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa1279d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555c75cb3bb0 con 0x555c7596d990 2024-09-21T02:46:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa1279d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555c75ca2fe0 con 0x555c75975b70 2024-09-21T02:46:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa1279d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555c75ca6df0 con 0x555c75996850 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d3d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.164+0000 7faa097c4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d3d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa097c4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d3d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:50340/0 (socket says 172.21.15.29:50340) 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa097c4640 1 -- 172.21.15.29:0/231971956 learned_addr learned my addr 172.21.15.29:0/231971956 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7fa9f6ffd640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9f00b6510 0x7fa9f00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa097c4640 1 -- 172.21.15.29:0/231971956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75996850 msgr2=0x555c75d3ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75996850 0x555c75d3ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa097c4640 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75996850 0x555c75d3ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa097c4640 1 -- 172.21.15.29:0/231971956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 msgr2=0x555c75d3d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa097c4640 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d3d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa097c4640 1 -- 172.21.15.29:0/231971956 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c75a10ba0 con 0x555c75975b70 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75996850 0x555c75d3ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9f00b6510 0x7fa9f00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7fa9f6ffd640 1 -- 172.21.15.29:0/1130158027 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faa00107800 con 0x7faa04104840 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.165+0000 7faa097c4640 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d3d8f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8011540 tx=0x7fa9f8011940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15e769fbc020eef7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.166+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9f00b6510 0x7fa9f00b8a20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc005c40 tx=0x7fa9fc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.166+0000 7fa9effff640 1 -- 172.21.15.29:0/231971956 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f801e070 con 0x555c75975b70 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.166+0000 7fa9effff640 1 -- 172.21.15.29:0/231971956 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9f800fa50 con 0x555c75975b70 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.166+0000 7fa9effff640 1 -- 172.21.15.29:0/231971956 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8019af0 con 0x555c75975b70 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 -- 172.21.15.29:0/231971956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 msgr2=0x555c75d3d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d3d8f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8011540 tx=0x7fa9f8011940 comp rx=0 tx=0).stop 2024-09-21T02:46:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 -- 172.21.15.29:0/231971956 shutdown_connections 2024-09-21T02:46:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d3d8f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d3d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 --2- 172.21.15.29:0/231971956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75996850 0x555c75d3ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 -- 172.21.15.29:0/231971956 >> 172.21.15.29:0/231971956 conn(0x555c7598ac40 msgr2=0x555c75d3ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 -- 172.21.15.29:0/231971956 shutdown_connections 2024-09-21T02:46:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 -- 172.21.15.29:0/231971956 wait complete. 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 -- start start 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.167+0000 7faa1279d740 1 Processor -- start 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa1279d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d489d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa1279d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75975b70 0x555c75d48f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa1279d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75996850 0x555c75d54940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa1279d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555c75b047b0 con 0x555c7596d990 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa1279d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555c75c3de90 con 0x555c75996850 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa1279d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555c75d4b650 con 0x555c75975b70 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d489d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33918/0 (socket says 172.21.15.29:33918) 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa09fc5640 1 -- 172.21.15.29:0/3764848454 learned_addr learned my addr 172.21.15.29:0/3764848454 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75996850 0x555c75d54940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa097c4640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75975b70 0x555c75d48f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa1279d740 10 client.0 _ll_register_callbacks cb 0x555c75ca0710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa0a7c6640 1 -- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75975b70 msgr2=0x555c75d48f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.168+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75975b70 0x555c75d48f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa0a7c6640 1 -- 172.21.15.29:0/3764848454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 msgr2=0x555c75d489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d489d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa0a7c6640 1 -- 172.21.15.29:0/3764848454 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c75d54eb0 con 0x555c75996850 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa097c4640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75975b70 0x555c75d48f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa09fc5640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75996850 0x555c75d54940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009e20 tx=0x7fa9fc009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8767bf743af809f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7fa9ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/3764848454 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc01cb60 con 0x555c75996850 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa1279d740 1 -- 172.21.15.29:0/3764848454 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9f800fa50 con 0x555c75996850 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/3764848454 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa9fc01cd00 con 0x555c75996850 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/3764848454 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa9fc027480 con 0x555c75996850 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa1279d740 10 client.26060 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.169+0000 7faa1279d740 1 -- 172.21.15.29:0/3764848454 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555c75d577c0 con 0x555c75996850 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/3764848454 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9f800fa50 con 0x555c75996850 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/3764848454 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fa9fc01e5d0 con 0x555c75996850 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7faa1279d740 10 client.26060 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:42.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7faa1279d740 10 client.26060 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7faa1279d740 20 client.26060 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7faa1279d740 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c759551d0 0x555c75d5ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7faa1279d740 4 client.26060 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7faa1279d740 1 -- 172.21.15.29:0/3764848454 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c75cb5b00 con 0x555c759551d0 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.170+0000 7faa09fc5640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c759551d0 0x555c75d5ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.171+0000 7faa09fc5640 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c759551d0 0x555c75d5ad50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6226417d7293920d server_cookie=6de9c59ce5b42f5f in_seq=0 out_seq=0 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.171+0000 7fa9ed7fa640 10 client.26060 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.172+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/3764848454 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555c75cb5b00 con 0x555c759551d0 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.172+0000 7fa9ed7fa640 10 client.26060 handle_command_reply: tid=0 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.172+0000 7faa1279d740 1 client.26060 shutdown 2024-09-21T02:46:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.172+0000 7faa1279d740 1 -- 172.21.15.29:0/3764848454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75996850 msgr2=0x555c75d54940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.172+0000 7faa1279d740 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75996850 0x555c75d54940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009e20 tx=0x7fa9fc009500 comp rx=0 tx=0).stop 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 1 -- 172.21.15.29:0/3764848454 shutdown_connections 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75996850 0x555c75d54940 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c75975b70 0x555c75d48f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c759551d0 0x555c75d5ad50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 1 --2- 172.21.15.29:0/3764848454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c7596d990 0x555c75d489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7fa9ed7fa640 0 client.26060 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 1 -- 172.21.15.29:0/3764848454 >> 172.21.15.29:0/3764848454 conn(0x555c7598ac40 msgr2=0x555c75d40000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 1 -- 172.21.15.29:0/3764848454 shutdown_connections 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 1 -- 172.21.15.29:0/3764848454 wait complete. 2024-09-21T02:46:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.173+0000 7faa1279d740 20 client.26060 trim_cache size 0 max 16384 2024-09-21T02:46:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.194+0000 7faa1279d740 1 Processor -- start 2024-09-21T02:46:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.194+0000 7faa1279d740 1 -- start start 2024-09-21T02:46:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa1279d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c759551d0 0x555c75d42f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa1279d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c7596d990 0x555c75d54840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa1279d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c75975b70 0x555c75d47050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa1279d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa041b2c90 con 0x555c75975b70 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa1279d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faa041b47d0 con 0x555c7596d990 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa1279d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa041b3360 con 0x555c759551d0 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa0a7c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c75975b70 0x555c75d47050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c759551d0 0x555c75d42f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa0a7c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c75975b70 0x555c75d47050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33920/0 (socket says 172.21.15.29:33920) 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c759551d0 0x555c75d42f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55828/0 (socket says 172.21.15.29:55828) 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa0a7c6640 1 -- 172.21.15.29:0/1645107562 learned_addr learned my addr 172.21.15.29:0/1645107562 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa097c4640 1 --2- 172.21.15.29:0/1645107562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c7596d990 0x555c75d54840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa09fc5640 1 -- 172.21.15.29:0/1645107562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c7596d990 msgr2=0x555c75d54840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1645107562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c7596d990 0x555c75d54840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa09fc5640 1 -- 172.21.15.29:0/1645107562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c75975b70 msgr2=0x555c75d47050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1645107562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c75975b70 0x555c75d47050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa09fc5640 1 -- 172.21.15.29:0/1645107562 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c75d3b040 con 0x555c759551d0 2024-09-21T02:46:42.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.195+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1645107562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c759551d0 0x555c75d42f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faa00104f80 tx=0x7faa000c9c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12245e7cdadd9b65 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:42.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.196+0000 7fa9ef7fe640 1 -- 172.21.15.29:0/1645107562 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00048070 con 0x555c759551d0 2024-09-21T02:46:42.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.196+0000 7fa9ef7fe640 1 -- 172.21.15.29:0/1645107562 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa0003b460 con 0x555c759551d0 2024-09-21T02:46:42.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.196+0000 7fa9ef7fe640 1 -- 172.21.15.29:0/1645107562 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0010d040 con 0x555c759551d0 2024-09-21T02:46:42.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 -- 172.21.15.29:0/1645107562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c759551d0 msgr2=0x555c75d42f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 --2- 172.21.15.29:0/1645107562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c759551d0 0x555c75d42f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faa00104f80 tx=0x7faa000c9c60 comp rx=0 tx=0).stop 2024-09-21T02:46:42.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 -- 172.21.15.29:0/1645107562 shutdown_connections 2024-09-21T02:46:42.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 --2- 172.21.15.29:0/1645107562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c75975b70 0x555c75d47050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 --2- 172.21.15.29:0/1645107562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c7596d990 0x555c75d54840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 --2- 172.21.15.29:0/1645107562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c759551d0 0x555c75d42f90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 -- 172.21.15.29:0/1645107562 >> 172.21.15.29:0/1645107562 conn(0x555c7598ac40 msgr2=0x555c75d43910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 -- 172.21.15.29:0/1645107562 shutdown_connections 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 -- 172.21.15.29:0/1645107562 wait complete. 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 -- start start 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.197+0000 7faa1279d740 1 Processor -- start 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa1279d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c759551d0 0x555c75d512b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa1279d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c7596d990 0x555c75d45950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa1279d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d46060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa1279d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa041b4100 con 0x555c759551d0 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa1279d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faa041b3a30 con 0x555c75975b70 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa1279d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa041b0c70 con 0x555c7596d990 2024-09-21T02:46:42.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c759551d0 0x555c75d512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa09fc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c759551d0 0x555c75d512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33936/0 (socket says 172.21.15.29:33936) 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa09fc5640 1 -- 172.21.15.29:0/1829701360 learned_addr learned my addr 172.21.15.29:0/1829701360 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa097c4640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c7596d990 0x555c75d45950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa1279d740 10 client.0 _ll_register_callbacks cb 0x7fa9fc009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d46060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.198+0000 7faa09fc5640 1 -- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c7596d990 msgr2=0x555c75d45950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c7596d990 0x555c75d45950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa09fc5640 1 -- 172.21.15.29:0/1829701360 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 msgr2=0x555c75d46060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d46060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa09fc5640 1 -- 172.21.15.29:0/1829701360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c75dc4710 con 0x555c759551d0 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa0a7c6640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d46060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:42.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa097c4640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c7596d990 0x555c75d45950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa09fc5640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c759551d0 0x555c75d512b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faa00006010 tx=0x7faa00003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ca9e6723a4304f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7fa9ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/1829701360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00048070 con 0x555c759551d0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/1829701360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faa000c9880 con 0x555c759551d0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa1279d740 1 -- 172.21.15.29:0/1829701360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555c75cb15b0 con 0x555c759551d0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/1829701360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa0010d040 con 0x555c759551d0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa1279d740 10 client.16581 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.199+0000 7faa1279d740 1 -- 172.21.15.29:0/1829701360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555c75dc5870 con 0x555c759551d0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/1829701360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x555c75cb15b0 con 0x555c759551d0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/1829701360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7faa0004b060 con 0x555c759551d0 2024-09-21T02:46:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7faa1279d740 10 client.16581 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:42.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7faa1279d740 10 client.16581 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:46:42.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7faa1279d740 20 client.16581 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:42.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7faa1279d740 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c75996850 0x555c75dc8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:42.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7faa1279d740 4 client.16581 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}] 2024-09-21T02:46:42.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7faa1279d740 1 -- 172.21.15.29:0/1829701360 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}) -- 0x555c75cae9e0 con 0x555c75996850 2024-09-21T02:46:42.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.200+0000 7faa097c4640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c75996850 0x555c75dc8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:42.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.201+0000 7fa9ed7fa640 10 client.16581 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:42.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.201+0000 7faa097c4640 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c75996850 0x555c75dc8000 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60ee4e5ef6feae7c server_cookie=e4013b098994a942 in_seq=0 out_seq=0 2024-09-21T02:46:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.317+0000 7fa9ed7fa640 1 -- 172.21.15.29:0/1829701360 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555c75cae9e0 con 0x555c75996850 2024-09-21T02:46:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.317+0000 7fa9ed7fa640 10 client.16581 handle_command_reply: tid=0 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.317+0000 7faa1279d740 1 client.16581 shutdown 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.317+0000 7faa1279d740 1 -- 172.21.15.29:0/1829701360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c759551d0 msgr2=0x555c75d512b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.317+0000 7faa1279d740 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c759551d0 0x555c75d512b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faa00006010 tx=0x7faa00003960 comp rx=0 tx=0).stop 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 1 -- 172.21.15.29:0/1829701360 shutdown_connections 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c75975b70 0x555c75d46060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c7596d990 0x555c75d45950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c75996850 0x555c75dc8000 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 1 --2- 172.21.15.29:0/1829701360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c759551d0 0x555c75d512b0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 1 -- 172.21.15.29:0/1829701360 >> 172.21.15.29:0/1829701360 conn(0x555c7598ac40 msgr2=0x555c75d43910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7fa9ed7fa640 0 client.16581 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 1 -- 172.21.15.29:0/1829701360 shutdown_connections 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 1 -- 172.21.15.29:0/1829701360 wait complete. 2024-09-21T02:46:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.318+0000 7faa1279d740 20 client.16581 trim_cache size 0 max 16384 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 30, 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "c79740bc": { 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 30, 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:46:42.321 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 141.1, 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:46:42.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:46:42.323 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:46:42.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 -- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9f00b6510 msgr2=0x7fa9f00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9f00b6510 0x7fa9f00b8a20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc005c40 tx=0x7fa9fc004a40 comp rx=0 tx=0).stop 2024-09-21T02:46:42.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 -- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 msgr2=0x7faa04102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:42.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 0x7faa04102750 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7faa00005940 tx=0x7faa00005e10 comp rx=0 tx=0).stop 2024-09-21T02:46:42.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 -- 172.21.15.29:0/1130158027 shutdown_connections 2024-09-21T02:46:42.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9f00b6510 0x7fa9f00b8a20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa0410ddf0 0x7faa041031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa04105270 0x7faa04102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 --2- 172.21.15.29:0/1130158027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04104840 0x7faa04102750 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:42.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 -- 172.21.15.29:0/1130158027 >> 172.21.15.29:0/1130158027 conn(0x7faa040796d0 msgr2=0x7faa041068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:42.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 -- 172.21.15.29:0/1130158027 shutdown_connections 2024-09-21T02:46:42.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:46:42.319+0000 7faa0c229640 1 -- 172.21.15.29:0/1130158027 wait complete. 2024-09-21T02:46:43.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.48 MB/sec execute 260 sec latency 849.890 ms 2024-09-21T02:46:43.178 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c79740bc', quiesce took 3.8 seconds. Will release after: 17.25 2024-09-21T02:46:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:43 smithi066 ceph-mon[33436]: pgmap v617: 129 pgs: 129 active+clean; 36 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T02:46:44.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.47 MB/sec execute 261 sec latency 1850.001 ms 2024-09-21T02:46:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:43 smithi190 ceph-mon[33871]: pgmap v617: 129 pgs: 129 active+clean; 36 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T02:46:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:43 smithi029 ceph-mon[31006]: pgmap v617: 129 pgs: 129 active+clean; 36 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T02:46:45.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.46 MB/sec execute 262 sec latency 2850.109 ms 2024-09-21T02:46:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:45 smithi066 ceph-mon[33436]: pgmap v618: 129 pgs: 129 active+clean; 38 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-09-21T02:46:46.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 -- 172.21.15.29:0/2724451118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a41025c0 msgr2=0x7f71a4102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 --2- 172.21.15.29:0/2724451118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a41025c0 0x7f71a4102a30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7194009900 tx=0x7f719402a410 comp rx=0 tx=0).stop 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 -- 172.21.15.29:0/2724451118 shutdown_connections 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 --2- 172.21.15.29:0/2724451118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71a4103fe0 0x7f71a410a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 --2- 172.21.15.29:0/2724451118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71a4103640 0x7f71a4103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 --2- 172.21.15.29:0/2724451118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a41025c0 0x7f71a4102a30 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 -- 172.21.15.29:0/2724451118 >> 172.21.15.29:0/2724451118 conn(0x7f71a40fdf00 msgr2=0x7f71a41003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 -- 172.21.15.29:0/2724451118 shutdown_connections 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.058+0000 7f71aa92b640 1 -- 172.21.15.29:0/2724451118 wait complete. 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71aa92b640 1 Processor -- start 2024-09-21T02:46:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71aa92b640 1 -- start start 2024-09-21T02:46:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71aa92b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71a41025c0 0x7f71a4199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71aa92b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a4103640 0x7f71a419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71aa92b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71a4103fe0 0x7f71a419b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71aa92b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f71a410e100 con 0x7f71a4103640 2024-09-21T02:46:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71aa92b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f71a410df80 con 0x7f71a41025c0 2024-09-21T02:46:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71aa92b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f71a410e280 con 0x7f71a4103fe0 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71a37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a4103640 0x7f71a419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71a37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a4103640 0x7f71a419a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33820/0 (socket says 172.21.15.29:33820) 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71a37fe640 1 -- 172.21.15.29:0/1618654255 learned_addr learned my addr 172.21.15.29:0/1618654255 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71a3fff640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71a41025c0 0x7f71a4199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.059+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71a4103fe0 0x7f71a419b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a37fe640 1 -- 172.21.15.29:0/1618654255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71a4103fe0 msgr2=0x7f71a419b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a37fe640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71a4103fe0 0x7f71a419b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a37fe640 1 -- 172.21.15.29:0/1618654255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71a41025c0 msgr2=0x7f71a4199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a37fe640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71a41025c0 0x7f71a4199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a37fe640 1 -- 172.21.15.29:0/1618654255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a419b8d0 con 0x7f71a4103640 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a37fe640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a4103640 0x7f71a419a500 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f719800d950 tx=0x7f719800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a17fa640 1 -- 172.21.15.29:0/1618654255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198017070 con 0x7f71a4103640 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a17fa640 1 -- 172.21.15.29:0/1618654255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7198009c30 con 0x7f71a4103640 2024-09-21T02:46:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71aa92b640 1 -- 172.21.15.29:0/1618654255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a419bbc0 con 0x7f71a4103640 2024-09-21T02:46:46.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71a17fa640 1 -- 172.21.15.29:0/1618654255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719800f620 con 0x7f71a4103640 2024-09-21T02:46:46.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.060+0000 7f71aa92b640 1 -- 172.21.15.29:0/1618654255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a41a22b0 con 0x7f71a4103640 2024-09-21T02:46:46.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.063+0000 7f71b189a740 1 Processor -- start 2024-09-21T02:46:46.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.063+0000 7f71b189a740 1 -- start start 2024-09-21T02:46:46.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71b189a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e37e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71b189a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71b189a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e39010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71b189a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dbc5daa6f0 con 0x55dbc5a672f0 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71b189a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55dbc5d99b20 con 0x55dbc5a901b0 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71b189a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55dbc5d9d930 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33830/0 (socket says 172.21.15.29:33830) 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 -- 172.21.15.29:0/421984731 learned_addr learned my addr 172.21.15.29:0/421984731 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a17fa640 1 -- 172.21.15.29:0/1618654255 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f7198010460 con 0x7f71a4103640 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/421984731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e37e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 -- 172.21.15.29:0/421984731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 msgr2=0x55dbc5e39010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 --2- 172.21.15.29:0/421984731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e39010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 -- 172.21.15.29:0/421984731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 msgr2=0x55dbc5e37e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 --2- 172.21.15.29:0/421984731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e37e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 -- 172.21.15.29:0/421984731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbc5d5f530 con 0x55dbc5a672f0 2024-09-21T02:46:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.064+0000 7f71a3fff640 1 --2- 172.21.15.29:0/421984731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e384c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f719402aa00 tx=0x7f7194004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22d0f54efe15f77 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.065+0000 7f718a7fc640 1 -- 172.21.15.29:0/421984731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194035030 con 0x55dbc5a672f0 2024-09-21T02:46:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.065+0000 7f718a7fc640 1 -- 172.21.15.29:0/421984731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7194040070 con 0x55dbc5a672f0 2024-09-21T02:46:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.065+0000 7f718a7fc640 1 -- 172.21.15.29:0/421984731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71940319a0 con 0x55dbc5a672f0 2024-09-21T02:46:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.065+0000 7f71a17fa640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f718c0b6510 0x7f718c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.065+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f718c0b6510 0x7f718c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.065+0000 7f71a17fa640 1 -- 172.21.15.29:0/1618654255 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f71980e72d0 con 0x7f71a4103640 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 -- 172.21.15.29:0/421984731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 msgr2=0x55dbc5e384c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 --2- 172.21.15.29:0/421984731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e384c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f719402aa00 tx=0x7f7194004170 comp rx=0 tx=0).stop 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f718c0b6510 0x7f718c0b8a20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f719c005fd0 tx=0x7f719c009580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 -- 172.21.15.29:0/421984731 shutdown_connections 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 --2- 172.21.15.29:0/421984731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e39010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 --2- 172.21.15.29:0/421984731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e384c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 --2- 172.21.15.29:0/421984731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e37e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 -- 172.21.15.29:0/421984731 >> 172.21.15.29:0/421984731 conn(0x55dbc5a84300 msgr2=0x55dbc5e338e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 -- 172.21.15.29:0/421984731 shutdown_connections 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 -- 172.21.15.29:0/421984731 wait complete. 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 -- start start 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.066+0000 7f71b189a740 1 Processor -- start 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71b189a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e416c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71b189a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e41d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71b189a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e42880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71b189a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dbc5cd1ab0 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71b189a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55dbc5d3f730 con 0x55dbc5a901b0 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71b189a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55dbc5e39ad0 con 0x55dbc5a672f0 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71b189a740 10 client.0 _ll_register_callbacks cb 0x55dbc5d936b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e41d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e41d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33846/0 (socket says 172.21.15.29:33846) 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 -- 172.21.15.29:0/204027039 learned_addr learned my addr 172.21.15.29:0/204027039 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e42880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a3fff640 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 -- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 msgr2=0x55dbc5e416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 -- 172.21.15.29:0/204027039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 msgr2=0x55dbc5e42880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e42880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 -- 172.21.15.29:0/204027039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbc5afb790 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a3fff640 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71a37fe640 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e41d80 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f71980aa350 tx=0x7f71980a95e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8995d4937376fc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.067+0000 7f71837fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71837fe640 1 -- 172.21.15.29:0/204027039 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198020070 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71837fe640 1 -- 172.21.15.29:0/204027039 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f71980a8a30 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 1 -- 172.21.15.29:0/204027039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7194040070 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71837fe640 1 -- 172.21.15.29:0/204027039 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f71980acbf0 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 10 client.16593 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 1 -- 172.21.15.29:0/204027039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dbc5b74620 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71837fe640 1 -- 172.21.15.29:0/204027039 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7194040070 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71837fe640 1 -- 172.21.15.29:0/204027039 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f71980a8c30 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 10 client.16593 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 10 client.16593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 20 client.16593 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dbc5a4ed80 0x55dbc5e4ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 4 client.16593 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.068+0000 7f71b189a740 1 -- 172.21.15.29:0/204027039 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbc5dac640 con 0x55dbc5a4ed80 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.069+0000 7f71a3fff640 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dbc5a4ed80 0x55dbc5e4ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.069+0000 7f71a3fff640 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dbc5a4ed80 0x55dbc5e4ecb0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=614f7af04ce4056f server_cookie=5cba3da71b63eea2 in_seq=0 out_seq=0 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.069+0000 7f71837fe640 10 client.16593 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71837fe640 1 -- 172.21.15.29:0/204027039 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dbc5dac640 con 0x55dbc5a4ed80 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71837fe640 10 client.16593 handle_command_reply: tid=0 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 client.16593 shutdown 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 -- 172.21.15.29:0/204027039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a6f4d0 msgr2=0x55dbc5e41d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e41d80 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f71980aa350 tx=0x7f71980a95e0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 -- 172.21.15.29:0/204027039 shutdown_connections 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e42880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dbc5a4ed80 0x55dbc5e4ecb0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e41d80 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 --2- 172.21.15.29:0/204027039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71837fe640 0 client.16593 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 -- 172.21.15.29:0/204027039 >> 172.21.15.29:0/204027039 conn(0x55dbc5a84300 msgr2=0x55dbc5e344e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 -- 172.21.15.29:0/204027039 shutdown_connections 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 1 -- 172.21.15.29:0/204027039 wait complete. 2024-09-21T02:46:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.070+0000 7f71b189a740 20 client.16593 trim_cache size 0 max 16384 2024-09-21T02:46:46.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.45 MB/sec execute 263 sec latency 3850.174 ms 2024-09-21T02:46:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.085+0000 7f71b189a740 1 Processor -- start 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.085+0000 7f71b189a740 1 -- start start 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.085+0000 7f71b189a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e40320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.085+0000 7f71b189a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e34480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.085+0000 7f71b189a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e349c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.085+0000 7f71b189a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f71a41ac540 con 0x55dbc5a672f0 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.085+0000 7f71b189a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f71a41ae080 con 0x55dbc5a901b0 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.085+0000 7f71b189a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f71a41acc10 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e40320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e40320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33854/0 (socket says 172.21.15.29:33854) 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 -- 172.21.15.29:0/2209509345 learned_addr learned my addr 172.21.15.29:0/2209509345 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a37fe640 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e34480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 -- 172.21.15.29:0/2209509345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 msgr2=0x55dbc5e34480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e34480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 -- 172.21.15.29:0/2209509345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 msgr2=0x55dbc5e349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 -- 172.21.15.29:0/2209509345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbc5e3cef0 con 0x55dbc5a672f0 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e349c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f71a3fff640 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e40320 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f719402a930 tx=0x7f7194003830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c82760d004c897 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f7189ffb640 1 -- 172.21.15.29:0/2209509345 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194035040 con 0x55dbc5a672f0 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f7189ffb640 1 -- 172.21.15.29:0/2209509345 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7194040070 con 0x55dbc5a672f0 2024-09-21T02:46:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.086+0000 7f7189ffb640 1 -- 172.21.15.29:0/2209509345 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194031d80 con 0x55dbc5a672f0 2024-09-21T02:46:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 -- 172.21.15.29:0/2209509345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 msgr2=0x55dbc5e40320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e40320 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f719402a930 tx=0x7f7194003830 comp rx=0 tx=0).stop 2024-09-21T02:46:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 -- 172.21.15.29:0/2209509345 shutdown_connections 2024-09-21T02:46:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a901b0 0x55dbc5e349c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e34480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 --2- 172.21.15.29:0/2209509345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a672f0 0x55dbc5e40320 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 -- 172.21.15.29:0/2209509345 >> 172.21.15.29:0/2209509345 conn(0x55dbc5a84300 msgr2=0x55dbc5e3fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 -- 172.21.15.29:0/2209509345 shutdown_connections 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 -- 172.21.15.29:0/2209509345 wait complete. 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.087+0000 7f71b189a740 1 -- start start 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71b189a740 1 Processor -- start 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71b189a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e48c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71b189a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e47950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71b189a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a901b0 0x55dbc5e47e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71b189a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f71a41ad9b0 con 0x55dbc5a901b0 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71b189a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f71a41ad2e0 con 0x55dbc5a6f4d0 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71b189a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f71a41aa520 con 0x55dbc5a672f0 2024-09-21T02:46:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71b189a740 10 client.0 _ll_register_callbacks cb 0x7f719800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a8ec8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a901b0 0x55dbc5e47e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a8ec8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a901b0 0x55dbc5e47e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33870/0 (socket says 172.21.15.29:33870) 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a8ec8640 1 -- 172.21.15.29:0/2169927400 learned_addr learned my addr 172.21.15.29:0/2169927400 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a3fff640 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e48c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a37fe640 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e47950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a8ec8640 1 -- 172.21.15.29:0/2169927400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 msgr2=0x55dbc5e48c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e48c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a8ec8640 1 -- 172.21.15.29:0/2169927400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a6f4d0 msgr2=0x55dbc5e47950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e47950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.088+0000 7f71a8ec8640 1 -- 172.21.15.29:0/2169927400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbc5e48450 con 0x55dbc5a901b0 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71a8ec8640 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a901b0 0x55dbc5e47e90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f719c012a10 tx=0x7f719c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160d1681924d1e34 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71837fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71837fe640 1 -- 172.21.15.29:0/2169927400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c020c00 con 0x55dbc5a901b0 2024-09-21T02:46:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71837fe640 1 -- 172.21.15.29:0/2169927400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f719c020da0 con 0x55dbc5a901b0 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71b189a740 1 -- 172.21.15.29:0/2169927400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7194040070 con 0x55dbc5a901b0 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71837fe640 1 -- 172.21.15.29:0/2169927400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f719c033050 con 0x55dbc5a901b0 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71b189a740 10 client.16599 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71b189a740 1 -- 172.21.15.29:0/2169927400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dbc5eb96b0 con 0x55dbc5a901b0 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71837fe640 1 -- 172.21.15.29:0/2169927400 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7194040070 con 0x55dbc5a901b0 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71837fe640 1 -- 172.21.15.29:0/2169927400 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f719c00f040 con 0x55dbc5a901b0 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71b189a740 10 client.16599 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71b189a740 10 client.16599 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.089+0000 7f71b189a740 20 client.16599 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.090+0000 7f71b189a740 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dbc5a241c0 0x55dbc5ebbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.090+0000 7f71b189a740 4 client.16599 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.090+0000 7f71b189a740 1 -- 172.21.15.29:0/2169927400 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbc5da80f0 con 0x55dbc5a241c0 2024-09-21T02:46:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.090+0000 7f71a3fff640 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dbc5a241c0 0x55dbc5ebbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.090+0000 7f71a3fff640 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dbc5a241c0 0x55dbc5ebbf90 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=753616c38f4a5f7c server_cookie=e5876461a184a642 in_seq=0 out_seq=0 2024-09-21T02:46:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.090+0000 7f71837fe640 10 client.16599 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:46.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:45 smithi190 ceph-mon[33871]: pgmap v618: 129 pgs: 129 active+clean; 38 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.122+0000 7f71837fe640 1 -- 172.21.15.29:0/2169927400 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbc5da80f0 con 0x55dbc5a241c0 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.122+0000 7f71837fe640 10 client.16599 handle_command_reply: tid=0 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.122+0000 7f71b189a740 1 client.16599 shutdown 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 -- 172.21.15.29:0/2169927400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a901b0 msgr2=0x55dbc5e47e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a901b0 0x55dbc5e47e90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f719c012a10 tx=0x7f719c004a40 comp rx=0 tx=0).stop 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 -- 172.21.15.29:0/2169927400 shutdown_connections 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dbc5a241c0 0x55dbc5ebbf90 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dbc5a901b0 0x55dbc5e47e90 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dbc5a6f4d0 0x55dbc5e47950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 --2- 172.21.15.29:0/2169927400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dbc5a672f0 0x55dbc5e48c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71837fe640 0 client.16599 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 -- 172.21.15.29:0/2169927400 >> 172.21.15.29:0/2169927400 conn(0x55dbc5a84300 msgr2=0x55dbc5e3bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 -- 172.21.15.29:0/2169927400 shutdown_connections 2024-09-21T02:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 1 -- 172.21.15.29:0/2169927400 wait complete. 2024-09-21T02:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71b189a740 20 client.16599 trim_cache size 0 max 16384 2024-09-21T02:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.123+0000 7f71aa92b640 1 -- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f718c0b6510 msgr2=0x7f718c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f718c0b6510 0x7f718c0b8a20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f719c005fd0 tx=0x7f719c009580 comp rx=0 tx=0).stop 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 -- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a4103640 msgr2=0x7f71a419a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a4103640 0x7f71a419a500 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f719800d950 tx=0x7f719800de10 comp rx=0 tx=0).stop 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 -- 172.21.15.29:0/1618654255 shutdown_connections 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f718c0b6510 0x7f718c0b8a20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71a4103fe0 0x7f71a419b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a4103640 0x7f71a419a500 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 --2- 172.21.15.29:0/1618654255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71a41025c0 0x7f71a4199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 -- 172.21.15.29:0/1618654255 >> 172.21.15.29:0/1618654255 conn(0x7f71a40fdf00 msgr2=0x7f71a40ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 -- 172.21.15.29:0/1618654255 shutdown_connections 2024-09-21T02:46:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:46.124+0000 7f71aa92b640 1 -- 172.21.15.29:0/1618654255 wait complete. 2024-09-21T02:46:46.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:45 smithi029 ceph-mon[31006]: pgmap v618: 129 pgs: 129 active+clean; 38 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-09-21T02:46:47.085 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.44 MB/sec execute 264 sec latency 4850.274 ms 2024-09-21T02:46:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:47 smithi066 ceph-mon[33436]: pgmap v619: 129 pgs: 129 active+clean; 37 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-09-21T02:46:48.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.43 MB/sec execute 265 sec latency 5850.382 ms 2024-09-21T02:46:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:47 smithi190 ceph-mon[33871]: pgmap v619: 129 pgs: 129 active+clean; 37 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-09-21T02:46:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:47 smithi029 ceph-mon[31006]: pgmap v619: 129 pgs: 129 active+clean; 37 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-09-21T02:46:49.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.42 MB/sec execute 266 sec latency 6850.487 ms 2024-09-21T02:46:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:49 smithi066 ceph-mon[33436]: pgmap v620: 129 pgs: 129 active+clean; 36 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 963 KiB/s wr, 110 op/s 2024-09-21T02:46:50.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.41 MB/sec execute 267 sec latency 7850.608 ms 2024-09-21T02:46:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:49 smithi190 ceph-mon[33871]: pgmap v620: 129 pgs: 129 active+clean; 36 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 963 KiB/s wr, 110 op/s 2024-09-21T02:46:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:49 smithi029 ceph-mon[31006]: pgmap v620: 129 pgs: 129 active+clean; 36 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 963 KiB/s wr, 110 op/s 2024-09-21T02:46:51.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.41 MB/sec execute 268 sec latency 8850.731 ms 2024-09-21T02:46:52.002 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:51 smithi029 ceph-mon[31006]: pgmap v621: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 802 KiB/s wr, 84 op/s 2024-09-21T02:46:52.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:51 smithi066 ceph-mon[33436]: pgmap v621: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 802 KiB/s wr, 84 op/s 2024-09-21T02:46:52.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.40 MB/sec execute 269 sec latency 9850.819 ms 2024-09-21T02:46:52.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:51 smithi190 ceph-mon[33871]: pgmap v621: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 802 KiB/s wr, 84 op/s 2024-09-21T02:46:53.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.39 MB/sec execute 270 sec latency 10850.941 ms 2024-09-21T02:46:54.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:53 smithi066 ceph-mon[33436]: pgmap v622: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 636 KiB/s wr, 69 op/s 2024-09-21T02:46:54.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.38 MB/sec execute 271 sec latency 11851.072 ms 2024-09-21T02:46:54.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:53 smithi190 ceph-mon[33871]: pgmap v622: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 636 KiB/s wr, 69 op/s 2024-09-21T02:46:54.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:53 smithi029 ceph-mon[31006]: pgmap v622: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 636 KiB/s wr, 69 op/s 2024-09-21T02:46:55.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.37 MB/sec execute 272 sec latency 12851.172 ms 2024-09-21T02:46:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:55 smithi066 ceph-mon[33436]: pgmap v623: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 359 KiB/s wr, 45 op/s 2024-09-21T02:46:56.086 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.36 MB/sec execute 273 sec latency 13851.252 ms 2024-09-21T02:46:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:55 smithi190 ceph-mon[33871]: pgmap v623: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 359 KiB/s wr, 45 op/s 2024-09-21T02:46:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:55 smithi029 ceph-mon[31006]: pgmap v623: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 359 KiB/s wr, 45 op/s 2024-09-21T02:46:57.087 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.35 MB/sec execute 274 sec latency 14851.382 ms 2024-09-21T02:46:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:57 smithi066 ceph-mon[33436]: pgmap v624: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 149 KiB/s wr, 28 op/s 2024-09-21T02:46:58.087 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.34 MB/sec execute 275 sec latency 15851.489 ms 2024-09-21T02:46:58.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:57 smithi190 ceph-mon[33871]: pgmap v624: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 149 KiB/s wr, 28 op/s 2024-09-21T02:46:58.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:57 smithi029 ceph-mon[31006]: pgmap v624: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 149 KiB/s wr, 28 op/s 2024-09-21T02:46:59.087 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.34 MB/sec execute 276 sec latency 16851.616 ms 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 -- 172.21.15.29:0/1962174696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc106c00 msgr2=0x7f03bc109020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 --2- 172.21.15.29:0/1962174696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc106c00 0x7f03bc109020 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009900 tx=0x7f03a402a410 comp rx=0 tx=0).stop 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 -- 172.21.15.29:0/1962174696 shutdown_connections 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 --2- 172.21.15.29:0/1962174696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03bc109560 0x7f03bc10b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 --2- 172.21.15.29:0/1962174696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc106c00 0x7f03bc109020 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 --2- 172.21.15.29:0/1962174696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03bc101ab0 0x7f03bc101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 -- 172.21.15.29:0/1962174696 >> 172.21.15.29:0/1962174696 conn(0x7f03bc0fd690 msgr2=0x7f03bc0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 -- 172.21.15.29:0/1962174696 shutdown_connections 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.237+0000 7f03c2fba640 1 -- 172.21.15.29:0/1962174696 wait complete. 2024-09-21T02:46:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c2fba640 1 Processor -- start 2024-09-21T02:46:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c2fba640 1 -- start start 2024-09-21T02:46:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c2fba640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03bc101ab0 0x7f03bc104240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c2fba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03bc106c00 0x7f03bc102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c2fba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc109560 0x7f03bc103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c2fba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc10f1a0 con 0x7f03bc109560 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c2fba640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc10f020 con 0x7f03bc101ab0 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c2fba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc10f320 con 0x7f03bc106c00 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c1557640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc109560 0x7f03bc103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c1557640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc109560 0x7f03bc103040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32898/0 (socket says 172.21.15.29:32898) 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c1557640 1 -- 172.21.15.29:0/133923161 learned_addr learned my addr 172.21.15.29:0/133923161 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03c0d56640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03bc101ab0 0x7f03bc104240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.238+0000 7f03bbfff640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03bc106c00 0x7f03bc102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03c1557640 1 -- 172.21.15.29:0/133923161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03bc106c00 msgr2=0x7f03bc102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03c1557640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03bc106c00 0x7f03bc102b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03c1557640 1 -- 172.21.15.29:0/133923161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03bc101ab0 msgr2=0x7f03bc104240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03c1557640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03bc101ab0 0x7f03bc104240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03c1557640 1 -- 172.21.15.29:0/133923161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03bc103770 con 0x7f03bc109560 2024-09-21T02:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03bbfff640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03bc106c00 0x7f03bc102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03c1557640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc109560 0x7f03bc103040 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f03b4007ee0 tx=0x7f03b400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03b9ffb640 1 -- 172.21.15.29:0/133923161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b400dd90 con 0x7f03bc109560 2024-09-21T02:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03b9ffb640 1 -- 172.21.15.29:0/133923161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03b4013880 con 0x7f03bc109560 2024-09-21T02:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03c2fba640 1 -- 172.21.15.29:0/133923161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03bc1a3f30 con 0x7f03bc109560 2024-09-21T02:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03b9ffb640 1 -- 172.21.15.29:0/133923161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b4012770 con 0x7f03bc109560 2024-09-21T02:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.239+0000 7f03c2fba640 1 -- 172.21.15.29:0/133923161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03bc1a4340 con 0x7f03bc109560 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03b9ffb640 1 -- 172.21.15.29:0/133923161 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f03b401a020 con 0x7f03bc109560 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03ca000740 1 Processor -- start 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03ca000740 1 -- start start 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03ca000740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 0x560092f2ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03ca000740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f2d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03ca000740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f2e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03ca000740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560092e9f6f0 con 0x560092b644d0 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03ca000740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560092e8eb20 con 0x560092b5c2f0 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03ca000740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560092e92930 con 0x560092b851b0 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f2e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f2e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32906/0 (socket says 172.21.15.29:32906) 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.242+0000 7f03bbfff640 1 -- 172.21.15.29:0/3435163996 learned_addr learned my addr 172.21.15.29:0/3435163996 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c0d56640 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f2d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03b9ffb640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03a80b6510 0x7f03a80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03bbfff640 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 0x560092f2ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c0d56640 1 -- 172.21.15.29:0/3435163996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 msgr2=0x560092f2ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c0d56640 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 0x560092f2ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c1557640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03a80b6510 0x7f03a80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c0d56640 1 -- 172.21.15.29:0/3435163996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 msgr2=0x560092f2e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c0d56640 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f2e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c0d56640 1 -- 172.21.15.29:0/3435163996 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560092e54530 con 0x560092b5c2f0 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03bbfff640 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f2e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03b9ffb640 1 -- 172.21.15.29:0/133923161 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f03b40e7300 con 0x7f03bc109560 2024-09-21T02:46:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c0d56640 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f2d4c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00bbf0 tx=0x7f03ac0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=713cb546cc5177ec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03c1557640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03a80b6510 0x7f03a80b8a20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f03b40ad0a0 tx=0x7f03b40ddbb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.243+0000 7f03b2ffd640 1 -- 172.21.15.29:0/3435163996 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac004030 con 0x560092b5c2f0 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03b2ffd640 1 -- 172.21.15.29:0/3435163996 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03ac0035f0 con 0x560092b5c2f0 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03b2ffd640 1 -- 172.21.15.29:0/3435163996 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac011660 con 0x560092b5c2f0 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 -- 172.21.15.29:0/3435163996 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 msgr2=0x560092f2d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f2d4c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00bbf0 tx=0x7f03ac0107a0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 -- 172.21.15.29:0/3435163996 shutdown_connections 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f2e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f2d4c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 --2- 172.21.15.29:0/3435163996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 0x560092f2ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 -- 172.21.15.29:0/3435163996 >> 172.21.15.29:0/3435163996 conn(0x560092b79300 msgr2=0x560092f288e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 -- 172.21.15.29:0/3435163996 shutdown_connections 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.244+0000 7f03ca000740 1 -- 172.21.15.29:0/3435163996 wait complete. 2024-09-21T02:46:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.245+0000 7f03ca000740 1 -- start start 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.245+0000 7f03ca000740 1 Processor -- start 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.245+0000 7f03ca000740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 0x560092f36600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.245+0000 7f03ca000740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 0x560092f36cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.245+0000 7f03ca000740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f37830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03ca000740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560092dc6ab0 con 0x560092b851b0 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03ca000740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560092e34730 con 0x560092b644d0 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03ca000740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560092f2eb30 con 0x560092b5c2f0 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03ca000740 10 client.0 _ll_register_callbacks cb 0x560092e886b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f37830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f37830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32920/0 (socket says 172.21.15.29:32920) 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 -- 172.21.15.29:0/1457495632 learned_addr learned my addr 172.21.15.29:0/1457495632 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03c0d56640 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 0x560092f36cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 0x560092f36600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 -- 172.21.15.29:0/1457495632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 msgr2=0x560092f36600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 0x560092f36600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 -- 172.21.15.29:0/1457495632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 msgr2=0x560092f36cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 0x560092f36cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 -- 172.21.15.29:0/1457495632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560092bf0790 con 0x560092b851b0 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03bbfff640 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f37830 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f03a4032d20 tx=0x7f03a402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cc59aa74f78524 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f039bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f039bfff640 1 -- 172.21.15.29:0/1457495632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4035070 con 0x560092b851b0 2024-09-21T02:46:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f039bfff640 1 -- 172.21.15.29:0/1457495632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f03a4004450 con 0x560092b851b0 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.246+0000 7f03ca000740 1 -- 172.21.15.29:0/1457495632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03ac0035f0 con 0x560092b851b0 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f039bfff640 1 -- 172.21.15.29:0/1457495632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f03a4053050 con 0x560092b851b0 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f03ca000740 10 client.16611 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f03ca000740 1 -- 172.21.15.29:0/1457495632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560092c69620 con 0x560092b851b0 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f039bfff640 1 -- 172.21.15.29:0/1457495632 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03ac0035f0 con 0x560092b851b0 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f039bfff640 1 -- 172.21.15.29:0/1457495632 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f03a403a040 con 0x560092b851b0 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f03ca000740 10 client.16611 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f03ca000740 10 client.16611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f03ca000740 20 client.16611 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f03ca000740 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560092b43d80 0x560092f43da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f03ca000740 4 client.16611 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.247+0000 7f03ca000740 1 -- 172.21.15.29:0/1457495632 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560092ea1640 con 0x560092b43d80 2024-09-21T02:46:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.248+0000 7f03c0d56640 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560092b43d80 0x560092f43da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.248+0000 7f03c0d56640 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560092b43d80 0x560092f43da0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c5290744fcbbaa8 server_cookie=eebd2904ceaf3178 in_seq=0 out_seq=0 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.248+0000 7f039bfff640 10 client.16611 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f039bfff640 1 -- 172.21.15.29:0/1457495632 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560092ea1640 con 0x560092b43d80 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f039bfff640 10 client.16611 handle_command_reply: tid=0 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 client.16611 shutdown 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 -- 172.21.15.29:0/1457495632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 msgr2=0x560092f37830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f37830 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f03a4032d20 tx=0x7f03a402a410 comp rx=0 tx=0).stop 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 -- 172.21.15.29:0/1457495632 shutdown_connections 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560092b43d80 0x560092f43da0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f37830 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 0x560092f36cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 --2- 172.21.15.29:0/1457495632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 0x560092f36600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 -- 172.21.15.29:0/1457495632 >> 172.21.15.29:0/1457495632 conn(0x560092b79300 msgr2=0x560092f26ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f039bfff640 0 client.16611 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 -- 172.21.15.29:0/1457495632 shutdown_connections 2024-09-21T02:46:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 1 -- 172.21.15.29:0/1457495632 wait complete. 2024-09-21T02:46:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.249+0000 7f03ca000740 20 client.16611 trim_cache size 0 max 16384 2024-09-21T02:46:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.264+0000 7f03ca000740 1 Processor -- start 2024-09-21T02:46:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.264+0000 7f03ca000740 1 -- start start 2024-09-21T02:46:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.264+0000 7f03ca000740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f31650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.264+0000 7f03ca000740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f32490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.264+0000 7f03ca000740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 0x560092f329d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.264+0000 7f03ca000740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc1ae7f0 con 0x560092b644d0 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.264+0000 7f03ca000740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc1b0330 con 0x560092b5c2f0 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.264+0000 7f03ca000740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc1aeec0 con 0x560092b851b0 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f32490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f32490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32924/0 (socket says 172.21.15.29:32924) 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 -- 172.21.15.29:0/2274529565 learned_addr learned my addr 172.21.15.29:0/2274529565 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03bbfff640 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f31650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03bbfff640 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 0x560092f329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 -- 172.21.15.29:0/2274529565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 msgr2=0x560092f329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 0x560092f329d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 -- 172.21.15.29:0/2274529565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 msgr2=0x560092f31650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f31650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 -- 172.21.15.29:0/2274529565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560092f36970 con 0x560092b644d0 2024-09-21T02:46:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03bbfff640 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f31650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:46:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03c0d56640 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f32490 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00bbf0 tx=0x7f03ac00b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5110a10e713a5371 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03b27fc640 1 -- 172.21.15.29:0/2274529565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac005240 con 0x560092b644d0 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03b27fc640 1 -- 172.21.15.29:0/2274529565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03ac0053e0 con 0x560092b644d0 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.265+0000 7f03b27fc640 1 -- 172.21.15.29:0/2274529565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac027470 con 0x560092b644d0 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 -- 172.21.15.29:0/2274529565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 msgr2=0x560092f32490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f32490 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00bbf0 tx=0x7f03ac00b570 comp rx=0 tx=0).stop 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 -- 172.21.15.29:0/2274529565 shutdown_connections 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b851b0 0x560092f329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b644d0 0x560092f32490 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 --2- 172.21.15.29:0/2274529565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b5c2f0 0x560092f31650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 -- 172.21.15.29:0/2274529565 >> 172.21.15.29:0/2274529565 conn(0x560092b79300 msgr2=0x560092f30f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 -- 172.21.15.29:0/2274529565 shutdown_connections 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.266+0000 7f03ca000740 1 -- 172.21.15.29:0/2274529565 wait complete. 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 1 -- start start 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 1 Processor -- start 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 0x560092f3d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 0x560092f3ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f3c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc1afc60 con 0x560092b851b0 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc1af590 con 0x560092b644d0 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc1ac7d0 con 0x560092b5c2f0 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03ca000740 10 client.0 _ll_register_callbacks cb 0x7f03a4031920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:46:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f3c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f3c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32940/0 (socket says 172.21.15.29:32940) 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03bbfff640 1 -- 172.21.15.29:0/1009046608 learned_addr learned my addr 172.21.15.29:0/1009046608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.267+0000 7f03c0d56640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 0x560092f3ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03bbfff640 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 0x560092f3d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03bbfff640 1 -- 172.21.15.29:0/1009046608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 msgr2=0x560092f3d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03bbfff640 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 0x560092f3d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03bbfff640 1 -- 172.21.15.29:0/1009046608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 msgr2=0x560092f3ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03bbfff640 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 0x560092f3ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03bbfff640 1 -- 172.21.15.29:0/1009046608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560092f3c930 con 0x560092b851b0 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03bbfff640 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f3c3f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f03a403d5c0 tx=0x7f03a4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667f18d62a74613d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f039bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f039bfff640 1 -- 172.21.15.29:0/1009046608 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4035070 con 0x560092b851b0 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f039bfff640 1 -- 172.21.15.29:0/1009046608 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f03a4004450 con 0x560092b851b0 2024-09-21T02:46:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03ca000740 1 -- 172.21.15.29:0/1009046608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03ac0053e0 con 0x560092b851b0 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f039bfff640 1 -- 172.21.15.29:0/1009046608 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f03a4046040 con 0x560092b851b0 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03ca000740 10 client.16617 fetch_fsmap learned FSMap version 31 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.268+0000 7f03ca000740 1 -- 172.21.15.29:0/1009046608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560092faed10 con 0x560092b851b0 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f039bfff640 1 -- 172.21.15.29:0/1009046608 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03ac0053e0 con 0x560092b851b0 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f039bfff640 1 -- 172.21.15.29:0/1009046608 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f03a40035f0 con 0x560092b851b0 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f03ca000740 10 client.16617 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f03ca000740 10 client.16617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f03ca000740 20 client.16617 populate_metadata read hostname 'smithi029' 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f03ca000740 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560092b191c0 0x560092fb1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f03ca000740 4 client.16617 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f03ca000740 1 -- 172.21.15.29:0/1009046608 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560092f29090 con 0x560092b191c0 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.269+0000 7f03c0d56640 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560092b191c0 0x560092fb1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:46:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.270+0000 7f03c0d56640 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560092b191c0 0x560092fb1730 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e448dc6f45e04162 server_cookie=f0d0cb7bf47f0eff in_seq=0 out_seq=0 2024-09-21T02:46:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.270+0000 7f039bfff640 10 client.16617 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f039bfff640 1 -- 172.21.15.29:0/1009046608 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560092f29090 con 0x560092b191c0 2024-09-21T02:46:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f039bfff640 10 client.16617 handle_command_reply: tid=0 2024-09-21T02:46:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 client.16617 shutdown 2024-09-21T02:46:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 -- 172.21.15.29:0/1009046608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 msgr2=0x560092f3c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f3c3f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f03a403d5c0 tx=0x7f03a4000bc0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 -- 172.21.15.29:0/1009046608 shutdown_connections 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560092b191c0 0x560092fb1730 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560092b851b0 0x560092f3c3f0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560092b644d0 0x560092f3ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f039bfff640 0 client.16617 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 --2- 172.21.15.29:0/1009046608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560092b5c2f0 0x560092f3d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 -- 172.21.15.29:0/1009046608 >> 172.21.15.29:0/1009046608 conn(0x560092b79300 msgr2=0x560092f275b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 -- 172.21.15.29:0/1009046608 shutdown_connections 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 1 -- 172.21.15.29:0/1009046608 wait complete. 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03ca000740 20 client.16617 trim_cache size 0 max 16384 2024-09-21T02:46:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:46:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:46:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:46:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:46:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03c2fba640 1 -- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03a80b6510 msgr2=0x7f03a80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03c2fba640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03a80b6510 0x7f03a80b8a20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f03b40ad0a0 tx=0x7f03b40ddbb0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03c2fba640 1 -- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc109560 msgr2=0x7f03bc103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.291+0000 7f03c2fba640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc109560 0x7f03bc103040 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f03b4007ee0 tx=0x7f03b400d4a0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.292+0000 7f03c2fba640 1 -- 172.21.15.29:0/133923161 shutdown_connections 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.292+0000 7f03c2fba640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03a80b6510 0x7f03a80b8a20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.292+0000 7f03c2fba640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc109560 0x7f03bc103040 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.292+0000 7f03c2fba640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03bc106c00 0x7f03bc102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.292+0000 7f03c2fba640 1 --2- 172.21.15.29:0/133923161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03bc101ab0 0x7f03bc104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.292+0000 7f03c2fba640 1 -- 172.21.15.29:0/133923161 >> 172.21.15.29:0/133923161 conn(0x7f03bc0fd690 msgr2=0x7f03bc107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.292+0000 7f03c2fba640 1 -- 172.21.15.29:0/133923161 shutdown_connections 2024-09-21T02:46:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:46:59.292+0000 7f03c2fba640 1 -- 172.21.15.29:0/133923161 wait complete. 2024-09-21T02:47:00.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:46:59 smithi066 ceph-mon[33436]: pgmap v625: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 27 op/s 2024-09-21T02:47:00.087 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.33 MB/sec execute 277 sec latency 17851.739 ms 2024-09-21T02:47:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:46:59 smithi190 ceph-mon[33871]: pgmap v625: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 27 op/s 2024-09-21T02:47:00.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:46:59 smithi029 ceph-mon[31006]: pgmap v625: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 27 op/s 2024-09-21T02:47:00.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:47:00.702 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:47:01.087 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.32 MB/sec execute 278 sec latency 18851.806 ms 2024-09-21T02:47:02.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:01 smithi066 ceph-mon[33436]: pgmap v626: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 19 op/s 2024-09-21T02:47:02.087 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.31 MB/sec execute 279 sec latency 19851.910 ms 2024-09-21T02:47:02.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:01 smithi190 ceph-mon[33871]: pgmap v626: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 19 op/s 2024-09-21T02:47:02.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:01 smithi029 ceph-mon[31006]: pgmap v626: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 19 op/s 2024-09-21T02:47:03.087 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.30 MB/sec execute 280 sec latency 20852.025 ms 2024-09-21T02:47:03.779 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:03 smithi029 ceph-mon[31006]: pgmap v627: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 13 op/s 2024-09-21T02:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 -- 172.21.15.29:0/1924625719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8103b60 msgr2=0x7f4df8103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 --2- 172.21.15.29:0/1924625719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8103b60 0x7f4df8103f70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4de8009900 tx=0x7f4de802a410 comp rx=0 tx=0).stop 2024-09-21T02:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 -- 172.21.15.29:0/1924625719 shutdown_connections 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 --2- 172.21.15.29:0/1924625719 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4df8109df0 0x7f4df8110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 --2- 172.21.15.29:0/1924625719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4df81094a0 0x7f4df81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 --2- 172.21.15.29:0/1924625719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8103b60 0x7f4df8103f70 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 -- 172.21.15.29:0/1924625719 >> 172.21.15.29:0/1924625719 conn(0x7f4df807b850 msgr2=0x7f4df807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 -- 172.21.15.29:0/1924625719 shutdown_connections 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.777+0000 7f4dfe921640 1 -- 172.21.15.29:0/1924625719 wait complete. 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.778+0000 7f4dfe921640 1 Processor -- start 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.778+0000 7f4dfe921640 1 -- start start 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.778+0000 7f4dfe921640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4df8103b60 0x7f4df819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.778+0000 7f4dfe921640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4df81094a0 0x7f4df819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.778+0000 7f4dfe921640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8109df0 0x7f4df819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.778+0000 7f4dfe921640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4df8113960 con 0x7f4df8109df0 2024-09-21T02:47:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.778+0000 7f4dfe921640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4df81137e0 con 0x7f4df8103b60 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.778+0000 7f4dfe921640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4df8113ae0 con 0x7f4df81094a0 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8109df0 0x7f4df819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4df7fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4df8103b60 0x7f4df819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8109df0 0x7f4df819fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32958/0 (socket says 172.21.15.29:32958) 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 -- 172.21.15.29:0/2669379120 learned_addr learned my addr 172.21.15.29:0/2669379120 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4df77fe640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4df81094a0 0x7f4df819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 -- 172.21.15.29:0/2669379120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4df81094a0 msgr2=0x7f4df819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4df81094a0 0x7f4df819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 -- 172.21.15.29:0/2669379120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4df8103b60 msgr2=0x7f4df819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4df8103b60 0x7f4df819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 -- 172.21.15.29:0/2669379120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df81a0300 con 0x7f4df8109df0 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4df7fff640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4df8103b60 0x7f4df819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfcebe640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8109df0 0x7f4df819fc90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4df000cce0 tx=0x7f4df000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4df57fa640 1 -- 172.21.15.29:0/2669379120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df000f450 con 0x7f4df8109df0 2024-09-21T02:47:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df81a05f0 con 0x7f4df8109df0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4df57fa640 1 -- 172.21.15.29:0/2669379120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df000fa80 con 0x7f4df8109df0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4df57fa640 1 -- 172.21.15.29:0/2669379120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df0018a70 con 0x7f4df8109df0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.779+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df81a6e10 con 0x7f4df8109df0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.780+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df8103f70 con 0x7f4df8109df0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.782+0000 7f4df57fa640 1 -- 172.21.15.29:0/2669379120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4df0018c10 con 0x7f4df8109df0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.783+0000 7f4df57fa640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4de00b6540 0x7f4de00b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.783+0000 7f4df57fa640 1 -- 172.21.15.29:0/2669379120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4df00e7b50 con 0x7f4df8109df0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.783+0000 7f4df7fff640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4de00b6540 0x7f4de00b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.784+0000 7f4df7fff640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4de00b6540 0x7f4de00b8a50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4de80057c0 tx=0x7f4de80058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:03.786+0000 7f4df57fa640 1 -- 172.21.15.29:0/2669379120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4df0056710 con 0x7f4df8109df0 2024-09-21T02:47:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:03 smithi066 ceph-mon[33436]: pgmap v627: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 13 op/s 2024-09-21T02:47:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.012+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4df819b580 con 0x7f4df8109df0 2024-09-21T02:47:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.013+0000 7f4df57fa640 1 -- 172.21.15.29:0/2669379120 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f4df819b580 con 0x7f4df8109df0 2024-09-21T02:47:04.014 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:47:04.015 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:47:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:47:04.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4de00b6540 msgr2=0x7f4de00b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4de00b6540 0x7f4de00b8a50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4de80057c0 tx=0x7f4de80058d0 comp rx=0 tx=0).stop 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8109df0 msgr2=0x7f4df819fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8109df0 0x7f4df819fc90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4df000cce0 tx=0x7f4df000a620 comp rx=0 tx=0).stop 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 shutdown_connections 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4de00b6540 0x7f4de00b8a50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4df8109df0 0x7f4df819fc90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4df81094a0 0x7f4df819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 --2- 172.21.15.29:0/2669379120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4df8103b60 0x7f4df819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 >> 172.21.15.29:0/2669379120 conn(0x7f4df807b850 msgr2=0x7f4df8107650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 shutdown_connections 2024-09-21T02:47:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:04.015+0000 7f4dfe921640 1 -- 172.21.15.29:0/2669379120 wait complete. 2024-09-21T02:47:04.088 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.29 MB/sec execute 281 sec latency 21852.076 ms 2024-09-21T02:47:04.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:03 smithi190 ceph-mon[33871]: pgmap v627: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 13 op/s 2024-09-21T02:47:04.856 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id c79740bc --release --await' 2024-09-21T02:47:04.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id c79740bc --release --await 2024-09-21T02:47:05.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:04 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2669379120' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:05.088 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149598 2.29 MB/sec execute 282 sec latency 22852.484 ms 2024-09-21T02:47:05.123 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:47:05.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:04 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2669379120' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:05.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:04 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2669379120' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:05.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 -- 172.21.15.29:0/2035311722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 msgr2=0x7f1d30103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 --2- 172.21.15.29:0/2035311722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 0x7f1d30103f70 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009900 tx=0x7f1d2c02a410 comp rx=0 tx=0).stop 2024-09-21T02:47:05.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 -- 172.21.15.29:0/2035311722 shutdown_connections 2024-09-21T02:47:05.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 --2- 172.21.15.29:0/2035311722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d30109df0 0x7f1d30110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 --2- 172.21.15.29:0/2035311722 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1d301094a0 0x7f1d301098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 --2- 172.21.15.29:0/2035311722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 0x7f1d30103f70 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 -- 172.21.15.29:0/2035311722 >> 172.21.15.29:0/2035311722 conn(0x7f1d3007b850 msgr2=0x7f1d3007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 -- 172.21.15.29:0/2035311722 shutdown_connections 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 -- 172.21.15.29:0/2035311722 wait complete. 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 Processor -- start 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.686+0000 7f1d36b76640 1 -- start start 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d36b76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 0x7f1d30075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d36b76640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1d301094a0 0x7f1d30075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d36b76640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d30109df0 0x7f1d30076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d36b76640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301139b0 con 0x7f1d30103b60 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d36b76640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30113830 con 0x7f1d301094a0 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d36b76640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30113b30 con 0x7f1d30109df0 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 0x7f1d30075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 0x7f1d30075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60002/0 (socket says 172.21.15.29:60002) 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 -- 172.21.15.29:0/882693946 learned_addr learned my addr 172.21.15.29:0/882693946 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d27fff640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1d301094a0 0x7f1d30075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d35113640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d30109df0 0x7f1d30076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 -- 172.21.15.29:0/882693946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d30109df0 msgr2=0x7f1d30076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d30109df0 0x7f1d30076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 -- 172.21.15.29:0/882693946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1d301094a0 msgr2=0x7f1d30075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1d301094a0 0x7f1d30075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 -- 172.21.15.29:0/882693946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d301a8610 con 0x7f1d30103b60 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d27fff640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1d301094a0 0x7f1d30075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.687+0000 7f1d34912640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 0x7f1d30075710 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c02a930 tx=0x7f1d2c003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.688+0000 7f1d25ffb640 1 -- 172.21.15.29:0/882693946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c03f070 con 0x7f1d30103b60 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.688+0000 7f1d36b76640 1 -- 172.21.15.29:0/882693946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d301a88a0 con 0x7f1d30103b60 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.688+0000 7f1d25ffb640 1 -- 172.21.15.29:0/882693946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1d2c03bdc0 con 0x7f1d30103b60 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.688+0000 7f1d25ffb640 1 -- 172.21.15.29:0/882693946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c035030 con 0x7f1d30103b60 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.688+0000 7f1d36b76640 1 -- 172.21.15.29:0/882693946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d301a8dd0 con 0x7f1d30103b60 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.690+0000 7f1d3d0ea740 1 Processor -- start 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d3d0ea740 1 -- start start 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d25ffb640 1 -- 172.21.15.29:0/882693946 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f1d2c042070 con 0x7f1d30103b60 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 0x562ffc1a8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1a8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562ffc093dc0 con 0x562ffbde1840 2024-09-21T02:47:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562ffc1201f0 con 0x562ffbdd9660 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562ffc10f620 con 0x562ffbe02520 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1a8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1a8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60014/0 (socket says 172.21.15.29:60014) 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d27fff640 1 -- 172.21.15.29:0/2633395924 learned_addr learned my addr 172.21.15.29:0/2633395924 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d35113640 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 0x562ffc1a8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d34912640 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d27fff640 1 -- 172.21.15.29:0/2633395924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 msgr2=0x562ffc1a8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 0x562ffc1a8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d27fff640 1 -- 172.21.15.29:0/2633395924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 msgr2=0x562ffc1a8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.691+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1a8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d27fff640 1 -- 172.21.15.29:0/2633395924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ffc0d7fd0 con 0x562ffbde1840 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d34912640 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1a8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d25ffb640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1d180b6510 0x7f1d180b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d35113640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1d180b6510 0x7f1d180b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1a8f90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000cc00 tx=0x7f1d2000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0f822de01d0209 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d25ffb640 1 -- 172.21.15.29:0/882693946 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1d2c108180 con 0x7f1d30103b60 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d16ffd640 1 -- 172.21.15.29:0/2633395924 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20005220 con 0x562ffbde1840 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d16ffd640 1 -- 172.21.15.29:0/2633395924 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1d20004c30 con 0x562ffbde1840 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.692+0000 7f1d16ffd640 1 -- 172.21.15.29:0/2633395924 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20010720 con 0x562ffbde1840 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2633395924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 msgr2=0x562ffc1a8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1a8f90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000cc00 tx=0x7f1d2000e720 comp rx=0 tx=0).stop 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d35113640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1d180b6510 0x7f1d180b8a20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1d28007ed0 tx=0x7f1d28007680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2633395924 shutdown_connections 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1a8f90 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1a8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2633395924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 0x562ffc1a8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2633395924 >> 172.21.15.29:0/2633395924 conn(0x562ffbdf6910 msgr2=0x562ffc1a6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2633395924 shutdown_connections 2024-09-21T02:47:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.693+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2633395924 wait complete. 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 1 -- start start 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 1 Processor -- start 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 0x562ffc1b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1bfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562ffc0aa300 con 0x562ffbe02520 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562ffc113430 con 0x562ffbde1840 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562ffbf70bb0 con 0x562ffbdd9660 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d3d0ea740 10 client.0 _ll_register_callbacks cb 0x562ffc0c4120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d35113640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1bfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d35113640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1bfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60016/0 (socket says 172.21.15.29:60016) 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d35113640 1 -- 172.21.15.29:0/2111417542 learned_addr learned my addr 172.21.15.29:0/2111417542 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d34912640 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.694+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 0x562ffc1b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d34912640 1 -- 172.21.15.29:0/2111417542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 msgr2=0x562ffc1b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d34912640 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 0x562ffc1b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d34912640 1 -- 172.21.15.29:0/2111417542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 msgr2=0x562ffc1bfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d34912640 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1bfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d34912640 1 -- 172.21.15.29:0/2111417542 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ffc099780 con 0x562ffbdd9660 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d35113640 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1bfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d34912640 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1b3b80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c105950 tx=0x7f1d2c0ca6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8c40229b5c6e8db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2111417542 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c049030 con 0x562ffbdd9660 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2111417542 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1d2c0d7520 con 0x562ffbdd9660 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2111417542 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d20004c30 con 0x562ffbdd9660 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2111417542 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1d2c0cd700 con 0x562ffbdd9660 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d3d0ea740 10 client.26218 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.695+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2111417542 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562ffbe6db00 con 0x562ffbdd9660 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2111417542 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1d20004c30 con 0x562ffbdd9660 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2111417542 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f1d2c0b91f0 con 0x562ffbdd9660 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d3d0ea740 10 client.26218 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d3d0ea740 10 client.26218 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d3d0ea740 20 client.26218 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ffbdc0ea0 0x562ffc1c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d3d0ea740 4 client.26218 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2111417542 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ffc122140 con 0x562ffbdc0ea0 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.696+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ffbdc0ea0 0x562ffc1c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.697+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ffbdc0ea0 0x562ffc1c5eb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=108185a88cd62eca server_cookie=69402b18c29bb2c9 in_seq=0 out_seq=0 2024-09-21T02:47:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.697+0000 7f1d0ffff640 10 client.26218 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2111417542 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562ffc122140 con 0x562ffbdc0ea0 2024-09-21T02:47:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d0ffff640 10 client.26218 handle_command_reply: tid=0 2024-09-21T02:47:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 client.26218 shutdown 2024-09-21T02:47:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 msgr2=0x562ffc1b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1b3b80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c105950 tx=0x7f1d2c0ca6c0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2111417542 shutdown_connections 2024-09-21T02:47:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ffbdc0ea0 0x562ffc1c5eb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1bfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 0x562ffc1b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d0ffff640 0 client.26218 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2111417542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1b3b80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2111417542 >> 172.21.15.29:0/2111417542 conn(0x562ffbdf6910 msgr2=0x562ffc1ab6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2111417542 shutdown_connections 2024-09-21T02:47:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2111417542 wait complete. 2024-09-21T02:47:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.698+0000 7f1d3d0ea740 20 client.26218 trim_cache size 0 max 16384 2024-09-21T02:47:05.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.715+0000 7f1d3d0ea740 1 Processor -- start 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.715+0000 7f1d3d0ea740 1 -- start start 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 0x562ffc1bfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301b2ac0 con 0x562ffbe02520 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301b4600 con 0x562ffbde1840 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301b3190 con 0x562ffbdd9660 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1b2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60022/0 (socket says 172.21.15.29:60022) 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 -- 172.21.15.29:0/3960688133 learned_addr learned my addr 172.21.15.29:0/3960688133 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d34912640 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d27fff640 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 0x562ffc1bfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 -- 172.21.15.29:0/3960688133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 msgr2=0x562ffc1a9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1a9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 -- 172.21.15.29:0/3960688133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 msgr2=0x562ffc1bfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 0x562ffc1bfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 -- 172.21.15.29:0/3960688133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ffc1b5f50 con 0x562ffbe02520 2024-09-21T02:47:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.716+0000 7f1d35113640 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1b2b10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800e940 tx=0x7f1d28014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4218bde0c4520eca server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.717+0000 7f1d167fc640 1 -- 172.21.15.29:0/3960688133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2801c790 con 0x562ffbe02520 2024-09-21T02:47:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.717+0000 7f1d167fc640 1 -- 172.21.15.29:0/3960688133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1d2801cdc0 con 0x562ffbe02520 2024-09-21T02:47:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.717+0000 7f1d167fc640 1 -- 172.21.15.29:0/3960688133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28026e00 con 0x562ffbe02520 2024-09-21T02:47:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/3960688133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 msgr2=0x562ffc1b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1b2b10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800e940 tx=0x7f1d28014000 comp rx=0 tx=0).stop 2024-09-21T02:47:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/3960688133 shutdown_connections 2024-09-21T02:47:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbe02520 0x562ffc1b2b10 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbde1840 0x562ffc1bfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/3960688133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbdd9660 0x562ffc1a9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/3960688133 >> 172.21.15.29:0/3960688133 conn(0x562ffbdf6910 msgr2=0x562ffc1b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/3960688133 shutdown_connections 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/3960688133 wait complete. 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 -- start start 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.718+0000 7f1d3d0ea740 1 Processor -- start 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1bc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1b87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d3d0ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 0x562ffc1b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301b3f30 con 0x562ffbde1840 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301b3860 con 0x562ffbdd9660 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d3d0ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301b0aa0 con 0x562ffbe02520 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1b87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d3d0ea740 10 client.0 _ll_register_callbacks cb 0x7f1d2c031400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1b87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60034/0 (socket says 172.21.15.29:60034) 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 -- 172.21.15.29:0/2873975163 learned_addr learned my addr 172.21.15.29:0/2873975163 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d34912640 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1bc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d35113640 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 0x562ffc1b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 -- 172.21.15.29:0/2873975163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 msgr2=0x562ffc1b8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 0x562ffc1b8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 -- 172.21.15.29:0/2873975163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 msgr2=0x562ffc1bc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1bc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 -- 172.21.15.29:0/2873975163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ffbee4960 con 0x562ffbde1840 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d27fff640 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1b87b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000ec20 tx=0x7f1d2000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f971d7bd5c65215 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.719+0000 7f1d0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2873975163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20005450 con 0x562ffbde1840 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2873975163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1d20004660 con 0x562ffbde1840 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2873975163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d2801cdc0 con 0x562ffbde1840 2024-09-21T02:47:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2873975163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1d20004860 con 0x562ffbde1840 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 10 client.16641 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2873975163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562ffc231320 con 0x562ffbde1840 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2873975163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1d2801cdc0 con 0x562ffbde1840 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2873975163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f1d2001a920 con 0x562ffbde1840 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 10 client.16641 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 10 client.16641 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 20 client.16641 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ffbd96250 0x562ffc233a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 4 client.16641 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c79740bc", "release": true, "await": true}] 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.720+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2873975163 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c79740bc", "release": true, "await": true}) -- 0x562ffc11dbf0 con 0x562ffbd96250 2024-09-21T02:47:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.721+0000 7f1d34912640 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ffbd96250 0x562ffc233a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:05.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.721+0000 7f1d0ffff640 10 client.16641 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:05.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.721+0000 7f1d34912640 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ffbd96250 0x562ffc233a80 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b221633493722df server_cookie=942ed6114d17391a in_seq=0 out_seq=0 2024-09-21T02:47:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d0ffff640 1 -- 172.21.15.29:0/2873975163 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562ffc11dbf0 con 0x562ffbd96250 2024-09-21T02:47:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d0ffff640 10 client.16641 handle_command_reply: tid=0 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 client.16641 shutdown 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2873975163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 msgr2=0x562ffc1b87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1b87b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000ec20 tx=0x7f1d2000cb60 comp rx=0 tx=0).stop 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2873975163 shutdown_connections 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ffbe02520 0x562ffc1b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ffbd96250 0x562ffc233a80 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ffbde1840 0x562ffc1b87b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 --2- 172.21.15.29:0/2873975163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ffbdd9660 0x562ffc1bc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2873975163 >> 172.21.15.29:0/2873975163 conn(0x562ffbdf6910 msgr2=0x562ffc1b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d0ffff640 0 client.16641 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2873975163 shutdown_connections 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 1 -- 172.21.15.29:0/2873975163 wait complete. 2024-09-21T02:47:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.730+0000 7f1d3d0ea740 20 client.16641 trim_cache size 0 max 16384 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 32, 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "c79740bc": { 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 32, 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:47:05.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 141.1, 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:47:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 -- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1d180b6510 msgr2=0x7f1d180b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1d180b6510 0x7f1d180b8a20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1d28007ed0 tx=0x7f1d28007680 comp rx=0 tx=0).stop 2024-09-21T02:47:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 -- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 msgr2=0x7f1d30075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 0x7f1d30075710 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c02a930 tx=0x7f1d2c003d60 comp rx=0 tx=0).stop 2024-09-21T02:47:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 -- 172.21.15.29:0/882693946 shutdown_connections 2024-09-21T02:47:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1d180b6510 0x7f1d180b8a20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d30109df0 0x7f1d30076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1d301094a0 0x7f1d30075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 --2- 172.21.15.29:0/882693946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d30103b60 0x7f1d30075710 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 -- 172.21.15.29:0/882693946 >> 172.21.15.29:0/882693946 conn(0x7f1d3007b850 msgr2=0x7f1d30107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 -- 172.21.15.29:0/882693946 shutdown_connections 2024-09-21T02:47:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:05.731+0000 7f1d36b76640 1 -- 172.21.15.29:0/882693946 wait complete. 2024-09-21T02:47:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:05 smithi029 ceph-mon[31006]: pgmap v628: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-09-21T02:47:06.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:05 smithi066 ceph-mon[33436]: pgmap v628: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-09-21T02:47:06.088 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149808 2.28 MB/sec execute 283 sec latency 23499.697 ms 2024-09-21T02:47:06.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:05 smithi190 ceph-mon[33871]: pgmap v628: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-09-21T02:47:06.583 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c79740bc', total seconds elapsed: 27.2 2024-09-21T02:47:06.584 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.05 2024-09-21T02:47:06.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:47:06.864 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:47:07.088 INFO:tasks.workunit.client.0.smithi029.stdout: 1 150454 2.29 MB/sec execute 284 sec latency 16.000 ms 2024-09-21T02:47:07.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1312566941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 msgr2=0x7f27c410d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:07.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1312566941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 0x7f27c410d660 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f27b8009900 tx=0x7f27b802a410 comp rx=0 tx=0).stop 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1312566941 shutdown_connections 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1312566941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c410dc20 0x7f27c4110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1312566941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 0x7f27c410d660 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1312566941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27c41047c0 0x7f27c4104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1312566941 >> 172.21.15.29:0/1312566941 conn(0x7f27c40fe2c0 msgr2=0x7f27c4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1312566941 shutdown_connections 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.393+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1312566941 wait complete. 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.394+0000 7f27c8b9e640 1 Processor -- start 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.394+0000 7f27c8b9e640 1 -- start start 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c8b9e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27c41047c0 0x7f27c419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c8b9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 0x7f27c419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c8b9e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c410dc20 0x7f27c419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c8b9e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27c4113730 con 0x7f27c4105110 2024-09-21T02:47:07.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c8b9e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41135b0 con 0x7f27c41047c0 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c8b9e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41138b0 con 0x7f27c410dc20 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 0x7f27c419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c410dc20 0x7f27c419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 0x7f27c419a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60056/0 (socket says 172.21.15.29:60056) 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 -- 172.21.15.29:0/1719706939 learned_addr learned my addr 172.21.15.29:0/1719706939 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c37fe640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27c41047c0 0x7f27c419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 -- 172.21.15.29:0/1719706939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c410dc20 msgr2=0x7f27c419fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c410dc20 0x7f27c419fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 -- 172.21.15.29:0/1719706939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27c41047c0 msgr2=0x7f27c419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27c41047c0 0x7f27c419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 -- 172.21.15.29:0/1719706939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c41a01f0 con 0x7f27c4105110 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c3fff640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c410dc20 0x7f27c419fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:07.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.395+0000 7f27c2ffd640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 0x7f27c419a820 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f27b8004770 tx=0x7f27b8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.396+0000 7f27c0ff9640 1 -- 172.21.15.29:0/1719706939 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b803f030 con 0x7f27c4105110 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.396+0000 7f27c0ff9640 1 -- 172.21.15.29:0/1719706939 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27b8004440 con 0x7f27c4105110 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.396+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c41a0480 con 0x7f27c4105110 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.396+0000 7f27c0ff9640 1 -- 172.21.15.29:0/1719706939 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b8035070 con 0x7f27c4105110 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.396+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c41a6c40 con 0x7f27c4105110 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.397+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c4104bd0 con 0x7f27c4105110 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.399+0000 7f27c0ff9640 1 -- 172.21.15.29:0/1719706939 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f27b8042030 con 0x7f27c4105110 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.399+0000 7f27c0ff9640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27ac0b6590 0x7f27ac0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.400+0000 7f27c37fe640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27ac0b6590 0x7f27ac0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.400+0000 7f27c0ff9640 1 -- 172.21.15.29:0/1719706939 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f27b8107930 con 0x7f27c4105110 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.400+0000 7f27c37fe640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27ac0b6590 0x7f27ac0b8aa0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f27b400bbe0 tx=0x7f27b400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.404+0000 7f27c0ff9640 1 -- 172.21.15.29:0/1719706939 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f27b80d2220 con 0x7f27c4105110 2024-09-21T02:47:07.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.669+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27c419b3b0 con 0x7f27c4105110 2024-09-21T02:47:07.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.669+0000 7f27c0ff9640 1 -- 172.21.15.29:0/1719706939 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f27b80d7160 con 0x7f27c4105110 2024-09-21T02:47:07.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:47:07.671 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:47:07.671 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:47:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27ac0b6590 msgr2=0x7f27ac0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27ac0b6590 0x7f27ac0b8aa0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f27b400bbe0 tx=0x7f27b400b390 comp rx=0 tx=0).stop 2024-09-21T02:47:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 msgr2=0x7f27c419a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 0x7f27c419a820 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f27b8004770 tx=0x7f27b8004040 comp rx=0 tx=0).stop 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 shutdown_connections 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27ac0b6590 0x7f27ac0b8aa0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c410dc20 0x7f27c419fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c4105110 0x7f27c419a820 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 --2- 172.21.15.29:0/1719706939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27c41047c0 0x7f27c419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 >> 172.21.15.29:0/1719706939 conn(0x7f27c40fe2c0 msgr2=0x7f27c40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 shutdown_connections 2024-09-21T02:47:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:07.672+0000 7f27c8b9e640 1 -- 172.21.15.29:0/1719706939 wait complete. 2024-09-21T02:47:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:07 smithi066 ceph-mon[33436]: pgmap v629: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-09-21T02:47:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:07 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1719706939' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:08.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:07 smithi029 ceph-mon[31006]: pgmap v629: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-09-21T02:47:08.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:07 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1719706939' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:08.088 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151039 2.28 MB/sec execute 285 sec latency 64.091 ms 2024-09-21T02:47:08.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:07 smithi190 ceph-mon[33871]: pgmap v629: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-09-21T02:47:08.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:07 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1719706939' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:08.443 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0' 2024-09-21T02:47:08.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0 2024-09-21T02:47:08.740 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:47:09.090 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151607 2.28 MB/sec execute 286 sec latency 15.058 ms 2024-09-21T02:47:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.287+0000 7f6283989640 1 -- 172.21.15.29:0/467824522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c107b40 msgr2=0x7f627c107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.287+0000 7f6283989640 1 --2- 172.21.15.29:0/467824522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c107b40 0x7f627c107f50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f626c00aa10 tx=0x7f626c02b720 comp rx=0 tx=0).stop 2024-09-21T02:47:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.287+0000 7f6283989640 1 -- 172.21.15.29:0/467824522 shutdown_connections 2024-09-21T02:47:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.287+0000 7f6283989640 1 --2- 172.21.15.29:0/467824522 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f627c108490 0x7f627c10ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.287+0000 7f6283989640 1 --2- 172.21.15.29:0/467824522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c107b40 0x7f627c107f50 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.287+0000 7f6283989640 1 --2- 172.21.15.29:0/467824522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f627c102200 0x7f627c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.287+0000 7f6283989640 1 -- 172.21.15.29:0/467824522 >> 172.21.15.29:0/467824522 conn(0x7f627c079f50 msgr2=0x7f627c07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.288+0000 7f6283989640 1 -- 172.21.15.29:0/467824522 shutdown_connections 2024-09-21T02:47:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.288+0000 7f6283989640 1 -- 172.21.15.29:0/467824522 wait complete. 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6283989640 1 Processor -- start 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6283989640 1 -- start start 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6283989640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f627c102200 0x7f627c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6283989640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f627c107b40 0x7f627c19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6283989640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c108490 0x7f627c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6283989640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f627c111780 con 0x7f627c108490 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6283989640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f627c111600 con 0x7f627c102200 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6283989640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f627c111900 con 0x7f627c107b40 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6281f26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c108490 0x7f627c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6281f26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c108490 0x7f627c19fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60082/0 (socket says 172.21.15.29:60082) 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6281f26640 1 -- 172.21.15.29:0/1833260023 learned_addr learned my addr 172.21.15.29:0/1833260023 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6280f24640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f627c107b40 0x7f627c19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.289+0000 7f6281725640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f627c102200 0x7f627c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f6281f26640 1 -- 172.21.15.29:0/1833260023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f627c107b40 msgr2=0x7f627c19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f6281f26640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f627c107b40 0x7f627c19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f6281f26640 1 -- 172.21.15.29:0/1833260023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f627c102200 msgr2=0x7f627c19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f6281f26640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f627c102200 0x7f627c19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f6281f26640 1 -- 172.21.15.29:0/1833260023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627c1a03e0 con 0x7f627c108490 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f6281f26640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c108490 0x7f627c19fcb0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f627400eb10 tx=0x7f627400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f62727fc640 1 -- 172.21.15.29:0/1833260023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6274004b20 con 0x7f627c108490 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f62727fc640 1 -- 172.21.15.29:0/1833260023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6274004cc0 con 0x7f627c108490 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f6283989640 1 -- 172.21.15.29:0/1833260023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f627c1a06d0 con 0x7f627c108490 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f62727fc640 1 -- 172.21.15.29:0/1833260023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6274010690 con 0x7f627c108490 2024-09-21T02:47:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.290+0000 7f6283989640 1 -- 172.21.15.29:0/1833260023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f627c1a6e30 con 0x7f627c108490 2024-09-21T02:47:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.293+0000 7f6289efd740 1 Processor -- start 2024-09-21T02:47:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.294+0000 7f62727fc640 1 -- 172.21.15.29:0/1833260023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f6274005020 con 0x7f627c108490 2024-09-21T02:47:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.294+0000 7f6289efd740 1 -- start start 2024-09-21T02:47:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6289efd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe42850 0x55b5901ed280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6289efd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe19990 0x55b5901ed7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6289efd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901edd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6280f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe42850 0x55b5901ed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6289efd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b59015fbb0 con 0x55b58fe42850 2024-09-21T02:47:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6280f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe42850 0x55b5901ed280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60086/0 (socket says 172.21.15.29:60086) 2024-09-21T02:47:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6289efd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b59014efe0 con 0x55b58fe19990 2024-09-21T02:47:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6280f24640 1 -- 172.21.15.29:0/864176373 learned_addr learned my addr 172.21.15.29:0/864176373 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6289efd740 1 -- 172.21.15.29:0/864176373 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b590152df0 con 0x55b58fe21b70 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6281725640 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe19990 0x55b5901ed7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6280f24640 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f62727fc640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f62680b6590 0x7f62680b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.295+0000 7f6280f24640 1 -- 172.21.15.29:0/864176373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 msgr2=0x55b5901edd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f6280f24640 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901edd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f6280f24640 1 -- 172.21.15.29:0/864176373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe19990 msgr2=0x55b5901ed7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f6280f24640 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe19990 0x55b5901ed7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f6280f24640 1 -- 172.21.15.29:0/864176373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b58febcba0 con 0x55b58fe42850 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f6281725640 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe19990 0x55b5901ed7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f6281f26640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f62680b6590 0x7f62680b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f62727fc640 1 -- 172.21.15.29:0/1833260023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f62740e7b80 con 0x7f627c108490 2024-09-21T02:47:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f6280f24640 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe42850 0x55b5901ed280 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f626c000c00 tx=0x7f626c005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ff4c5849e0c81b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f62677fe640 1 -- 172.21.15.29:0/864176373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c036040 con 0x55b58fe42850 2024-09-21T02:47:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f6281f26640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f62680b6590 0x7f62680b8aa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f62740aac00 tx=0x7f62740a2240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f62677fe640 1 -- 172.21.15.29:0/864176373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f626c0044c0 con 0x55b58fe42850 2024-09-21T02:47:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.296+0000 7f62677fe640 1 -- 172.21.15.29:0/864176373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c003d80 con 0x55b58fe42850 2024-09-21T02:47:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.297+0000 7f6289efd740 1 -- 172.21.15.29:0/864176373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe42850 msgr2=0x55b5901ed280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.297+0000 7f6289efd740 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe42850 0x55b5901ed280 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f626c000c00 tx=0x7f626c005e30 comp rx=0 tx=0).stop 2024-09-21T02:47:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 -- 172.21.15.29:0/864176373 shutdown_connections 2024-09-21T02:47:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe19990 0x55b5901ed7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 --2- 172.21.15.29:0/864176373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe42850 0x55b5901ed280 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 -- 172.21.15.29:0/864176373 >> 172.21.15.29:0/864176373 conn(0x55b58fe36c40 msgr2=0x55b5901eb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 -- 172.21.15.29:0/864176373 shutdown_connections 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 -- 172.21.15.29:0/864176373 wait complete. 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 -- start start 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.298+0000 7f6289efd740 1 Processor -- start 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6289efd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b5901f8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6289efd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901f9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6289efd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 0x55b590204ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6289efd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b58ffb07b0 con 0x55b58fe19990 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6289efd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b5900e9e90 con 0x55b58fe42850 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6289efd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b5901fb9f0 con 0x55b58fe21b70 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6289efd740 10 client.0 _ll_register_callbacks cb 0x55b59014c710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6280f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b5901f8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6280f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b5901f8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60102/0 (socket says 172.21.15.29:60102) 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6280f24640 1 -- 172.21.15.29:0/3076795814 learned_addr learned my addr 172.21.15.29:0/3076795814 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6280f24640 1 -- 172.21.15.29:0/3076795814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 msgr2=0x55b5901f9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6280f24640 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901f9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6280f24640 1 -- 172.21.15.29:0/3076795814 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 msgr2=0x55b590204ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.299+0000 7f6280f24640 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 0x55b590204ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6280f24640 1 -- 172.21.15.29:0/3076795814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5902052f0 con 0x55b58fe19990 2024-09-21T02:47:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6280f24640 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b5901f8ef0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f626c006010 tx=0x7f626c0058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51bfa430b26ba1f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6264ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6264ff9640 1 -- 172.21.15.29:0/3076795814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c036030 con 0x55b58fe19990 2024-09-21T02:47:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6264ff9640 1 -- 172.21.15.29:0/3076795814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f626c0035f0 con 0x55b58fe19990 2024-09-21T02:47:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6289efd740 1 -- 172.21.15.29:0/3076795814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b590161b00 con 0x55b58fe19990 2024-09-21T02:47:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6264ff9640 1 -- 172.21.15.29:0/3076795814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f626c057050 con 0x55b58fe19990 2024-09-21T02:47:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6289efd740 10 client.16659 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.300+0000 7f6289efd740 1 -- 172.21.15.29:0/3076795814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b590207c20 con 0x55b58fe19990 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6264ff9640 1 -- 172.21.15.29:0/3076795814 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55b590161b00 con 0x55b58fe19990 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6264ff9640 1 -- 172.21.15.29:0/3076795814 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f626c03b040 con 0x55b58fe19990 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6289efd740 10 client.16659 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6289efd740 10 client.16659 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6289efd740 20 client.16659 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6289efd740 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b58fe011d0 0x55b59020af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6289efd740 4 client.16659 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6289efd740 1 -- 172.21.15.29:0/3076795814 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b59015d5b0 con 0x55b58fe011d0 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.301+0000 7f6281725640 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b58fe011d0 0x55b59020af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.302+0000 7f6264ff9640 10 client.16659 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.302+0000 7f6281725640 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b58fe011d0 0x55b59020af80 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c61ffdb42415cdbf server_cookie=97c929c05085fa62 in_seq=0 out_seq=0 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6264ff9640 1 -- 172.21.15.29:0/3076795814 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b59015d5b0 con 0x55b58fe011d0 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6264ff9640 10 client.16659 handle_command_reply: tid=0 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 client.16659 shutdown 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 -- 172.21.15.29:0/3076795814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 msgr2=0x55b5901f8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b5901f8ef0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f626c006010 tx=0x7f626c0058c0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 -- 172.21.15.29:0/3076795814 shutdown_connections 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 0x55b590204ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901f9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b58fe011d0 0x55b59020af80 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 --2- 172.21.15.29:0/3076795814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b5901f8ef0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6289efd740 1 -- 172.21.15.29:0/3076795814 >> 172.21.15.29:0/3076795814 conn(0x55b58fe36c40 msgr2=0x55b5901f0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.303+0000 7f6264ff9640 0 client.16659 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:09.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.304+0000 7f6289efd740 1 -- 172.21.15.29:0/3076795814 shutdown_connections 2024-09-21T02:47:09.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.304+0000 7f6289efd740 1 -- 172.21.15.29:0/3076795814 wait complete. 2024-09-21T02:47:09.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.304+0000 7f6289efd740 20 client.16659 trim_cache size 0 max 16384 2024-09-21T02:47:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6289efd740 1 Processor -- start 2024-09-21T02:47:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6289efd740 1 -- start start 2024-09-21T02:47:09.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6289efd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b590204c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6289efd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901f2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6289efd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 0x55b5901f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6289efd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1ae760 con 0x55b58fe19990 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6289efd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1b02a0 con 0x55b58fe42850 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6289efd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1aee30 con 0x55b58fe21b70 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6280f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b590204c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6280f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b590204c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60112/0 (socket says 172.21.15.29:60112) 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.326+0000 7f6280f24640 1 -- 172.21.15.29:0/3952598235 learned_addr learned my addr 172.21.15.29:0/3952598235 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6281725640 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6280f24640 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 0x55b5901f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6280f24640 1 -- 172.21.15.29:0/3952598235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 msgr2=0x55b5901f2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6280f24640 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901f2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6280f24640 1 -- 172.21.15.29:0/3952598235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 msgr2=0x55b5901f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6280f24640 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 0x55b5901f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6280f24640 1 -- 172.21.15.29:0/3952598235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5901f4580 con 0x55b58fe19990 2024-09-21T02:47:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6281725640 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901f2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6280f24640 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b590204c50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f626c001eb0 tx=0x7f626c005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09514dab4d7b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6266ffd640 1 -- 172.21.15.29:0/3952598235 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c036040 con 0x55b58fe19990 2024-09-21T02:47:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6266ffd640 1 -- 172.21.15.29:0/3952598235 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f626c004240 con 0x55b58fe19990 2024-09-21T02:47:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.327+0000 7f6266ffd640 1 -- 172.21.15.29:0/3952598235 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c05d420 con 0x55b58fe19990 2024-09-21T02:47:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.328+0000 7f6289efd740 1 -- 172.21.15.29:0/3952598235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 msgr2=0x55b590204c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.328+0000 7f6289efd740 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b590204c50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f626c001eb0 tx=0x7f626c005f30 comp rx=0 tx=0).stop 2024-09-21T02:47:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.328+0000 7f6289efd740 1 -- 172.21.15.29:0/3952598235 shutdown_connections 2024-09-21T02:47:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.328+0000 7f6289efd740 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe42850 0x55b5901f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.328+0000 7f6289efd740 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe21b70 0x55b5901f2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.328+0000 7f6289efd740 1 --2- 172.21.15.29:0/3952598235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b590204c50 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.328+0000 7f6289efd740 1 -- 172.21.15.29:0/3952598235 >> 172.21.15.29:0/3952598235 conn(0x55b58fe36c40 msgr2=0x55b5901f0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.328+0000 7f6289efd740 1 -- 172.21.15.29:0/3952598235 shutdown_connections 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 -- 172.21.15.29:0/3952598235 wait complete. 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 -- start start 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 Processor -- start 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b59020a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe21b70 0x55b59020ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe42850 0x55b5902746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1afbd0 con 0x55b58fe19990 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1af500 con 0x55b58fe21b70 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.329+0000 7f6289efd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1ac790 con 0x55b58fe42850 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6289efd740 10 client.0 _ll_register_callbacks cb 0x7f62780097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b59020a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b59020a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60124/0 (socket says 172.21.15.29:60124) 2024-09-21T02:47:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 -- 172.21.15.29:0/3811803899 learned_addr learned my addr 172.21.15.29:0/3811803899 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe42850 0x55b5902746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6281725640 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe21b70 0x55b59020ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 -- 172.21.15.29:0/3811803899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe21b70 msgr2=0x55b59020ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe21b70 0x55b59020ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 -- 172.21.15.29:0/3811803899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 msgr2=0x55b59020a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b59020a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 -- 172.21.15.29:0/3811803899 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b590274c80 con 0x55b58fe42850 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6280f24640 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe42850 0x55b5902746f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f626c03fa30 tx=0x7f626c005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e69376c21c8f8e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.330+0000 7f6264ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6264ff9640 1 -- 172.21.15.29:0/3811803899 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c036030 con 0x55b58fe42850 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 1 -- 172.21.15.29:0/3811803899 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f626c004240 con 0x55b58fe42850 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6264ff9640 1 -- 172.21.15.29:0/3811803899 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f626c0035f0 con 0x55b58fe42850 2024-09-21T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6264ff9640 1 -- 172.21.15.29:0/3811803899 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f626c0037f0 con 0x55b58fe42850 2024-09-21T02:47:09.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 10 client.26245 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:09.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 1 -- 172.21.15.29:0/3811803899 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b590275ec0 con 0x55b58fe42850 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6264ff9640 1 -- 172.21.15.29:0/3811803899 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f626c004240 con 0x55b58fe42850 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6264ff9640 1 -- 172.21.15.29:0/3811803899 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f626c00b040 con 0x55b58fe42850 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 10 client.26245 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 10 client.26245 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 20 client.26245 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b58fdd6580 0x55b590278500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 4 client.26245 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}] 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.331+0000 7f6289efd740 1 -- 172.21.15.29:0/3811803899 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}) -- 0x55b59015a9e0 con 0x55b58fdd6580 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.332+0000 7f6281725640 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b58fdd6580 0x55b590278500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.332+0000 7f6264ff9640 10 client.26245 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.332+0000 7f6281725640 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b58fdd6580 0x55b590278500 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d2546c3bffe8945 server_cookie=6424565b8ac4e9 in_seq=0 out_seq=0 2024-09-21T02:47:09.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6264ff9640 1 -- 172.21.15.29:0/3811803899 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b59015a9e0 con 0x55b58fdd6580 2024-09-21T02:47:09.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6264ff9640 10 client.26245 handle_command_reply: tid=0 2024-09-21T02:47:09.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 client.26245 shutdown 2024-09-21T02:47:09.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 -- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe42850 msgr2=0x55b5902746f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe42850 0x55b5902746f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f626c03fa30 tx=0x7f626c005970 comp rx=0 tx=0).stop 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 -- 172.21.15.29:0/3811803899 shutdown_connections 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b58fe42850 0x55b5902746f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b58fe21b70 0x55b59020ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b58fdd6580 0x55b590278500 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 --2- 172.21.15.29:0/3811803899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b58fe19990 0x55b59020a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6289efd740 1 -- 172.21.15.29:0/3811803899 >> 172.21.15.29:0/3811803899 conn(0x55b58fe36c40 msgr2=0x55b5901f7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.436+0000 7f6264ff9640 0 client.26245 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6289efd740 1 -- 172.21.15.29:0/3811803899 shutdown_connections 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6289efd740 1 -- 172.21.15.29:0/3811803899 wait complete. 2024-09-21T02:47:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6289efd740 20 client.26245 trim_cache size 0 max 16384 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 34, 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "3fbf52c0": { 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 34, 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:47:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 141.1, 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:09.441 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:47:09.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 -- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f62680b6590 msgr2=0x7f62680b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f62680b6590 0x7f62680b8aa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f62740aac00 tx=0x7f62740a2240 comp rx=0 tx=0).stop 2024-09-21T02:47:09.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 -- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c108490 msgr2=0x7f627c19fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:09.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c108490 0x7f627c19fcb0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f627400eb10 tx=0x7f627400c2d0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 -- 172.21.15.29:0/1833260023 shutdown_connections 2024-09-21T02:47:09.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f62680b6590 0x7f62680b8aa0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627c108490 0x7f627c19fcb0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f627c107b40 0x7f627c19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.437+0000 7f6283989640 1 --2- 172.21.15.29:0/1833260023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f627c102200 0x7f627c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:09.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.438+0000 7f6283989640 1 -- 172.21.15.29:0/1833260023 >> 172.21.15.29:0/1833260023 conn(0x7f627c079f50 msgr2=0x7f627c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:09.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.438+0000 7f6283989640 1 -- 172.21.15.29:0/1833260023 shutdown_connections 2024-09-21T02:47:09.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:09.438+0000 7f6283989640 1 -- 172.21.15.29:0/1833260023 wait complete. 2024-09-21T02:47:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:09 smithi066 ceph-mon[33436]: pgmap v630: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-09-21T02:47:10.090 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.29 MB/sec execute 287 sec latency 657.303 ms 2024-09-21T02:47:10.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:09 smithi190 ceph-mon[33871]: pgmap v630: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-09-21T02:47:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:09 smithi029 ceph-mon[31006]: pgmap v630: 129 pgs: 129 active+clean; 35 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-09-21T02:47:10.247 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3fbf52c0', quiesce took 3.7 seconds. Will release after: 17.35 2024-09-21T02:47:11.090 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.28 MB/sec execute 288 sec latency 1657.392 ms 2024-09-21T02:47:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:11 smithi066 ceph-mon[33436]: pgmap v631: 129 pgs: 129 active+clean; 36 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 182 KiB/s wr, 27 op/s 2024-09-21T02:47:12.090 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.27 MB/sec execute 289 sec latency 2657.517 ms 2024-09-21T02:47:12.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:11 smithi190 ceph-mon[33871]: pgmap v631: 129 pgs: 129 active+clean; 36 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 182 KiB/s wr, 27 op/s 2024-09-21T02:47:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:11 smithi029 ceph-mon[31006]: pgmap v631: 129 pgs: 129 active+clean; 36 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 182 KiB/s wr, 27 op/s 2024-09-21T02:47:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 -- 172.21.15.29:0/1984737955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 msgr2=0x7faa5810b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 --2- 172.21.15.29:0/1984737955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 0x7faa5810b9a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faa50009ca0 tx=0x7faa5002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 -- 172.21.15.29:0/1984737955 shutdown_connections 2024-09-21T02:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 --2- 172.21.15.29:0/1984737955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 0x7faa5810b9a0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 --2- 172.21.15.29:0/1984737955 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa58106c20 0x7faa58109040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 --2- 172.21.15.29:0/1984737955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa58101ad0 0x7faa58101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 -- 172.21.15.29:0/1984737955 >> 172.21.15.29:0/1984737955 conn(0x7faa580fd690 msgr2=0x7faa580ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 -- 172.21.15.29:0/1984737955 shutdown_connections 2024-09-21T02:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 -- 172.21.15.29:0/1984737955 wait complete. 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 Processor -- start 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.405+0000 7faa5eb39640 1 -- start start 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5eb39640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa58101ad0 0x7faa58104260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5eb39640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa58106c20 0x7faa58102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5eb39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 0x7faa58103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5eb39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa5810f180 con 0x7faa58109580 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5eb39640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faa5810f000 con 0x7faa58101ad0 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5eb39640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa5810f300 con 0x7faa58106c20 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 0x7faa58103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 0x7faa58103060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60134/0 (socket says 172.21.15.29:60134) 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 -- 172.21.15.29:0/391829330 learned_addr learned my addr 172.21.15.29:0/391829330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa58101ad0 0x7faa58104260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa57fff640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa58106c20 0x7faa58102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 -- 172.21.15.29:0/391829330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa58106c20 msgr2=0x7faa58102b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa58106c20 0x7faa58102b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 -- 172.21.15.29:0/391829330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa58101ad0 msgr2=0x7faa58104260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa58101ad0 0x7faa58104260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 -- 172.21.15.29:0/391829330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa58103790 con 0x7faa58109580 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.406+0000 7faa5d0d6640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 0x7faa58103060 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7faa50009dc0 tx=0x7faa50003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.407+0000 7faa55ffb640 1 -- 172.21.15.29:0/391829330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa50004240 con 0x7faa58109580 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.407+0000 7faa55ffb640 1 -- 172.21.15.29:0/391829330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa50040030 con 0x7faa58109580 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.407+0000 7faa5eb39640 1 -- 172.21.15.29:0/391829330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa581a3f60 con 0x7faa58109580 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.407+0000 7faa55ffb640 1 -- 172.21.15.29:0/391829330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5003b430 con 0x7faa58109580 2024-09-21T02:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.407+0000 7faa5eb39640 1 -- 172.21.15.29:0/391829330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa581a4310 con 0x7faa58109580 2024-09-21T02:47:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa65aad740 1 Processor -- start 2024-09-21T02:47:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa55ffb640 1 -- 172.21.15.29:0/391829330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7faa50030650 con 0x7faa58109580 2024-09-21T02:47:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa65aad740 1 -- start start 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa65aad740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7da1b0 0x55c34cb81e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa65aad740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7b12f0 0x55c34cb824c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa65aad740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb83010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa65aad740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c34caf46f0 con 0x55c34c7b94d0 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa65aad740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c34cae3b20 con 0x55c34c7b12f0 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa65aad740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c34cae7930 con 0x55c34c7da1b0 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb83010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb83010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60150/0 (socket says 172.21.15.29:60150) 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.410+0000 7faa57fff640 1 -- 172.21.15.29:0/768488632 learned_addr learned my addr 172.21.15.29:0/768488632 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7b12f0 0x55c34cb824c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa57fff640 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7da1b0 0x55c34cb81e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa57fff640 1 -- 172.21.15.29:0/768488632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7da1b0 msgr2=0x55c34cb81e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa57fff640 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7da1b0 0x55c34cb81e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa55ffb640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faa480b6510 0x7faa480b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa57fff640 1 -- 172.21.15.29:0/768488632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7b12f0 msgr2=0x55c34cb824c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa57fff640 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7b12f0 0x55c34cb824c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa57fff640 1 -- 172.21.15.29:0/768488632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c34caa9530 con 0x55c34c7b94d0 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7b12f0 0x55c34cb824c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa5d0d6640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faa480b6510 0x7faa480b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa57fff640 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb83010 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7faa44012520 tx=0x7faa4400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e45fc8f707acd76 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa55ffb640 1 -- 172.21.15.29:0/391829330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faa50108420 con 0x7faa58109580 2024-09-21T02:47:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa42ffd640 1 -- 172.21.15.29:0/768488632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa440106b0 con 0x55c34c7b94d0 2024-09-21T02:47:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa42ffd640 1 -- 172.21.15.29:0/768488632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa44010ce0 con 0x55c34c7b94d0 2024-09-21T02:47:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.411+0000 7faa42ffd640 1 -- 172.21.15.29:0/768488632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4400eb70 con 0x55c34c7b94d0 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 -- 172.21.15.29:0/768488632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 msgr2=0x55c34cb83010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa5d0d6640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faa480b6510 0x7faa480b8a20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faa500ca8b0 tx=0x7faa5009b720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb83010 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7faa44012520 tx=0x7faa4400b810 comp rx=0 tx=0).stop 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 -- 172.21.15.29:0/768488632 shutdown_connections 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb83010 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7b12f0 0x55c34cb824c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 --2- 172.21.15.29:0/768488632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7da1b0 0x55c34cb81e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 -- 172.21.15.29:0/768488632 >> 172.21.15.29:0/768488632 conn(0x55c34c7ce300 msgr2=0x55c34cb7d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 -- 172.21.15.29:0/768488632 shutdown_connections 2024-09-21T02:47:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.412+0000 7faa65aad740 1 -- 172.21.15.29:0/768488632 wait complete. 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 1 -- start start 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 1 Processor -- start 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb8b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb8c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c34ca1bab0 con 0x55c34c7b12f0 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c34ca89730 con 0x55c34c7da1b0 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c34cb83b30 con 0x55c34c7b94d0 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa65aad740 10 client.0 _ll_register_callbacks cb 0x55c34cadd6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb8b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb8b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60166/0 (socket says 172.21.15.29:60166) 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.413+0000 7faa57fff640 1 -- 172.21.15.29:0/2306229471 learned_addr learned my addr 172.21.15.29:0/2306229471 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa57fff640 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb8c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa57fff640 1 -- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 msgr2=0x55c34cb8bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa57fff640 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa57fff640 1 -- 172.21.15.29:0/2306229471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 msgr2=0x55c34cb8c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa57fff640 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb8c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa57fff640 1 -- 172.21.15.29:0/2306229471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c34c845790 con 0x55c34c7b12f0 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa57fff640 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb8b670 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7faa4400e680 tx=0x7faa4400e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11665eddea2359ac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa37fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa37fff640 1 -- 172.21.15.29:0/2306229471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4402c060 con 0x55c34c7b12f0 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa37fff640 1 -- 172.21.15.29:0/2306229471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faa44004460 con 0x55c34c7b12f0 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa65aad740 1 -- 172.21.15.29:0/2306229471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa44010ce0 con 0x55c34c7b12f0 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.414+0000 7faa37fff640 1 -- 172.21.15.29:0/2306229471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa44033050 con 0x55c34c7b12f0 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa65aad740 10 client.16677 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa65aad740 1 -- 172.21.15.29:0/2306229471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c34c8be620 con 0x55c34c7b12f0 2024-09-21T02:47:12.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa37fff640 1 -- 172.21.15.29:0/2306229471 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa44010ce0 con 0x55c34c7b12f0 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa37fff640 1 -- 172.21.15.29:0/2306229471 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7faa4401c020 con 0x55c34c7b12f0 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa65aad740 10 client.16677 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa65aad740 10 client.16677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa65aad740 20 client.16677 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa65aad740 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c34c798d80 0x55c34cb98aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa65aad740 4 client.16677 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa65aad740 1 -- 172.21.15.29:0/2306229471 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c34caf6640 con 0x55c34c798d80 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.415+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c34c798d80 0x55c34cb98aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.416+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c34c798d80 0x55c34cb98aa0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6db31251d824fa79 server_cookie=476e1643fb388b86 in_seq=0 out_seq=0 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.416+0000 7faa37fff640 10 client.16677 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa37fff640 1 -- 172.21.15.29:0/2306229471 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c34caf6640 con 0x55c34c798d80 2024-09-21T02:47:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa37fff640 10 client.16677 handle_command_reply: tid=0 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 client.16677 shutdown 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 -- 172.21.15.29:0/2306229471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 msgr2=0x55c34cb8b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb8b670 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7faa4400e680 tx=0x7faa4400e5f0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 -- 172.21.15.29:0/2306229471 shutdown_connections 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb8c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c34c798d80 0x55c34cb98aa0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 --2- 172.21.15.29:0/2306229471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb8b670 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa37fff640 0 client.16677 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 -- 172.21.15.29:0/2306229471 >> 172.21.15.29:0/2306229471 conn(0x55c34c7ce300 msgr2=0x55c34cb7bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 -- 172.21.15.29:0/2306229471 shutdown_connections 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 1 -- 172.21.15.29:0/2306229471 wait complete. 2024-09-21T02:47:12.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.417+0000 7faa65aad740 20 client.16677 trim_cache size 0 max 16384 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa65aad740 1 Processor -- start 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa65aad740 1 -- start start 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa65aad740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb89b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa65aad740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa65aad740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb7e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa65aad740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa581ae810 con 0x55c34c7b12f0 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa65aad740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faa581b0350 con 0x55c34c7da1b0 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa65aad740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa581aeee0 con 0x55c34c7b94d0 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb89b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb89b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60170/0 (socket says 172.21.15.29:60170) 2024-09-21T02:47:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa57fff640 1 -- 172.21.15.29:0/336201904 learned_addr learned my addr 172.21.15.29:0/336201904 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/336201904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa57fff640 1 -- 172.21.15.29:0/336201904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 msgr2=0x55c34cb8a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa57fff640 1 --2- 172.21.15.29:0/336201904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa57fff640 1 -- 172.21.15.29:0/336201904 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 msgr2=0x55c34cb7e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa57fff640 1 --2- 172.21.15.29:0/336201904 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb7e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.432+0000 7faa57fff640 1 -- 172.21.15.29:0/336201904 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c34cb93b40 con 0x55c34c7b12f0 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa57fff640 1 --2- 172.21.15.29:0/336201904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb89b80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faa44005c30 tx=0x7faa4400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c9ea1bb32b60be server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa427fc640 1 -- 172.21.15.29:0/336201904 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4402c070 con 0x55c34c7b12f0 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa427fc640 1 -- 172.21.15.29:0/336201904 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa440052c0 con 0x55c34c7b12f0 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa427fc640 1 -- 172.21.15.29:0/336201904 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44022070 con 0x55c34c7b12f0 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 -- 172.21.15.29:0/336201904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 msgr2=0x55c34cb89b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 --2- 172.21.15.29:0/336201904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb89b80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faa44005c30 tx=0x7faa4400b760 comp rx=0 tx=0).stop 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 -- 172.21.15.29:0/336201904 shutdown_connections 2024-09-21T02:47:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 --2- 172.21.15.29:0/336201904 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb7e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 --2- 172.21.15.29:0/336201904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b94d0 0x55c34cb8a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 --2- 172.21.15.29:0/336201904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b12f0 0x55c34cb89b80 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 -- 172.21.15.29:0/336201904 >> 172.21.15.29:0/336201904 conn(0x55c34c7ce300 msgr2=0x55c34cb81df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 -- 172.21.15.29:0/336201904 shutdown_connections 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.433+0000 7faa65aad740 1 -- 172.21.15.29:0/336201904 wait complete. 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 1 -- start start 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 1 Processor -- start 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b12f0 0x55c34cb90820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb90ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb91a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa581afc80 con 0x55c34c7b94d0 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7faa581af5b0 con 0x55c34c7da1b0 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa581ac7f0 con 0x55c34c7b12f0 2024-09-21T02:47:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa65aad740 10 client.0 _ll_register_callbacks cb 0x7faa4c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa5c8d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb90ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa5c8d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb90ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60172/0 (socket says 172.21.15.29:60172) 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.434+0000 7faa5c8d5640 1 -- 172.21.15.29:0/368582163 learned_addr learned my addr 172.21.15.29:0/368582163 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa57fff640 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b12f0 0x55c34cb90820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa57fff640 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb91a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa5c8d5640 1 -- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b12f0 msgr2=0x55c34cb90820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b12f0 0x55c34cb90820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa5c8d5640 1 -- 172.21.15.29:0/368582163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 msgr2=0x55c34cb91a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb91a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa5c8d5640 1 -- 172.21.15.29:0/368582163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c34cb921b0 con 0x55c34c7b94d0 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa57fff640 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b12f0 0x55c34cb90820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa5c8d5640 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb90ee0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x55c34cb991d0 tx=0x7faa4c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58252e6a81ab180 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa37fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa37fff640 1 -- 172.21.15.29:0/368582163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c005220 con 0x55c34c7b94d0 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa37fff640 1 -- 172.21.15.29:0/368582163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7faa4c0053c0 con 0x55c34c7b94d0 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa65aad740 1 -- 172.21.15.29:0/368582163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa440052c0 con 0x55c34c7b94d0 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa37fff640 1 -- 172.21.15.29:0/368582163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa4c02f050 con 0x55c34c7b94d0 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa65aad740 10 client.16683 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.435+0000 7faa65aad740 1 -- 172.21.15.29:0/368582163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c34cb93540 con 0x55c34c7b94d0 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa37fff640 1 -- 172.21.15.29:0/368582163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa440052c0 con 0x55c34c7b94d0 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa37fff640 1 -- 172.21.15.29:0/368582163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7faa4c011770 con 0x55c34c7b94d0 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa65aad740 10 client.16683 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa65aad740 10 client.16683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa65aad740 20 client.16683 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa65aad740 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c34c76e1c0 0x55c34cc05900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa65aad740 4 client.16683 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:47:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa65aad740 1 -- 172.21.15.29:0/368582163 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c34caf20f0 con 0x55c34c76e1c0 2024-09-21T02:47:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.436+0000 7faa57fff640 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c34c76e1c0 0x55c34cc05900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.437+0000 7faa57fff640 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c34c76e1c0 0x55c34cc05900 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f095225bdbd9fa9 server_cookie=b94793e82f6385f3 in_seq=0 out_seq=0 2024-09-21T02:47:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.437+0000 7faa37fff640 10 client.16683 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.466+0000 7faa37fff640 1 -- 172.21.15.29:0/368582163 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c34caf20f0 con 0x55c34c76e1c0 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.466+0000 7faa37fff640 10 client.16683 handle_command_reply: tid=0 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 client.16683 shutdown 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 -- 172.21.15.29:0/368582163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 msgr2=0x55c34cb90ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb90ee0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x55c34cb991d0 tx=0x7faa4c0039c0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 -- 172.21.15.29:0/368582163 shutdown_connections 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c34c7da1b0 0x55c34cb91a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c34c76e1c0 0x55c34cc05900 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c34c7b94d0 0x55c34cb90ee0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 --2- 172.21.15.29:0/368582163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c34c7b12f0 0x55c34cb90820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa37fff640 0 client.16683 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 -- 172.21.15.29:0/368582163 >> 172.21.15.29:0/368582163 conn(0x55c34c7ce300 msgr2=0x55c34cb89b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 -- 172.21.15.29:0/368582163 shutdown_connections 2024-09-21T02:47:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 1 -- 172.21.15.29:0/368582163 wait complete. 2024-09-21T02:47:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.467+0000 7faa65aad740 20 client.16683 trim_cache size 0 max 16384 2024-09-21T02:47:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:47:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:47:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:47:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:47:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 -- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faa480b6510 msgr2=0x7faa480b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faa480b6510 0x7faa480b8a20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faa500ca8b0 tx=0x7faa5009b720 comp rx=0 tx=0).stop 2024-09-21T02:47:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 -- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 msgr2=0x7faa58103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 0x7faa58103060 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7faa50009dc0 tx=0x7faa50003ba0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 -- 172.21.15.29:0/391829330 shutdown_connections 2024-09-21T02:47:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7faa480b6510 0x7faa480b8a20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa58109580 0x7faa58103060 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa58106c20 0x7faa58102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 --2- 172.21.15.29:0/391829330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa58101ad0 0x7faa58104260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 -- 172.21.15.29:0/391829330 >> 172.21.15.29:0/391829330 conn(0x7faa580fd690 msgr2=0x7faa58107b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 -- 172.21.15.29:0/391829330 shutdown_connections 2024-09-21T02:47:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:12.468+0000 7faa5eb39640 1 -- 172.21.15.29:0/391829330 wait complete. 2024-09-21T02:47:13.090 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.27 MB/sec execute 290 sec latency 3657.641 ms 2024-09-21T02:47:14.090 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.26 MB/sec execute 291 sec latency 4657.748 ms 2024-09-21T02:47:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:13 smithi190 ceph-mon[33871]: pgmap v632: 129 pgs: 129 active+clean; 36 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 335 KiB/s wr, 34 op/s 2024-09-21T02:47:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:13 smithi029 ceph-mon[31006]: pgmap v632: 129 pgs: 129 active+clean; 36 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 335 KiB/s wr, 34 op/s 2024-09-21T02:47:14.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:13 smithi066 ceph-mon[33436]: pgmap v632: 129 pgs: 129 active+clean; 36 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 335 KiB/s wr, 34 op/s 2024-09-21T02:47:15.090 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.25 MB/sec execute 292 sec latency 5657.859 ms 2024-09-21T02:47:16.090 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.24 MB/sec execute 293 sec latency 6657.943 ms 2024-09-21T02:47:16.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:15 smithi190 ceph-mon[33871]: pgmap v633: 129 pgs: 129 active+clean; 38 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 497 KiB/s wr, 53 op/s 2024-09-21T02:47:16.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:15 smithi029 ceph-mon[31006]: pgmap v633: 129 pgs: 129 active+clean; 38 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 497 KiB/s wr, 53 op/s 2024-09-21T02:47:16.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:15 smithi066 ceph-mon[33436]: pgmap v633: 129 pgs: 129 active+clean; 38 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 497 KiB/s wr, 53 op/s 2024-09-21T02:47:17.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.24 MB/sec execute 294 sec latency 7658.039 ms 2024-09-21T02:47:18.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.23 MB/sec execute 295 sec latency 8658.188 ms 2024-09-21T02:47:18.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:17 smithi190 ceph-mon[33871]: pgmap v634: 129 pgs: 129 active+clean; 38 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 497 KiB/s wr, 53 op/s 2024-09-21T02:47:18.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:17 smithi029 ceph-mon[31006]: pgmap v634: 129 pgs: 129 active+clean; 38 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 497 KiB/s wr, 53 op/s 2024-09-21T02:47:18.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:17 smithi066 ceph-mon[33436]: pgmap v634: 129 pgs: 129 active+clean; 38 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 497 KiB/s wr, 53 op/s 2024-09-21T02:47:19.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.22 MB/sec execute 296 sec latency 9658.288 ms 2024-09-21T02:47:20.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.21 MB/sec execute 297 sec latency 10658.434 ms 2024-09-21T02:47:20.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:19 smithi190 ceph-mon[33871]: pgmap v635: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 497 KiB/s wr, 55 op/s 2024-09-21T02:47:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:19 smithi029 ceph-mon[31006]: pgmap v635: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 497 KiB/s wr, 55 op/s 2024-09-21T02:47:20.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:19 smithi066 ceph-mon[33436]: pgmap v635: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 497 KiB/s wr, 55 op/s 2024-09-21T02:47:21.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.21 MB/sec execute 298 sec latency 11658.529 ms 2024-09-21T02:47:22.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.20 MB/sec execute 299 sec latency 12658.654 ms 2024-09-21T02:47:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:21 smithi190 ceph-mon[33871]: pgmap v636: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 449 KiB/s wr, 47 op/s 2024-09-21T02:47:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:21 smithi029 ceph-mon[31006]: pgmap v636: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 449 KiB/s wr, 47 op/s 2024-09-21T02:47:22.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:21 smithi066 ceph-mon[33436]: pgmap v636: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 449 KiB/s wr, 47 op/s 2024-09-21T02:47:23.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.19 MB/sec execute 300 sec latency 13658.747 ms 2024-09-21T02:47:24.062 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:23 smithi066 ceph-mon[33436]: pgmap v637: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 318 KiB/s wr, 41 op/s 2024-09-21T02:47:24.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.18 MB/sec execute 301 sec latency 14658.850 ms 2024-09-21T02:47:24.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:23 smithi190 ceph-mon[33871]: pgmap v637: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 318 KiB/s wr, 41 op/s 2024-09-21T02:47:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:23 smithi029 ceph-mon[31006]: pgmap v637: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 318 KiB/s wr, 41 op/s 2024-09-21T02:47:25.091 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.18 MB/sec execute 302 sec latency 15658.937 ms 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 -- 172.21.15.29:0/172260153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee901025b0 msgr2=0x7fee90102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 --2- 172.21.15.29:0/172260153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee901025b0 0x7fee90102a20 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fee88009ca0 tx=0x7fee8802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 -- 172.21.15.29:0/172260153 shutdown_connections 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 --2- 172.21.15.29:0/172260153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90103fd0 0x7fee9010a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 --2- 172.21.15.29:0/172260153 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee90103630 0x7fee90103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 --2- 172.21.15.29:0/172260153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee901025b0 0x7fee90102a20 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 -- 172.21.15.29:0/172260153 >> 172.21.15.29:0/172260153 conn(0x7fee900fdf30 msgr2=0x7fee901003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 -- 172.21.15.29:0/172260153 shutdown_connections 2024-09-21T02:47:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.581+0000 7fee94af9640 1 -- 172.21.15.29:0/172260153 wait complete. 2024-09-21T02:47:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.582+0000 7fee94af9640 1 Processor -- start 2024-09-21T02:47:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.582+0000 7fee94af9640 1 -- start start 2024-09-21T02:47:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.582+0000 7fee94af9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee901025b0 0x7fee901061a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.582+0000 7fee94af9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90103630 0x7fee90104a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.582+0000 7fee94af9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee90103fd0 0x7fee90104fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.582+0000 7fee94af9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fee9010e0c0 con 0x7fee90103fd0 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.582+0000 7fee94af9640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fee9010df40 con 0x7fee901025b0 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.582+0000 7fee94af9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee9010e240 con 0x7fee90103630 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee90103fd0 0x7fee90104fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee90103fd0 0x7fee90104fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58580/0 (socket says 172.21.15.29:58580) 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 -- 172.21.15.29:0/3454190556 learned_addr learned my addr 172.21.15.29:0/3454190556 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8e59c640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee901025b0 0x7fee901061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90103630 0x7fee90104a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 -- 172.21.15.29:0/3454190556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90103630 msgr2=0x7fee90104a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90103630 0x7fee90104a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 -- 172.21.15.29:0/3454190556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee901025b0 msgr2=0x7fee901061a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee901025b0 0x7fee901061a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 -- 172.21.15.29:0/3454190556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee901056d0 con 0x7fee90103fd0 2024-09-21T02:47:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee8ed9d640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee90103fd0 0x7fee90104fa0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fee80009fd0 tx=0x7fee8000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee877fe640 1 -- 172.21.15.29:0/3454190556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80011830 con 0x7fee90103fd0 2024-09-21T02:47:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee877fe640 1 -- 172.21.15.29:0/3454190556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee80016070 con 0x7fee90103fd0 2024-09-21T02:47:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee94af9640 1 -- 172.21.15.29:0/3454190556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee901a3fa0 con 0x7fee90103fd0 2024-09-21T02:47:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.583+0000 7fee877fe640 1 -- 172.21.15.29:0/3454190556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80010d20 con 0x7fee90103fd0 2024-09-21T02:47:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.584+0000 7fee94af9640 1 -- 172.21.15.29:0/3454190556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee901a43b0 con 0x7fee90103fd0 2024-09-21T02:47:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.586+0000 7fee9ba65740 1 Processor -- start 2024-09-21T02:47:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.586+0000 7fee877fe640 1 -- 172.21.15.29:0/3454190556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fee8001b460 con 0x7fee90103fd0 2024-09-21T02:47:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee9ba65740 1 -- start start 2024-09-21T02:47:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5ae1b0 0x55d13f955e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5852f0 0x55d13f9564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f957010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee9ba65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d13f8c86f0 con 0x55d13f58d4d0 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee9ba65740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d13f8b7b20 con 0x55d13f5ae1b0 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee9ba65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d13f8bb930 con 0x55d13f5852f0 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f957010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f957010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58582/0 (socket says 172.21.15.29:58582) 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 -- 172.21.15.29:0/44004970 learned_addr learned my addr 172.21.15.29:0/44004970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8e59c640 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5852f0 0x55d13f9564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5ae1b0 0x55d13f955e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee877fe640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee7c0b6510 0x7fee7c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 -- 172.21.15.29:0/44004970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5852f0 msgr2=0x55d13f9564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5852f0 0x55d13f9564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 -- 172.21.15.29:0/44004970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5ae1b0 msgr2=0x55d13f955e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5ae1b0 0x55d13f955e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.587+0000 7fee8dd9b640 1 -- 172.21.15.29:0/44004970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d13f87d530 con 0x55d13f58d4d0 2024-09-21T02:47:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.588+0000 7fee8ed9d640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee7c0b6510 0x7fee7c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.588+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f957010 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fee78011560 tx=0x7fee7800a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5138feb5a6bd73 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.588+0000 7fee877fe640 1 -- 172.21.15.29:0/3454190556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fee800e6aa0 con 0x7fee90103fd0 2024-09-21T02:47:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.588+0000 7fee84ff9640 1 -- 172.21.15.29:0/44004970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78004220 con 0x55d13f58d4d0 2024-09-21T02:47:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.588+0000 7fee84ff9640 1 -- 172.21.15.29:0/44004970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee780043c0 con 0x55d13f58d4d0 2024-09-21T02:47:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.588+0000 7fee84ff9640 1 -- 172.21.15.29:0/44004970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78004db0 con 0x55d13f58d4d0 2024-09-21T02:47:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 -- 172.21.15.29:0/44004970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 msgr2=0x55d13f957010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee8ed9d640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee7c0b6510 0x7fee7c0b8a20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fee800ac5b0 tx=0x7fee800b0040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f957010 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fee78011560 tx=0x7fee7800a790 comp rx=0 tx=0).stop 2024-09-21T02:47:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 -- 172.21.15.29:0/44004970 shutdown_connections 2024-09-21T02:47:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f957010 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5852f0 0x55d13f9564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 --2- 172.21.15.29:0/44004970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5ae1b0 0x55d13f955e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 -- 172.21.15.29:0/44004970 >> 172.21.15.29:0/44004970 conn(0x55d13f5a2300 msgr2=0x55d13f9518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 -- 172.21.15.29:0/44004970 shutdown_connections 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 -- 172.21.15.29:0/44004970 wait complete. 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.589+0000 7fee9ba65740 1 -- start start 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee9ba65740 1 Processor -- start 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 0x55d13f95f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f95fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 0x55d13f960850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee9ba65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d13f7efab0 con 0x55d13f58d4d0 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee9ba65740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d13f85d730 con 0x55d13f5852f0 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee9ba65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d13f957b30 con 0x55d13f5ae1b0 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee9ba65740 10 client.0 _ll_register_callbacks cb 0x55d13f8b16b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f95fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f95fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58588/0 (socket says 172.21.15.29:58588) 2024-09-21T02:47:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8e59c640 1 -- 172.21.15.29:0/2198641290 learned_addr learned my addr 172.21.15.29:0/2198641290 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 0x55d13f95f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 0x55d13f960850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8e59c640 1 -- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 msgr2=0x55d13f960850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8e59c640 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 0x55d13f960850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8e59c640 1 -- 172.21.15.29:0/2198641290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 msgr2=0x55d13f95f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8e59c640 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 0x55d13f95f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8e59c640 1 -- 172.21.15.29:0/2198641290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d13f619790 con 0x55d13f58d4d0 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.590+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 0x55d13f960850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee8e59c640 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f95fce0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fee88035000 tx=0x7fee88031cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc45a574afcfd332 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee75ffb640 1 -- 172.21.15.29:0/2198641290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8804b060 con 0x55d13f58d4d0 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee9ba65740 1 -- 172.21.15.29:0/2198641290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee780043c0 con 0x55d13f58d4d0 2024-09-21T02:47:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee75ffb640 1 -- 172.21.15.29:0/2198641290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fee88003b80 con 0x55d13f58d4d0 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee75ffb640 1 -- 172.21.15.29:0/2198641290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fee88052050 con 0x55d13f58d4d0 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee9ba65740 10 client.16695 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee9ba65740 1 -- 172.21.15.29:0/2198641290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d13f692620 con 0x55d13f58d4d0 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.591+0000 7fee75ffb640 1 -- 172.21.15.29:0/2198641290 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee780043c0 con 0x55d13f58d4d0 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.592+0000 7fee75ffb640 1 -- 172.21.15.29:0/2198641290 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fee880305c0 con 0x55d13f58d4d0 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.592+0000 7fee9ba65740 10 client.16695 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.592+0000 7fee9ba65740 10 client.16695 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.592+0000 7fee9ba65740 20 client.16695 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.592+0000 7fee9ba65740 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d13f56cd80 0x55d13f96cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.592+0000 7fee9ba65740 4 client.16695 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.592+0000 7fee9ba65740 1 -- 172.21.15.29:0/2198641290 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d13f8ca640 con 0x55d13f56cd80 2024-09-21T02:47:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.592+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d13f56cd80 0x55d13f96cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.593+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d13f56cd80 0x55d13f96cd50 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4766b3aab1a17e9f server_cookie=df1e56eb85435993 in_seq=0 out_seq=0 2024-09-21T02:47:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.593+0000 7fee75ffb640 10 client.16695 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.593+0000 7fee75ffb640 1 -- 172.21.15.29:0/2198641290 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d13f8ca640 con 0x55d13f56cd80 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.593+0000 7fee75ffb640 10 client.16695 handle_command_reply: tid=0 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.593+0000 7fee9ba65740 1 client.16695 shutdown 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 -- 172.21.15.29:0/2198641290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 msgr2=0x55d13f95fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f95fce0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fee88035000 tx=0x7fee88031cb0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 -- 172.21.15.29:0/2198641290 shutdown_connections 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 0x55d13f960850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d13f56cd80 0x55d13f96cd50 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f58d4d0 0x55d13f95fce0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee75ffb640 0 client.16695 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 --2- 172.21.15.29:0/2198641290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 0x55d13f95f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 -- 172.21.15.29:0/2198641290 >> 172.21.15.29:0/2198641290 conn(0x55d13f5a2300 msgr2=0x55d13f94fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 -- 172.21.15.29:0/2198641290 shutdown_connections 2024-09-21T02:47:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 1 -- 172.21.15.29:0/2198641290 wait complete. 2024-09-21T02:47:25.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.594+0000 7fee9ba65740 20 client.16695 trim_cache size 0 max 16384 2024-09-21T02:47:25.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee9ba65740 1 Processor -- start 2024-09-21T02:47:25.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee9ba65740 1 -- start start 2024-09-21T02:47:25.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5852f0 0x55d13f95e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f58d4d0 0x55d13f95f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 0x55d13f95fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee9ba65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fee901ae7d0 con 0x55d13f5852f0 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee9ba65740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fee901b0310 con 0x55d13f58d4d0 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee9ba65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee901aeea0 con 0x55d13f5ae1b0 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5852f0 0x55d13f95e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5852f0 0x55d13f95e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58598/0 (socket says 172.21.15.29:58598) 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8dd9b640 1 -- 172.21.15.29:0/4268496682 learned_addr learned my addr 172.21.15.29:0/4268496682 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8dd9b640 1 -- 172.21.15.29:0/4268496682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 msgr2=0x55d13f95fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 0x55d13f95fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8dd9b640 1 -- 172.21.15.29:0/4268496682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f58d4d0 msgr2=0x55d13f95f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8e59c640 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f58d4d0 0x55d13f95f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f58d4d0 0x55d13f95f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8dd9b640 1 -- 172.21.15.29:0/4268496682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d13f95d7f0 con 0x55d13f5852f0 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.609+0000 7fee8e59c640 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f58d4d0 0x55d13f95f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5852f0 0x55d13f95e280 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fee78011eb0 tx=0x7fee78004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fe99af323d4cab server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee77fff640 1 -- 172.21.15.29:0/4268496682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78004660 con 0x55d13f5852f0 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee77fff640 1 -- 172.21.15.29:0/4268496682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee78033070 con 0x55d13f5852f0 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee77fff640 1 -- 172.21.15.29:0/4268496682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7800f8f0 con 0x55d13f5852f0 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee9ba65740 1 -- 172.21.15.29:0/4268496682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5852f0 msgr2=0x55d13f95e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee9ba65740 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5852f0 0x55d13f95e280 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fee78011eb0 tx=0x7fee78004f90 comp rx=0 tx=0).stop 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee9ba65740 1 -- 172.21.15.29:0/4268496682 shutdown_connections 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee9ba65740 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f5ae1b0 0x55d13f95fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee9ba65740 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f58d4d0 0x55d13f95f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee9ba65740 1 --2- 172.21.15.29:0/4268496682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5852f0 0x55d13f95e280 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.610+0000 7fee9ba65740 1 -- 172.21.15.29:0/4268496682 >> 172.21.15.29:0/4268496682 conn(0x55d13f5a2300 msgr2=0x55d13f95db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 -- 172.21.15.29:0/4268496682 shutdown_connections 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 -- 172.21.15.29:0/4268496682 wait complete. 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 -- start start 2024-09-21T02:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 Processor -- start 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 0x55d13f964a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f58d4d0 0x55d13f9650c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5ae1b0 0x55d13f965c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fee901afc40 con 0x55d13f5ae1b0 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fee901af570 con 0x55d13f5852f0 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee901ac7b0 con 0x55d13f58d4d0 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee9ba65740 10 client.0 _ll_register_callbacks cb 0x7fee88009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5ae1b0 0x55d13f965c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5ae1b0 0x55d13f965c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58608/0 (socket says 172.21.15.29:58608) 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee8dd9b640 1 -- 172.21.15.29:0/1291400321 learned_addr learned my addr 172.21.15.29:0/1291400321 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.611+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 0x55d13f964a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee8e59c640 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f58d4d0 0x55d13f9650c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee8dd9b640 1 -- 172.21.15.29:0/1291400321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f58d4d0 msgr2=0x55d13f9650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f58d4d0 0x55d13f9650c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee8dd9b640 1 -- 172.21.15.29:0/1291400321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 msgr2=0x55d13f964a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 0x55d13f964a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee8dd9b640 1 -- 172.21.15.29:0/1291400321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d13f966390 con 0x55d13f5ae1b0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee8dd9b640 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5ae1b0 0x55d13f965c30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fee7800a410 tx=0x7fee78013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fb4a4e13a9b9ab server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee75ffb640 1 -- 172.21.15.29:0/1291400321 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7800f890 con 0x55d13f5ae1b0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee75ffb640 1 -- 172.21.15.29:0/1291400321 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fee7800fad0 con 0x55d13f5ae1b0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee9ba65740 1 -- 172.21.15.29:0/1291400321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee78033070 con 0x55d13f5ae1b0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee75ffb640 1 -- 172.21.15.29:0/1291400321 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fee7800fcd0 con 0x55d13f5ae1b0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee9ba65740 10 client.16701 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.612+0000 7fee9ba65740 1 -- 172.21.15.29:0/1291400321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d13f9675f0 con 0x55d13f5ae1b0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee75ffb640 1 -- 172.21.15.29:0/1291400321 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee78033070 con 0x55d13f5ae1b0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee75ffb640 1 -- 172.21.15.29:0/1291400321 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fee78044030 con 0x55d13f5ae1b0 2024-09-21T02:47:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee9ba65740 10 client.16701 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee9ba65740 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:47:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee9ba65740 20 client.16701 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee9ba65740 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d13f5421c0 0x55d13f9d9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee9ba65740 4 client.16701 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:47:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee9ba65740 1 -- 172.21.15.29:0/1291400321 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d13f8c60f0 con 0x55d13f5421c0 2024-09-21T02:47:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.613+0000 7fee8e59c640 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d13f5421c0 0x55d13f9d9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.614+0000 7fee75ffb640 10 client.16701 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.614+0000 7fee8e59c640 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d13f5421c0 0x55d13f9d9900 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bce5ea82c179a1d2 server_cookie=31ff8a293791a16f in_seq=0 out_seq=0 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee75ffb640 1 -- 172.21.15.29:0/1291400321 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d13f8c60f0 con 0x55d13f5421c0 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee75ffb640 10 client.16701 handle_command_reply: tid=0 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 client.16701 shutdown 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 -- 172.21.15.29:0/1291400321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5ae1b0 msgr2=0x55d13f965c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5ae1b0 0x55d13f965c30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fee7800a410 tx=0x7fee78013040 comp rx=0 tx=0).stop 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 -- 172.21.15.29:0/1291400321 shutdown_connections 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d13f5421c0 0x55d13f9d9900 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d13f5ae1b0 0x55d13f965c30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee75ffb640 0 client.16701 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d13f58d4d0 0x55d13f9650c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 --2- 172.21.15.29:0/1291400321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d13f5852f0 0x55d13f964a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 -- 172.21.15.29:0/1291400321 >> 172.21.15.29:0/1291400321 conn(0x55d13f5a2300 msgr2=0x55d13f95f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 -- 172.21.15.29:0/1291400321 shutdown_connections 2024-09-21T02:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 1 -- 172.21.15.29:0/1291400321 wait complete. 2024-09-21T02:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.626+0000 7fee9ba65740 20 client.16701 trim_cache size 0 max 16384 2024-09-21T02:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 -- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee7c0b6510 msgr2=0x7fee7c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee7c0b6510 0x7fee7c0b8a20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fee800ac5b0 tx=0x7fee800b0040 comp rx=0 tx=0).stop 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 -- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee90103fd0 msgr2=0x7fee90104fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee90103fd0 0x7fee90104fa0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fee80009fd0 tx=0x7fee8000b8e0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 -- 172.21.15.29:0/3454190556 shutdown_connections 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fee7c0b6510 0x7fee7c0b8a20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee90103fd0 0x7fee90104fa0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90103630 0x7fee90104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 --2- 172.21.15.29:0/3454190556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee901025b0 0x7fee901061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 -- 172.21.15.29:0/3454190556 >> 172.21.15.29:0/3454190556 conn(0x7fee900fdf30 msgr2=0x7fee900ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 -- 172.21.15.29:0/3454190556 shutdown_connections 2024-09-21T02:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:25.627+0000 7fee94af9640 1 -- 172.21.15.29:0/3454190556 wait complete. 2024-09-21T02:47:26.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.17 MB/sec execute 303 sec latency 16659.042 ms 2024-09-21T02:47:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:25 smithi190 ceph-mon[33871]: pgmap v638: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 166 KiB/s wr, 35 op/s 2024-09-21T02:47:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:25 smithi029 ceph-mon[31006]: pgmap v638: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 166 KiB/s wr, 35 op/s 2024-09-21T02:47:26.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:25 smithi066 ceph-mon[33436]: pgmap v638: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 166 KiB/s wr, 35 op/s 2024-09-21T02:47:27.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.16 MB/sec execute 304 sec latency 17659.137 ms 2024-09-21T02:47:27.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:47:27.874 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:47:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:27 smithi029 ceph-mon[31006]: pgmap v639: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-09-21T02:47:28.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.15 MB/sec execute 305 sec latency 18659.245 ms 2024-09-21T02:47:28.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:27 smithi190 ceph-mon[33871]: pgmap v639: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-09-21T02:47:28.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:27 smithi066 ceph-mon[33436]: pgmap v639: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-09-21T02:47:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 -- 172.21.15.29:0/1167932316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f0105d90 msgr2=0x7f75f01061a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 --2- 172.21.15.29:0/1167932316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f0105d90 0x7f75f01061a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009900 tx=0x7f75e002a410 comp rx=0 tx=0).stop 2024-09-21T02:47:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 -- 172.21.15.29:0/1167932316 shutdown_connections 2024-09-21T02:47:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 --2- 172.21.15.29:0/1167932316 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75f010c020 0x7f75f0112850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 --2- 172.21.15.29:0/1167932316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75f010b6d0 0x7f75f010bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 --2- 172.21.15.29:0/1167932316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f0105d90 0x7f75f01061a0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 -- 172.21.15.29:0/1167932316 >> 172.21.15.29:0/1167932316 conn(0x7f75f0101860 msgr2=0x7f75f0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 -- 172.21.15.29:0/1167932316 shutdown_connections 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.587+0000 7f75f50e0640 1 -- 172.21.15.29:0/1167932316 wait complete. 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75f50e0640 1 Processor -- start 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75f50e0640 1 -- start start 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75f50e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75f0105d90 0x7f75f010fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75f50e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f010b6d0 0x7f75f0110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75f50e0640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75f010c020 0x7f75f01107d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75f50e0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f75f0115b90 con 0x7f75f010b6d0 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75f50e0640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f75f0115a10 con 0x7f75f010c020 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75f50e0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f75f0115d10 con 0x7f75f0105d90 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f010b6d0 0x7f75f0110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f010b6d0 0x7f75f0110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58636/0 (socket says 172.21.15.29:58636) 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75e7fff640 1 -- 172.21.15.29:0/3336636742 learned_addr learned my addr 172.21.15.29:0/3336636742 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75ef59e640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75f010c020 0x7f75f01107d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.588+0000 7f75eed9d640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75f0105d90 0x7f75f010fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75e7fff640 1 -- 172.21.15.29:0/3336636742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75f0105d90 msgr2=0x7f75f010fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75e7fff640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75f0105d90 0x7f75f010fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75e7fff640 1 -- 172.21.15.29:0/3336636742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75f010c020 msgr2=0x7f75f01107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75e7fff640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75f010c020 0x7f75f01107d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75e7fff640 1 -- 172.21.15.29:0/3336636742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f01acaa0 con 0x7f75f010b6d0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75e7fff640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f010b6d0 0x7f75f0110290 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f75d800d740 tx=0x7f75d800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75ecd99640 1 -- 172.21.15.29:0/3336636742 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d8017070 con 0x7f75f010b6d0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f01acd90 con 0x7f75f010b6d0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75ecd99640 1 -- 172.21.15.29:0/3336636742 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75d800de50 con 0x7f75f010b6d0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75ecd99640 1 -- 172.21.15.29:0/3336636742 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d800f620 con 0x7f75f010b6d0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.589+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f01ad240 con 0x7f75f010b6d0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.590+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f01061a0 con 0x7f75f010b6d0 2024-09-21T02:47:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.592+0000 7f75ecd99640 1 -- 172.21.15.29:0/3336636742 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f75d801a070 con 0x7f75f010b6d0 2024-09-21T02:47:28.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.593+0000 7f75ecd99640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f75dc0b6510 0x7f75dc0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:28.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.593+0000 7f75eed9d640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f75dc0b6510 0x7f75dc0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:28.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.593+0000 7f75ecd99640 1 -- 172.21.15.29:0/3336636742 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f75d80e6fc0 con 0x7f75f010b6d0 2024-09-21T02:47:28.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.593+0000 7f75eed9d640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f75dc0b6510 0x7f75dc0b8a20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f75f01127b0 tx=0x7f75e00058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:28.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.596+0000 7f75ecd99640 1 -- 172.21.15.29:0/3336636742 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f75d80b18b0 con 0x7f75f010b6d0 2024-09-21T02:47:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.824+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75f0111050 con 0x7f75f010b6d0 2024-09-21T02:47:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.824+0000 7f75ecd99640 1 -- 172.21.15.29:0/3336636742 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f75d80b67f0 con 0x7f75f010b6d0 2024-09-21T02:47:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:47:28.826 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:47:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f75dc0b6510 msgr2=0x7f75dc0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f75dc0b6510 0x7f75dc0b8a20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f75f01127b0 tx=0x7f75e00058d0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f010b6d0 msgr2=0x7f75f0110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f010b6d0 0x7f75f0110290 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f75d800d740 tx=0x7f75d800dc00 comp rx=0 tx=0).stop 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 shutdown_connections 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f75dc0b6510 0x7f75dc0b8a20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75f010c020 0x7f75f01107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f010b6d0 0x7f75f0110290 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 --2- 172.21.15.29:0/3336636742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75f0105d90 0x7f75f010fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 >> 172.21.15.29:0/3336636742 conn(0x7f75f0101860 msgr2=0x7f75f0103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 shutdown_connections 2024-09-21T02:47:28.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:28.827+0000 7f75f50e0640 1 -- 172.21.15.29:0/3336636742 wait complete. 2024-09-21T02:47:29.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.15 MB/sec execute 306 sec latency 19659.346 ms 2024-09-21T02:47:29.989 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 3fbf52c0 --release --await' 2024-09-21T02:47:29.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 3fbf52c0 --release --await 2024-09-21T02:47:30.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 151917 2.14 MB/sec execute 307 sec latency 20659.389 ms 2024-09-21T02:47:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:29 smithi190 ceph-mon[33871]: pgmap v640: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-21T02:47:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:29 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3336636742' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:29 smithi029 ceph-mon[31006]: pgmap v640: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-21T02:47:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:29 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3336636742' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:47:30.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:29 smithi066 ceph-mon[33436]: pgmap v640: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-21T02:47:30.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:29 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3336636742' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:47:30.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 -- 172.21.15.29:0/1338102085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3341083a0 msgr2=0x7fb3341087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 --2- 172.21.15.29:0/1338102085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3341083a0 0x7fb3341087b0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb324009ca0 tx=0x7fb32402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 -- 172.21.15.29:0/1338102085 shutdown_connections 2024-09-21T02:47:30.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 --2- 172.21.15.29:0/1338102085 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb334108cf0 0x7fb33410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 --2- 172.21.15.29:0/1338102085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3341083a0 0x7fb3341087b0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 --2- 172.21.15.29:0/1338102085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb334102a60 0x7fb334102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 -- 172.21.15.29:0/1338102085 >> 172.21.15.29:0/1338102085 conn(0x7fb334079730 msgr2=0x7fb334079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 -- 172.21.15.29:0/1338102085 shutdown_connections 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.777+0000 7fb33ad00640 1 -- 172.21.15.29:0/1338102085 wait complete. 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33ad00640 1 Processor -- start 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33ad00640 1 -- start start 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33ad00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb334102a60 0x7fb33419c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33ad00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3341083a0 0x7fb33419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33ad00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334108cf0 0x7fb3341a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33ad00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb334112860 con 0x7fb334108cf0 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33ad00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341126e0 con 0x7fb3341083a0 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33ad00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341129e0 con 0x7fb334102a60 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33929d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334108cf0 0x7fb3341a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33929d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334108cf0 0x7fb3341a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58650/0 (socket says 172.21.15.29:58650) 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.778+0000 7fb33929d640 1 -- 172.21.15.29:0/522700852 learned_addr learned my addr 172.21.15.29:0/522700852 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:30.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb338a9c640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb334102a60 0x7fb33419c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb333fff640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3341083a0 0x7fb33419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb33929d640 1 -- 172.21.15.29:0/522700852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb334102a60 msgr2=0x7fb33419c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb33929d640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb334102a60 0x7fb33419c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb33929d640 1 -- 172.21.15.29:0/522700852 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3341083a0 msgr2=0x7fb33419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb33929d640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3341083a0 0x7fb33419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb33929d640 1 -- 172.21.15.29:0/522700852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3341a24b0 con 0x7fb334108cf0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb338a9c640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb334102a60 0x7fb33419c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb333fff640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3341083a0 0x7fb33419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb33929d640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334108cf0 0x7fb3341a1e40 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00c970 tx=0x7fb32c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb331ffb640 1 -- 172.21.15.29:0/522700852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c007bb0 con 0x7fb334108cf0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb33ad00640 1 -- 172.21.15.29:0/522700852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3341a27a0 con 0x7fb334108cf0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb331ffb640 1 -- 172.21.15.29:0/522700852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb32c007d50 con 0x7fb334108cf0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb331ffb640 1 -- 172.21.15.29:0/522700852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c018660 con 0x7fb334108cf0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.779+0000 7fb33ad00640 1 -- 172.21.15.29:0/522700852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3341a8fc0 con 0x7fb334108cf0 2024-09-21T02:47:30.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb341274740 1 Processor -- start 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb331ffb640 1 -- 172.21.15.29:0/522700852 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb32c00f970 con 0x7fb334108cf0 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb341274740 1 -- start start 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb341274740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 0x555c5e897510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb341274740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e897a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb341274740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e897f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb341274740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555c5e782dc0 con 0x555c5e4c8660 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb341274740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555c5e80f1f0 con 0x555c5e4f1520 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb341274740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555c5e7fe620 con 0x555c5e4d0840 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.782+0000 7fb338a9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e897a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb338a9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e897a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58660/0 (socket says 172.21.15.29:58660) 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb338a9c640 1 -- 172.21.15.29:0/2192364466 learned_addr learned my addr 172.21.15.29:0/2192364466 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb333fff640 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 0x555c5e897510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb333fff640 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e897f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb338a9c640 1 -- 172.21.15.29:0/2192364466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 msgr2=0x555c5e897f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb331ffb640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3200b6540 0x7fb3200b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb338a9c640 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e897f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb338a9c640 1 -- 172.21.15.29:0/2192364466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 msgr2=0x555c5e897510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb338a9c640 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 0x555c5e897510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb338a9c640 1 -- 172.21.15.29:0/2192364466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c5e7c6fd0 con 0x555c5e4c8660 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb333fff640 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e897f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb33929d640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3200b6540 0x7fb3200b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb338a9c640 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e897a50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00bd40 tx=0x7fb31c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0f38905a5aa888 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb331ffb640 1 -- 172.21.15.29:0/522700852 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb32c0e7710 con 0x7fb334108cf0 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb32affd640 1 -- 172.21.15.29:0/2192364466 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c015070 con 0x555c5e4c8660 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb32affd640 1 -- 172.21.15.29:0/2192364466 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb31c00be30 con 0x555c5e4c8660 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.783+0000 7fb32affd640 1 -- 172.21.15.29:0/2192364466 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c011890 con 0x555c5e4c8660 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 -- 172.21.15.29:0/2192364466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 msgr2=0x555c5e897a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e897a50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00bd40 tx=0x7fb31c00f730 comp rx=0 tx=0).stop 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb33929d640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3200b6540 0x7fb3200b8a50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0ad220 tx=0x7fb32c0a1dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:30.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 -- 172.21.15.29:0/2192364466 shutdown_connections 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e897f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e897a50 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 --2- 172.21.15.29:0/2192364466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 0x555c5e897510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 -- 172.21.15.29:0/2192364466 >> 172.21.15.29:0/2192364466 conn(0x555c5e4e5910 msgr2=0x555c5e895400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 -- 172.21.15.29:0/2192364466 shutdown_connections 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.784+0000 7fb341274740 1 -- 172.21.15.29:0/2192364466 wait complete. 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.785+0000 7fb341274740 1 -- start start 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.785+0000 7fb341274740 1 Processor -- start 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.785+0000 7fb341274740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4c8660 0x555c5e8a2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.785+0000 7fb341274740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4d0840 0x555c5e8a3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.785+0000 7fb341274740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4f1520 0x555c5e8aee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.785+0000 7fb341274740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555c5e799300 con 0x555c5e4f1520 2024-09-21T02:47:30.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.785+0000 7fb341274740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555c5e802430 con 0x555c5e4d0840 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.785+0000 7fb341274740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555c5e65fbb0 con 0x555c5e4c8660 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb341274740 10 client.0 _ll_register_callbacks cb 0x555c5e7b3120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb333fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4f1520 0x555c5e8aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb338a9c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4d0840 0x555c5e8a3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb333fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4f1520 0x555c5e8aee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58670/0 (socket says 172.21.15.29:58670) 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb333fff640 1 -- 172.21.15.29:0/359145255 learned_addr learned my addr 172.21.15.29:0/359145255 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb333fff640 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4c8660 0x555c5e8a2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb338a9c640 1 -- 172.21.15.29:0/359145255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4c8660 msgr2=0x555c5e8a2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb338a9c640 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4c8660 0x555c5e8a2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb338a9c640 1 -- 172.21.15.29:0/359145255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4f1520 msgr2=0x555c5e8aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb338a9c640 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4f1520 0x555c5e8aee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb338a9c640 1 -- 172.21.15.29:0/359145255 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c5e788780 con 0x555c5e4d0840 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb333fff640 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4f1520 0x555c5e8aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:30.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb338a9c640 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4d0840 0x555c5e8a3100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb31c011570 tx=0x7fb31c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0ebc326c9072789 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.786+0000 7fb313fff640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.789+0000 7fb313fff640 1 -- 172.21.15.29:0/359145255 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c028060 con 0x555c5e4d0840 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.789+0000 7fb313fff640 1 -- 172.21.15.29:0/359145255 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb31c0055f0 con 0x555c5e4d0840 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.789+0000 7fb341274740 1 -- 172.21.15.29:0/359145255 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb31c00be30 con 0x555c5e4d0840 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.789+0000 7fb313fff640 1 -- 172.21.15.29:0/359145255 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb31c02f050 con 0x555c5e4d0840 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.789+0000 7fb341274740 10 client.26183 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.789+0000 7fb341274740 1 -- 172.21.15.29:0/359145255 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555c5e55cb00 con 0x555c5e4d0840 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb313fff640 1 -- 172.21.15.29:0/359145255 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb31c00be30 con 0x555c5e4d0840 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb313fff640 1 -- 172.21.15.29:0/359145255 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb31c0118a0 con 0x555c5e4d0840 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb341274740 10 client.26183 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb341274740 10 client.26183 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb341274740 20 client.26183 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb341274740 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c5e4afea0 0x555c5e8b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb341274740 4 client.26183 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:47:30.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb341274740 1 -- 172.21.15.29:0/359145255 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c5e811140 con 0x555c5e4afea0 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.790+0000 7fb333fff640 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c5e4afea0 0x555c5e8b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.791+0000 7fb313fff640 10 client.26183 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.791+0000 7fb333fff640 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c5e4afea0 0x555c5e8b4d80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fdc131833642573 server_cookie=bc3dbc1001175ae7 in_seq=0 out_seq=0 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.791+0000 7fb313fff640 1 -- 172.21.15.29:0/359145255 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555c5e811140 con 0x555c5e4afea0 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.791+0000 7fb313fff640 10 client.26183 handle_command_reply: tid=0 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 client.26183 shutdown 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 -- 172.21.15.29:0/359145255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4d0840 msgr2=0x555c5e8a3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4d0840 0x555c5e8a3100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb31c011570 tx=0x7fb31c005270 comp rx=0 tx=0).stop 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 -- 172.21.15.29:0/359145255 shutdown_connections 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c5e4afea0 0x555c5e8b4d80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4f1520 0x555c5e8aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4d0840 0x555c5e8a3100 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb313fff640 0 client.26183 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:30.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 --2- 172.21.15.29:0/359145255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4c8660 0x555c5e8a2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 -- 172.21.15.29:0/359145255 >> 172.21.15.29:0/359145255 conn(0x555c5e4e5910 msgr2=0x555c5e89a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:30.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 -- 172.21.15.29:0/359145255 shutdown_connections 2024-09-21T02:47:30.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 1 -- 172.21.15.29:0/359145255 wait complete. 2024-09-21T02:47:30.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.792+0000 7fb341274740 20 client.26183 trim_cache size 0 max 16384 2024-09-21T02:47:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb341274740 1 Processor -- start 2024-09-21T02:47:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb341274740 1 -- start start 2024-09-21T02:47:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb341274740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4c8660 0x555c5e89e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb341274740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4d0840 0x555c5e89ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb341274740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4f1520 0x555c5e898c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb341274740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341b2ad0 con 0x555c5e4d0840 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb341274740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341b4610 con 0x555c5e4c8660 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb341274740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341b31a0 con 0x555c5e4f1520 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb338a9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4d0840 0x555c5e89ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb338a9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4d0840 0x555c5e89ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58686/0 (socket says 172.21.15.29:58686) 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb338a9c640 1 -- 172.21.15.29:0/636134445 learned_addr learned my addr 172.21.15.29:0/636134445 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb333fff640 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4f1520 0x555c5e898c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb338a9c640 1 -- 172.21.15.29:0/636134445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4f1520 msgr2=0x555c5e898c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb338a9c640 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4f1520 0x555c5e898c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb338a9c640 1 -- 172.21.15.29:0/636134445 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4c8660 msgr2=0x555c5e89e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb338a9c640 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4c8660 0x555c5e89e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb338a9c640 1 -- 172.21.15.29:0/636134445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c5e5d3960 con 0x555c5e4d0840 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.810+0000 7fb333fff640 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4f1520 0x555c5e898c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.811+0000 7fb338a9c640 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4d0840 0x555c5e89ac80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0059f0 tx=0x7fb31c00fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622fd94a5e075a59 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.811+0000 7fb32a7fc640 1 -- 172.21.15.29:0/636134445 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c038070 con 0x555c5e4d0840 2024-09-21T02:47:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.811+0000 7fb32a7fc640 1 -- 172.21.15.29:0/636134445 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb31c011a30 con 0x555c5e4d0840 2024-09-21T02:47:30.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.811+0000 7fb32a7fc640 1 -- 172.21.15.29:0/636134445 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c028070 con 0x555c5e4d0840 2024-09-21T02:47:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 -- 172.21.15.29:0/636134445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4d0840 msgr2=0x555c5e89ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4d0840 0x555c5e89ac80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0059f0 tx=0x7fb31c00fd60 comp rx=0 tx=0).stop 2024-09-21T02:47:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 -- 172.21.15.29:0/636134445 shutdown_connections 2024-09-21T02:47:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4f1520 0x555c5e898c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4d0840 0x555c5e89ac80 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 --2- 172.21.15.29:0/636134445 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4c8660 0x555c5e89e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 -- 172.21.15.29:0/636134445 >> 172.21.15.29:0/636134445 conn(0x555c5e4e5910 msgr2=0x555c5e898540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 -- 172.21.15.29:0/636134445 shutdown_connections 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 -- 172.21.15.29:0/636134445 wait complete. 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 -- start start 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.812+0000 7fb341274740 1 Processor -- start 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb341274740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e8b8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb341274740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e8b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb341274740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 0x555c5e8b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb341274740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341b3f40 con 0x555c5e4c8660 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb341274740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341b3870 con 0x555c5e4f1520 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb341274740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341b0ab0 con 0x555c5e4d0840 2024-09-21T02:47:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb341274740 10 client.0 _ll_register_callbacks cb 0x7fb31c00bbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb333fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e8b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb333fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e8b8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58702/0 (socket says 172.21.15.29:58702) 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb333fff640 1 -- 172.21.15.29:0/2065718436 learned_addr learned my addr 172.21.15.29:0/2065718436 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb338a9c640 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e8b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb333fff640 1 -- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 msgr2=0x555c5e8b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb333fff640 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e8b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb333fff640 1 -- 172.21.15.29:0/2065718436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 msgr2=0x555c5e8b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb333fff640 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 0x555c5e8b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb333fff640 1 -- 172.21.15.29:0/2065718436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c5e91e4f0 con 0x555c5e4c8660 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.813+0000 7fb338a9c640 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e8b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb333fff640 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e8b8730 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb324031720 tx=0x7fb324031980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84b7b4945eab7eb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb313fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb313fff640 1 -- 172.21.15.29:0/2065718436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324030910 con 0x555c5e4c8660 2024-09-21T02:47:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb313fff640 1 -- 172.21.15.29:0/2065718436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb324030ab0 con 0x555c5e4c8660 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 1 -- 172.21.15.29:0/2065718436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb31c011a30 con 0x555c5e4c8660 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb313fff640 1 -- 172.21.15.29:0/2065718436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb324030cb0 con 0x555c5e4c8660 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 10 client.16725 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 1 -- 172.21.15.29:0/2065718436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555c5e91f5d0 con 0x555c5e4c8660 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb313fff640 1 -- 172.21.15.29:0/2065718436 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb31c011a30 con 0x555c5e4c8660 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb313fff640 1 -- 172.21.15.29:0/2065718436 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb32403b400 con 0x555c5e4c8660 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 10 client.16725 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 10 client.16725 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 20 client.16725 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c5e485250 0x555c5e921d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 4 client.16725 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3fbf52c0", "release": true, "await": true}] 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.814+0000 7fb341274740 1 -- 172.21.15.29:0/2065718436 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3fbf52c0", "release": true, "await": true}) -- 0x555c5e80cbf0 con 0x555c5e485250 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.815+0000 7fb338a9c640 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c5e485250 0x555c5e921d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.815+0000 7fb338a9c640 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c5e485250 0x555c5e921d30 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=306edfd0bc9518bf server_cookie=b3a487891eee980 in_seq=0 out_seq=0 2024-09-21T02:47:30.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.815+0000 7fb313fff640 10 client.16725 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.826+0000 7fb313fff640 1 -- 172.21.15.29:0/2065718436 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555c5e80cbf0 con 0x555c5e485250 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.826+0000 7fb313fff640 10 client.16725 handle_command_reply: tid=0 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.826+0000 7fb341274740 1 client.16725 shutdown 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 -- 172.21.15.29:0/2065718436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 msgr2=0x555c5e8b8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e8b8730 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb324031720 tx=0x7fb324031980 comp rx=0 tx=0).stop 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 -- 172.21.15.29:0/2065718436 shutdown_connections 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555c5e4f1520 0x555c5e8b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555c5e4d0840 0x555c5e8b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555c5e485250 0x555c5e921d30 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 --2- 172.21.15.29:0/2065718436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c5e4c8660 0x555c5e8b8730 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 -- 172.21.15.29:0/2065718436 >> 172.21.15.29:0/2065718436 conn(0x555c5e4e5910 msgr2=0x555c5e8a3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb313fff640 0 client.16725 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 -- 172.21.15.29:0/2065718436 shutdown_connections 2024-09-21T02:47:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 1 -- 172.21.15.29:0/2065718436 wait complete. 2024-09-21T02:47:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.827+0000 7fb341274740 20 client.16725 trim_cache size 0 max 16384 2024-09-21T02:47:30.829 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 36, 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "3fbf52c0": { 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 36, 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:47:30.830 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 141.1, 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:47:30.831 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 -- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3200b6540 msgr2=0x7fb3200b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3200b6540 0x7fb3200b8a50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0ad220 tx=0x7fb32c0a1dd0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 -- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334108cf0 msgr2=0x7fb3341a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334108cf0 0x7fb3341a1e40 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00c970 tx=0x7fb32c00ce30 comp rx=0 tx=0).stop 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 -- 172.21.15.29:0/522700852 shutdown_connections 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3200b6540 0x7fb3200b8a50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334108cf0 0x7fb3341a1e40 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3341083a0 0x7fb33419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 --2- 172.21.15.29:0/522700852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb334102a60 0x7fb33419c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 -- 172.21.15.29:0/522700852 >> 172.21.15.29:0/522700852 conn(0x7fb334079730 msgr2=0x7fb3341044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 -- 172.21.15.29:0/522700852 shutdown_connections 2024-09-21T02:47:30.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:47:30.828+0000 7fb33ad00640 1 -- 172.21.15.29:0/522700852 wait complete. 2024-09-21T02:47:31.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 152189 2.14 MB/sec execute 308 sec latency 21391.102 ms 2024-09-21T02:47:31.616 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3fbf52c0', total seconds elapsed: 25.0 2024-09-21T02:47:32.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 152715 2.14 MB/sec execute 309 sec latency 12.384 ms 2024-09-21T02:47:32.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:31 smithi190 ceph-mon[33871]: pgmap v641: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-21T02:47:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:31 smithi029 ceph-mon[31006]: pgmap v641: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-21T02:47:32.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:31 smithi066 ceph-mon[33436]: pgmap v641: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-21T02:47:33.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 153000 2.14 MB/sec execute 310 sec latency 13.051 ms 2024-09-21T02:47:34.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 153491 2.14 MB/sec execute 311 sec latency 16.765 ms 2024-09-21T02:47:34.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:33 smithi190 ceph-mon[33871]: pgmap v642: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-09-21T02:47:34.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:47:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:47:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:47:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:33 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:47:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:33 smithi029 ceph-mon[31006]: pgmap v642: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-09-21T02:47:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:47:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:47:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:47:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:33 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:47:34.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:33 smithi066 ceph-mon[33436]: pgmap v642: 129 pgs: 129 active+clean; 34 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-09-21T02:47:34.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:47:34.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:47:34.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:47:34.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:33 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:47:35.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 154122 2.14 MB/sec execute 312 sec latency 14.663 ms 2024-09-21T02:47:36.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 154754 2.14 MB/sec execute 313 sec latency 18.758 ms 2024-09-21T02:47:36.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:35 smithi190 ceph-mon[33871]: pgmap v643: 129 pgs: 129 active+clean; 35 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 102 KiB/s wr, 29 op/s 2024-09-21T02:47:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:35 smithi029 ceph-mon[31006]: pgmap v643: 129 pgs: 129 active+clean; 35 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 102 KiB/s wr, 29 op/s 2024-09-21T02:47:36.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:35 smithi066 ceph-mon[33436]: pgmap v643: 129 pgs: 129 active+clean; 35 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 102 KiB/s wr, 29 op/s 2024-09-21T02:47:37.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 155512 2.15 MB/sec execute 314 sec latency 14.501 ms 2024-09-21T02:47:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:36 smithi190 ceph-mon[33871]: pgmap v644: 129 pgs: 129 active+clean; 35 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 360 KiB/s wr, 31 op/s 2024-09-21T02:47:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:36 smithi029 ceph-mon[31006]: pgmap v644: 129 pgs: 129 active+clean; 35 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 360 KiB/s wr, 31 op/s 2024-09-21T02:47:37.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:36 smithi066 ceph-mon[33436]: pgmap v644: 129 pgs: 129 active+clean; 35 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 360 KiB/s wr, 31 op/s 2024-09-21T02:47:38.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 156116 2.16 MB/sec execute 315 sec latency 14.100 ms 2024-09-21T02:47:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 -- 172.21.15.29:0/707187075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b401025a0 msgr2=0x7f9b40102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 --2- 172.21.15.29:0/707187075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b401025a0 0x7f9b40102a10 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c02a410 comp rx=0 tx=0).stop 2024-09-21T02:47:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 -- 172.21.15.29:0/707187075 shutdown_connections 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 --2- 172.21.15.29:0/707187075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b40103fc0 0x7f9b4010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 --2- 172.21.15.29:0/707187075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9b40103620 0x7f9b40103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 --2- 172.21.15.29:0/707187075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b401025a0 0x7f9b40102a10 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 -- 172.21.15.29:0/707187075 >> 172.21.15.29:0/707187075 conn(0x7f9b400fdee0 msgr2=0x7f9b40100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 -- 172.21.15.29:0/707187075 shutdown_connections 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.737+0000 7f9b4818a640 1 -- 172.21.15.29:0/707187075 wait complete. 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b4818a640 1 Processor -- start 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b4818a640 1 -- start start 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b4818a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b401025a0 0x7f9b40199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b4818a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b40103620 0x7f9b4019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b4818a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9b40103fc0 0x7f9b4019b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b4818a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4010e0c0 con 0x7f9b40103620 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b4818a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4010df40 con 0x7f9b40103fc0 2024-09-21T02:47:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b4818a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4010e240 con 0x7f9b401025a0 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b45725640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b40103620 0x7f9b4019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b45725640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b40103620 0x7f9b4019a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37958/0 (socket says 172.21.15.29:37958) 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b45725640 1 -- 172.21.15.29:0/463444258 learned_addr learned my addr 172.21.15.29:0/463444258 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b45f26640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b401025a0 0x7f9b40199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.738+0000 7f9b46727640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9b40103fc0 0x7f9b4019b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b45725640 1 -- 172.21.15.29:0/463444258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b401025a0 msgr2=0x7f9b40199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b45725640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b401025a0 0x7f9b40199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b45725640 1 -- 172.21.15.29:0/463444258 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9b40103fc0 msgr2=0x7f9b4019b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b45725640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9b40103fc0 0x7f9b4019b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b45725640 1 -- 172.21.15.29:0/463444258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4019b810 con 0x7f9b40103620 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b45725640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b40103620 0x7f9b4019a4d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000c920 tx=0x7f9b3000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b36ffd640 1 -- 172.21.15.29:0/463444258 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b30007c70 con 0x7f9b40103620 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b36ffd640 1 -- 172.21.15.29:0/463444258 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b30007e10 con 0x7f9b40103620 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b36ffd640 1 -- 172.21.15.29:0/463444258 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b30018710 con 0x7f9b40103620 2024-09-21T02:47:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b4818a640 1 -- 172.21.15.29:0/463444258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4019bb00 con 0x7f9b40103620 2024-09-21T02:47:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.739+0000 7f9b4818a640 1 -- 172.21.15.29:0/463444258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b401a20a0 con 0x7f9b40103620 2024-09-21T02:47:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.742+0000 7f9b4f1ce740 1 Processor -- start 2024-09-21T02:47:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.742+0000 7f9b4f1ce740 1 -- start start 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.742+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557feced2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.742+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb022f0 0x557feced34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.742+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557feced4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.742+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557fece456f0 con 0x557fecb2b1b0 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.742+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x557fece34b20 con 0x557fecb0a4d0 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.742+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x557fece38930 con 0x557fecb022f0 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557feced2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557feced2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37968/0 (socket says 172.21.15.29:37968) 2024-09-21T02:47:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 -- 172.21.15.29:0/2937365330 learned_addr learned my addr 172.21.15.29:0/2937365330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b36ffd640 1 -- 172.21.15.29:0/463444258 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9b3000f450 con 0x7f9b40103620 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b45f26640 1 --2- 172.21.15.29:0/2937365330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb022f0 0x557feced34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 -- 172.21.15.29:0/2937365330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb022f0 msgr2=0x557feced34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 --2- 172.21.15.29:0/2937365330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb022f0 0x557feced34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 -- 172.21.15.29:0/2937365330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 msgr2=0x557feced4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 --2- 172.21.15.29:0/2937365330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557feced4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 -- 172.21.15.29:0/2937365330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fecdfa530 con 0x557fecb2b1b0 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b46727640 1 --2- 172.21.15.29:0/2937365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557feced2e50 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9b38011470 tx=0x7f9b38011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75dfb3cf8bd8f80 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b2bfff640 1 -- 172.21.15.29:0/2937365330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3801e070 con 0x557fecb2b1b0 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b2bfff640 1 -- 172.21.15.29:0/2937365330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b3800f9d0 con 0x557fecb2b1b0 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b2bfff640 1 -- 172.21.15.29:0/2937365330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38019b20 con 0x557fecb2b1b0 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.743+0000 7f9b36ffd640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9b2c0b6510 0x7f9b2c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b45f26640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9b2c0b6510 0x7f9b2c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b36ffd640 1 -- 172.21.15.29:0/463444258 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9b300e77a0 con 0x7f9b40103620 2024-09-21T02:47:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/2937365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 msgr2=0x557feced2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b45f26640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9b2c0b6510 0x7f9b2c0b8a20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005760 tx=0x7f9b3c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/2937365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557feced2e50 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9b38011470 tx=0x7f9b38011930 comp rx=0 tx=0).stop 2024-09-21T02:47:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/2937365330 shutdown_connections 2024-09-21T02:47:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/2937365330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557feced4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/2937365330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb022f0 0x557feced34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/2937365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557feced2e50 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/2937365330 >> 172.21.15.29:0/2937365330 conn(0x557fecb1f300 msgr2=0x557fecece8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/2937365330 shutdown_connections 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.744+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/2937365330 wait complete. 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 1 -- start start 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 1 Processor -- start 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557fecedcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fecedd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557fecd6cab0 con 0x557fecb022f0 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x557fecdda730 con 0x557fecb0a4d0 2024-09-21T02:47:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x557feced4b30 con 0x557fecb2b1b0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b4f1ce740 10 client.0 _ll_register_callbacks cb 0x557fece2e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.745+0000 7f9b45f26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45f26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37972/0 (socket says 172.21.15.29:37972) 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45f26640 1 -- 172.21.15.29:0/1147459630 learned_addr learned my addr 172.21.15.29:0/1147459630 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45725640 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557fecedcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b46727640 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fecedd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45f26640 1 -- 172.21.15.29:0/1147459630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 msgr2=0x557fecedd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45f26640 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fecedd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45f26640 1 -- 172.21.15.29:0/1147459630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 msgr2=0x557fecedcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45f26640 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557fecedcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45f26640 1 -- 172.21.15.29:0/1147459630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fecb96790 con 0x557fecb022f0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b45f26640 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedc6f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2932027882cc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b297fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b297fa640 1 -- 172.21.15.29:0/1147459630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c035040 con 0x557fecb022f0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b297fa640 1 -- 172.21.15.29:0/1147459630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9b3c031500 con 0x557fecb022f0 2024-09-21T02:47:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.746+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1147459630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3800f9d0 con 0x557fecb022f0 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b297fa640 1 -- 172.21.15.29:0/1147459630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9b3c031700 con 0x557fecb022f0 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b4f1ce740 10 client.16737 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1147459630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557fecc0f620 con 0x557fecb022f0 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b297fa640 1 -- 172.21.15.29:0/1147459630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b3800f9d0 con 0x557fecb022f0 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b297fa640 1 -- 172.21.15.29:0/1147459630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f9b3c044030 con 0x557fecb022f0 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b4f1ce740 10 client.16737 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b4f1ce740 10 client.16737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b4f1ce740 20 client.16737 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557fecae9d80 0x557fecee9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b4f1ce740 4 client.16737 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.747+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1147459630 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fece47640 con 0x557fecae9d80 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.748+0000 7f9b45725640 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557fecae9d80 0x557fecee9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.748+0000 7f9b45725640 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557fecae9d80 0x557fecee9dc0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a95905a501a1cb98 server_cookie=e4138d61ef037ec4 in_seq=0 out_seq=0 2024-09-21T02:47:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.748+0000 7f9b297fa640 10 client.16737 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b297fa640 1 -- 172.21.15.29:0/1147459630 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557fece47640 con 0x557fecae9d80 2024-09-21T02:47:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b297fa640 10 client.16737 handle_command_reply: tid=0 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 client.16737 shutdown 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1147459630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 msgr2=0x557fecedc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedc6f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c009260 comp rx=0 tx=0).stop 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1147459630 shutdown_connections 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fecedd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557fecedcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557fecae9d80 0x557fecee9dc0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1147459630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedc6f0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b297fa640 0 client.16737 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1147459630 >> 172.21.15.29:0/1147459630 conn(0x557fecb1f300 msgr2=0x557fececcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1147459630 shutdown_connections 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1147459630 wait complete. 2024-09-21T02:47:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.749+0000 7f9b4f1ce740 20 client.16737 trim_cache size 0 max 16384 2024-09-21T02:47:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.765+0000 7f9b4f1ce740 1 Processor -- start 2024-09-21T02:47:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.765+0000 7f9b4f1ce740 1 -- start start 2024-09-21T02:47:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557feced3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fececf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9b401ac510 con 0x557fecb022f0 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9b401ae050 con 0x557fecb0a4d0 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9b401acbe0 con 0x557fecb2b1b0 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37984/0 (socket says 172.21.15.29:37984) 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 -- 172.21.15.29:0/1832810946 learned_addr learned my addr 172.21.15.29:0/1832810946 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45725640 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557feced3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b46727640 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fececf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 -- 172.21.15.29:0/1832810946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 msgr2=0x557fececf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fececf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 -- 172.21.15.29:0/1832810946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 msgr2=0x557feced3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557feced3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 -- 172.21.15.29:0/1832810946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fececa080 con 0x557fecb022f0 2024-09-21T02:47:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45725640 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557feced3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:47:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b46727640 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fececf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:47:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b45f26640 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedafc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c03d470 tx=0x7f9b3c0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e2621c6da65dc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.766+0000 7f9b2b7fe640 1 -- 172.21.15.29:0/1832810946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c035070 con 0x557fecb022f0 2024-09-21T02:47:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b2b7fe640 1 -- 172.21.15.29:0/1832810946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b3c04c070 con 0x557fecb022f0 2024-09-21T02:47:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b2b7fe640 1 -- 172.21.15.29:0/1832810946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c03d4a0 con 0x557fecb022f0 2024-09-21T02:47:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1832810946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 msgr2=0x557fecedafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedafc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c03d470 tx=0x7f9b3c0058d0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1832810946 shutdown_connections 2024-09-21T02:47:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb2b1b0 0x557fececf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb0a4d0 0x557feced3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/1832810946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb022f0 0x557fecedafc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.767+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1832810946 >> 172.21.15.29:0/1832810946 conn(0x557fecb1f300 msgr2=0x557fecece290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1832810946 shutdown_connections 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/1832810946 wait complete. 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 -- start start 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 Processor -- start 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb022f0 0x557fecee45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb0a4d0 0x557fecee3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557fecee38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9b401ad980 con 0x557fecb2b1b0 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9b401ad2b0 con 0x557fecb022f0 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9b401aa4f0 con 0x557fecb0a4d0 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.768+0000 7f9b4f1ce740 10 client.0 _ll_register_callbacks cb 0x7f9b3c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557fecee38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557fecee38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37990/0 (socket says 172.21.15.29:37990) 2024-09-21T02:47:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 -- 172.21.15.29:0/4002271733 learned_addr learned my addr 172.21.15.29:0/4002271733 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b45f26640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb022f0 0x557fecee45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b45725640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb0a4d0 0x557fecee3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 -- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb0a4d0 msgr2=0x557fecee3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb0a4d0 0x557fecee3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 -- 172.21.15.29:0/4002271733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb022f0 msgr2=0x557fecee45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb022f0 0x557fecee45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 -- 172.21.15.29:0/4002271733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fecee4070 con 0x557fecb2b1b0 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b45725640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb0a4d0 0x557fecee3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b45f26640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb022f0 0x557fecee45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b46727640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557fecee38e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9b38000f80 tx=0x7f9b38003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951dd005dc13983c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b297fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b297fa640 1 -- 172.21.15.29:0/4002271733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38028020 con 0x557fecb2b1b0 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b297fa640 1 -- 172.21.15.29:0/4002271733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9b3800f750 con 0x557fecb2b1b0 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.769+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/4002271733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c04c070 con 0x557fecb2b1b0 2024-09-21T02:47:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b297fa640 1 -- 172.21.15.29:0/4002271733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9b38033050 con 0x557fecb2b1b0 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b4f1ce740 10 client.16743 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/4002271733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557fecf547c0 con 0x557fecb2b1b0 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b297fa640 1 -- 172.21.15.29:0/4002271733 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b3c04c070 con 0x557fecb2b1b0 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b297fa640 1 -- 172.21.15.29:0/4002271733 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f9b380247f0 con 0x557fecb2b1b0 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b4f1ce740 10 client.16743 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b4f1ce740 10 client.16743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b4f1ce740 20 client.16743 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557fecabf1c0 0x557fecf56fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b4f1ce740 4 client.16743 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/4002271733 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557fece430f0 con 0x557fecabf1c0 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.770+0000 7f9b45f26640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557fecabf1c0 0x557fecf56fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.771+0000 7f9b297fa640 10 client.16743 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.771+0000 7f9b45f26640 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557fecabf1c0 0x557fecf56fc0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80c871e74c3c716a server_cookie=d88b0247ba5cef6c in_seq=0 out_seq=0 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b297fa640 1 -- 172.21.15.29:0/4002271733 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557fece430f0 con 0x557fecabf1c0 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b297fa640 10 client.16743 handle_command_reply: tid=0 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 client.16743 shutdown 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/4002271733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 msgr2=0x557fecee38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557fecee38e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9b38000f80 tx=0x7f9b38003d50 comp rx=0 tx=0).stop 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/4002271733 shutdown_connections 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x557fecabf1c0 0x557fecf56fc0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557fecb2b1b0 0x557fecee38e0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557fecb0a4d0 0x557fecee3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b297fa640 0 client.16743 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 --2- 172.21.15.29:0/4002271733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x557fecb022f0 0x557fecee45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/4002271733 >> 172.21.15.29:0/4002271733 conn(0x557fecb1f300 msgr2=0x557fecece220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/4002271733 shutdown_connections 2024-09-21T02:47:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 1 -- 172.21.15.29:0/4002271733 wait complete. 2024-09-21T02:47:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.814+0000 7f9b4f1ce740 20 client.16743 trim_cache size 0 max 16384 2024-09-21T02:47:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:47:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:47:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:47:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 -- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9b2c0b6510 msgr2=0x7f9b2c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9b2c0b6510 0x7f9b2c0b8a20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005760 tx=0x7f9b3c031400 comp rx=0 tx=0).stop 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 -- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b40103620 msgr2=0x7f9b4019a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b40103620 0x7f9b4019a4d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000c920 tx=0x7f9b3000cde0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 -- 172.21.15.29:0/463444258 shutdown_connections 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9b2c0b6510 0x7f9b2c0b8a20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9b40103fc0 0x7f9b4019b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b40103620 0x7f9b4019a4d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 --2- 172.21.15.29:0/463444258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b401025a0 0x7f9b40199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 -- 172.21.15.29:0/463444258 >> 172.21.15.29:0/463444258 conn(0x7f9b400fdee0 msgr2=0x7f9b400ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 -- 172.21.15.29:0/463444258 shutdown_connections 2024-09-21T02:47:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:38.815+0000 7f9b4818a640 1 -- 172.21.15.29:0/463444258 wait complete. 2024-09-21T02:47:39.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 156476 2.16 MB/sec execute 316 sec latency 35.146 ms 2024-09-21T02:47:39.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:39 smithi066 ceph-mon[33436]: pgmap v645: 129 pgs: 129 active+clean; 35 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 553 KiB/s wr, 59 op/s 2024-09-21T02:47:39.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:39 smithi190 ceph-mon[33871]: pgmap v645: 129 pgs: 129 active+clean; 35 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 553 KiB/s wr, 59 op/s 2024-09-21T02:47:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:39 smithi029 ceph-mon[31006]: pgmap v645: 129 pgs: 129 active+clean; 35 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 553 KiB/s wr, 59 op/s 2024-09-21T02:47:40.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 156905 2.16 MB/sec execute 317 sec latency 17.470 ms 2024-09-21T02:47:41.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 157548 2.16 MB/sec execute 318 sec latency 17.267 ms 2024-09-21T02:47:41.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:41 smithi066 ceph-mon[33436]: pgmap v646: 129 pgs: 129 active+clean; 36 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 880 KiB/s wr, 71 op/s 2024-09-21T02:47:41.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:41 smithi190 ceph-mon[33871]: pgmap v646: 129 pgs: 129 active+clean; 36 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 880 KiB/s wr, 71 op/s 2024-09-21T02:47:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:41 smithi029 ceph-mon[31006]: pgmap v646: 129 pgs: 129 active+clean; 36 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 880 KiB/s wr, 71 op/s 2024-09-21T02:47:42.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 158176 2.16 MB/sec execute 319 sec latency 23.232 ms 2024-09-21T02:47:43.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 158748 2.16 MB/sec execute 320 sec latency 38.123 ms 2024-09-21T02:47:43.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:43 smithi066 ceph-mon[33436]: pgmap v647: 129 pgs: 129 active+clean; 36 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1021 KiB/s wr, 84 op/s 2024-09-21T02:47:43.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:43 smithi190 ceph-mon[33871]: pgmap v647: 129 pgs: 129 active+clean; 36 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1021 KiB/s wr, 84 op/s 2024-09-21T02:47:43.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:43 smithi029 ceph-mon[31006]: pgmap v647: 129 pgs: 129 active+clean; 36 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1021 KiB/s wr, 84 op/s 2024-09-21T02:47:44.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 159433 2.17 MB/sec execute 321 sec latency 15.162 ms 2024-09-21T02:47:45.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 159965 2.18 MB/sec execute 322 sec latency 17.362 ms 2024-09-21T02:47:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:45 smithi029 ceph-mon[31006]: pgmap v648: 129 pgs: 129 active+clean; 37 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-09-21T02:47:45.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:45 smithi066 ceph-mon[33436]: pgmap v648: 129 pgs: 129 active+clean; 37 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-09-21T02:47:45.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:45 smithi190 ceph-mon[33871]: pgmap v648: 129 pgs: 129 active+clean; 37 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-09-21T02:47:46.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 160297 2.17 MB/sec execute 323 sec latency 26.135 ms 2024-09-21T02:47:47.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 160926 2.17 MB/sec execute 324 sec latency 14.701 ms 2024-09-21T02:47:47.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:47 smithi066 ceph-mon[33436]: pgmap v649: 129 pgs: 129 active+clean; 36 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-09-21T02:47:47.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:47 smithi190 ceph-mon[33871]: pgmap v649: 129 pgs: 129 active+clean; 36 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-09-21T02:47:47.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:47 smithi029 ceph-mon[31006]: pgmap v649: 129 pgs: 129 active+clean; 36 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-09-21T02:47:48.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 161618 2.18 MB/sec execute 325 sec latency 17.561 ms 2024-09-21T02:47:49.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 162222 2.18 MB/sec execute 326 sec latency 23.491 ms 2024-09-21T02:47:49.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:49 smithi066 ceph-mon[33436]: pgmap v650: 129 pgs: 129 active+clean; 39 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:47:49.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:49 smithi190 ceph-mon[33871]: pgmap v650: 129 pgs: 129 active+clean; 39 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:47:49.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:49 smithi029 ceph-mon[31006]: pgmap v650: 129 pgs: 129 active+clean; 39 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-09-21T02:47:50.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 162968 2.19 MB/sec execute 327 sec latency 16.461 ms 2024-09-21T02:47:51.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 163481 2.20 MB/sec execute 328 sec latency 17.239 ms 2024-09-21T02:47:51.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:51 smithi066 ceph-mon[33436]: pgmap v651: 129 pgs: 129 active+clean; 39 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:47:51.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:51 smithi190 ceph-mon[33871]: pgmap v651: 129 pgs: 129 active+clean; 39 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:47:51.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:51 smithi029 ceph-mon[31006]: pgmap v651: 129 pgs: 129 active+clean; 39 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:47:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.921+0000 7f165c357640 1 -- 172.21.15.29:0/2503695842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103fc0 msgr2=0x7f165410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.921+0000 7f165c357640 1 --2- 172.21.15.29:0/2503695842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103fc0 0x7f165410a960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f164c009c90 tx=0x7f164c02a860 comp rx=0 tx=0).stop 2024-09-21T02:47:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 -- 172.21.15.29:0/2503695842 shutdown_connections 2024-09-21T02:47:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 --2- 172.21.15.29:0/2503695842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103fc0 0x7f165410a960 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 --2- 172.21.15.29:0/2503695842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1654103620 0x7f1654103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 --2- 172.21.15.29:0/2503695842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16541025a0 0x7f1654102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 -- 172.21.15.29:0/2503695842 >> 172.21.15.29:0/2503695842 conn(0x7f16540fdee0 msgr2=0x7f1654100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 -- 172.21.15.29:0/2503695842 shutdown_connections 2024-09-21T02:47:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 -- 172.21.15.29:0/2503695842 wait complete. 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 Processor -- start 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.922+0000 7f165c357640 1 -- start start 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f165c357640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16541025a0 0x7f1654199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f165c357640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103620 0x7f165419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f165c357640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1654103fc0 0x7f165419b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f165c357640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f165410e0b0 con 0x7f1654103620 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f165c357640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f165410df30 con 0x7f16541025a0 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f165c357640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f165410e230 con 0x7f1654103fc0 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103620 0x7f165419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103620 0x7f165419a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36638/0 (socket says 172.21.15.29:36638) 2024-09-21T02:47:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 -- 172.21.15.29:0/227545313 learned_addr learned my addr 172.21.15.29:0/227545313 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f165a8f4640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1654103fc0 0x7f165419b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f165a0f3640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16541025a0 0x7f1654199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 -- 172.21.15.29:0/227545313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1654103fc0 msgr2=0x7f165419b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1654103fc0 0x7f165419b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 -- 172.21.15.29:0/227545313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16541025a0 msgr2=0x7f1654199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16541025a0 0x7f1654199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 -- 172.21.15.29:0/227545313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165419b8d0 con 0x7f1654103620 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.923+0000 7f16598f2640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103620 0x7f165419a500 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f164400d950 tx=0x7f164400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.924+0000 7f164b7fe640 1 -- 172.21.15.29:0/227545313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1644017030 con 0x7f1654103620 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.924+0000 7f165c357640 1 -- 172.21.15.29:0/227545313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165419bbc0 con 0x7f1654103620 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.924+0000 7f164b7fe640 1 -- 172.21.15.29:0/227545313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1644004020 con 0x7f1654103620 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.924+0000 7f164b7fe640 1 -- 172.21.15.29:0/227545313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16440055d0 con 0x7f1654103620 2024-09-21T02:47:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.924+0000 7f165c357640 1 -- 172.21.15.29:0/227545313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16541a22b0 con 0x7f1654103620 2024-09-21T02:47:51.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.926+0000 7f16632c4740 1 Processor -- start 2024-09-21T02:47:51.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f16632c4740 1 -- start start 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f16632c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 0x55b1c89e4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f164b7fe640 1 -- 172.21.15.29:0/227545313 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f1644009a30 con 0x7f1654103620 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f16632c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c860f2f0 0x55b1c89e48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f16632c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89e5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f16632c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b1c89526f0 con 0x55b1c860f2f0 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f16632c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b1c8941b20 con 0x55b1c86174d0 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f16632c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b1c8945930 con 0x55b1c86381b0 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f165a0f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c860f2f0 0x55b1c89e48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f165a0f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c860f2f0 0x55b1c89e48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36650/0 (socket says 172.21.15.29:36650) 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f165a0f3640 1 -- 172.21.15.29:0/3184838741 learned_addr learned my addr 172.21.15.29:0/3184838741 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f16598f2640 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89e5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.927+0000 7f165a8f4640 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 0x55b1c89e4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f165a0f3640 1 -- 172.21.15.29:0/3184838741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 msgr2=0x55b1c89e4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f165a0f3640 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 0x55b1c89e4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f165a0f3640 1 -- 172.21.15.29:0/3184838741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 msgr2=0x55b1c89e5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f165a0f3640 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89e5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f165a0f3640 1 -- 172.21.15.29:0/3184838741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1c8907530 con 0x55b1c860f2f0 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f16598f2640 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89e5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f165a0f3640 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c860f2f0 0x55b1c89e48d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1650009d20 tx=0x7f1650011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae949e2b83e8bb57 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f1648bfe640 1 -- 172.21.15.29:0/3184838741 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165001e070 con 0x55b1c860f2f0 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f164b7fe640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16400b6510 0x7f16400b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f1648bfe640 1 -- 172.21.15.29:0/3184838741 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1650019ac0 con 0x55b1c860f2f0 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f1648bfe640 1 -- 172.21.15.29:0/3184838741 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165000e9c0 con 0x55b1c860f2f0 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f165a8f4640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16400b6510 0x7f16400b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.928+0000 7f164b7fe640 1 -- 172.21.15.29:0/227545313 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f16440e6fa0 con 0x7f1654103620 2024-09-21T02:47:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 -- 172.21.15.29:0/3184838741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c860f2f0 msgr2=0x55b1c89e48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f165a8f4640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16400b6510 0x7f16400b8a20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f164c009c60 tx=0x7f164c031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c860f2f0 0x55b1c89e48d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1650009d20 tx=0x7f1650011810 comp rx=0 tx=0).stop 2024-09-21T02:47:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 -- 172.21.15.29:0/3184838741 shutdown_connections 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89e5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c860f2f0 0x55b1c89e48d0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 --2- 172.21.15.29:0/3184838741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 0x55b1c89e4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 -- 172.21.15.29:0/3184838741 >> 172.21.15.29:0/3184838741 conn(0x55b1c862c300 msgr2=0x55b1c89db8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 -- 172.21.15.29:0/3184838741 shutdown_connections 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.929+0000 7f16632c4740 1 -- 172.21.15.29:0/3184838741 wait complete. 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 1 -- start start 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 1 Processor -- start 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89eda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86174d0 0x55b1c89ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 0x55b1c89eec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b1c8879ab0 con 0x55b1c86174d0 2024-09-21T02:47:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b1c88e7730 con 0x55b1c860f2f0 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b1c89e5f40 con 0x55b1c86381b0 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16632c4740 10 client.0 _ll_register_callbacks cb 0x55b1c893b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16598f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86174d0 0x55b1c89ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f165a0f3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89eda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f165a0f3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89eda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42898/0 (socket says 172.21.15.29:42898) 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f16598f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86174d0 0x55b1c89ee0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36660/0 (socket says 172.21.15.29:36660) 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.930+0000 7f165a0f3640 1 -- 172.21.15.29:0/3834077602 learned_addr learned my addr 172.21.15.29:0/3834077602 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f165a8f4640 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 0x55b1c89eec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f165a0f3640 1 -- 172.21.15.29:0/3834077602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 msgr2=0x55b1c89eec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f165a0f3640 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 0x55b1c89eec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f165a0f3640 1 -- 172.21.15.29:0/3834077602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86174d0 msgr2=0x55b1c89ee0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f165a0f3640 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86174d0 0x55b1c89ee0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f165a0f3640 1 -- 172.21.15.29:0/3834077602 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1c86a3790 con 0x55b1c860f2f0 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f16598f2640 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86174d0 0x55b1c89ee0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f165a0f3640 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89eda20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f16500057c0 tx=0x7f1650004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c026e40df1f429fd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f163dffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f163dffb640 1 -- 172.21.15.29:0/3834077602 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165001e050 con 0x55b1c860f2f0 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f16632c4740 1 -- 172.21.15.29:0/3834077602 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1650019ac0 con 0x55b1c860f2f0 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.931+0000 7f163dffb640 1 -- 172.21.15.29:0/3834077602 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1650019430 con 0x55b1c860f2f0 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f163dffb640 1 -- 172.21.15.29:0/3834077602 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1650032050 con 0x55b1c860f2f0 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16632c4740 10 client.26204 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16632c4740 1 -- 172.21.15.29:0/3834077602 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b1c871c620 con 0x55b1c860f2f0 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f163dffb640 1 -- 172.21.15.29:0/3834077602 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1650019ac0 con 0x55b1c860f2f0 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f163dffb640 1 -- 172.21.15.29:0/3834077602 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f16500035f0 con 0x55b1c860f2f0 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16632c4740 10 client.26204 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16632c4740 10 client.26204 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16632c4740 20 client.26204 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16632c4740 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1c85f6d80 0x55b1c89faf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16632c4740 4 client.26204 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:47:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16632c4740 1 -- 172.21.15.29:0/3834077602 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1c8954640 con 0x55b1c85f6d80 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.932+0000 7f16598f2640 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1c85f6d80 0x55b1c89faf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.933+0000 7f163dffb640 10 client.26204 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.933+0000 7f16598f2640 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1c85f6d80 0x55b1c89faf20 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f64bb5b1eff6366 server_cookie=dc0aac20112dd518 in_seq=0 out_seq=0 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.933+0000 7f163dffb640 1 -- 172.21.15.29:0/3834077602 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b1c8954640 con 0x55b1c85f6d80 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.933+0000 7f163dffb640 10 client.26204 handle_command_reply: tid=0 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.933+0000 7f16632c4740 1 client.26204 shutdown 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 -- 172.21.15.29:0/3834077602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 msgr2=0x55b1c89eda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89eda20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f16500057c0 tx=0x7f1650004270 comp rx=0 tx=0).stop 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 -- 172.21.15.29:0/3834077602 shutdown_connections 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86381b0 0x55b1c89eec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1c85f6d80 0x55b1c89faf20 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86174d0 0x55b1c89ee0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 --2- 172.21.15.29:0/3834077602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89eda20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f163dffb640 0 client.26204 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 -- 172.21.15.29:0/3834077602 >> 172.21.15.29:0/3834077602 conn(0x55b1c862c300 msgr2=0x55b1c89d9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 -- 172.21.15.29:0/3834077602 shutdown_connections 2024-09-21T02:47:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 1 -- 172.21.15.29:0/3834077602 wait complete. 2024-09-21T02:47:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.934+0000 7f16632c4740 20 client.26204 trim_cache size 0 max 16384 2024-09-21T02:47:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16632c4740 1 Processor -- start 2024-09-21T02:47:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16632c4740 1 -- start start 2024-09-21T02:47:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16632c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c860f2f0 0x55b1c89ebf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16632c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16632c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89ec470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16632c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f16541aa410 con 0x55b1c86381b0 2024-09-21T02:47:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16632c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f16541abf50 con 0x55b1c86174d0 2024-09-21T02:47:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16632c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f16541aaae0 con 0x55b1c860f2f0 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f16598f2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f165a0f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c860f2f0 0x55b1c89ebf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f165a0f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c860f2f0 0x55b1c89ebf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47350/0 (socket says 172.21.15.29:47350) 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.951+0000 7f165a0f3640 1 -- 172.21.15.29:0/1530525569 learned_addr learned my addr 172.21.15.29:0/1530525569 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f165a8f4640 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89ec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f16598f2640 1 -- 172.21.15.29:0/1530525569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c860f2f0 msgr2=0x55b1c89ebf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f16598f2640 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c860f2f0 0x55b1c89ebf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f16598f2640 1 -- 172.21.15.29:0/1530525569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 msgr2=0x55b1c89ec470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f16598f2640 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89ec470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f16598f2640 1 -- 172.21.15.29:0/1530525569 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1c89eefc0 con 0x55b1c86174d0 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f165a8f4640 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89ec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:47:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f165a0f3640 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c860f2f0 0x55b1c89ebf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f16598f2640 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89f5b30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f16440e4770 tx=0x7f16440d0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=415424cbf1280eee server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f163ffff640 1 -- 172.21.15.29:0/1530525569 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16440ac8c0 con 0x55b1c86174d0 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f163ffff640 1 -- 172.21.15.29:0/1530525569 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16440ad4b0 con 0x55b1c86174d0 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.952+0000 7f163ffff640 1 -- 172.21.15.29:0/1530525569 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16440a9020 con 0x55b1c86174d0 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 -- 172.21.15.29:0/1530525569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 msgr2=0x55b1c89f5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89f5b30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f16440e4770 tx=0x7f16440d0040 comp rx=0 tx=0).stop 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 -- 172.21.15.29:0/1530525569 shutdown_connections 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89ec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c86174d0 0x55b1c89f5b30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 --2- 172.21.15.29:0/1530525569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c860f2f0 0x55b1c89ebf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 -- 172.21.15.29:0/1530525569 >> 172.21.15.29:0/1530525569 conn(0x55b1c862c300 msgr2=0x55b1c89e4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 -- 172.21.15.29:0/1530525569 shutdown_connections 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.953+0000 7f16632c4740 1 -- 172.21.15.29:0/1530525569 wait complete. 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 1 -- start start 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 1 Processor -- start 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89f5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86174d0 0x55b1c89f4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89f4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f16541ab880 con 0x55b1c86381b0 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f16541ab1b0 con 0x55b1c860f2f0 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f16541a83f0 con 0x55b1c86174d0 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16632c4740 10 client.0 _ll_register_callbacks cb 0x7f165001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f165a0f3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16598f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86174d0 0x55b1c89f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f165a8f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f165a0f3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89f5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42912/0 (socket says 172.21.15.29:42912) 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f16598f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86174d0 0x55b1c89f4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47356/0 (socket says 172.21.15.29:47356) 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f165a0f3640 1 -- 172.21.15.29:0/3060519807 learned_addr learned my addr 172.21.15.29:0/3060519807 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.954+0000 7f165a8f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89f4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36670/0 (socket says 172.21.15.29:36670) 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16598f2640 1 -- 172.21.15.29:0/3060519807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 msgr2=0x55b1c89f5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16598f2640 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89f5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16598f2640 1 -- 172.21.15.29:0/3060519807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 msgr2=0x55b1c89f4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16598f2640 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89f4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16598f2640 1 -- 172.21.15.29:0/3060519807 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1c89f50b0 con 0x55b1c86174d0 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16598f2640 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86174d0 0x55b1c89f4430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f16440e4a90 tx=0x7f16440ac950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67f4abbfc09e56bb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f163dffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f163dffb640 1 -- 172.21.15.29:0/3060519807 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1644098020 con 0x55b1c86174d0 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f163dffb640 1 -- 172.21.15.29:0/3060519807 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f16440a9020 con 0x55b1c86174d0 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16632c4740 1 -- 172.21.15.29:0/3060519807 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16440ad4b0 con 0x55b1c86174d0 2024-09-21T02:47:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f163dffb640 1 -- 172.21.15.29:0/3060519807 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f16440b1890 con 0x55b1c86174d0 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16632c4740 10 client.26323 fetch_fsmap learned FSMap version 31 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f16632c4740 1 -- 172.21.15.29:0/3060519807 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b1c8a65920 con 0x55b1c86174d0 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.955+0000 7f163dffb640 1 -- 172.21.15.29:0/3060519807 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16440ad4b0 con 0x55b1c86174d0 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f163dffb640 1 -- 172.21.15.29:0/3060519807 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f16440e4be0 con 0x55b1c86174d0 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f16632c4740 10 client.26323 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f16632c4740 10 client.26323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f16632c4740 20 client.26323 populate_metadata read hostname 'smithi029' 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f16632c4740 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1c85cc1c0 0x55b1c8a68200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f16632c4740 4 client.26323 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f16632c4740 1 -- 172.21.15.29:0/3060519807 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b1c89500f0 con 0x55b1c85cc1c0 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f165a0f3640 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1c85cc1c0 0x55b1c8a68200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f165a0f3640 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1c85cc1c0 0x55b1c8a68200 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2523c85b8bdf6658 server_cookie=f5c75f451165a730 in_seq=0 out_seq=0 2024-09-21T02:47:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.956+0000 7f163dffb640 10 client.26323 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f163dffb640 1 -- 172.21.15.29:0/3060519807 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b1c89500f0 con 0x55b1c85cc1c0 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f163dffb640 10 client.26323 handle_command_reply: tid=0 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 client.26323 shutdown 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 -- 172.21.15.29:0/3060519807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86174d0 msgr2=0x55b1c89f4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86174d0 0x55b1c89f4430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f16440e4a90 tx=0x7f16440ac950 comp rx=0 tx=0).stop 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 -- 172.21.15.29:0/3060519807 shutdown_connections 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1c85cc1c0 0x55b1c8a68200 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1c86381b0 0x55b1c89f4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1c86174d0 0x55b1c89f4430 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 --2- 172.21.15.29:0/3060519807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1c860f2f0 0x55b1c89f5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 -- 172.21.15.29:0/3060519807 >> 172.21.15.29:0/3060519807 conn(0x55b1c862c300 msgr2=0x55b1c89e4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f163dffb640 0 client.26323 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 -- 172.21.15.29:0/3060519807 shutdown_connections 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.996+0000 7f16632c4740 1 -- 172.21.15.29:0/3060519807 wait complete. 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f16632c4740 20 client.26323 trim_cache size 0 max 16384 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:47:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 -- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16400b6510 msgr2=0x7f16400b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16400b6510 0x7f16400b8a20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f164c009c60 tx=0x7f164c031000 comp rx=0 tx=0).stop 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 -- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103620 msgr2=0x7f165419a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103620 0x7f165419a500 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f164400d950 tx=0x7f164400de10 comp rx=0 tx=0).stop 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 -- 172.21.15.29:0/227545313 shutdown_connections 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16400b6510 0x7f16400b8a20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1654103fc0 0x7f165419b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1654103620 0x7f165419a500 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 --2- 172.21.15.29:0/227545313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16541025a0 0x7f1654199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:47:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 -- 172.21.15.29:0/227545313 >> 172.21.15.29:0/227545313 conn(0x7f16540fdee0 msgr2=0x7f16540ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:47:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 -- 172.21.15.29:0/227545313 shutdown_connections 2024-09-21T02:47:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:47:51.997+0000 7f165c357640 1 -- 172.21.15.29:0/227545313 wait complete. 2024-09-21T02:47:52.092 INFO:tasks.workunit.client.0.smithi029.stdout: 1 163762 2.19 MB/sec execute 329 sec latency 43.172 ms 2024-09-21T02:47:53.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 164359 2.19 MB/sec execute 330 sec latency 17.269 ms 2024-09-21T02:47:53.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:53 smithi066 ceph-mon[33436]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-09-21T02:47:53.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:53 smithi190 ceph-mon[33871]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-09-21T02:47:53.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:53 smithi029 ceph-mon[31006]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-09-21T02:47:54.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 165030 2.20 MB/sec execute 331 sec latency 14.236 ms 2024-09-21T02:47:55.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 165664 2.20 MB/sec execute 332 sec latency 15.587 ms 2024-09-21T02:47:55.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:55 smithi066 ceph-mon[33436]: pgmap v653: 129 pgs: 129 active+clean; 36 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-09-21T02:47:55.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:55 smithi190 ceph-mon[33871]: pgmap v653: 129 pgs: 129 active+clean; 36 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-09-21T02:47:55.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:55 smithi029 ceph-mon[31006]: pgmap v653: 129 pgs: 129 active+clean; 36 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-09-21T02:47:56.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 166492 2.21 MB/sec execute 333 sec latency 13.623 ms 2024-09-21T02:47:57.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 166995 2.22 MB/sec execute 334 sec latency 18.500 ms 2024-09-21T02:47:57.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:57 smithi066 ceph-mon[33436]: pgmap v654: 129 pgs: 129 active+clean; 36 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:47:57.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:57 smithi190 ceph-mon[33871]: pgmap v654: 129 pgs: 129 active+clean; 36 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:47:57.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:57 smithi029 ceph-mon[31006]: pgmap v654: 129 pgs: 129 active+clean; 36 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T02:47:58.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 167268 2.21 MB/sec execute 335 sec latency 14.562 ms 2024-09-21T02:47:59.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 167856 2.21 MB/sec execute 336 sec latency 18.780 ms 2024-09-21T02:47:59.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:47:59 smithi066 ceph-mon[33436]: pgmap v655: 129 pgs: 129 active+clean; 37 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-09-21T02:47:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:47:59 smithi190 ceph-mon[33871]: pgmap v655: 129 pgs: 129 active+clean; 37 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-09-21T02:47:59.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:47:59 smithi029 ceph-mon[31006]: pgmap v655: 129 pgs: 129 active+clean; 37 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-09-21T02:48:00.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 168544 2.21 MB/sec execute 337 sec latency 18.320 ms 2024-09-21T02:48:01.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 169157 2.21 MB/sec execute 338 sec latency 15.670 ms 2024-09-21T02:48:01.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:01 smithi066 ceph-mon[33436]: pgmap v656: 129 pgs: 129 active+clean; 38 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-09-21T02:48:01.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:01 smithi190 ceph-mon[33871]: pgmap v656: 129 pgs: 129 active+clean; 38 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-09-21T02:48:01.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:01 smithi029 ceph-mon[31006]: pgmap v656: 129 pgs: 129 active+clean; 38 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-09-21T02:48:02.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 170014 2.23 MB/sec execute 339 sec latency 15.419 ms 2024-09-21T02:48:03.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 170504 2.23 MB/sec execute 340 sec latency 16.589 ms 2024-09-21T02:48:03.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:03 smithi066 ceph-mon[33436]: pgmap v657: 129 pgs: 129 active+clean; 39 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.7 MiB/s wr, 141 op/s 2024-09-21T02:48:03.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:03 smithi190 ceph-mon[33871]: pgmap v657: 129 pgs: 129 active+clean; 39 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.7 MiB/s wr, 141 op/s 2024-09-21T02:48:03.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:03 smithi029 ceph-mon[31006]: pgmap v657: 129 pgs: 129 active+clean; 39 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.7 MiB/s wr, 141 op/s 2024-09-21T02:48:04.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 170749 2.23 MB/sec execute 341 sec latency 14.613 ms 2024-09-21T02:48:05.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 171328 2.23 MB/sec execute 342 sec latency 16.858 ms 2024-09-21T02:48:05.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 -- 172.21.15.29:0/3700257424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7541025a0 msgr2=0x7fe754102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 --2- 172.21.15.29:0/3700257424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7541025a0 0x7fe754102a10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe744009900 tx=0x7fe74402a410 comp rx=0 tx=0).stop 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 -- 172.21.15.29:0/3700257424 shutdown_connections 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 --2- 172.21.15.29:0/3700257424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe754103fc0 0x7fe75410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 --2- 172.21.15.29:0/3700257424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe754103620 0x7fe754103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 --2- 172.21.15.29:0/3700257424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7541025a0 0x7fe754102a10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 -- 172.21.15.29:0/3700257424 >> 172.21.15.29:0/3700257424 conn(0x7fe7540fdee0 msgr2=0x7fe754100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 -- 172.21.15.29:0/3700257424 shutdown_connections 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.110+0000 7fe759869640 1 -- 172.21.15.29:0/3700257424 wait complete. 2024-09-21T02:48:05.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe759869640 1 Processor -- start 2024-09-21T02:48:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe759869640 1 -- start start 2024-09-21T02:48:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe759869640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7541025a0 0x7fe754199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe759869640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe754103620 0x7fe75419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe759869640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe754103fc0 0x7fe75419b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe759869640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe75410e0a0 con 0x7fe754103fc0 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe759869640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe75410df20 con 0x7fe754103620 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe759869640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe75410e220 con 0x7fe7541025a0 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe7537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe754103fc0 0x7fe75419b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe752ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7541025a0 0x7fe754199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe7527fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe754103620 0x7fe75419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe7537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe754103fc0 0x7fe75419b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47430/0 (socket says 172.21.15.29:47430) 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.111+0000 7fe7537fe640 1 -- 172.21.15.29:0/3339064413 learned_addr learned my addr 172.21.15.29:0/3339064413 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe7527fc640 1 -- 172.21.15.29:0/3339064413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7541025a0 msgr2=0x7fe754199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe7527fc640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7541025a0 0x7fe754199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe7527fc640 1 -- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe754103fc0 msgr2=0x7fe75419b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe7527fc640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe754103fc0 0x7fe75419b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe7527fc640 1 -- 172.21.15.29:0/3339064413 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75419b8c0 con 0x7fe754103620 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe7537fe640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe754103fc0 0x7fe75419b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe7527fc640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe754103620 0x7fe75419a4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe74800d6d0 tx=0x7fe74800db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:05.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe758867640 1 -- 172.21.15.29:0/3339064413 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748017070 con 0x7fe754103620 2024-09-21T02:48:05.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe759869640 1 -- 172.21.15.29:0/3339064413 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe75419bbb0 con 0x7fe754103620 2024-09-21T02:48:05.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe758867640 1 -- 172.21.15.29:0/3339064413 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe748009820 con 0x7fe754103620 2024-09-21T02:48:05.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe758867640 1 -- 172.21.15.29:0/3339064413 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748004bc0 con 0x7fe754103620 2024-09-21T02:48:05.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.112+0000 7fe759869640 1 -- 172.21.15.29:0/3339064413 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7541a22a0 con 0x7fe754103620 2024-09-21T02:48:05.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.115+0000 7fe7607d8740 1 Processor -- start 2024-09-21T02:48:05.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7607d8740 1 -- start start 2024-09-21T02:48:05.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe758867640 1 -- 172.21.15.29:0/3339064413 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fe748005220 con 0x7fe754103620 2024-09-21T02:48:05.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd47a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd47a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd47b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7607d8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55afbd3e86f0 con 0x55afbd0ad4d0 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7607d8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55afbd3d7b20 con 0x55afbd0a52f0 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7607d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55afbd3db930 con 0x55afbd0ce1b0 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe752ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd47a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7537fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd47a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe752ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd47a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46658/0 (socket says 172.21.15.29:46658) 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe752ffd640 1 -- 172.21.15.29:0/395691786 learned_addr learned my addr 172.21.15.29:0/395691786 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7537fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd47a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42536/0 (socket says 172.21.15.29:42536) 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe7527fc640 1 --2- 172.21.15.29:0/395691786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd47b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe752ffd640 1 -- 172.21.15.29:0/395691786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 msgr2=0x55afbd47a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe752ffd640 1 --2- 172.21.15.29:0/395691786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd47a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe752ffd640 1 -- 172.21.15.29:0/395691786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 msgr2=0x55afbd47b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe752ffd640 1 --2- 172.21.15.29:0/395691786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd47b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.116+0000 7fe752ffd640 1 -- 172.21.15.29:0/395691786 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afbd39d530 con 0x55afbd0a52f0 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.117+0000 7fe752ffd640 1 --2- 172.21.15.29:0/395691786 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd47a8d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe74402a930 tx=0x7fe744005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86c55936f1b544af server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.117+0000 7fe758867640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe73c0b6510 0x7fe73c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.117+0000 7fe7537fe640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe73c0b6510 0x7fe73c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.117+0000 7fe739ffb640 1 -- 172.21.15.29:0/395691786 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe744035030 con 0x55afbd0a52f0 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.117+0000 7fe739ffb640 1 -- 172.21.15.29:0/395691786 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe744040070 con 0x55afbd0a52f0 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.117+0000 7fe739ffb640 1 -- 172.21.15.29:0/395691786 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe744003850 con 0x55afbd0a52f0 2024-09-21T02:48:05.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.117+0000 7fe758867640 1 -- 172.21.15.29:0/3339064413 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe7480e7100 con 0x7fe754103620 2024-09-21T02:48:05.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 -- 172.21.15.29:0/395691786 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 msgr2=0x55afbd47a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 --2- 172.21.15.29:0/395691786 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd47a8d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe74402a930 tx=0x7fe744005fd0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7537fe640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe73c0b6510 0x7fe73c0b8a20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00f9e0 tx=0x7fe74c0094e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:05.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 -- 172.21.15.29:0/395691786 shutdown_connections 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 --2- 172.21.15.29:0/395691786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd47b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 --2- 172.21.15.29:0/395691786 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd47a8d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 --2- 172.21.15.29:0/395691786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd47a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 -- 172.21.15.29:0/395691786 >> 172.21.15.29:0/395691786 conn(0x55afbd0c2300 msgr2=0x55afbd4718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 -- 172.21.15.29:0/395691786 shutdown_connections 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 -- 172.21.15.29:0/395691786 wait complete. 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.118+0000 7fe7607d8740 1 -- start start 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7607d8740 1 Processor -- start 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0a52f0 0x55afbd483960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ad4d0 0x55afbd484020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd484b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7607d8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55afbd30fab0 con 0x55afbd0a52f0 2024-09-21T02:48:05.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7607d8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55afbd37d730 con 0x55afbd0ad4d0 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7607d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55afbd47bee0 con 0x55afbd0ce1b0 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7607d8740 10 client.0 _ll_register_callbacks cb 0x55afbd3d16b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe752ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0a52f0 0x55afbd483960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7527fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ad4d0 0x55afbd484020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe752ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0a52f0 0x55afbd483960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47442/0 (socket says 172.21.15.29:47442) 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe752ffd640 1 -- 172.21.15.29:0/4020054231 learned_addr learned my addr 172.21.15.29:0/4020054231 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7537fe640 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd484b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.119+0000 7fe7527fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ad4d0 0x55afbd484020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46660/0 (socket says 172.21.15.29:46660) 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe752ffd640 1 -- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 msgr2=0x55afbd484b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe752ffd640 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd484b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe752ffd640 1 -- 172.21.15.29:0/4020054231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ad4d0 msgr2=0x55afbd484020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe752ffd640 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ad4d0 0x55afbd484020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe752ffd640 1 -- 172.21.15.29:0/4020054231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afbd139790 con 0x55afbd0a52f0 2024-09-21T02:48:05.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe7527fc640 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ad4d0 0x55afbd484020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe7537fe640 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd484b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe752ffd640 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0a52f0 0x55afbd483960 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe744005760 tx=0x7fe7440318b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30ae9cc56cf060d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe732ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe732ffd640 1 -- 172.21.15.29:0/4020054231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe744052060 con 0x55afbd0a52f0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe7607d8740 1 -- 172.21.15.29:0/4020054231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe744040070 con 0x55afbd0a52f0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe732ffd640 1 -- 172.21.15.29:0/4020054231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe74403c440 con 0x55afbd0a52f0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe732ffd640 1 -- 172.21.15.29:0/4020054231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe744059050 con 0x55afbd0a52f0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe7607d8740 10 client.16773 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe7607d8740 1 -- 172.21.15.29:0/4020054231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55afbd1b2620 con 0x55afbd0a52f0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.120+0000 7fe732ffd640 1 -- 172.21.15.29:0/4020054231 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe744040070 con 0x55afbd0a52f0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe732ffd640 1 -- 172.21.15.29:0/4020054231 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fe74404c030 con 0x55afbd0a52f0 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe7607d8740 10 client.16773 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe7607d8740 10 client.16773 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:05.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe7607d8740 20 client.16773 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:05.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe7607d8740 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55afbd08cd80 0x55afbd490c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe7607d8740 4 client.16773 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:48:05.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe7607d8740 1 -- 172.21.15.29:0/4020054231 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afbd3ea640 con 0x55afbd08cd80 2024-09-21T02:48:05.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe7527fc640 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55afbd08cd80 0x55afbd490c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe732ffd640 10 client.16773 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:05.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.121+0000 7fe7527fc640 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55afbd08cd80 0x55afbd490c70 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=935dc2681397ccc4 server_cookie=9e90869fcaad3871 in_seq=0 out_seq=0 2024-09-21T02:48:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.123+0000 7fe732ffd640 1 -- 172.21.15.29:0/4020054231 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55afbd3ea640 con 0x55afbd08cd80 2024-09-21T02:48:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.123+0000 7fe732ffd640 10 client.16773 handle_command_reply: tid=0 2024-09-21T02:48:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.123+0000 7fe7607d8740 1 client.16773 shutdown 2024-09-21T02:48:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 -- 172.21.15.29:0/4020054231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0a52f0 msgr2=0x55afbd483960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0a52f0 0x55afbd483960 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe744005760 tx=0x7fe7440318b0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 -- 172.21.15.29:0/4020054231 shutdown_connections 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ce1b0 0x55afbd484b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ad4d0 0x55afbd484020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55afbd08cd80 0x55afbd490c70 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 --2- 172.21.15.29:0/4020054231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0a52f0 0x55afbd483960 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 -- 172.21.15.29:0/4020054231 >> 172.21.15.29:0/4020054231 conn(0x55afbd0c2300 msgr2=0x55afbd46fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe732ffd640 0 client.16773 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 -- 172.21.15.29:0/4020054231 shutdown_connections 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 1 -- 172.21.15.29:0/4020054231 wait complete. 2024-09-21T02:48:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.124+0000 7fe7607d8740 20 client.16773 trim_cache size 0 max 16384 2024-09-21T02:48:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.142+0000 7fe7607d8740 1 Processor -- start 2024-09-21T02:48:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7607d8740 1 -- start start 2024-09-21T02:48:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd470650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ad4d0 0x55afbd48ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ce1b0 0x55afbd470b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7607d8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe7541aa350 con 0x55afbd0ce1b0 2024-09-21T02:48:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7607d8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe7541abe90 con 0x55afbd0a52f0 2024-09-21T02:48:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7607d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe7541aaa20 con 0x55afbd0ad4d0 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ce1b0 0x55afbd470b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ce1b0 0x55afbd470b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47444/0 (socket says 172.21.15.29:47444) 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7527fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ad4d0 0x55afbd48ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7537fe640 1 -- 172.21.15.29:0/3079148490 learned_addr learned my addr 172.21.15.29:0/3079148490 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe752ffd640 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd470650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7537fe640 1 -- 172.21.15.29:0/3079148490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ad4d0 msgr2=0x55afbd48ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7537fe640 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ad4d0 0x55afbd48ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7537fe640 1 -- 172.21.15.29:0/3079148490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 msgr2=0x55afbd470650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7537fe640 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd470650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7537fe640 1 -- 172.21.15.29:0/3079148490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afbd481b70 con 0x55afbd0ce1b0 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.143+0000 7fe7527fc640 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ad4d0 0x55afbd48ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.144+0000 7fe7537fe640 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ce1b0 0x55afbd470b90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe74c01d760 tx=0x7fe74c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c48567a312b142b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.144+0000 7fe7397fa640 1 -- 172.21.15.29:0/3079148490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c01de10 con 0x55afbd0ce1b0 2024-09-21T02:48:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.144+0000 7fe7397fa640 1 -- 172.21.15.29:0/3079148490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe74c02e070 con 0x55afbd0ce1b0 2024-09-21T02:48:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.144+0000 7fe7397fa640 1 -- 172.21.15.29:0/3079148490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c027620 con 0x55afbd0ce1b0 2024-09-21T02:48:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 -- 172.21.15.29:0/3079148490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ce1b0 msgr2=0x55afbd470b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ce1b0 0x55afbd470b90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe74c01d760 tx=0x7fe74c01b000 comp rx=0 tx=0).stop 2024-09-21T02:48:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 -- 172.21.15.29:0/3079148490 shutdown_connections 2024-09-21T02:48:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ce1b0 0x55afbd470b90 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0ad4d0 0x55afbd48ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3079148490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0a52f0 0x55afbd470650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 -- 172.21.15.29:0/3079148490 >> 172.21.15.29:0/3079148490 conn(0x55afbd0c2300 msgr2=0x55afbd482b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 -- 172.21.15.29:0/3079148490 shutdown_connections 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 -- 172.21.15.29:0/3079148490 wait complete. 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.145+0000 7fe7607d8740 1 -- start start 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7607d8740 1 Processor -- start 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0a52f0 0x55afbd4888a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd488f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7607d8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ce1b0 0x55afbd489a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7607d8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe7541ab7c0 con 0x55afbd0ad4d0 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7607d8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe7541ab0f0 con 0x55afbd0ce1b0 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7607d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe7541a8330 con 0x55afbd0a52f0 2024-09-21T02:48:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7607d8740 10 client.0 _ll_register_callbacks cb 0x7fe7480eaab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7527fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd488f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7527fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd488f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47454/0 (socket says 172.21.15.29:47454) 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe752ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0a52f0 0x55afbd4888a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7527fc640 1 -- 172.21.15.29:0/3304376504 learned_addr learned my addr 172.21.15.29:0/3304376504 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7537fe640 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ce1b0 0x55afbd489a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7527fc640 1 -- 172.21.15.29:0/3304376504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0a52f0 msgr2=0x55afbd4888a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7527fc640 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0a52f0 0x55afbd4888a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7527fc640 1 -- 172.21.15.29:0/3304376504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ce1b0 msgr2=0x55afbd489a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7527fc640 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ce1b0 0x55afbd489a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7527fc640 1 -- 172.21.15.29:0/3304376504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afbd48a1c0 con 0x55afbd0ad4d0 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe7537fe640 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ce1b0 0x55afbd489a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.146+0000 7fe752ffd640 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0a52f0 0x55afbd4888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe7527fc640 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd488f60 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe748000c00 tx=0x7fe7480d0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56117a55d2df1b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe732ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe732ffd640 1 -- 172.21.15.29:0/3304376504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748020040 con 0x55afbd0ad4d0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe732ffd640 1 -- 172.21.15.29:0/3304376504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe7480d4470 con 0x55afbd0ad4d0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe7607d8740 1 -- 172.21.15.29:0/3304376504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe74c02e070 con 0x55afbd0ad4d0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe732ffd640 1 -- 172.21.15.29:0/3304376504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe7480acad0 con 0x55afbd0ad4d0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe7607d8740 10 client.16779 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe7607d8740 1 -- 172.21.15.29:0/3304376504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55afbd48b580 con 0x55afbd0ad4d0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.147+0000 7fe732ffd640 1 -- 172.21.15.29:0/3304376504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe74c02e070 con 0x55afbd0ad4d0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe732ffd640 1 -- 172.21.15.29:0/3304376504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fe7480a9230 con 0x55afbd0ad4d0 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe7607d8740 10 client.16779 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe7607d8740 10 client.16779 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe7607d8740 20 client.16779 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55afbd0621c0 0x55afbd4fd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe7607d8740 4 client.16779 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:48:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe7607d8740 1 -- 172.21.15.29:0/3304376504 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55afbd3e60f0 con 0x55afbd0621c0 2024-09-21T02:48:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe752ffd640 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55afbd0621c0 0x55afbd4fd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe752ffd640 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55afbd0621c0 0x55afbd4fd5c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9da8659bbeaea05b server_cookie=89a8684df810694b in_seq=0 out_seq=0 2024-09-21T02:48:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.148+0000 7fe732ffd640 10 client.16779 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe732ffd640 1 -- 172.21.15.29:0/3304376504 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55afbd3e60f0 con 0x55afbd0621c0 2024-09-21T02:48:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe732ffd640 10 client.16779 handle_command_reply: tid=0 2024-09-21T02:48:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 client.16779 shutdown 2024-09-21T02:48:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 -- 172.21.15.29:0/3304376504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 msgr2=0x55afbd488f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd488f60 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe748000c00 tx=0x7fe7480d0040 comp rx=0 tx=0).stop 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 -- 172.21.15.29:0/3304376504 shutdown_connections 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55afbd0ce1b0 0x55afbd489a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55afbd0621c0 0x55afbd4fd5c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55afbd0ad4d0 0x55afbd488f60 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 --2- 172.21.15.29:0/3304376504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55afbd0a52f0 0x55afbd4888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe732ffd640 0 client.16779 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 -- 172.21.15.29:0/3304376504 >> 172.21.15.29:0/3304376504 conn(0x55afbd0c2300 msgr2=0x55afbd47a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 -- 172.21.15.29:0/3304376504 shutdown_connections 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 1 -- 172.21.15.29:0/3304376504 wait complete. 2024-09-21T02:48:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.199+0000 7fe7607d8740 20 client.16779 trim_cache size 0 max 16384 2024-09-21T02:48:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:48:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:48:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:48:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:48:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 -- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe73c0b6510 msgr2=0x7fe73c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe73c0b6510 0x7fe73c0b8a20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00f9e0 tx=0x7fe74c0094e0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 -- 172.21.15.29:0/3339064413 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe754103620 msgr2=0x7fe75419a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe754103620 0x7fe75419a4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe74800d6d0 tx=0x7fe74800db90 comp rx=0 tx=0).stop 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 -- 172.21.15.29:0/3339064413 shutdown_connections 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe73c0b6510 0x7fe73c0b8a20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe754103fc0 0x7fe75419b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe754103620 0x7fe75419a4f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 --2- 172.21.15.29:0/3339064413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7541025a0 0x7fe754199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 -- 172.21.15.29:0/3339064413 >> 172.21.15.29:0/3339064413 conn(0x7fe7540fdee0 msgr2=0x7fe7540ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 -- 172.21.15.29:0/3339064413 shutdown_connections 2024-09-21T02:48:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:05.200+0000 7fe759869640 1 -- 172.21.15.29:0/3339064413 wait complete. 2024-09-21T02:48:05.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:05 smithi066 ceph-mon[33436]: pgmap v658: 129 pgs: 129 active+clean; 38 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.5 MiB/s wr, 132 op/s 2024-09-21T02:48:05.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:05 smithi190 ceph-mon[33871]: pgmap v658: 129 pgs: 129 active+clean; 38 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.5 MiB/s wr, 132 op/s 2024-09-21T02:48:05.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:05 smithi029 ceph-mon[31006]: pgmap v658: 129 pgs: 129 active+clean; 38 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.5 MiB/s wr, 132 op/s 2024-09-21T02:48:06.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 171923 2.23 MB/sec execute 343 sec latency 19.925 ms 2024-09-21T02:48:07.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 172602 2.23 MB/sec execute 344 sec latency 22.329 ms 2024-09-21T02:48:07.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:07 smithi066 ceph-mon[33436]: pgmap v659: 129 pgs: 129 active+clean; 39 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.6 MiB/s wr, 109 op/s 2024-09-21T02:48:07.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:07 smithi190 ceph-mon[33871]: pgmap v659: 129 pgs: 129 active+clean; 39 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.6 MiB/s wr, 109 op/s 2024-09-21T02:48:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:07 smithi029 ceph-mon[31006]: pgmap v659: 129 pgs: 129 active+clean; 39 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.6 MiB/s wr, 109 op/s 2024-09-21T02:48:08.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 173443 2.24 MB/sec execute 345 sec latency 17.999 ms 2024-09-21T02:48:09.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 173949 2.25 MB/sec execute 346 sec latency 15.469 ms 2024-09-21T02:48:09.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:09 smithi066 ceph-mon[33436]: pgmap v660: 129 pgs: 129 active+clean; 38 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:48:09.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:09 smithi190 ceph-mon[33871]: pgmap v660: 129 pgs: 129 active+clean; 38 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:48:09.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:09 smithi029 ceph-mon[31006]: pgmap v660: 129 pgs: 129 active+clean; 38 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:48:10.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 174233 2.24 MB/sec execute 347 sec latency 60.492 ms 2024-09-21T02:48:11.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 174684 2.24 MB/sec execute 348 sec latency 17.685 ms 2024-09-21T02:48:11.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:11 smithi066 ceph-mon[33436]: pgmap v661: 129 pgs: 129 active+clean; 40 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-09-21T02:48:11.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:11 smithi190 ceph-mon[33871]: pgmap v661: 129 pgs: 129 active+clean; 40 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-09-21T02:48:11.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:11 smithi029 ceph-mon[31006]: pgmap v661: 129 pgs: 129 active+clean; 40 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-09-21T02:48:12.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 175319 2.25 MB/sec execute 349 sec latency 15.278 ms 2024-09-21T02:48:13.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 175947 2.24 MB/sec execute 350 sec latency 17.709 ms 2024-09-21T02:48:13.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:13 smithi066 ceph-mon[33436]: pgmap v662: 129 pgs: 129 active+clean; 40 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:48:13.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:13 smithi190 ceph-mon[33871]: pgmap v662: 129 pgs: 129 active+clean; 40 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:48:13.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:13 smithi029 ceph-mon[31006]: pgmap v662: 129 pgs: 129 active+clean; 40 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:48:14.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 176694 2.25 MB/sec execute 351 sec latency 16.418 ms 2024-09-21T02:48:15.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 177339 2.26 MB/sec execute 352 sec latency 15.725 ms 2024-09-21T02:48:15.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:15 smithi066 ceph-mon[33436]: pgmap v663: 129 pgs: 129 active+clean; 40 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-09-21T02:48:15.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:15 smithi190 ceph-mon[33871]: pgmap v663: 129 pgs: 129 active+clean; 40 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-09-21T02:48:15.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:15 smithi029 ceph-mon[31006]: pgmap v663: 129 pgs: 129 active+clean; 40 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-09-21T02:48:16.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 177735 2.26 MB/sec execute 353 sec latency 14.535 ms 2024-09-21T02:48:17.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 178110 2.25 MB/sec execute 354 sec latency 19.524 ms 2024-09-21T02:48:17.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:17 smithi066 ceph-mon[33436]: pgmap v664: 129 pgs: 129 active+clean; 41 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-09-21T02:48:17.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:17 smithi190 ceph-mon[33871]: pgmap v664: 129 pgs: 129 active+clean; 41 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-09-21T02:48:17.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:17 smithi029 ceph-mon[31006]: pgmap v664: 129 pgs: 129 active+clean; 41 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-09-21T02:48:18.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 178769 2.26 MB/sec execute 355 sec latency 16.605 ms 2024-09-21T02:48:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.310+0000 7fb62217b640 1 -- 172.21.15.29:0/159746406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103620 msgr2=0x7fb61c103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.310+0000 7fb62217b640 1 --2- 172.21.15.29:0/159746406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103620 0x7fb61c103a30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb610009900 tx=0x7fb61002a410 comp rx=0 tx=0).stop 2024-09-21T02:48:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.310+0000 7fb62217b640 1 -- 172.21.15.29:0/159746406 shutdown_connections 2024-09-21T02:48:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.310+0000 7fb62217b640 1 --2- 172.21.15.29:0/159746406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb61c103fc0 0x7fb61c10a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.310+0000 7fb62217b640 1 --2- 172.21.15.29:0/159746406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103620 0x7fb61c103a30 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.310+0000 7fb62217b640 1 --2- 172.21.15.29:0/159746406 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb61c1025a0 0x7fb61c102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.310+0000 7fb62217b640 1 -- 172.21.15.29:0/159746406 >> 172.21.15.29:0/159746406 conn(0x7fb61c0fdee0 msgr2=0x7fb61c100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:18.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.311+0000 7fb62217b640 1 -- 172.21.15.29:0/159746406 shutdown_connections 2024-09-21T02:48:18.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.311+0000 7fb62217b640 1 -- 172.21.15.29:0/159746406 wait complete. 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.311+0000 7fb62217b640 1 Processor -- start 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.311+0000 7fb62217b640 1 -- start start 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb62217b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb61c1025a0 0x7fb61c199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb62217b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb61c103620 0x7fb61c19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb62217b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103fc0 0x7fb61c19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb62217b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c10e090 con 0x7fb61c103fc0 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb62217b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c10df10 con 0x7fb61c1025a0 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb62217b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c10e210 con 0x7fb61c103620 2024-09-21T02:48:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb61c1025a0 0x7fb61c199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb61c103620 0x7fb61c19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103fc0 0x7fb61c19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb61c103620 0x7fb61c19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52434/0 (socket says 172.21.15.29:52434) 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb61c1025a0 0x7fb61c199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:44440/0 (socket says 172.21.15.29:44440) 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61affd640 1 -- 172.21.15.29:0/1615464763 learned_addr learned my addr 172.21.15.29:0/1615464763 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61bfff640 1 -- 172.21.15.29:0/1615464763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb61c103620 msgr2=0x7fb61c19a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61bfff640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb61c103620 0x7fb61c19a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61bfff640 1 -- 172.21.15.29:0/1615464763 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb61c1025a0 msgr2=0x7fb61c199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61bfff640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb61c1025a0 0x7fb61c199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61bfff640 1 -- 172.21.15.29:0/1615464763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61c19b8c0 con 0x7fb61c103fc0 2024-09-21T02:48:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61affd640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb61c103620 0x7fb61c19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:18.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb61c1025a0 0x7fb61c199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:18.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.312+0000 7fb61bfff640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103fc0 0x7fb61c19b0a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb61400eb20 tx=0x7fb61400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:18.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.313+0000 7fb618ff9640 1 -- 172.21.15.29:0/1615464763 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140039c0 con 0x7fb61c103fc0 2024-09-21T02:48:18.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.313+0000 7fb62217b640 1 -- 172.21.15.29:0/1615464763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb61c19bbb0 con 0x7fb61c103fc0 2024-09-21T02:48:18.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.313+0000 7fb618ff9640 1 -- 172.21.15.29:0/1615464763 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb614004020 con 0x7fb61c103fc0 2024-09-21T02:48:18.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.313+0000 7fb618ff9640 1 -- 172.21.15.29:0/1615464763 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614010690 con 0x7fb61c103fc0 2024-09-21T02:48:18.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.313+0000 7fb62217b640 1 -- 172.21.15.29:0/1615464763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb61c1a22a0 con 0x7fb61c103fc0 2024-09-21T02:48:18.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.316+0000 7fb6291c6740 1 Processor -- start 2024-09-21T02:48:18.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb618ff9640 1 -- 172.21.15.29:0/1615464763 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb614010830 con 0x7fb61c103fc0 2024-09-21T02:48:18.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb6291c6740 1 -- start start 2024-09-21T02:48:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e440550e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e4405514c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440552010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb6291c6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e4404c36f0 con 0x55e4401884d0 2024-09-21T02:48:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb6291c6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e4404b2b20 con 0x55e4401a91b0 2024-09-21T02:48:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb6291c6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e4404b6930 con 0x55e4401802f0 2024-09-21T02:48:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb61b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e4405514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb61b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e4405514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52444/0 (socket says 172.21.15.29:52444) 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.317+0000 7fb61b7fe640 1 -- 172.21.15.29:0/2411919678 learned_addr learned my addr 172.21.15.29:0/2411919678 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61affd640 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e440550e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61affd640 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440552010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61b7fe640 1 -- 172.21.15.29:0/2411919678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 msgr2=0x55e440550e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e440550e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61b7fe640 1 -- 172.21.15.29:0/2411919678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 msgr2=0x55e440552010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440552010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61b7fe640 1 -- 172.21.15.29:0/2411919678 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e440478530 con 0x55e4401802f0 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61affd640 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440552010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb618ff9640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6040b6510 0x7fb6040b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e4405514c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb60c0115f0 tx=0x7fb60c011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9402e05ac1a5ceac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb61affd640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6040b6510 0x7fb6040b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb618ff9640 1 -- 172.21.15.29:0/1615464763 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb6140e7d60 con 0x7fb61c103fc0 2024-09-21T02:48:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb601ffb640 1 -- 172.21.15.29:0/2411919678 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60c01e070 con 0x55e4401802f0 2024-09-21T02:48:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb601ffb640 1 -- 172.21.15.29:0/2411919678 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb60c00fad0 con 0x55e4401802f0 2024-09-21T02:48:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.318+0000 7fb601ffb640 1 -- 172.21.15.29:0/2411919678 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60c019a50 con 0x55e4401802f0 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.319+0000 7fb6291c6740 1 -- 172.21.15.29:0/2411919678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 msgr2=0x55e4405514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.319+0000 7fb6291c6740 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e4405514c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb60c0115f0 tx=0x7fb60c011ab0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.319+0000 7fb61affd640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6040b6510 0x7fb6040b8a20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb610005760 tx=0x7fb610031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.319+0000 7fb6291c6740 1 -- 172.21.15.29:0/2411919678 shutdown_connections 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440552010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e4405514c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 --2- 172.21.15.29:0/2411919678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e440550e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 -- 172.21.15.29:0/2411919678 >> 172.21.15.29:0/2411919678 conn(0x55e44019d300 msgr2=0x55e44054c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 -- 172.21.15.29:0/2411919678 shutdown_connections 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 -- 172.21.15.29:0/2411919678 wait complete. 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 -- start start 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 Processor -- start 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.320+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e44055e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e44055f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e44055fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb6291c6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e4403eaab0 con 0x55e4401884d0 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb6291c6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e440458730 con 0x55e4401802f0 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb6291c6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e440552b30 con 0x55e4401a91b0 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb6291c6740 10 client.0 _ll_register_callbacks cb 0x55e4404ac6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e44055f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e44055fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e44055f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56702/0 (socket says 172.21.15.29:56702) 2024-09-21T02:48:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61affd640 1 -- 172.21.15.29:0/3527401376 learned_addr learned my addr 172.21.15.29:0/3527401376 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e44055e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61bfff640 1 -- 172.21.15.29:0/3527401376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 msgr2=0x55e44055e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61bfff640 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e44055e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61bfff640 1 -- 172.21.15.29:0/3527401376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 msgr2=0x55e44055f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61bfff640 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e44055f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61bfff640 1 -- 172.21.15.29:0/3527401376 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e440214790 con 0x55e4401a91b0 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61affd640 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e44055f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb61bfff640 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e44055fc00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb614009ac0 tx=0x7fb6140ad680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bea651adb7e2770c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb5faffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.321+0000 7fb5faffd640 1 -- 172.21.15.29:0/3527401376 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140b2650 con 0x55e4401a91b0 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 1 -- 172.21.15.29:0/3527401376 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb60c00fad0 con 0x55e4401a91b0 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb5faffd640 1 -- 172.21.15.29:0/3527401376 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb6140cfb80 con 0x55e4401a91b0 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb5faffd640 1 -- 172.21.15.29:0/3527401376 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb6140b7590 con 0x55e4401a91b0 2024-09-21T02:48:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 10 client.26353 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 1 -- 172.21.15.29:0/3527401376 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e44028d620 con 0x55e4401a91b0 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb5faffd640 1 -- 172.21.15.29:0/3527401376 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb60c00fad0 con 0x55e4401a91b0 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb5faffd640 1 -- 172.21.15.29:0/3527401376 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb6140b21c0 con 0x55e4401a91b0 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 10 client.26353 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 10 client.26353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 20 client.26353 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e440167d80 0x55e44056c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 4 client.26353 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb6291c6740 1 -- 172.21.15.29:0/3527401376 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4404c5640 con 0x55e440167d80 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.322+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e440167d80 0x55e44056c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.323+0000 7fb5faffd640 10 client.26353 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.323+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e440167d80 0x55e44056c0a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4def5104fda450fc server_cookie=a01d85f76418cfd3 in_seq=0 out_seq=0 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb5faffd640 1 -- 172.21.15.29:0/3527401376 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e4404c5640 con 0x55e440167d80 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb5faffd640 10 client.26353 handle_command_reply: tid=0 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 client.26353 shutdown 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 -- 172.21.15.29:0/3527401376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 msgr2=0x55e44055fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e44055fc00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb614009ac0 tx=0x7fb6140ad680 comp rx=0 tx=0).stop 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 -- 172.21.15.29:0/3527401376 shutdown_connections 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e44055fc00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e440167d80 0x55e44056c0a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e44055f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 --2- 172.21.15.29:0/3527401376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e44055e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb5faffd640 0 client.26353 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 -- 172.21.15.29:0/3527401376 >> 172.21.15.29:0/3527401376 conn(0x55e44019d300 msgr2=0x55e44054aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 -- 172.21.15.29:0/3527401376 shutdown_connections 2024-09-21T02:48:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 1 -- 172.21.15.29:0/3527401376 wait complete. 2024-09-21T02:48:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.324+0000 7fb6291c6740 20 client.26353 trim_cache size 0 max 16384 2024-09-21T02:48:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb6291c6740 1 Processor -- start 2024-09-21T02:48:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb6291c6740 1 -- start start 2024-09-21T02:48:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e440550df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440551330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e44055d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb6291c6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c1aa340 con 0x55e4401884d0 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb6291c6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c1abe80 con 0x55e4401a91b0 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb6291c6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c1aaa10 con 0x55e4401802f0 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440551330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e440550df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440551330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56704/0 (socket says 172.21.15.29:56704) 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61affd640 1 -- 172.21.15.29:0/2054857489 learned_addr learned my addr 172.21.15.29:0/2054857489 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e440550df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52456/0 (socket says 172.21.15.29:52456) 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61bfff640 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e44055d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61b7fe640 1 -- 172.21.15.29:0/2054857489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 msgr2=0x55e44055d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e44055d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61b7fe640 1 -- 172.21.15.29:0/2054857489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 msgr2=0x55e440551330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440551330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61b7fe640 1 -- 172.21.15.29:0/2054857489 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e44055f400 con 0x55e4401802f0 2024-09-21T02:48:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61affd640 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440551330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.341+0000 7fb61bfff640 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e44055d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:48:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.342+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e440550df0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb60c000c00 tx=0x7fb60c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e931c99762908da1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.342+0000 7fb6017fa640 1 -- 172.21.15.29:0/2054857489 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60c01e040 con 0x55e4401802f0 2024-09-21T02:48:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.342+0000 7fb6017fa640 1 -- 172.21.15.29:0/2054857489 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb60c029070 con 0x55e4401802f0 2024-09-21T02:48:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.342+0000 7fb6017fa640 1 -- 172.21.15.29:0/2054857489 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60c003850 con 0x55e4401802f0 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 -- 172.21.15.29:0/2054857489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 msgr2=0x55e440550df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e440550df0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb60c000c00 tx=0x7fb60c014000 comp rx=0 tx=0).stop 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 -- 172.21.15.29:0/2054857489 shutdown_connections 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401a91b0 0x55e44055d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440551330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 --2- 172.21.15.29:0/2054857489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401802f0 0x55e440550df0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 -- 172.21.15.29:0/2054857489 >> 172.21.15.29:0/2054857489 conn(0x55e44019d300 msgr2=0x55e44054c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 -- 172.21.15.29:0/2054857489 shutdown_connections 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 -- 172.21.15.29:0/2054857489 wait complete. 2024-09-21T02:48:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.343+0000 7fb6291c6740 1 -- start start 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb6291c6740 1 Processor -- start 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e440566270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440564f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb6291c6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e440565470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb6291c6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c1ab7b0 con 0x55e4401884d0 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb6291c6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c1ab0e0 con 0x55e4401802f0 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb6291c6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c1a8320 con 0x55e4401a91b0 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb6291c6740 10 client.0 _ll_register_callbacks cb 0x7fb6140ad4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440564f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e440566270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440564f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56718/0 (socket says 172.21.15.29:56718) 2024-09-21T02:48:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61affd640 1 -- 172.21.15.29:0/1040818173 learned_addr learned my addr 172.21.15.29:0/1040818173 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61bfff640 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e440565470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61affd640 1 -- 172.21.15.29:0/1040818173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 msgr2=0x55e440565470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61affd640 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e440565470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61affd640 1 -- 172.21.15.29:0/1040818173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 msgr2=0x55e440566270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61affd640 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e440566270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61affd640 1 -- 172.21.15.29:0/1040818173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e440565be0 con 0x55e4401884d0 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.344+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e440566270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb61affd640 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440564f30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb610003870 tx=0x7fb6100318b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd039341777cb74 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb5faffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb5faffd640 1 -- 172.21.15.29:0/1040818173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb610057060 con 0x55e4401884d0 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb6291c6740 1 -- 172.21.15.29:0/1040818173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb60c029070 con 0x55e4401884d0 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb5faffd640 1 -- 172.21.15.29:0/1040818173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb610048400 con 0x55e4401884d0 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb5faffd640 1 -- 172.21.15.29:0/1040818173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb610035040 con 0x55e4401884d0 2024-09-21T02:48:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb6291c6740 10 client.16794 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb6291c6740 1 -- 172.21.15.29:0/1040818173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e4405d6a70 con 0x55e4401884d0 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb5faffd640 1 -- 172.21.15.29:0/1040818173 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb60c029070 con 0x55e4401884d0 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.345+0000 7fb5faffd640 1 -- 172.21.15.29:0/1040818173 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb610051070 con 0x55e4401884d0 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb6291c6740 10 client.16794 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb6291c6740 10 client.16794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb6291c6740 20 client.16794 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb6291c6740 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e44013d1c0 0x55e4405d93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb6291c6740 4 client.16794 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb6291c6740 1 -- 172.21.15.29:0/1040818173 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e4404c10f0 con 0x55e44013d1c0 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e44013d1c0 0x55e4405d93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb5faffd640 10 client.16794 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.346+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e44013d1c0 0x55e4405d93c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d5295abdc76f982 server_cookie=ee741ca5583cbf12 in_seq=0 out_seq=0 2024-09-21T02:48:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.390+0000 7fb5faffd640 1 -- 172.21.15.29:0/1040818173 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e4404c10f0 con 0x55e44013d1c0 2024-09-21T02:48:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.390+0000 7fb5faffd640 10 client.16794 handle_command_reply: tid=0 2024-09-21T02:48:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.390+0000 7fb6291c6740 1 client.16794 shutdown 2024-09-21T02:48:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 -- 172.21.15.29:0/1040818173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 msgr2=0x55e440564f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440564f30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb610003870 tx=0x7fb6100318b0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 -- 172.21.15.29:0/1040818173 shutdown_connections 2024-09-21T02:48:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e4401a91b0 0x55e440565470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e44013d1c0 0x55e4405d93c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e4401884d0 0x55e440564f30 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 --2- 172.21.15.29:0/1040818173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e4401802f0 0x55e440566270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 -- 172.21.15.29:0/1040818173 >> 172.21.15.29:0/1040818173 conn(0x55e44019d300 msgr2=0x55e44054c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb5faffd640 0 client.16794 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 -- 172.21.15.29:0/1040818173 shutdown_connections 2024-09-21T02:48:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.392+0000 7fb6291c6740 1 -- 172.21.15.29:0/1040818173 wait complete. 2024-09-21T02:48:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.393+0000 7fb6291c6740 20 client.16794 trim_cache size 0 max 16384 2024-09-21T02:48:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:48:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:48:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:48:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.393+0000 7fb62217b640 1 -- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6040b6510 msgr2=0x7fb6040b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.393+0000 7fb62217b640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6040b6510 0x7fb6040b8a20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb610005760 tx=0x7fb610031400 comp rx=0 tx=0).stop 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 -- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103fc0 msgr2=0x7fb61c19b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103fc0 0x7fb61c19b0a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb61400eb20 tx=0x7fb61400c2d0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 -- 172.21.15.29:0/1615464763 shutdown_connections 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6040b6510 0x7fb6040b8a20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c103fc0 0x7fb61c19b0a0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb61c103620 0x7fb61c19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 --2- 172.21.15.29:0/1615464763 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb61c1025a0 0x7fb61c199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 -- 172.21.15.29:0/1615464763 >> 172.21.15.29:0/1615464763 conn(0x7fb61c0fdee0 msgr2=0x7fb61c0ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 -- 172.21.15.29:0/1615464763 shutdown_connections 2024-09-21T02:48:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:18.394+0000 7fb62217b640 1 -- 172.21.15.29:0/1615464763 wait complete. 2024-09-21T02:48:19.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 179369 2.26 MB/sec execute 356 sec latency 35.748 ms 2024-09-21T02:48:19.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:19 smithi066 ceph-mon[33436]: pgmap v665: 129 pgs: 129 active+clean; 42 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-09-21T02:48:19.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:19 smithi190 ceph-mon[33871]: pgmap v665: 129 pgs: 129 active+clean; 42 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-09-21T02:48:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:19 smithi029 ceph-mon[31006]: pgmap v665: 129 pgs: 129 active+clean; 42 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-09-21T02:48:20.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 179959 2.26 MB/sec execute 357 sec latency 15.361 ms 2024-09-21T02:48:21.093 INFO:tasks.workunit.client.0.smithi029.stdout: 1 180686 2.27 MB/sec execute 358 sec latency 17.062 ms 2024-09-21T02:48:21.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:21 smithi066 ceph-mon[33436]: pgmap v666: 129 pgs: 129 active+clean; 43 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-09-21T02:48:21.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:21 smithi190 ceph-mon[33871]: pgmap v666: 129 pgs: 129 active+clean; 43 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-09-21T02:48:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:21 smithi029 ceph-mon[31006]: pgmap v666: 129 pgs: 129 active+clean; 43 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-09-21T02:48:22.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 181201 2.27 MB/sec execute 359 sec latency 16.104 ms 2024-09-21T02:48:23.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 181495 2.27 MB/sec execute 360 sec latency 14.992 ms 2024-09-21T02:48:23.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:23 smithi066 ceph-mon[33436]: pgmap v667: 129 pgs: 129 active+clean; 37 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-09-21T02:48:23.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:23 smithi190 ceph-mon[33871]: pgmap v667: 129 pgs: 129 active+clean; 37 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-09-21T02:48:23.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:23 smithi029 ceph-mon[31006]: pgmap v667: 129 pgs: 129 active+clean; 37 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-09-21T02:48:24.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 182089 2.27 MB/sec execute 361 sec latency 18.628 ms 2024-09-21T02:48:25.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 182759 2.27 MB/sec execute 362 sec latency 16.308 ms 2024-09-21T02:48:25.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:25 smithi066 ceph-mon[33436]: pgmap v668: 129 pgs: 129 active+clean; 38 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:48:25.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:25 smithi190 ceph-mon[33871]: pgmap v668: 129 pgs: 129 active+clean; 38 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:48:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:25 smithi029 ceph-mon[31006]: pgmap v668: 129 pgs: 129 active+clean; 38 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:48:26.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 183361 2.27 MB/sec execute 363 sec latency 14.880 ms 2024-09-21T02:48:27.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 184209 2.28 MB/sec execute 364 sec latency 18.694 ms 2024-09-21T02:48:27.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:27 smithi066 ceph-mon[33436]: pgmap v669: 129 pgs: 129 active+clean; 40 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 136 op/s 2024-09-21T02:48:27.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:27 smithi190 ceph-mon[33871]: pgmap v669: 129 pgs: 129 active+clean; 40 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 136 op/s 2024-09-21T02:48:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:27 smithi029 ceph-mon[31006]: pgmap v669: 129 pgs: 129 active+clean; 40 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 136 op/s 2024-09-21T02:48:28.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 184698 2.29 MB/sec execute 365 sec latency 29.548 ms 2024-09-21T02:48:29.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 184939 2.28 MB/sec execute 366 sec latency 15.015 ms 2024-09-21T02:48:29.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:29 smithi066 ceph-mon[33436]: pgmap v670: 129 pgs: 129 active+clean; 41 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:48:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:29 smithi190 ceph-mon[33871]: pgmap v670: 129 pgs: 129 active+clean; 41 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:48:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:29 smithi029 ceph-mon[31006]: pgmap v670: 129 pgs: 129 active+clean; 41 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:48:30.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 185545 2.28 MB/sec execute 367 sec latency 17.766 ms 2024-09-21T02:48:31.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 186227 2.29 MB/sec execute 368 sec latency 16.975 ms 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 -- 172.21.15.29:0/1447403990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9440103fc0 msgr2=0x7f944010a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 --2- 172.21.15.29:0/1447403990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9440103fc0 0x7f944010a960 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9438009c90 tx=0x7f943802a860 comp rx=0 tx=0).stop 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 -- 172.21.15.29:0/1447403990 shutdown_connections 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 --2- 172.21.15.29:0/1447403990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9440103fc0 0x7f944010a960 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 --2- 172.21.15.29:0/1447403990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9440103620 0x7f9440103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 --2- 172.21.15.29:0/1447403990 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94401025a0 0x7f9440102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 -- 172.21.15.29:0/1447403990 >> 172.21.15.29:0/1447403990 conn(0x7f94400fdee0 msgr2=0x7f9440100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 -- 172.21.15.29:0/1447403990 shutdown_connections 2024-09-21T02:48:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.502+0000 7f9448721640 1 -- 172.21.15.29:0/1447403990 wait complete. 2024-09-21T02:48:31.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.503+0000 7f9448721640 1 Processor -- start 2024-09-21T02:48:31.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.503+0000 7f9448721640 1 -- start start 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.503+0000 7f9448721640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94401025a0 0x7f9440198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.503+0000 7f9448721640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9440103620 0x7f9440199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.503+0000 7f9448721640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9440103fc0 0x7f9440199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.503+0000 7f9448721640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f944010e090 con 0x7f94401025a0 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.503+0000 7f9448721640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f944010df10 con 0x7f9440103fc0 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.503+0000 7f9448721640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f944010e210 con 0x7f9440103620 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94401025a0 0x7f9440198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94401025a0 0x7f9440198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55862/0 (socket says 172.21.15.29:55862) 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 -- 172.21.15.29:0/880207926 learned_addr learned my addr 172.21.15.29:0/880207926 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f9445cbc640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9440103620 0x7f9440199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f9446cbe640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9440103fc0 0x7f9440199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 -- 172.21.15.29:0/880207926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9440103620 msgr2=0x7f9440199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9440103620 0x7f9440199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 -- 172.21.15.29:0/880207926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9440103fc0 msgr2=0x7f9440199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9440103fc0 0x7f9440199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 -- 172.21.15.29:0/880207926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944019a760 con 0x7f94401025a0 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94464bd640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94401025a0 0x7f9440198d20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f943c00e9f0 tx=0x7f943c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94377fe640 1 -- 172.21.15.29:0/880207926 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c00cd80 con 0x7f94401025a0 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94377fe640 1 -- 172.21.15.29:0/880207926 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f943c0108c0 con 0x7f94401025a0 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f9448721640 1 -- 172.21.15.29:0/880207926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f944019aa50 con 0x7f94401025a0 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f94377fe640 1 -- 172.21.15.29:0/880207926 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c0187c0 con 0x7f94401025a0 2024-09-21T02:48:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.504+0000 7f9448721640 1 -- 172.21.15.29:0/880207926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f944006adc0 con 0x7f94401025a0 2024-09-21T02:48:31.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:31 smithi066 ceph-mon[33436]: pgmap v671: 129 pgs: 129 active+clean; 40 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:48:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.507+0000 7f944f688740 1 Processor -- start 2024-09-21T02:48:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.507+0000 7f944f688740 1 -- start start 2024-09-21T02:48:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f944f688740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c751b0 0x55e62201ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f944f688740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 0x55e62201d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f944f688740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c544d0 0x55e62201e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f944f688740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e621f8f6f0 con 0x55e621c544d0 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f944f688740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e621f7eb20 con 0x55e621c4c2f0 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f944f688740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e621f82930 con 0x55e621c751b0 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9446cbe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c751b0 0x55e62201ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9445cbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c544d0 0x55e62201e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9445cbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c544d0 0x55e62201e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55868/0 (socket says 172.21.15.29:55868) 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9446cbe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c751b0 0x55e62201ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38264/0 (socket says 172.21.15.29:38264) 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9445cbc640 1 -- 172.21.15.29:0/775668701 learned_addr learned my addr 172.21.15.29:0/775668701 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f94377fe640 1 -- 172.21.15.29:0/880207926 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f943c010430 con 0x7f94401025a0 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9446cbe640 1 -- 172.21.15.29:0/775668701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 msgr2=0x55e62201d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9446cbe640 1 --2- 172.21.15.29:0/775668701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 0x55e62201d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9446cbe640 1 -- 172.21.15.29:0/775668701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c544d0 msgr2=0x55e62201e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9446cbe640 1 --2- 172.21.15.29:0/775668701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c544d0 0x55e62201e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9446cbe640 1 -- 172.21.15.29:0/775668701 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e621f44530 con 0x55e621c751b0 2024-09-21T02:48:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.508+0000 7f9446cbe640 1 --2- 172.21.15.29:0/775668701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c751b0 0x55e62201ce50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f94380092e0 tx=0x7f94380093d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4504f4fb7cccd098 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.509+0000 7f9434ff9640 1 -- 172.21.15.29:0/775668701 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94380316f0 con 0x55e621c751b0 2024-09-21T02:48:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.509+0000 7f9434ff9640 1 -- 172.21.15.29:0/775668701 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9438036070 con 0x55e621c751b0 2024-09-21T02:48:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.509+0000 7f9434ff9640 1 -- 172.21.15.29:0/775668701 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9438003e60 con 0x55e621c751b0 2024-09-21T02:48:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.509+0000 7f94377fe640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f942c0b6590 0x7f942c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f94464bd640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f942c0b6590 0x7f942c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f94377fe640 1 -- 172.21.15.29:0/880207926 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f943c0e7950 con 0x7f94401025a0 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 -- 172.21.15.29:0/775668701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c751b0 msgr2=0x55e62201ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 --2- 172.21.15.29:0/775668701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c751b0 0x55e62201ce50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f94380092e0 tx=0x7f94380093d0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 -- 172.21.15.29:0/775668701 shutdown_connections 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 --2- 172.21.15.29:0/775668701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c544d0 0x55e62201e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 --2- 172.21.15.29:0/775668701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 0x55e62201d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 --2- 172.21.15.29:0/775668701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c751b0 0x55e62201ce50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 -- 172.21.15.29:0/775668701 >> 172.21.15.29:0/775668701 conn(0x55e621c69300 msgr2=0x55e6220188e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 -- 172.21.15.29:0/775668701 shutdown_connections 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f944f688740 1 -- 172.21.15.29:0/775668701 wait complete. 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.510+0000 7f94464bd640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f942c0b6590 0x7f942c0b8aa0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f943c00c6a0 tx=0x7f943c07a5d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 1 -- start start 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 1 Processor -- start 2024-09-21T02:48:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c4c2f0 0x55e622026590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e622026c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c751b0 0x55e6220277c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e621eb6ab0 con 0x55e621c751b0 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e621f24730 con 0x55e621c544d0 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e62201ead0 con 0x55e621c4c2f0 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f944f688740 10 client.0 _ll_register_callbacks cb 0x55e621f786b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f9446cbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c751b0 0x55e6220277c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f9446cbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c751b0 0x55e6220277c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55884/0 (socket says 172.21.15.29:55884) 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f9445cbc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e622026c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f9446cbe640 1 -- 172.21.15.29:0/3009973352 learned_addr learned my addr 172.21.15.29:0/3009973352 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.511+0000 7f9446cbe640 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c4c2f0 0x55e622026590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9445cbc640 1 -- 172.21.15.29:0/3009973352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c4c2f0 msgr2=0x55e622026590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9445cbc640 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c4c2f0 0x55e622026590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9445cbc640 1 -- 172.21.15.29:0/3009973352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c751b0 msgr2=0x55e6220277c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9445cbc640 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c751b0 0x55e6220277c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9445cbc640 1 -- 172.21.15.29:0/3009973352 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e621ce0790 con 0x55e621c544d0 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9445cbc640 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e622026c50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f943000bf30 tx=0x7f94300118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca0031137d2e89d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9429ffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9429ffb640 1 -- 172.21.15.29:0/3009973352 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430004020 con 0x55e621c544d0 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f9429ffb640 1 -- 172.21.15.29:0/3009973352 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f94300041c0 con 0x55e621c544d0 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.512+0000 7f944f688740 1 -- 172.21.15.29:0/3009973352 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9438036070 con 0x55e621c544d0 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f9429ffb640 1 -- 172.21.15.29:0/3009973352 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f94300043c0 con 0x55e621c544d0 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f944f688740 10 client.26255 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f944f688740 1 -- 172.21.15.29:0/3009973352 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e621d59620 con 0x55e621c544d0 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f9429ffb640 1 -- 172.21.15.29:0/3009973352 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9438036070 con 0x55e621c544d0 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f9429ffb640 1 -- 172.21.15.29:0/3009973352 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f9430004ba0 con 0x55e621c544d0 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f944f688740 10 client.26255 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f944f688740 10 client.26255 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f944f688740 20 client.26255 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f944f688740 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e621c33d80 0x55e622033d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f944f688740 4 client.26255 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.513+0000 7f944f688740 1 -- 172.21.15.29:0/3009973352 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e621f91640 con 0x55e621c33d80 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.514+0000 7f9446cbe640 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e621c33d80 0x55e622033d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.514+0000 7f9446cbe640 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e621c33d80 0x55e622033d50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad8cf6b64d017377 server_cookie=7619727228ed6dfc in_seq=0 out_seq=0 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.514+0000 7f9429ffb640 10 client.26255 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f9429ffb640 1 -- 172.21.15.29:0/3009973352 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e621f91640 con 0x55e621c33d80 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f9429ffb640 10 client.26255 handle_command_reply: tid=0 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 client.26255 shutdown 2024-09-21T02:48:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 -- 172.21.15.29:0/3009973352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 msgr2=0x55e622026c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e622026c50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f943000bf30 tx=0x7f94300118d0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 -- 172.21.15.29:0/3009973352 shutdown_connections 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e621c33d80 0x55e622033d50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c751b0 0x55e6220277c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e622026c50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f9429ffb640 0 client.26255 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 --2- 172.21.15.29:0/3009973352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c4c2f0 0x55e622026590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 -- 172.21.15.29:0/3009973352 >> 172.21.15.29:0/3009973352 conn(0x55e621c69300 msgr2=0x55e622016ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 -- 172.21.15.29:0/3009973352 shutdown_connections 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 1 -- 172.21.15.29:0/3009973352 wait complete. 2024-09-21T02:48:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.515+0000 7f944f688740 20 client.26255 trim_cache size 0 max 16384 2024-09-21T02:48:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f944f688740 1 Processor -- start 2024-09-21T02:48:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f944f688740 1 -- start start 2024-09-21T02:48:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f944f688740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c33d80 0x55e62201dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f944f688740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 0x55e622024b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f944f688740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c544d0 0x55e622025060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f944f688740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94401ac4c0 con 0x55e621c33d80 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f944f688740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f94401ae000 con 0x55e621c4c2f0 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f944f688740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f94401acb90 con 0x55e621c544d0 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f9446cbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c33d80 0x55e62201dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f9446cbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c33d80 0x55e62201dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55898/0 (socket says 172.21.15.29:55898) 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.530+0000 7f9446cbe640 1 -- 172.21.15.29:0/744616355 learned_addr learned my addr 172.21.15.29:0/744616355 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f9445cbc640 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 0x55e622024b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f9446cbe640 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c544d0 0x55e622025060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f9446cbe640 1 -- 172.21.15.29:0/744616355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c544d0 msgr2=0x55e622025060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f9446cbe640 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c544d0 0x55e622025060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f9446cbe640 1 -- 172.21.15.29:0/744616355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 msgr2=0x55e622024b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f9446cbe640 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 0x55e622024b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f9446cbe640 1 -- 172.21.15.29:0/744616355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e62202ea60 con 0x55e621c33d80 2024-09-21T02:48:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f9446cbe640 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c33d80 0x55e62201dce0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9438000f80 tx=0x7f9438031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2746d44e991e35c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f942bfff640 1 -- 172.21.15.29:0/744616355 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9438031c60 con 0x55e621c33d80 2024-09-21T02:48:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f942bfff640 1 -- 172.21.15.29:0/744616355 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9438036070 con 0x55e621c33d80 2024-09-21T02:48:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.531+0000 7f942bfff640 1 -- 172.21.15.29:0/744616355 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9438030bf0 con 0x55e621c33d80 2024-09-21T02:48:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 -- 172.21.15.29:0/744616355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c33d80 msgr2=0x55e62201dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c33d80 0x55e62201dce0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9438000f80 tx=0x7f9438031000 comp rx=0 tx=0).stop 2024-09-21T02:48:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 -- 172.21.15.29:0/744616355 shutdown_connections 2024-09-21T02:48:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c544d0 0x55e622025060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c4c2f0 0x55e622024b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 --2- 172.21.15.29:0/744616355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c33d80 0x55e62201dce0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 -- 172.21.15.29:0/744616355 >> 172.21.15.29:0/744616355 conn(0x55e621c69300 msgr2=0x55e622025790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 -- 172.21.15.29:0/744616355 shutdown_connections 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 -- 172.21.15.29:0/744616355 wait complete. 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.532+0000 7f944f688740 1 -- start start 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f944f688740 1 Processor -- start 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f944f688740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c33d80 0x55e622029860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f944f688740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c4c2f0 0x55e622029f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f944f688740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e62202aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f944f688740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94401ad930 con 0x55e621c4c2f0 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f944f688740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f94401ad260 con 0x55e621c544d0 2024-09-21T02:48:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f944f688740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f94401aa4a0 con 0x55e621c33d80 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f944f688740 10 client.0 _ll_register_callbacks cb 0x7f943000bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9445cbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c4c2f0 0x55e622029f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9445cbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c4c2f0 0x55e622029f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55912/0 (socket says 172.21.15.29:55912) 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9445cbc640 1 -- 172.21.15.29:0/3274546351 learned_addr learned my addr 172.21.15.29:0/3274546351 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9446cbe640 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c33d80 0x55e622029860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9446cbe640 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e62202aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9445cbc640 1 -- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c33d80 msgr2=0x55e622029860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9445cbc640 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c33d80 0x55e622029860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9445cbc640 1 -- 172.21.15.29:0/3274546351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 msgr2=0x55e62202aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9445cbc640 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e62202aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9445cbc640 1 -- 172.21.15.29:0/3274546351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e62202b180 con 0x55e621c4c2f0 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.533+0000 7f9446cbe640 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c33d80 0x55e622029860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:48:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f9445cbc640 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c4c2f0 0x55e622029f20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9430010810 tx=0x7f9430005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8589c0d5978152bc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f9429ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f9429ffb640 1 -- 172.21.15.29:0/3274546351 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943000b480 con 0x55e621c4c2f0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f9429ffb640 1 -- 172.21.15.29:0/3274546351 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9430004660 con 0x55e621c4c2f0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 1 -- 172.21.15.29:0/3274546351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9438036070 con 0x55e621c4c2f0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f9429ffb640 1 -- 172.21.15.29:0/3274546351 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9430005850 con 0x55e621c4c2f0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 10 client.16812 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 1 -- 172.21.15.29:0/3274546351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e62202e580 con 0x55e621c4c2f0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f9429ffb640 1 -- 172.21.15.29:0/3274546351 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9438036070 con 0x55e621c4c2f0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f9429ffb640 1 -- 172.21.15.29:0/3274546351 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f9430029520 con 0x55e621c4c2f0 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 10 client.16812 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 10 client.16812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 20 client.16812 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e621c751b0 0x55e6220a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 4 client.16812 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:48:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.534+0000 7f944f688740 1 -- 172.21.15.29:0/3274546351 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e621f8d0f0 con 0x55e621c751b0 2024-09-21T02:48:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.535+0000 7f9446cbe640 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e621c751b0 0x55e6220a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.535+0000 7f9429ffb640 10 client.16812 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.535+0000 7f9446cbe640 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e621c751b0 0x55e6220a50e0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfa8dee490e3131b server_cookie=2cccadf1b0305fc in_seq=0 out_seq=0 2024-09-21T02:48:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f9429ffb640 1 -- 172.21.15.29:0/3274546351 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e621f8d0f0 con 0x55e621c751b0 2024-09-21T02:48:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f9429ffb640 10 client.16812 handle_command_reply: tid=0 2024-09-21T02:48:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 client.16812 shutdown 2024-09-21T02:48:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 -- 172.21.15.29:0/3274546351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c4c2f0 msgr2=0x55e622029f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c4c2f0 0x55e622029f20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9430010810 tx=0x7f9430005270 comp rx=0 tx=0).stop 2024-09-21T02:48:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 -- 172.21.15.29:0/3274546351 shutdown_connections 2024-09-21T02:48:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e621c544d0 0x55e62202aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e621c751b0 0x55e6220a50e0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621c4c2f0 0x55e622029f20 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f9429ffb640 0 client.16812 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 --2- 172.21.15.29:0/3274546351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e621c33d80 0x55e622029860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 -- 172.21.15.29:0/3274546351 >> 172.21.15.29:0/3274546351 conn(0x55e621c69300 msgr2=0x55e62201dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 -- 172.21.15.29:0/3274546351 shutdown_connections 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 1 -- 172.21.15.29:0/3274546351 wait complete. 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.601+0000 7f944f688740 20 client.16812 trim_cache size 0 max 16384 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:48:31.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:48:31.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 -- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f942c0b6590 msgr2=0x7f942c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f942c0b6590 0x7f942c0b8aa0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f943c00c6a0 tx=0x7f943c07a5d0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 -- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94401025a0 msgr2=0x7f9440198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:31.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94401025a0 0x7f9440198d20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f943c00e9f0 tx=0x7f943c00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 -- 172.21.15.29:0/880207926 shutdown_connections 2024-09-21T02:48:31.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f942c0b6590 0x7f942c0b8aa0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9440103fc0 0x7f9440199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9440103620 0x7f9440199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 --2- 172.21.15.29:0/880207926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94401025a0 0x7f9440198d20 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:31.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 -- 172.21.15.29:0/880207926 >> 172.21.15.29:0/880207926 conn(0x7f94400fdee0 msgr2=0x7f9440106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:31.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 -- 172.21.15.29:0/880207926 shutdown_connections 2024-09-21T02:48:31.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:31.602+0000 7f9448721640 1 -- 172.21.15.29:0/880207926 wait complete. 2024-09-21T02:48:31.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:31 smithi190 ceph-mon[33871]: pgmap v671: 129 pgs: 129 active+clean; 40 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:48:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:31 smithi029 ceph-mon[31006]: pgmap v671: 129 pgs: 129 active+clean; 40 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:48:32.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 186819 2.28 MB/sec execute 369 sec latency 46.863 ms 2024-09-21T02:48:33.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 187665 2.30 MB/sec execute 370 sec latency 19.502 ms 2024-09-21T02:48:33.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:33 smithi066 ceph-mon[33436]: pgmap v672: 129 pgs: 129 active+clean; 42 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:48:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:33 smithi190 ceph-mon[33871]: pgmap v672: 129 pgs: 129 active+clean; 42 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:48:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:33 smithi029 ceph-mon[31006]: pgmap v672: 129 pgs: 129 active+clean; 42 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:48:34.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 188147 2.30 MB/sec execute 371 sec latency 15.655 ms 2024-09-21T02:48:34.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:34 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:48:34.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:34 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:48:34.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:34 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:48:35.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 188412 2.30 MB/sec execute 372 sec latency 14.290 ms 2024-09-21T02:48:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:35 smithi190 ceph-mon[33871]: pgmap v673: 129 pgs: 129 active+clean; 41 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-09-21T02:48:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:35 smithi029 ceph-mon[31006]: pgmap v673: 129 pgs: 129 active+clean; 41 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-09-21T02:48:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:35 smithi066 ceph-mon[33436]: pgmap v673: 129 pgs: 129 active+clean; 41 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-09-21T02:48:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:36.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 188839 2.29 MB/sec execute 373 sec latency 17.072 ms 2024-09-21T02:48:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:48:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:48:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:36.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:48:36.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:48:36.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:36.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:48:36.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:48:36.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:48:37.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 189513 2.30 MB/sec execute 374 sec latency 17.834 ms 2024-09-21T02:48:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:37 smithi066 ceph-mon[33436]: pgmap v674: 129 pgs: 129 active+clean; 38 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.7 MiB/s wr, 125 op/s 2024-09-21T02:48:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:37 smithi190 ceph-mon[33871]: pgmap v674: 129 pgs: 129 active+clean; 38 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.7 MiB/s wr, 125 op/s 2024-09-21T02:48:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:37 smithi029 ceph-mon[31006]: pgmap v674: 129 pgs: 129 active+clean; 38 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.7 MiB/s wr, 125 op/s 2024-09-21T02:48:38.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 190149 2.30 MB/sec execute 375 sec latency 23.059 ms 2024-09-21T02:48:39.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 190904 2.31 MB/sec execute 376 sec latency 15.540 ms 2024-09-21T02:48:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:39 smithi066 ceph-mon[33436]: pgmap v675: 129 pgs: 129 active+clean; 40 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:48:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:39 smithi190 ceph-mon[33871]: pgmap v675: 129 pgs: 129 active+clean; 40 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:48:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:39 smithi029 ceph-mon[31006]: pgmap v675: 129 pgs: 129 active+clean; 40 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-09-21T02:48:40.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 191399 2.31 MB/sec execute 377 sec latency 21.848 ms 2024-09-21T02:48:41.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 191906 2.31 MB/sec execute 378 sec latency 41.882 ms 2024-09-21T02:48:41.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:41 smithi029 ceph-mon[31006]: pgmap v676: 129 pgs: 129 active+clean; 38 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:48:41.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:41 smithi066 ceph-mon[33436]: pgmap v676: 129 pgs: 129 active+clean; 38 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:48:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:41 smithi190 ceph-mon[33871]: pgmap v676: 129 pgs: 129 active+clean; 38 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:48:42.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 192239 2.31 MB/sec execute 379 sec latency 19.077 ms 2024-09-21T02:48:43.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 192935 2.31 MB/sec execute 380 sec latency 16.152 ms 2024-09-21T02:48:43.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:43 smithi066 ceph-mon[33436]: pgmap v677: 129 pgs: 129 active+clean; 39 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-09-21T02:48:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:43 smithi190 ceph-mon[33871]: pgmap v677: 129 pgs: 129 active+clean; 39 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-09-21T02:48:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:43 smithi029 ceph-mon[31006]: pgmap v677: 129 pgs: 129 active+clean; 39 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-09-21T02:48:44.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 193550 2.31 MB/sec execute 381 sec latency 14.117 ms 2024-09-21T02:48:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 -- 172.21.15.29:0/946550618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81050e0 msgr2=0x7fccf810ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/946550618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81050e0 0x7fccf810ba80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009be0 tx=0x7fccf002a6c0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 -- 172.21.15.29:0/946550618 shutdown_connections 2024-09-21T02:48:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/946550618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81050e0 0x7fccf810ba80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/946550618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccf8104740 0x7fccf8104b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/946550618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccf81036c0 0x7fccf8103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 -- 172.21.15.29:0/946550618 >> 172.21.15.29:0/946550618 conn(0x7fccf80ff020 msgr2=0x7fccf81014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 -- 172.21.15.29:0/946550618 shutdown_connections 2024-09-21T02:48:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.710+0000 7fccfcaf9640 1 -- 172.21.15.29:0/946550618 wait complete. 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccfcaf9640 1 Processor -- start 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccfcaf9640 1 -- start start 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccfcaf9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81036c0 0x7fccf819c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccfcaf9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccf8104740 0x7fccf819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccfcaf9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccf81050e0 0x7fccf819d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccfcaf9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccf810f1a0 con 0x7fccf81036c0 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccfcaf9640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fccf810f020 con 0x7fccf8104740 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccfcaf9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fccf810f320 con 0x7fccf81050e0 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81036c0 0x7fccf819c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81036c0 0x7fccf819c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43694/0 (socket says 172.21.15.29:43694) 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf659c640 1 -- 172.21.15.29:0/4149351803 learned_addr learned my addr 172.21.15.29:0/4149351803 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fcceffff640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccf8104740 0x7fccf819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccf81050e0 0x7fccf819d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf659c640 1 -- 172.21.15.29:0/4149351803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccf81050e0 msgr2=0x7fccf819d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf659c640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccf81050e0 0x7fccf819d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf659c640 1 -- 172.21.15.29:0/4149351803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccf8104740 msgr2=0x7fccf819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf659c640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccf8104740 0x7fccf819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fccf659c640 1 -- 172.21.15.29:0/4149351803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf819da60 con 0x7fccf81036c0 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.711+0000 7fcceffff640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccf8104740 0x7fccf819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.712+0000 7fccf659c640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81036c0 0x7fccf819c020 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fcce800e9f0 tx=0x7fcce800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.712+0000 7fccef7fe640 1 -- 172.21.15.29:0/4149351803 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce800ccc0 con 0x7fccf81036c0 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.712+0000 7fccfcaf9640 1 -- 172.21.15.29:0/4149351803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf819dd50 con 0x7fccf81036c0 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.712+0000 7fccef7fe640 1 -- 172.21.15.29:0/4149351803 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcce80108c0 con 0x7fccf81036c0 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.712+0000 7fccef7fe640 1 -- 172.21.15.29:0/4149351803 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8018660 con 0x7fccf81036c0 2024-09-21T02:48:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.712+0000 7fccfcaf9640 1 -- 172.21.15.29:0/4149351803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf81a4440 con 0x7fccf81036c0 2024-09-21T02:48:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fcd03a66740 1 Processor -- start 2024-09-21T02:48:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fcd03a66740 1 -- start start 2024-09-21T02:48:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fccef7fe640 1 -- 172.21.15.29:0/4149351803 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fcce8010430 con 0x7fccf81036c0 2024-09-21T02:48:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b505e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b5064c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b507010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fcd03a66740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e9b4786f0 con 0x564e9b15e1b0 2024-09-21T02:48:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fcd03a66740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564e9b467b20 con 0x564e9b13d4d0 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.715+0000 7fcd03a66740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564e9b46b930 con 0x564e9b1352f0 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b505e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b5064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b505e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43710/0 (socket says 172.21.15.29:43710) 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf6d9d640 1 -- 172.21.15.29:0/314362384 learned_addr learned my addr 172.21.15.29:0/314362384 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fcceffff640 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b507010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf659c640 1 -- 172.21.15.29:0/314362384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 msgr2=0x564e9b507010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf659c640 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b507010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf659c640 1 -- 172.21.15.29:0/314362384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 msgr2=0x564e9b505e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf659c640 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b505e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf659c640 1 -- 172.21.15.29:0/314362384 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9b42d530 con 0x564e9b1352f0 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b505e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fcceffff640 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b507010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccf659c640 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b5064c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcce80e51b0 tx=0x7fcce80ebe80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aba1ce97697ffb89 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccef7fe640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcce40b6510 0x7fcce40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.716+0000 7fccecff9640 1 -- 172.21.15.29:0/314362384 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce80fb070 con 0x564e9b1352f0 2024-09-21T02:48:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.717+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcce40b6510 0x7fcce40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.717+0000 7fccecff9640 1 -- 172.21.15.29:0/314362384 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcce80cf630 con 0x564e9b1352f0 2024-09-21T02:48:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.717+0000 7fccecff9640 1 -- 172.21.15.29:0/314362384 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce80acfe0 con 0x564e9b1352f0 2024-09-21T02:48:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.717+0000 7fccef7fe640 1 -- 172.21.15.29:0/4149351803 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fcce80e7a60 con 0x7fccf81036c0 2024-09-21T02:48:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcce40b6510 0x7fcce40b8a20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fccf0031040 tx=0x7fccf0030860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 -- 172.21.15.29:0/314362384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 msgr2=0x564e9b5064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b5064c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcce80e51b0 tx=0x7fcce80ebe80 comp rx=0 tx=0).stop 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 -- 172.21.15.29:0/314362384 shutdown_connections 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b507010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b5064c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 --2- 172.21.15.29:0/314362384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b505e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 -- 172.21.15.29:0/314362384 >> 172.21.15.29:0/314362384 conn(0x564e9b152300 msgr2=0x564e9b5018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 -- 172.21.15.29:0/314362384 shutdown_connections 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 -- 172.21.15.29:0/314362384 wait complete. 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 -- start start 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.718+0000 7fcd03a66740 1 Processor -- start 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b1352f0 0x564e9b50f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b13d4d0 0x564e9b50fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b510800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fcd03a66740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e9b39fab0 con 0x564e9b15e1b0 2024-09-21T02:48:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fcd03a66740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564e9b40d730 con 0x564e9b1352f0 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fcd03a66740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564e9b507b30 con 0x564e9b13d4d0 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fcd03a66740 10 client.0 _ll_register_callbacks cb 0x564e9b4616b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b510800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b510800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43716/0 (socket says 172.21.15.29:43716) 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 -- 172.21.15.29:0/1439441436 learned_addr learned my addr 172.21.15.29:0/1439441436 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fcceffff640 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b13d4d0 0x564e9b50fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf659c640 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b1352f0 0x564e9b50f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 -- 172.21.15.29:0/1439441436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b13d4d0 msgr2=0x564e9b50fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b13d4d0 0x564e9b50fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 -- 172.21.15.29:0/1439441436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b1352f0 msgr2=0x564e9b50f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b1352f0 0x564e9b50f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 -- 172.21.15.29:0/1439441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9b1c9790 con 0x564e9b15e1b0 2024-09-21T02:48:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b510800 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fccf0006010 tx=0x7fccf002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341482518c9e18c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.719+0000 7fccddffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fccddffb640 1 -- 172.21.15.29:0/1439441436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0050020 con 0x564e9b15e1b0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fcd03a66740 1 -- 172.21.15.29:0/1439441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcce80cf630 con 0x564e9b15e1b0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fccddffb640 1 -- 172.21.15.29:0/1439441436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fccf0004450 con 0x564e9b15e1b0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fccddffb640 1 -- 172.21.15.29:0/1439441436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fccf005b050 con 0x564e9b15e1b0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fcd03a66740 10 client.16824 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fcd03a66740 1 -- 172.21.15.29:0/1439441436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564e9b242620 con 0x564e9b15e1b0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fccddffb640 1 -- 172.21.15.29:0/1439441436 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcce80cf630 con 0x564e9b15e1b0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fccddffb640 1 -- 172.21.15.29:0/1439441436 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fccf003c040 con 0x564e9b15e1b0 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fcd03a66740 10 client.16824 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fcd03a66740 10 client.16824 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fcd03a66740 20 client.16824 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fcd03a66740 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e9b11cd80 0x564e9b51cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.720+0000 7fcd03a66740 4 client.16824 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.721+0000 7fcd03a66740 1 -- 172.21.15.29:0/1439441436 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e9b47a640 con 0x564e9b11cd80 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.721+0000 7fccf659c640 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e9b11cd80 0x564e9b51cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.721+0000 7fccddffb640 10 client.16824 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.721+0000 7fccf659c640 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e9b11cd80 0x564e9b51cd40 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f936a9e902c9101 server_cookie=4fc83e2a87b31038 in_seq=0 out_seq=0 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fccddffb640 1 -- 172.21.15.29:0/1439441436 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564e9b47a640 con 0x564e9b11cd80 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fccddffb640 10 client.16824 handle_command_reply: tid=0 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 client.16824 shutdown 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 -- 172.21.15.29:0/1439441436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 msgr2=0x564e9b510800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b510800 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fccf0006010 tx=0x7fccf002a6c0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 -- 172.21.15.29:0/1439441436 shutdown_connections 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e9b11cd80 0x564e9b51cd40 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b15e1b0 0x564e9b510800 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b13d4d0 0x564e9b50fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 --2- 172.21.15.29:0/1439441436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b1352f0 0x564e9b50f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 -- 172.21.15.29:0/1439441436 >> 172.21.15.29:0/1439441436 conn(0x564e9b152300 msgr2=0x564e9b4ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fccddffb640 0 client.16824 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 -- 172.21.15.29:0/1439441436 shutdown_connections 2024-09-21T02:48:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 1 -- 172.21.15.29:0/1439441436 wait complete. 2024-09-21T02:48:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.722+0000 7fcd03a66740 20 client.16824 trim_cache size 0 max 16384 2024-09-21T02:48:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcd03a66740 1 Processor -- start 2024-09-21T02:48:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcd03a66740 1 -- start start 2024-09-21T02:48:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b50dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b13d4d0 0x564e9b501290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b15e1b0 0x564e9b50e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcd03a66740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81ae7b0 con 0x564e9b13d4d0 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcd03a66740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81b02f0 con 0x564e9b15e1b0 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcd03a66740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81aee80 con 0x564e9b1352f0 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b13d4d0 0x564e9b501290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b13d4d0 0x564e9b501290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43718/0 (socket says 172.21.15.29:43718) 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fcceffff640 1 -- 172.21.15.29:0/3689382833 learned_addr learned my addr 172.21.15.29:0/3689382833 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b15e1b0 0x564e9b50e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.739+0000 7fccf659c640 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b50dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fcceffff640 1 -- 172.21.15.29:0/3689382833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 msgr2=0x564e9b50dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fcceffff640 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b50dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fcceffff640 1 -- 172.21.15.29:0/3689382833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b15e1b0 msgr2=0x564e9b50e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fcceffff640 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b15e1b0 0x564e9b50e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fcceffff640 1 -- 172.21.15.29:0/3689382833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9b50d750 con 0x564e9b13d4d0 2024-09-21T02:48:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fcceffff640 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b13d4d0 0x564e9b501290 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fcce00126f0 tx=0x7fcce0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5025518143802cbe server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fccdffff640 1 -- 172.21.15.29:0/3689382833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce0022070 con 0x564e9b13d4d0 2024-09-21T02:48:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fccdffff640 1 -- 172.21.15.29:0/3689382833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcce0012e50 con 0x564e9b13d4d0 2024-09-21T02:48:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.740+0000 7fccdffff640 1 -- 172.21.15.29:0/3689382833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce000fea0 con 0x564e9b13d4d0 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 -- 172.21.15.29:0/3689382833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b13d4d0 msgr2=0x564e9b501290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b13d4d0 0x564e9b501290 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fcce00126f0 tx=0x7fcce0009260 comp rx=0 tx=0).stop 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 -- 172.21.15.29:0/3689382833 shutdown_connections 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b15e1b0 0x564e9b50e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b13d4d0 0x564e9b501290 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 --2- 172.21.15.29:0/3689382833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b1352f0 0x564e9b50dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 -- 172.21.15.29:0/3689382833 >> 172.21.15.29:0/3689382833 conn(0x564e9b152300 msgr2=0x564e9b51cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 -- 172.21.15.29:0/3689382833 shutdown_connections 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 -- 172.21.15.29:0/3689382833 wait complete. 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 -- start start 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.741+0000 7fcd03a66740 1 Processor -- start 2024-09-21T02:48:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b1352f0 0x564e9b512c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b5132e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcd03a66740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b15e1b0 0x564e9b516ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcd03a66740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81afc20 con 0x564e9b1352f0 2024-09-21T02:48:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcd03a66740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81af550 con 0x564e9b13d4d0 2024-09-21T02:48:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcd03a66740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81ac790 con 0x564e9b15e1b0 2024-09-21T02:48:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcd03a66740 10 client.0 _ll_register_callbacks cb 0x7fccf0043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b1352f0 0x564e9b512c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b1352f0 0x564e9b512c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43720/0 (socket says 172.21.15.29:43720) 2024-09-21T02:48:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 -- 172.21.15.29:0/397170429 learned_addr learned my addr 172.21.15.29:0/397170429 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b15e1b0 0x564e9b516ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcceffff640 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b5132e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 -- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b15e1b0 msgr2=0x564e9b516ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b15e1b0 0x564e9b516ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 -- 172.21.15.29:0/397170429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 msgr2=0x564e9b5132e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b5132e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 -- 172.21.15.29:0/397170429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9b517630 con 0x564e9b1352f0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf6d9d640 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b15e1b0 0x564e9b516ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccf659c640 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b1352f0 0x564e9b512c20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fcce800c6a0 tx=0x7fcce8010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9871a0103659e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccddffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccddffb640 1 -- 172.21.15.29:0/397170429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce809d390 con 0x564e9b1352f0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fccddffb640 1 -- 172.21.15.29:0/397170429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fcce80eb050 con 0x564e9b1352f0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.742+0000 7fcd03a66740 1 -- 172.21.15.29:0/397170429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcce0012e50 con 0x564e9b1352f0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fccddffb640 1 -- 172.21.15.29:0/397170429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcce809e010 con 0x564e9b1352f0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcd03a66740 10 client.16830 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcd03a66740 1 -- 172.21.15.29:0/397170429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564e9b514a70 con 0x564e9b1352f0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fccddffb640 1 -- 172.21.15.29:0/397170429 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcce0012e50 con 0x564e9b1352f0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fccddffb640 1 -- 172.21.15.29:0/397170429 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fcce809c040 con 0x564e9b1352f0 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcd03a66740 10 client.16830 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcd03a66740 10 client.16830 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcd03a66740 20 client.16830 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcd03a66740 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e9b0f21c0 0x564e9b5899f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcd03a66740 4 client.16830 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:48:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcd03a66740 1 -- 172.21.15.29:0/397170429 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e9b4760f0 con 0x564e9b0f21c0 2024-09-21T02:48:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.743+0000 7fcceffff640 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e9b0f21c0 0x564e9b5899f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.744+0000 7fccddffb640 10 client.16830 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.744+0000 7fcceffff640 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e9b0f21c0 0x564e9b5899f0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41331e4b01486c28 server_cookie=89ac45f2f39aaf0 in_seq=0 out_seq=0 2024-09-21T02:48:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fccddffb640 1 -- 172.21.15.29:0/397170429 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564e9b4760f0 con 0x564e9b0f21c0 2024-09-21T02:48:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fccddffb640 10 client.16830 handle_command_reply: tid=0 2024-09-21T02:48:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 client.16830 shutdown 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 -- 172.21.15.29:0/397170429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b1352f0 msgr2=0x564e9b512c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b1352f0 0x564e9b512c20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fcce800c6a0 tx=0x7fcce8010ce0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 -- 172.21.15.29:0/397170429 shutdown_connections 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e9b15e1b0 0x564e9b516ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e9b13d4d0 0x564e9b5132e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e9b0f21c0 0x564e9b5899f0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 --2- 172.21.15.29:0/397170429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e9b1352f0 0x564e9b512c20 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 -- 172.21.15.29:0/397170429 >> 172.21.15.29:0/397170429 conn(0x564e9b152300 msgr2=0x564e9b51cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fccddffb640 0 client.16830 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 -- 172.21.15.29:0/397170429 shutdown_connections 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.791+0000 7fcd03a66740 1 -- 172.21.15.29:0/397170429 wait complete. 2024-09-21T02:48:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fcd03a66740 20 client.16830 trim_cache size 0 max 16384 2024-09-21T02:48:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:48:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:48:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:48:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 -- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcce40b6510 msgr2=0x7fcce40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcce40b6510 0x7fcce40b8a20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fccf0031040 tx=0x7fccf0030860 comp rx=0 tx=0).stop 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 -- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81036c0 msgr2=0x7fccf819c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81036c0 0x7fccf819c020 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fcce800e9f0 tx=0x7fcce800edf0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 -- 172.21.15.29:0/4149351803 shutdown_connections 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcce40b6510 0x7fcce40b8a20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccf81050e0 0x7fccf819d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccf8104740 0x7fccf819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 --2- 172.21.15.29:0/4149351803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf81036c0 0x7fccf819c020 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 -- 172.21.15.29:0/4149351803 >> 172.21.15.29:0/4149351803 conn(0x7fccf80ff020 msgr2=0x7fccf80755f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 -- 172.21.15.29:0/4149351803 shutdown_connections 2024-09-21T02:48:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:44.792+0000 7fccfcaf9640 1 -- 172.21.15.29:0/4149351803 wait complete. 2024-09-21T02:48:45.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 194132 2.31 MB/sec execute 382 sec latency 23.300 ms 2024-09-21T02:48:45.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:45 smithi066 ceph-mon[33436]: pgmap v678: 129 pgs: 129 active+clean; 41 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-09-21T02:48:45.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:45 smithi190 ceph-mon[33871]: pgmap v678: 129 pgs: 129 active+clean; 41 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-09-21T02:48:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:45 smithi029 ceph-mon[31006]: pgmap v678: 129 pgs: 129 active+clean; 41 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-09-21T02:48:46.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 194871 2.32 MB/sec execute 383 sec latency 19.454 ms 2024-09-21T02:48:47.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 195392 2.33 MB/sec execute 384 sec latency 14.683 ms 2024-09-21T02:48:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:47 smithi066 ceph-mon[33436]: pgmap v679: 129 pgs: 129 active+clean; 40 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-09-21T02:48:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:47 smithi190 ceph-mon[33871]: pgmap v679: 129 pgs: 129 active+clean; 40 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-09-21T02:48:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:47 smithi029 ceph-mon[31006]: pgmap v679: 129 pgs: 129 active+clean; 40 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-09-21T02:48:48.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 195697 2.32 MB/sec execute 385 sec latency 18.381 ms 2024-09-21T02:48:49.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 196307 2.32 MB/sec execute 386 sec latency 14.755 ms 2024-09-21T02:48:49.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:49 smithi066 ceph-mon[33436]: pgmap v680: 129 pgs: 129 active+clean; 41 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:48:49.803 INFO:tasks.quiescer.fs.[cephfs]:Total duration (49.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:48:49.804 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-09-21T02:48:49.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:48:49.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:49 smithi029 ceph-mon[31006]: pgmap v680: 129 pgs: 129 active+clean; 41 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:48:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:49 smithi190 ceph-mon[33871]: pgmap v680: 129 pgs: 129 active+clean; 41 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-09-21T02:48:50.087 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:48:50.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 196957 2.32 MB/sec execute 387 sec latency 17.480 ms 2024-09-21T02:48:50.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 -- 172.21.15.29:0/798658654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 msgr2=0x7f8964103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 --2- 172.21.15.29:0/798658654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 0x7f8964103fd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8958009900 tx=0x7f895802a410 comp rx=0 tx=0).stop 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 -- 172.21.15.29:0/798658654 shutdown_connections 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 --2- 172.21.15.29:0/798658654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8964109e00 0x7f8964110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 --2- 172.21.15.29:0/798658654 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89641094b0 0x7f89641098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 --2- 172.21.15.29:0/798658654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 0x7f8964103fd0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 -- 172.21.15.29:0/798658654 >> 172.21.15.29:0/798658654 conn(0x7f896407b910 msgr2=0x7f896407bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 -- 172.21.15.29:0/798658654 shutdown_connections 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.632+0000 7f896b09e640 1 -- 172.21.15.29:0/798658654 wait complete. 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.633+0000 7f896b09e640 1 Processor -- start 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.633+0000 7f896b09e640 1 -- start start 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.633+0000 7f896b09e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 0x7f896419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:50.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.633+0000 7f896b09e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89641094b0 0x7f896419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.633+0000 7f896b09e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8964109e00 0x7f896419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f896b09e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8964113970 con 0x7f8964103bc0 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f896b09e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f89641137f0 con 0x7f8964109e00 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f896b09e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8964113af0 con 0x7f89641094b0 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 0x7f896419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 0x7f896419a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43742/0 (socket says 172.21.15.29:43742) 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 -- 172.21.15.29:0/2723353275 learned_addr learned my addr 172.21.15.29:0/2723353275 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8963fff640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89641094b0 0x7f896419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f896963b640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8964109e00 0x7f896419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 -- 172.21.15.29:0/2723353275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89641094b0 msgr2=0x7f896419a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89641094b0 0x7f896419a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 -- 172.21.15.29:0/2723353275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8964109e00 msgr2=0x7f896419fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:50.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8964109e00 0x7f896419fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 -- 172.21.15.29:0/2723353275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89641a0350 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8963fff640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89641094b0 0x7f896419a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8968e3a640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 0x7f896419a3e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8958009230 tx=0x7f89580319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.634+0000 7f8961ffb640 1 -- 172.21.15.29:0/2723353275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895803f030 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.635+0000 7f8961ffb640 1 -- 172.21.15.29:0/2723353275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89580043d0 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.635+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89641a05e0 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.635+0000 7f8961ffb640 1 -- 172.21.15.29:0/2723353275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958035070 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.635+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89641a6da0 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.636+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8964103fd0 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.638+0000 7f8961ffb640 1 -- 172.21.15.29:0/2723353275 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f8958042030 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.639+0000 7f8961ffb640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89540b6590 0x7f89540b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.639+0000 7f8961ffb640 1 -- 172.21.15.29:0/2723353275 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8958107d40 con 0x7f8964103bc0 2024-09-21T02:48:50.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.639+0000 7f8963fff640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89540b6590 0x7f89540b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:50.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.640+0000 7f8963fff640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89540b6590 0x7f89540b8aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8964105bf0 tx=0x7f8950005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:50.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.643+0000 7f8961ffb640 1 -- 172.21.15.29:0/2723353275 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f89580d2630 con 0x7f8964103bc0 2024-09-21T02:48:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.898+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f896419b510 con 0x7f8964103bc0 2024-09-21T02:48:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.898+0000 7f8961ffb640 1 -- 172.21.15.29:0/2723353275 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f89580d7570 con 0x7f8964103bc0 2024-09-21T02:48:50.900 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:48:50.900 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:48:50.900 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:48:50.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89540b6590 msgr2=0x7f89540b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:50.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89540b6590 0x7f89540b8aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8964105bf0 tx=0x7f8950005f90 comp rx=0 tx=0).stop 2024-09-21T02:48:50.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 msgr2=0x7f896419a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 0x7f896419a3e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8958009230 tx=0x7f89580319a0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 shutdown_connections 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f89540b6590 0x7f89540b8aa0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8964109e00 0x7f896419fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89641094b0 0x7f896419a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 --2- 172.21.15.29:0/2723353275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8964103bc0 0x7f896419a3e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 >> 172.21.15.29:0/2723353275 conn(0x7f896407b910 msgr2=0x7f89641075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 shutdown_connections 2024-09-21T02:48:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:50.901+0000 7f896b09e640 1 -- 172.21.15.29:0/2723353275 wait complete. 2024-09-21T02:48:51.094 INFO:tasks.workunit.client.0.smithi029.stdout: 1 197550 2.32 MB/sec execute 388 sec latency 13.861 ms 2024-09-21T02:48:51.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:51 smithi066 ceph-mon[33436]: pgmap v681: 129 pgs: 129 active+clean; 42 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-09-21T02:48:51.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:51 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2723353275' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:48:51.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:51 smithi029 ceph-mon[31006]: pgmap v681: 129 pgs: 129 active+clean; 42 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-09-21T02:48:51.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:51 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2723353275' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:48:51.765 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0' 2024-09-21T02:48:51.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0 2024-09-21T02:48:51.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:51 smithi190 ceph-mon[33871]: pgmap v681: 129 pgs: 129 active+clean; 42 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-09-21T02:48:51.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:51 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2723353275' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:48:52.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198384 2.33 MB/sec execute 389 sec latency 16.790 ms 2024-09-21T02:48:52.100 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:48:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 -- 172.21.15.29:0/80423657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec1094d0 msgr2=0x7fa4ec1098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/80423657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec1094d0 0x7fa4ec1098e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009900 tx=0x7fa4d402a410 comp rx=0 tx=0).stop 2024-09-21T02:48:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 -- 172.21.15.29:0/80423657 shutdown_connections 2024-09-21T02:48:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/80423657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4ec109e20 0x7fa4ec110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/80423657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec1094d0 0x7fa4ec1098e0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/80423657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec103b90 0x7fa4ec103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 -- 172.21.15.29:0/80423657 >> 172.21.15.29:0/80423657 conn(0x7fa4ec0ff620 msgr2=0x7fa4ec101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 -- 172.21.15.29:0/80423657 shutdown_connections 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.662+0000 7fa4f22b2640 1 -- 172.21.15.29:0/80423657 wait complete. 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f22b2640 1 Processor -- start 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f22b2640 1 -- start start 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f22b2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4ec103b90 0x7fa4ec19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f22b2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec1094d0 0x7fa4ec19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f22b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec109e20 0x7fa4ec19fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f22b2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec113990 con 0x7fa4ec109e20 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f22b2640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec113810 con 0x7fa4ec1094d0 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f22b2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec113b10 con 0x7fa4ec103b90 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4ebfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4ec103b90 0x7fa4ec19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4ebfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4ec103b90 0x7fa4ec19a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:35858/0 (socket says 172.21.15.29:35858) 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4ebfff640 1 -- 172.21.15.29:0/3856585941 learned_addr learned my addr 172.21.15.29:0/3856585941 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:52.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4e3fff640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec1094d0 0x7fa4ec19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.663+0000 7fa4f084f640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec109e20 0x7fa4ec19fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4ebfff640 1 -- 172.21.15.29:0/3856585941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec1094d0 msgr2=0x7fa4ec19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec1094d0 0x7fa4ec19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4ebfff640 1 -- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec109e20 msgr2=0x7fa4ec19fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec109e20 0x7fa4ec19fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4ebfff640 1 -- 172.21.15.29:0/3856585941 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ec1a0310 con 0x7fa4ec103b90 2024-09-21T02:48:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4ec103b90 0x7fa4ec19a3a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00e9f0 tx=0x7fa4dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:52.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4e9ffb640 1 -- 172.21.15.29:0/3856585941 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc00cd80 con 0x7fa4ec103b90 2024-09-21T02:48:52.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4f22b2640 1 -- 172.21.15.29:0/3856585941 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ec1a0600 con 0x7fa4ec103b90 2024-09-21T02:48:52.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4e9ffb640 1 -- 172.21.15.29:0/3856585941 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4dc0108c0 con 0x7fa4ec103b90 2024-09-21T02:48:52.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4e9ffb640 1 -- 172.21.15.29:0/3856585941 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0187c0 con 0x7fa4ec103b90 2024-09-21T02:48:52.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.664+0000 7fa4f22b2640 1 -- 172.21.15.29:0/3856585941 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ec1a6d60 con 0x7fa4ec103b90 2024-09-21T02:48:52.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.667+0000 7fa4f8826740 1 Processor -- start 2024-09-21T02:48:52.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.667+0000 7fa4f8826740 1 -- start start 2024-09-21T02:48:52.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.667+0000 7fa4e9ffb640 1 -- 172.21.15.29:0/3856585941 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fa4dc010430 con 0x7fa4ec103b90 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e842600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e842b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 0x56229e843080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4f8826740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56229e7b8b90 con 0x56229e472980 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4f8826740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56229e7a7fc0 con 0x56229e47ab60 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4f8826740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56229e7abdd0 con 0x56229e49b840 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e842b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4f084f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e842600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e842b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43764/0 (socket says 172.21.15.29:43764) 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4f084f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e842600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:35872/0 (socket says 172.21.15.29:35872) 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4ebfff640 1 -- 172.21.15.29:0/2216710446 learned_addr learned my addr 172.21.15.29:0/2216710446 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4e3fff640 1 --2- 172.21.15.29:0/2216710446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 0x56229e843080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4ebfff640 1 -- 172.21.15.29:0/2216710446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 msgr2=0x56229e842600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2216710446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e842600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4ebfff640 1 -- 172.21.15.29:0/2216710446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 msgr2=0x56229e843080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:52.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2216710446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 0x56229e843080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.668+0000 7fa4ebfff640 1 -- 172.21.15.29:0/2216710446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56229e72f060 con 0x56229e472980 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.669+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2216710446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e842b40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0edbc0 tx=0x7fa4dc0ef610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6341ea4f6bf92b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.669+0000 7fa4e9ffb640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4d80b6510 0x7fa4d80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.669+0000 7fa4e27fc640 1 -- 172.21.15.29:0/2216710446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0f7020 con 0x56229e472980 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.669+0000 7fa4f084f640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4d80b6510 0x7fa4d80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.669+0000 7fa4e27fc640 1 -- 172.21.15.29:0/2216710446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4dc0efe40 con 0x56229e472980 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.669+0000 7fa4e27fc640 1 -- 172.21.15.29:0/2216710446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0a9730 con 0x56229e472980 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.669+0000 7fa4e9ffb640 1 -- 172.21.15.29:0/3856585941 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa4dc0e7a30 con 0x7fa4ec103b90 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 -- 172.21.15.29:0/2216710446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 msgr2=0x56229e842b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f084f640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4d80b6510 0x7fa4d80b8a20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4005fd0 tx=0x7fa4e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2216710446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e842b40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0edbc0 tx=0x7fa4dc0ef610 comp rx=0 tx=0).stop 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 -- 172.21.15.29:0/2216710446 shutdown_connections 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2216710446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 0x56229e843080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2216710446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e842b40 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2216710446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e842600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 -- 172.21.15.29:0/2216710446 >> 172.21.15.29:0/2216710446 conn(0x56229e48f990 msgr2=0x56229e8404f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 -- 172.21.15.29:0/2216710446 shutdown_connections 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.670+0000 7fa4f8826740 1 -- 172.21.15.29:0/2216710446 wait complete. 2024-09-21T02:48:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.671+0000 7fa4f8826740 1 -- start start 2024-09-21T02:48:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.671+0000 7fa4f8826740 1 Processor -- start 2024-09-21T02:48:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.671+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e84e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.671+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 0x56229e84e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.671+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e85a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f8826740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56229e6097a0 con 0x56229e472980 2024-09-21T02:48:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f8826740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56229e74d0a0 con 0x56229e47ab60 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f8826740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56229e850de0 con 0x56229e49b840 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f8826740 10 client.0 _ll_register_callbacks cb 0x56229e72a3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e85a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e85a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:35886/0 (socket says 172.21.15.29:35886) 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 -- 172.21.15.29:0/2543589793 learned_addr learned my addr 172.21.15.29:0/2543589793 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e84e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4e3fff640 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 0x56229e84e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 -- 172.21.15.29:0/2543589793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 msgr2=0x56229e84e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 0x56229e84e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 -- 172.21.15.29:0/2543589793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 msgr2=0x56229e84e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e84e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 -- 172.21.15.29:0/2543589793 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56229e73b880 con 0x56229e49b840 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4f084f640 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e85a0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4003fd0 tx=0x7fa4e4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db300f5bf3c3f5ea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.672+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2543589793 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400f8b0 con 0x56229e49b840 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2543589793 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa4e400fa50 con 0x56229e49b840 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 1 -- 172.21.15.29:0/2543589793 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56229e7baae0 con 0x56229e49b840 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2543589793 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa4e400fc50 con 0x56229e49b840 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 10 client.26413 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 1 -- 172.21.15.29:0/2543589793 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56229e515b90 con 0x56229e49b840 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2543589793 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56229e7baae0 con 0x56229e49b840 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2543589793 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fa4e4026070 con 0x56229e49b840 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 10 client.26413 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 10 client.26413 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 20 client.26413 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56229e45a1c0 0x56229e860190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 4 client.26413 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.673+0000 7fa4f8826740 1 -- 172.21.15.29:0/2543589793 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56229e7b6590 con 0x56229e45a1c0 2024-09-21T02:48:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.674+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56229e45a1c0 0x56229e860190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.674+0000 7fa4cb7fe640 10 client.26413 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:52.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.674+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56229e45a1c0 0x56229e860190 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=840bd802baf80b36 server_cookie=3de58b1cda8ad02c in_seq=0 out_seq=0 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2543589793 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56229e7b6590 con 0x56229e45a1c0 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4cb7fe640 10 client.26413 handle_command_reply: tid=0 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 client.26413 shutdown 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 -- 172.21.15.29:0/2543589793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 msgr2=0x56229e85a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e85a0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4003fd0 tx=0x7fa4e4004ab0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 -- 172.21.15.29:0/2543589793 shutdown_connections 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e49b840 0x56229e85a0d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e47ab60 0x56229e84e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56229e45a1c0 0x56229e860190 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2543589793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e84e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4cb7fe640 0 client.26413 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:52.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 -- 172.21.15.29:0/2543589793 >> 172.21.15.29:0/2543589793 conn(0x56229e48f990 msgr2=0x56229e845790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:52.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 -- 172.21.15.29:0/2543589793 shutdown_connections 2024-09-21T02:48:52.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 1 -- 172.21.15.29:0/2543589793 wait complete. 2024-09-21T02:48:52.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.675+0000 7fa4f8826740 20 client.26413 trim_cache size 0 max 16384 2024-09-21T02:48:52.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f8826740 1 Processor -- start 2024-09-21T02:48:52.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f8826740 1 -- start start 2024-09-21T02:48:52.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e45a1c0 0x56229e843d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e472980 0x56229e83faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e47ab60 0x56229e83ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f8826740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1b0a40 con 0x56229e47ab60 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f8826740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1b2580 con 0x56229e472980 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f8826740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1b1110 con 0x56229e45a1c0 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f084f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e47ab60 0x56229e83ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4ebfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e45a1c0 0x56229e843d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f084f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e47ab60 0x56229e83ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43790/0 (socket says 172.21.15.29:43790) 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.695+0000 7fa4f084f640 1 -- 172.21.15.29:0/1107478051 learned_addr learned my addr 172.21.15.29:0/1107478051 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4e3fff640 1 --2- 172.21.15.29:0/1107478051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e472980 0x56229e83faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4ebfff640 1 -- 172.21.15.29:0/1107478051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e472980 msgr2=0x56229e83faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/1107478051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e472980 0x56229e83faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4ebfff640 1 -- 172.21.15.29:0/1107478051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e47ab60 msgr2=0x56229e83ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:52.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/1107478051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e47ab60 0x56229e83ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4ebfff640 1 -- 172.21.15.29:0/1107478051 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56229e84e870 con 0x56229e45a1c0 2024-09-21T02:48:52.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/1107478051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e45a1c0 0x56229e843d40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009380 tx=0x7fa4dc0a7f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=895ac893a8e3cd92 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:52.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4e1ffb640 1 -- 172.21.15.29:0/1107478051 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0f7030 con 0x56229e45a1c0 2024-09-21T02:48:52.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4e1ffb640 1 -- 172.21.15.29:0/1107478051 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4dc0ad3a0 con 0x56229e45a1c0 2024-09-21T02:48:52.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.696+0000 7fa4e1ffb640 1 -- 172.21.15.29:0/1107478051 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc010720 con 0x56229e45a1c0 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 -- 172.21.15.29:0/1107478051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e45a1c0 msgr2=0x56229e843d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 --2- 172.21.15.29:0/1107478051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e45a1c0 0x56229e843d40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009380 tx=0x7fa4dc0a7f90 comp rx=0 tx=0).stop 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 -- 172.21.15.29:0/1107478051 shutdown_connections 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 --2- 172.21.15.29:0/1107478051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e47ab60 0x56229e83ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 --2- 172.21.15.29:0/1107478051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e472980 0x56229e83faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 --2- 172.21.15.29:0/1107478051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e45a1c0 0x56229e843d40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 -- 172.21.15.29:0/1107478051 >> 172.21.15.29:0/1107478051 conn(0x56229e48f990 msgr2=0x56229e843630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 -- 172.21.15.29:0/1107478051 shutdown_connections 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.697+0000 7fa4f8826740 1 -- 172.21.15.29:0/1107478051 wait complete. 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.698+0000 7fa4f8826740 1 -- start start 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.698+0000 7fa4f8826740 1 Processor -- start 2024-09-21T02:48:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.698+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e45a1c0 0x56229e852f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.698+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e8534b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.698+0000 7fa4f8826740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e47ab60 0x56229e8539f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f8826740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1b1eb0 con 0x56229e472980 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f8826740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1b17e0 con 0x56229e45a1c0 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f8826740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1aea20 con 0x56229e47ab60 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e47ab60 0x56229e8539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e8534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e47ab60 0x56229e8539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:35896/0 (socket says 172.21.15.29:35896) 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 -- 172.21.15.29:0/2658850826 learned_addr learned my addr 172.21.15.29:0/2658850826 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f8826740 10 client.0 _ll_register_callbacks cb 0x7fa4e4009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e45a1c0 0x56229e852f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 -- 172.21.15.29:0/2658850826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e45a1c0 msgr2=0x56229e852f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e45a1c0 0x56229e852f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 -- 172.21.15.29:0/2658850826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 msgr2=0x56229e8534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e8534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 -- 172.21.15.29:0/2658850826 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56229e8cab60 con 0x56229e47ab60 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4e3fff640 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e8534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4f084f640 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e47ab60 0x56229e8539f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009690 tx=0x7fa4e4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=745e8de185f1274b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.699+0000 7fa4cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2658850826 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4028800 con 0x56229e47ab60 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 1 -- 172.21.15.29:0/2658850826 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4dc0ad3a0 con 0x56229e47ab60 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2658850826 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa4e40289a0 con 0x56229e47ab60 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2658850826 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa4e4028ba0 con 0x56229e47ab60 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 10 client.26419 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 1 -- 172.21.15.29:0/2658850826 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56229e8cbd20 con 0x56229e47ab60 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2658850826 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4dc0ad3a0 con 0x56229e47ab60 2024-09-21T02:48:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2658850826 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fa4e4026070 con 0x56229e47ab60 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 10 client.26419 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 10 client.26419 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 20 client.26419 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56229e49b840 0x56229e8ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 4 client.26419 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}] 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.700+0000 7fa4f8826740 1 -- 172.21.15.29:0/2658850826 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}) -- 0x56229e7b39c0 con 0x56229e49b840 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.701+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56229e49b840 0x56229e8ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.701+0000 7fa4cb7fe640 10 client.26419 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.701+0000 7fa4ebfff640 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56229e49b840 0x56229e8ce5d0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f35a82d8a92267d6 server_cookie=febb13b8dd2f35e7 in_seq=0 out_seq=0 2024-09-21T02:48:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4cb7fe640 1 -- 172.21.15.29:0/2658850826 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56229e7b39c0 con 0x56229e49b840 2024-09-21T02:48:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4cb7fe640 10 client.26419 handle_command_reply: tid=0 2024-09-21T02:48:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 client.26419 shutdown 2024-09-21T02:48:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 -- 172.21.15.29:0/2658850826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e47ab60 msgr2=0x56229e8539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e47ab60 0x56229e8539f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009690 tx=0x7fa4e4004a40 comp rx=0 tx=0).stop 2024-09-21T02:48:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 -- 172.21.15.29:0/2658850826 shutdown_connections 2024-09-21T02:48:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56229e47ab60 0x56229e8539f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56229e49b840 0x56229e8ce5d0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4cb7fe640 0 client.26419 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56229e472980 0x56229e8534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 --2- 172.21.15.29:0/2658850826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56229e45a1c0 0x56229e852f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.831+0000 7fa4f8826740 1 -- 172.21.15.29:0/2658850826 >> 172.21.15.29:0/2658850826 conn(0x56229e48f990 msgr2=0x56229e845700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.832+0000 7fa4f8826740 1 -- 172.21.15.29:0/2658850826 shutdown_connections 2024-09-21T02:48:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.832+0000 7fa4f8826740 1 -- 172.21.15.29:0/2658850826 wait complete. 2024-09-21T02:48:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.832+0000 7fa4f8826740 20 client.26419 trim_cache size 0 max 16384 2024-09-21T02:48:52.834 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 38, 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "fc8b81d5": { 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 38, 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 144.5, 2024-09-21T02:48:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:48:52.836 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.832+0000 7fa4f22b2640 1 -- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4d80b6510 msgr2=0x7fa4d80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.832+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4d80b6510 0x7fa4d80b8a20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4005fd0 tx=0x7fa4e4009500 comp rx=0 tx=0).stop 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.832+0000 7fa4f22b2640 1 -- 172.21.15.29:0/3856585941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4ec103b90 msgr2=0x7fa4ec19a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.832+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4ec103b90 0x7fa4ec19a3a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00e9f0 tx=0x7fa4dc00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.833+0000 7fa4f22b2640 1 -- 172.21.15.29:0/3856585941 shutdown_connections 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.833+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4d80b6510 0x7fa4d80b8a20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.833+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec109e20 0x7fa4ec19fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.833+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec1094d0 0x7fa4ec19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.833+0000 7fa4f22b2640 1 --2- 172.21.15.29:0/3856585941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4ec103b90 0x7fa4ec19a3a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.833+0000 7fa4f22b2640 1 -- 172.21.15.29:0/3856585941 >> 172.21.15.29:0/3856585941 conn(0x7fa4ec0ff620 msgr2=0x7fa4ec101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.833+0000 7fa4f22b2640 1 -- 172.21.15.29:0/3856585941 shutdown_connections 2024-09-21T02:48:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:48:52.833+0000 7fa4f22b2640 1 -- 172.21.15.29:0/3856585941 wait complete. 2024-09-21T02:48:53.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.34 MB/sec execute 390 sec latency 254.486 ms 2024-09-21T02:48:53.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:53 smithi029 ceph-mon[31006]: pgmap v682: 129 pgs: 129 active+clean; 41 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-09-21T02:48:53.687 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'fc8b81d5', quiesce took 3.9 seconds. Will release after: 20.6 2024-09-21T02:48:53.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:53 smithi066 ceph-mon[33436]: pgmap v682: 129 pgs: 129 active+clean; 41 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-09-21T02:48:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:53 smithi190 ceph-mon[33871]: pgmap v682: 129 pgs: 129 active+clean; 41 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-09-21T02:48:54.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.33 MB/sec execute 391 sec latency 1254.585 ms 2024-09-21T02:48:55.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.32 MB/sec execute 392 sec latency 2254.693 ms 2024-09-21T02:48:55.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:55 smithi066 ceph-mon[33436]: pgmap v683: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-09-21T02:48:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:55 smithi190 ceph-mon[33871]: pgmap v683: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-09-21T02:48:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:55 smithi029 ceph-mon[31006]: pgmap v683: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-09-21T02:48:56.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.32 MB/sec execute 393 sec latency 3254.796 ms 2024-09-21T02:48:57.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.31 MB/sec execute 394 sec latency 4254.864 ms 2024-09-21T02:48:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:57 smithi066 ceph-mon[33436]: pgmap v684: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:48:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:57 smithi190 ceph-mon[33871]: pgmap v684: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:48:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:57 smithi029 ceph-mon[31006]: pgmap v684: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:48:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4205012123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103fe0 msgr2=0x7ff79c10a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4205012123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103fe0 0x7ff79c10a980 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff794009ca0 tx=0x7ff79402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4205012123 shutdown_connections 2024-09-21T02:48:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4205012123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103fe0 0x7ff79c10a980 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4205012123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff79c103640 0x7ff79c103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4205012123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff79c1025c0 0x7ff79c102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4205012123 >> 172.21.15.29:0/4205012123 conn(0x7ff79c0fdf00 msgr2=0x7ff79c1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4205012123 shutdown_connections 2024-09-21T02:48:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.903+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4205012123 wait complete. 2024-09-21T02:48:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a2efa640 1 Processor -- start 2024-09-21T02:48:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a2efa640 1 -- start start 2024-09-21T02:48:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a2efa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff79c1025c0 0x7ff79c06c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a2efa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103640 0x7ff79c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a2efa640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff79c103fe0 0x7ff79c1a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a2efa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c10e060 con 0x7ff79c103640 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a2efa640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c10dee0 con 0x7ff79c103fe0 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a2efa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c10e1e0 con 0x7ff79c1025c0 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff79bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103640 0x7ff79c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a0c96640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff79c1025c0 0x7ff79c06c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff79bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103640 0x7ff79c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39616/0 (socket says 172.21.15.29:39616) 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a0c96640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff79c1025c0 0x7ff79c06c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47760/0 (socket says 172.21.15.29:47760) 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff79bfff640 1 -- 172.21.15.29:0/4174172496 learned_addr learned my addr 172.21.15.29:0/4174172496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.904+0000 7ff7a1497640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff79c103fe0 0x7ff79c1a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff7a0c96640 1 -- 172.21.15.29:0/4174172496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff79c103fe0 msgr2=0x7ff79c1a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff7a0c96640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff79c103fe0 0x7ff79c1a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff7a0c96640 1 -- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103640 msgr2=0x7ff79c19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff7a0c96640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103640 0x7ff79c19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff7a0c96640 1 -- 172.21.15.29:0/4174172496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79c1a18d0 con 0x7ff79c1025c0 2024-09-21T02:48:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff79bfff640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103640 0x7ff79c19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:48:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff7a0c96640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff79c1025c0 0x7ff79c06c210 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00d6d0 tx=0x7ff78c00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff799ffb640 1 -- 172.21.15.29:0/4174172496 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c017070 con 0x7ff79c1025c0 2024-09-21T02:48:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4174172496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79c1a1bc0 con 0x7ff79c1025c0 2024-09-21T02:48:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff799ffb640 1 -- 172.21.15.29:0/4174172496 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff78c009820 con 0x7ff79c1025c0 2024-09-21T02:48:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff799ffb640 1 -- 172.21.15.29:0/4174172496 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c004bc0 con 0x7ff79c1025c0 2024-09-21T02:48:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.905+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4174172496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79c1a2160 con 0x7ff79c1025c0 2024-09-21T02:48:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.908+0000 7ff7a9e64740 1 Processor -- start 2024-09-21T02:48:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a9e64740 1 -- start start 2024-09-21T02:48:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb5b1b0 0x55718ff02e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 0x55718ff034c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff04010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55718fe756f0 con 0x55718fb5b1b0 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55718fe64b20 con 0x55718fb322f0 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55718fe68930 con 0x55718fb3a4d0 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a1497640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb5b1b0 0x55718ff02e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a1497640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb5b1b0 0x55718ff02e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39620/0 (socket says 172.21.15.29:39620) 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a1497640 1 -- 172.21.15.29:0/3440249199 learned_addr learned my addr 172.21.15.29:0/3440249199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff799ffb640 1 -- 172.21.15.29:0/4174172496 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff78c005220 con 0x7ff79c1025c0 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff79bfff640 1 --2- 172.21.15.29:0/3440249199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff04010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a1497640 1 -- 172.21.15.29:0/3440249199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 msgr2=0x55718ff04010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a1497640 1 --2- 172.21.15.29:0/3440249199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff04010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a1497640 1 -- 172.21.15.29:0/3440249199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 msgr2=0x55718ff034c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a1497640 1 --2- 172.21.15.29:0/3440249199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 0x55718ff034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.909+0000 7ff7a1497640 1 -- 172.21.15.29:0/3440249199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55718fe2a530 con 0x55718fb5b1b0 2024-09-21T02:48:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.910+0000 7ff7a1497640 1 --2- 172.21.15.29:0/3440249199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb5b1b0 0x55718ff02e50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff794004af0 tx=0x7ff794009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cddc1e642531588e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.910+0000 7ff792ffd640 1 -- 172.21.15.29:0/3440249199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794031a80 con 0x55718fb5b1b0 2024-09-21T02:48:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.910+0000 7ff792ffd640 1 -- 172.21.15.29:0/3440249199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff794036070 con 0x55718fb5b1b0 2024-09-21T02:48:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.910+0000 7ff792ffd640 1 -- 172.21.15.29:0/3440249199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794004420 con 0x55718fb5b1b0 2024-09-21T02:48:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.910+0000 7ff799ffb640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7880b6510 0x7ff7880b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.910+0000 7ff7a0c96640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7880b6510 0x7ff7880b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.910+0000 7ff799ffb640 1 -- 172.21.15.29:0/4174172496 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff78c0e7160 con 0x7ff79c1025c0 2024-09-21T02:48:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 -- 172.21.15.29:0/3440249199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb5b1b0 msgr2=0x55718ff02e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/3440249199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb5b1b0 0x55718ff02e50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff794004af0 tx=0x7ff794009880 comp rx=0 tx=0).stop 2024-09-21T02:48:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 -- 172.21.15.29:0/3440249199 shutdown_connections 2024-09-21T02:48:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/3440249199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff04010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/3440249199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 0x55718ff034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/3440249199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb5b1b0 0x55718ff02e50 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 -- 172.21.15.29:0/3440249199 >> 172.21.15.29:0/3440249199 conn(0x55718fb4f300 msgr2=0x55718fefe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a0c96640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7880b6510 0x7ff7880b8a20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff78c000c00 tx=0x7ff78c079cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 -- 172.21.15.29:0/3440249199 shutdown_connections 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 -- 172.21.15.29:0/3440249199 wait complete. 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 -- start start 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.911+0000 7ff7a9e64740 1 Processor -- start 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb3a4d0 0x55718ff0cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb5b1b0 0x55718ff0d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55718fd9cab0 con 0x55718fb3a4d0 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55718fe0a730 con 0x55718fb322f0 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55718ff04950 con 0x55718fb5b1b0 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a9e64740 10 client.0 _ll_register_callbacks cb 0x55718fe5e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb3a4d0 0x55718ff0cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb3a4d0 0x55718ff0cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39626/0 (socket says 172.21.15.29:39626) 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 -- 172.21.15.29:0/2951364339 learned_addr learned my addr 172.21.15.29:0/2951364339 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb5b1b0 0x55718ff0d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 -- 172.21.15.29:0/2951364339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb5b1b0 msgr2=0x55718ff0d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb5b1b0 0x55718ff0d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 -- 172.21.15.29:0/2951364339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 msgr2=0x55718ff0c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 -- 172.21.15.29:0/2951364339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55718fbc6790 con 0x55718fb3a4d0 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff79bfff640 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb3a4d0 0x55718ff0cb00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff784012d20 tx=0x7ff78400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2b2611ecf6f70c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.912+0000 7ff77bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff77bfff640 1 -- 172.21.15.29:0/2951364339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784004050 con 0x55718fb3a4d0 2024-09-21T02:48:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff77bfff640 1 -- 172.21.15.29:0/2951364339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff7840041f0 con 0x55718fb3a4d0 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2951364339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff794036070 con 0x55718fb3a4d0 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff77bfff640 1 -- 172.21.15.29:0/2951364339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff784025050 con 0x55718fb3a4d0 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 10 client.16866 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2951364339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55718fc3f620 con 0x55718fb3a4d0 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff77bfff640 1 -- 172.21.15.29:0/2951364339 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff794036070 con 0x55718fb3a4d0 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff77bfff640 1 -- 172.21.15.29:0/2951364339 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff784010430 con 0x55718fb3a4d0 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 10 client.16866 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 10 client.16866 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 20 client.16866 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55718fb19d80 0x55718ff19c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 4 client.16866 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.913+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2951364339 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55718fe77640 con 0x55718fb19d80 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.914+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55718fb19d80 0x55718ff19c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.914+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55718fb19d80 0x55718ff19c70 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c9fd0800de465a9 server_cookie=bf4a07852d424e7f in_seq=0 out_seq=0 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.914+0000 7ff77bfff640 10 client.16866 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.915+0000 7ff77bfff640 1 -- 172.21.15.29:0/2951364339 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55718fe77640 con 0x55718fb19d80 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.915+0000 7ff77bfff640 10 client.16866 handle_command_reply: tid=0 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.915+0000 7ff7a9e64740 1 client.16866 shutdown 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.915+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2951364339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb3a4d0 msgr2=0x55718ff0cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.915+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb3a4d0 0x55718ff0cb00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff784012d20 tx=0x7ff78400f790 comp rx=0 tx=0).stop 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.915+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2951364339 shutdown_connections 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.915+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb5b1b0 0x55718ff0d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.915+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55718fb19d80 0x55718ff19c70 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.916+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb3a4d0 0x55718ff0cb00 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.916+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2951364339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.916+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2951364339 >> 172.21.15.29:0/2951364339 conn(0x55718fb4f300 msgr2=0x55718fefcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.916+0000 7ff77bfff640 0 client.16866 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.916+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2951364339 shutdown_connections 2024-09-21T02:48:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.916+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2951364339 wait complete. 2024-09-21T02:48:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.916+0000 7ff7a9e64740 20 client.16866 trim_cache size 0 max 16384 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.930+0000 7ff7a9e64740 1 Processor -- start 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.930+0000 7ff7a9e64740 1 -- start start 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff0c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 0x55718ff06f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1ac480 con 0x55718fb322f0 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1adfc0 con 0x55718fb5b1b0 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1acb50 con 0x55718fb3a4d0 2024-09-21T02:48:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39632/0 (socket says 172.21.15.29:39632) 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 -- 172.21.15.29:0/621094061 learned_addr learned my addr 172.21.15.29:0/621094061 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff79bfff640 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff0c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 0x55718ff06f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 -- 172.21.15.29:0/621094061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 msgr2=0x55718ff0c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff0c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 -- 172.21.15.29:0/621094061 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 msgr2=0x55718ff06f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 0x55718ff06f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 -- 172.21.15.29:0/621094061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55718ff04d20 con 0x55718fb322f0 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7a1497640 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff0c440 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff794005e10 tx=0x7ff794009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b33ccd65ed7a63 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7927fc640 1 -- 172.21.15.29:0/621094061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794031b80 con 0x55718fb322f0 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7927fc640 1 -- 172.21.15.29:0/621094061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff794036050 con 0x55718fb322f0 2024-09-21T02:48:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.931+0000 7ff7927fc640 1 -- 172.21.15.29:0/621094061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794004710 con 0x55718fb322f0 2024-09-21T02:48:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 -- 172.21.15.29:0/621094061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 msgr2=0x55718ff0c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff0c440 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff794005e10 tx=0x7ff794009a20 comp rx=0 tx=0).stop 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 -- 172.21.15.29:0/621094061 shutdown_connections 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 0x55718ff06f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff0c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/621094061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff0c440 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 -- 172.21.15.29:0/621094061 >> 172.21.15.29:0/621094061 conn(0x55718fb4f300 msgr2=0x55718fef95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 -- 172.21.15.29:0/621094061 shutdown_connections 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.932+0000 7ff7a9e64740 1 -- 172.21.15.29:0/621094061 wait complete. 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 1 -- start start 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 1 Processor -- start 2024-09-21T02:48:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff13fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff12c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 0x55718ff131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1ad8f0 con 0x55718fb322f0 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1ad220 con 0x55718fb5b1b0 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1aa460 con 0x55718fb3a4d0 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a9e64740 10 client.0 _ll_register_callbacks cb 0x7ff784009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a1497640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff13fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a1497640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff13fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39638/0 (socket says 172.21.15.29:39638) 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff7a1497640 1 -- 172.21.15.29:0/2785930572 learned_addr learned my addr 172.21.15.29:0/2785930572 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.933+0000 7ff79bfff640 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff12c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 0x55718ff131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a1497640 1 -- 172.21.15.29:0/2785930572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 msgr2=0x55718ff12c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff12c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a1497640 1 -- 172.21.15.29:0/2785930572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 msgr2=0x55718ff131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 0x55718ff131b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a1497640 1 -- 172.21.15.29:0/2785930572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55718ff136f0 con 0x55718fb322f0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a1497640 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff13fb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff794031f30 tx=0x7ff794005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548a6a73447e858 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff77bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff77bfff640 1 -- 172.21.15.29:0/2785930572 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794003970 con 0x55718fb322f0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff77bfff640 1 -- 172.21.15.29:0/2785930572 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff794003b10 con 0x55718fb322f0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2785930572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff794036050 con 0x55718fb322f0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff77bfff640 1 -- 172.21.15.29:0/2785930572 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff794003f20 con 0x55718fb322f0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a9e64740 10 client.16872 fetch_fsmap learned FSMap version 31 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2785930572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55718ff12660 con 0x55718fb322f0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.934+0000 7ff77bfff640 1 -- 172.21.15.29:0/2785930572 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff794036050 con 0x55718fb322f0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff77bfff640 1 -- 172.21.15.29:0/2785930572 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff794043020 con 0x55718fb322f0 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff7a9e64740 10 client.16872 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:48:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff7a9e64740 10 client.16872 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:48:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff7a9e64740 20 client.16872 populate_metadata read hostname 'smithi029' 2024-09-21T02:48:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55718faef1c0 0x55718ff86a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:48:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff7a9e64740 4 client.16872 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:48:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2785930572 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55718fe730f0 con 0x55718faef1c0 2024-09-21T02:48:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff79bfff640 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55718faef1c0 0x55718ff86a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:48:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff77bfff640 10 client.16872 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.935+0000 7ff79bfff640 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55718faef1c0 0x55718ff86a70 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cb413c5122d3dc server_cookie=66a5495940517042 in_seq=0 out_seq=0 2024-09-21T02:48:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff77bfff640 1 -- 172.21.15.29:0/2785930572 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55718fe730f0 con 0x55718faef1c0 2024-09-21T02:48:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff77bfff640 10 client.16872 handle_command_reply: tid=0 2024-09-21T02:48:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 client.16872 shutdown 2024-09-21T02:48:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2785930572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 msgr2=0x55718ff13fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff13fb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff794031f30 tx=0x7ff794005fd0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2785930572 shutdown_connections 2024-09-21T02:48:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55718fb5b1b0 0x55718ff131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55718fb3a4d0 0x55718ff12c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55718faef1c0 0x55718ff86a70 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 --2- 172.21.15.29:0/2785930572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55718fb322f0 0x55718ff13fb0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff77bfff640 0 client.16872 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.973+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2785930572 >> 172.21.15.29:0/2785930572 conn(0x55718fb4f300 msgr2=0x55718ff0c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2785930572 shutdown_connections 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a9e64740 1 -- 172.21.15.29:0/2785930572 wait complete. 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a9e64740 20 client.16872 trim_cache size 0 max 16384 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:48:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:48:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:48:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7880b6510 msgr2=0x7ff7880b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7880b6510 0x7ff7880b8a20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff78c000c00 tx=0x7ff78c079cc0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4174172496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff79c1025c0 msgr2=0x7ff79c06c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:48:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff79c1025c0 0x7ff79c06c210 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00d6d0 tx=0x7ff78c00db90 comp rx=0 tx=0).stop 2024-09-21T02:48:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4174172496 shutdown_connections 2024-09-21T02:48:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7880b6510 0x7ff7880b8a20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff79c103fe0 0x7ff79c1a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff79c103640 0x7ff79c19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 --2- 172.21.15.29:0/4174172496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff79c1025c0 0x7ff79c06c210 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:48:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4174172496 >> 172.21.15.29:0/4174172496 conn(0x7ff79c0fdf00 msgr2=0x7ff79c106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:48:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4174172496 shutdown_connections 2024-09-21T02:48:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:48:57.974+0000 7ff7a2efa640 1 -- 172.21.15.29:0/4174172496 wait complete. 2024-09-21T02:48:58.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.31 MB/sec execute 395 sec latency 5254.968 ms 2024-09-21T02:48:59.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.30 MB/sec execute 396 sec latency 6255.086 ms 2024-09-21T02:48:59.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:48:59 smithi066 ceph-mon[33436]: pgmap v685: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-09-21T02:48:59.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:48:59 smithi190 ceph-mon[33871]: pgmap v685: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-09-21T02:48:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:48:59 smithi029 ceph-mon[31006]: pgmap v685: 129 pgs: 129 active+clean; 43 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-09-21T02:49:00.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.29 MB/sec execute 397 sec latency 7255.212 ms 2024-09-21T02:49:01.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.29 MB/sec execute 398 sec latency 8255.313 ms 2024-09-21T02:49:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:01 smithi066 ceph-mon[33436]: pgmap v686: 129 pgs: 129 active+clean; 43 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 960 KiB/s wr, 92 op/s 2024-09-21T02:49:01.793 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:01 smithi029 ceph-mon[31006]: pgmap v686: 129 pgs: 129 active+clean; 43 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 960 KiB/s wr, 92 op/s 2024-09-21T02:49:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:01 smithi190 ceph-mon[33871]: pgmap v686: 129 pgs: 129 active+clean; 43 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 960 KiB/s wr, 92 op/s 2024-09-21T02:49:02.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.28 MB/sec execute 399 sec latency 9255.351 ms 2024-09-21T02:49:03.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.28 MB/sec execute 400 sec latency 10255.447 ms 2024-09-21T02:49:03.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:03 smithi066 ceph-mon[33436]: pgmap v687: 129 pgs: 129 active+clean; 39 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 665 KiB/s wr, 68 op/s 2024-09-21T02:49:03.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:03 smithi029 ceph-mon[31006]: pgmap v687: 129 pgs: 129 active+clean; 39 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 665 KiB/s wr, 68 op/s 2024-09-21T02:49:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:03 smithi190 ceph-mon[33871]: pgmap v687: 129 pgs: 129 active+clean; 39 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 665 KiB/s wr, 68 op/s 2024-09-21T02:49:04.095 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.27 MB/sec execute 401 sec latency 11255.516 ms 2024-09-21T02:49:05.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.27 MB/sec execute 402 sec latency 12255.620 ms 2024-09-21T02:49:05.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:05 smithi066 ceph-mon[33436]: pgmap v688: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 473 KiB/s wr, 56 op/s 2024-09-21T02:49:05.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:05 smithi190 ceph-mon[33871]: pgmap v688: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 473 KiB/s wr, 56 op/s 2024-09-21T02:49:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:05 smithi029 ceph-mon[31006]: pgmap v688: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 473 KiB/s wr, 56 op/s 2024-09-21T02:49:06.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.26 MB/sec execute 403 sec latency 13255.750 ms 2024-09-21T02:49:07.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.25 MB/sec execute 404 sec latency 14255.858 ms 2024-09-21T02:49:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:07 smithi190 ceph-mon[33871]: pgmap v689: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 203 KiB/s wr, 29 op/s 2024-09-21T02:49:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:07 smithi029 ceph-mon[31006]: pgmap v689: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 203 KiB/s wr, 29 op/s 2024-09-21T02:49:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:07 smithi066 ceph-mon[33436]: pgmap v689: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 203 KiB/s wr, 29 op/s 2024-09-21T02:49:08.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.25 MB/sec execute 405 sec latency 15255.964 ms 2024-09-21T02:49:09.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.24 MB/sec execute 406 sec latency 16256.045 ms 2024-09-21T02:49:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:09 smithi190 ceph-mon[33871]: pgmap v690: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 26 op/s 2024-09-21T02:49:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:09 smithi029 ceph-mon[31006]: pgmap v690: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 26 op/s 2024-09-21T02:49:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:09 smithi066 ceph-mon[33436]: pgmap v690: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 26 op/s 2024-09-21T02:49:10.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.24 MB/sec execute 407 sec latency 17256.173 ms 2024-09-21T02:49:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 -- 172.21.15.29:0/932187771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 msgr2=0x7ffa20102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/932187771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 0x7ffa20102a40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009900 tx=0x7ffa1402a410 comp rx=0 tx=0).stop 2024-09-21T02:49:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 -- 172.21.15.29:0/932187771 shutdown_connections 2024-09-21T02:49:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/932187771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20103ff0 0x7ffa2010a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/932187771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa20103650 0x7ffa20103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/932187771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 0x7ffa20102a40 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 -- 172.21.15.29:0/932187771 >> 172.21.15.29:0/932187771 conn(0x7ffa200fdf30 msgr2=0x7ffa201003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 -- 172.21.15.29:0/932187771 shutdown_connections 2024-09-21T02:49:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 -- 172.21.15.29:0/932187771 wait complete. 2024-09-21T02:49:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 Processor -- start 2024-09-21T02:49:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.088+0000 7ffa26d2b640 1 -- start start 2024-09-21T02:49:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa26d2b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 0x7ffa2006c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa26d2b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20103650 0x7ffa2019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa26d2b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa20103ff0 0x7ffa201a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa26d2b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2010e0c0 con 0x7ffa201025d0 2024-09-21T02:49:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa26d2b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2010df40 con 0x7ffa20103650 2024-09-21T02:49:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa26d2b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2010e240 con 0x7ffa20103ff0 2024-09-21T02:49:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 0x7ffa2006c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 0x7ffa2006c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56426/0 (socket says 172.21.15.29:56426) 2024-09-21T02:49:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 -- 172.21.15.29:0/1114743566 learned_addr learned my addr 172.21.15.29:0/1114743566 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa252c8640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa20103ff0 0x7ffa201a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 -- 172.21.15.29:0/1114743566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa20103ff0 msgr2=0x7ffa201a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20103650 0x7ffa2019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa20103ff0 0x7ffa201a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 -- 172.21.15.29:0/1114743566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20103650 msgr2=0x7ffa2019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20103650 0x7ffa2019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 -- 172.21.15.29:0/1114743566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa201a1940 con 0x7ffa201025d0 2024-09-21T02:49:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20103650 0x7ffa2019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:49:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.089+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 0x7ffa2006c1f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009230 tx=0x7ffa14004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.090+0000 7ffa1dffb640 1 -- 172.21.15.29:0/1114743566 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1403f030 con 0x7ffa201025d0 2024-09-21T02:49:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.090+0000 7ffa1dffb640 1 -- 172.21.15.29:0/1114743566 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa14003a80 con 0x7ffa201025d0 2024-09-21T02:49:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.090+0000 7ffa26d2b640 1 -- 172.21.15.29:0/1114743566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa201a1bd0 con 0x7ffa201025d0 2024-09-21T02:49:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.090+0000 7ffa1dffb640 1 -- 172.21.15.29:0/1114743566 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14035070 con 0x7ffa201025d0 2024-09-21T02:49:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.090+0000 7ffa26d2b640 1 -- 172.21.15.29:0/1114743566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa201a2110 con 0x7ffa201025d0 2024-09-21T02:49:11.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.092+0000 7ffa2dc9a740 1 Processor -- start 2024-09-21T02:49:11.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa2dc9a740 1 -- start start 2024-09-21T02:49:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1dffb640 1 -- 172.21.15.29:0/1114743566 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ffa14042030 con 0x7ffa201025d0 2024-09-21T02:49:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198763e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987644c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198765010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b1986d66f0 con 0x55b19839b4d0 2024-09-21T02:49:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b1986c5b20 con 0x55b1983932f0 2024-09-21T02:49:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b1986c9930 con 0x55b1983bc1b0 2024-09-21T02:49:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198765010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198765010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56442/0 (socket says 172.21.15.29:56442) 2024-09-21T02:49:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1ffff640 1 -- 172.21.15.29:0/225262899 learned_addr learned my addr 172.21.15.29:0/225262899 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa252c8640 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198763e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987644c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1ffff640 1 -- 172.21.15.29:0/225262899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 msgr2=0x55b198763e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198763e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1ffff640 1 -- 172.21.15.29:0/225262899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 msgr2=0x55b1987644c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987644c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.093+0000 7ffa1ffff640 1 -- 172.21.15.29:0/225262899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b19868b530 con 0x55b19839b4d0 2024-09-21T02:49:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.094+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198765010 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009c50 tx=0x7ffa0c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441dacd360053a75 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.094+0000 7ffa1dffb640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa100b6590 0x7ffa100b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.094+0000 7ffa0affd640 1 -- 172.21.15.29:0/225262899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c016070 con 0x55b19839b4d0 2024-09-21T02:49:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.094+0000 7ffa252c8640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa100b6590 0x7ffa100b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.094+0000 7ffa0affd640 1 -- 172.21.15.29:0/225262899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa0c011900 con 0x55b19839b4d0 2024-09-21T02:49:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.094+0000 7ffa0affd640 1 -- 172.21.15.29:0/225262899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c010750 con 0x55b19839b4d0 2024-09-21T02:49:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.094+0000 7ffa1dffb640 1 -- 172.21.15.29:0/1114743566 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffa14107d00 con 0x7ffa201025d0 2024-09-21T02:49:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/225262899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 msgr2=0x55b198765010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198765010 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009c50 tx=0x7ffa0c00e6c0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa252c8640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa100b6590 0x7ffa100b8aa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ffa18005fd0 tx=0x7ffa180097e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/225262899 shutdown_connections 2024-09-21T02:49:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198765010 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987644c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/225262899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198763e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/225262899 >> 172.21.15.29:0/225262899 conn(0x55b1983b0300 msgr2=0x55b19875f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/225262899 shutdown_connections 2024-09-21T02:49:11.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/225262899 wait complete. 2024-09-21T02:49:11.109 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.23 MB/sec execute 408 sec latency 18256.205 ms 2024-09-21T02:49:11.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 -- start start 2024-09-21T02:49:11.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.095+0000 7ffa2dc9a740 1 Processor -- start 2024-09-21T02:49:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198772060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198772bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b1985fdab0 con 0x55b19839b4d0 2024-09-21T02:49:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b19866b730 con 0x55b1983932f0 2024-09-21T02:49:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b198765ad0 con 0x55b1983bc1b0 2024-09-21T02:49:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa2dc9a740 10 client.0 _ll_register_callbacks cb 0x55b1986bf6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198772060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198772060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56448/0 (socket says 172.21.15.29:56448) 2024-09-21T02:49:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa1ffff640 1 -- 172.21.15.29:0/1715856012 learned_addr learned my addr 172.21.15.29:0/1715856012 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa252c8640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198772bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa1ffff640 1 -- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 msgr2=0x55b198772bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:11.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198772bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa1ffff640 1 -- 172.21.15.29:0/1715856012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 msgr2=0x55b1987719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:11.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa1ffff640 1 -- 172.21.15.29:0/1715856012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b198427790 con 0x55b19839b4d0 2024-09-21T02:49:11.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa252c8640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198772bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:49:11.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.096+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:49:11.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198772060 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c005e90 tx=0x7ffa0c011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4448c028f7875e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:11.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ff9fffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:11.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ff9fffff640 1 -- 172.21.15.29:0/1715856012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c016030 con 0x55b19839b4d0 2024-09-21T02:49:11.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ff9fffff640 1 -- 172.21.15.29:0/1715856012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffa0c005250 con 0x55b19839b4d0 2024-09-21T02:49:11.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/1715856012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa0c011900 con 0x55b19839b4d0 2024-09-21T02:49:11.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ff9fffff640 1 -- 172.21.15.29:0/1715856012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ffa0c005450 con 0x55b19839b4d0 2024-09-21T02:49:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ffa2dc9a740 10 client.16884 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/1715856012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b1984a0620 con 0x55b19839b4d0 2024-09-21T02:49:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ff9fffff640 1 -- 172.21.15.29:0/1715856012 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa0c011900 con 0x55b19839b4d0 2024-09-21T02:49:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ff9fffff640 1 -- 172.21.15.29:0/1715856012 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ffa0c00f040 con 0x55b19839b4d0 2024-09-21T02:49:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ffa2dc9a740 10 client.16884 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ffa2dc9a740 10 client.16884 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:49:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ffa2dc9a740 20 client.16884 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.097+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b19837ad80 0x55b19877f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.098+0000 7ffa2dc9a740 4 client.16884 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:49:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.098+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/1715856012 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1986d8640 con 0x55b19837ad80 2024-09-21T02:49:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.098+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b19837ad80 0x55b19877f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.098+0000 7ff9fffff640 10 client.16884 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.098+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b19837ad80 0x55b19877f050 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=666eff5d68c45ad8 server_cookie=dd5c6a78c7362587 in_seq=0 out_seq=0 2024-09-21T02:49:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ff9fffff640 1 -- 172.21.15.29:0/1715856012 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b1986d8640 con 0x55b19837ad80 2024-09-21T02:49:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ff9fffff640 10 client.16884 handle_command_reply: tid=0 2024-09-21T02:49:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 client.16884 shutdown 2024-09-21T02:49:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/1715856012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 msgr2=0x55b198772060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198772060 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c005e90 tx=0x7ffa0c011850 comp rx=0 tx=0).stop 2024-09-21T02:49:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/1715856012 shutdown_connections 2024-09-21T02:49:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983bc1b0 0x55b198772bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b19837ad80 0x55b19877f050 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b198772060 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/1715856012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b1987719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ff9fffff640 0 client.16884 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/1715856012 >> 172.21.15.29:0/1715856012 conn(0x55b1983b0300 msgr2=0x55b19875dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:11.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/1715856012 shutdown_connections 2024-09-21T02:49:11.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/1715856012 wait complete. 2024-09-21T02:49:11.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.099+0000 7ffa2dc9a740 20 client.16884 trim_cache size 0 max 16384 2024-09-21T02:49:11.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.114+0000 7ffa2dc9a740 1 Processor -- start 2024-09-21T02:49:11.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.114+0000 7ffa2dc9a740 1 -- start start 2024-09-21T02:49:11.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.114+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983932f0 0x55b198764ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.114+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b19875a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.114+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983bc1b0 0x55b1987719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.114+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa201aa2b0 con 0x55b19839b4d0 2024-09-21T02:49:11.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.114+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ffa201abdf0 con 0x55b1983bc1b0 2024-09-21T02:49:11.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.114+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffa201aa980 con 0x55b1983932f0 2024-09-21T02:49:11.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b19875a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b19875a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56462/0 (socket says 172.21.15.29:56462) 2024-09-21T02:49:11.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 -- 172.21.15.29:0/20440168 learned_addr learned my addr 172.21.15.29:0/20440168 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983932f0 0x55b198764ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa252c8640 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983bc1b0 0x55b1987719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 -- 172.21.15.29:0/20440168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983932f0 msgr2=0x55b198764ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983932f0 0x55b198764ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 -- 172.21.15.29:0/20440168 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983bc1b0 msgr2=0x55b1987719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983bc1b0 0x55b1987719a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 -- 172.21.15.29:0/20440168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b198779e70 con 0x55b19839b4d0 2024-09-21T02:49:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b19875a5c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00ef60 tx=0x7ffa0c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543a67d23c05e2f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa0a7fc640 1 -- 172.21.15.29:0/20440168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c037070 con 0x55b19839b4d0 2024-09-21T02:49:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa0a7fc640 1 -- 172.21.15.29:0/20440168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa0c009aa0 con 0x55b19839b4d0 2024-09-21T02:49:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.115+0000 7ffa0a7fc640 1 -- 172.21.15.29:0/20440168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c016070 con 0x55b19839b4d0 2024-09-21T02:49:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/20440168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 msgr2=0x55b19875a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b19875a5c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00ef60 tx=0x7ffa0c00ed10 comp rx=0 tx=0).stop 2024-09-21T02:49:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/20440168 shutdown_connections 2024-09-21T02:49:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983bc1b0 0x55b1987719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b19839b4d0 0x55b19875a5c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/20440168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1983932f0 0x55b198764ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/20440168 >> 172.21.15.29:0/20440168 conn(0x55b1983b0300 msgr2=0x55b198770bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/20440168 shutdown_connections 2024-09-21T02:49:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/20440168 wait complete. 2024-09-21T02:49:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.116+0000 7ffa2dc9a740 1 -- start start 2024-09-21T02:49:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa2dc9a740 1 Processor -- start 2024-09-21T02:49:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b198776a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b19839b4d0 0x55b198777150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa2dc9a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1983bc1b0 0x55b198777c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa201ab720 con 0x55b1983bc1b0 2024-09-21T02:49:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ffa201ab050 con 0x55b1983932f0 2024-09-21T02:49:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa2dc9a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffa201a8290 con 0x55b19839b4d0 2024-09-21T02:49:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa2dc9a740 10 client.0 _ll_register_callbacks cb 0x7ffa18009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa252c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1983bc1b0 0x55b198777c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa252c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1983bc1b0 0x55b198777c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56472/0 (socket says 172.21.15.29:56472) 2024-09-21T02:49:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa252c8640 1 -- 172.21.15.29:0/319730461 learned_addr learned my addr 172.21.15.29:0/319730461 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b198776a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa1ffff640 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b19839b4d0 0x55b198777150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa252c8640 1 -- 172.21.15.29:0/319730461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b19839b4d0 msgr2=0x55b198777150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.117+0000 7ffa252c8640 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b19839b4d0 0x55b198777150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ffa252c8640 1 -- 172.21.15.29:0/319730461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 msgr2=0x55b198776a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ffa252c8640 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b198776a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ffa252c8640 1 -- 172.21.15.29:0/319730461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1987783b0 con 0x55b1983bc1b0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b198776a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ffa252c8640 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1983bc1b0 0x55b198777c50 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ffa18003f40 tx=0x7ffa18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc472ae357879e94 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ff9fffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ff9fffff640 1 -- 172.21.15.29:0/319730461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18010b00 con 0x55b1983bc1b0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ff9fffff640 1 -- 172.21.15.29:0/319730461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ffa18010ca0 con 0x55b1983bc1b0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/319730461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa0c009aa0 con 0x55b1983bc1b0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ff9fffff640 1 -- 172.21.15.29:0/319730461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ffa18033050 con 0x55b1983bc1b0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ffa2dc9a740 10 client.16890 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/319730461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b1987797a0 con 0x55b1983bc1b0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.118+0000 7ff9fffff640 1 -- 172.21.15.29:0/319730461 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa0c009aa0 con 0x55b1983bc1b0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ff9fffff640 1 -- 172.21.15.29:0/319730461 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ffa18021440 con 0x55b1983bc1b0 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ffa2dc9a740 10 client.16890 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ffa2dc9a740 10 client.16890 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:49:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ffa2dc9a740 20 client.16890 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1983501c0 0x55b1987ebd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ffa2dc9a740 4 client.16890 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:49:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/319730461 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b1986d40f0 con 0x55b1983501c0 2024-09-21T02:49:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1983501c0 0x55b1987ebd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ff9fffff640 10 client.16890 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.119+0000 7ffa24ac7640 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1983501c0 0x55b1987ebd60 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82b134e7f54e54aa server_cookie=4237e1b73ca92e6b in_seq=0 out_seq=0 2024-09-21T02:49:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.135+0000 7ff9fffff640 1 -- 172.21.15.29:0/319730461 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b1986d40f0 con 0x55b1983501c0 2024-09-21T02:49:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.135+0000 7ff9fffff640 10 client.16890 handle_command_reply: tid=0 2024-09-21T02:49:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.135+0000 7ffa2dc9a740 1 client.16890 shutdown 2024-09-21T02:49:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/319730461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1983bc1b0 msgr2=0x55b198777c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1983bc1b0 0x55b198777c50 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ffa18003f40 tx=0x7ffa18004a40 comp rx=0 tx=0).stop 2024-09-21T02:49:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/319730461 shutdown_connections 2024-09-21T02:49:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b1983501c0 0x55b1987ebd60 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1983bc1b0 0x55b198777c50 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b19839b4d0 0x55b198777150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ff9fffff640 0 client.16890 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 --2- 172.21.15.29:0/319730461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b1983932f0 0x55b198776a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/319730461 >> 172.21.15.29:0/319730461 conn(0x55b1983b0300 msgr2=0x55b19875a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/319730461 shutdown_connections 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 1 -- 172.21.15.29:0/319730461 wait complete. 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.136+0000 7ffa2dc9a740 20 client.16890 trim_cache size 0 max 16384 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:49:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:49:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:49:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 -- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa100b6590 msgr2=0x7ffa100b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa100b6590 0x7ffa100b8aa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ffa18005fd0 tx=0x7ffa180097e0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 -- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 msgr2=0x7ffa2006c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 0x7ffa2006c1f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009230 tx=0x7ffa14004270 comp rx=0 tx=0).stop 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 -- 172.21.15.29:0/1114743566 shutdown_connections 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffa100b6590 0x7ffa100b8aa0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa20103ff0 0x7ffa201a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20103650 0x7ffa2019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 --2- 172.21.15.29:0/1114743566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa201025d0 0x7ffa2006c1f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 -- 172.21.15.29:0/1114743566 >> 172.21.15.29:0/1114743566 conn(0x7ffa200fdf30 msgr2=0x7ffa20106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 -- 172.21.15.29:0/1114743566 shutdown_connections 2024-09-21T02:49:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:11.137+0000 7ffa26d2b640 1 -- 172.21.15.29:0/1114743566 wait complete. 2024-09-21T02:49:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:11 smithi190 ceph-mon[33871]: pgmap v691: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.6 KiB/s wr, 23 op/s 2024-09-21T02:49:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:11 smithi029 ceph-mon[31006]: pgmap v691: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.6 KiB/s wr, 23 op/s 2024-09-21T02:49:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:11 smithi066 ceph-mon[33436]: pgmap v691: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.6 KiB/s wr, 23 op/s 2024-09-21T02:49:12.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.23 MB/sec execute 409 sec latency 19256.306 ms 2024-09-21T02:49:13.096 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.22 MB/sec execute 410 sec latency 20256.417 ms 2024-09-21T02:49:13.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:13 smithi190 ceph-mon[33871]: pgmap v692: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.9 KiB/s wr, 17 op/s 2024-09-21T02:49:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:13 smithi029 ceph-mon[31006]: pgmap v692: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.9 KiB/s wr, 17 op/s 2024-09-21T02:49:14.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:13 smithi066 ceph-mon[33436]: pgmap v692: 129 pgs: 129 active+clean; 39 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.9 KiB/s wr, 17 op/s 2024-09-21T02:49:14.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.22 MB/sec execute 411 sec latency 21256.482 ms 2024-09-21T02:49:14.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:49:14.569 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:49:15.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.21 MB/sec execute 412 sec latency 22256.532 ms 2024-09-21T02:49:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:15 smithi190 ceph-mon[33871]: pgmap v693: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-09-21T02:49:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:15 smithi029 ceph-mon[31006]: pgmap v693: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-09-21T02:49:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:15 smithi066 ceph-mon[33436]: pgmap v693: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-09-21T02:49:16.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.21 MB/sec execute 413 sec latency 23256.609 ms 2024-09-21T02:49:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 -- 172.21.15.29:0/3303018386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981094a0 msgr2=0x7f56981098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 --2- 172.21.15.29:0/3303018386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981094a0 0x7f56981098b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5688009900 tx=0x7f568802a410 comp rx=0 tx=0).stop 2024-09-21T02:49:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 -- 172.21.15.29:0/3303018386 shutdown_connections 2024-09-21T02:49:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 --2- 172.21.15.29:0/3303018386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5698109df0 0x7f5698110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 --2- 172.21.15.29:0/3303018386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981094a0 0x7f56981098b0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 --2- 172.21.15.29:0/3303018386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698103b60 0x7f5698103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 -- 172.21.15.29:0/3303018386 >> 172.21.15.29:0/3303018386 conn(0x7f569807b850 msgr2=0x7f569807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 -- 172.21.15.29:0/3303018386 shutdown_connections 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.702+0000 7f569f1a6640 1 -- 172.21.15.29:0/3303018386 wait complete. 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569f1a6640 1 Processor -- start 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569f1a6640 1 -- start start 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569f1a6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698103b60 0x7f5698075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569f1a6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56981094a0 0x7f5698075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569f1a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698109df0 0x7f5698076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569f1a6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5698113960 con 0x7f5698109df0 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569f1a6640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f56981137e0 con 0x7f56981094a0 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569f1a6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5698113ae0 con 0x7f5698103b60 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569d743640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698109df0 0x7f5698076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569d743640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698109df0 0x7f5698076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40538/0 (socket says 172.21.15.29:40538) 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569d743640 1 -- 172.21.15.29:0/1460210452 learned_addr learned my addr 172.21.15.29:0/1460210452 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f569cf42640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698103b60 0x7f5698075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:17.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.703+0000 7f568ffff640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56981094a0 0x7f5698075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:17.071 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:49:17.071 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:49:17.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f569d743640 1 -- 172.21.15.29:0/1460210452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698103b60 msgr2=0x7f5698075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:17.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f569d743640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698103b60 0x7f5698075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f569d743640 1 -- 172.21.15.29:0/1460210452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56981094a0 msgr2=0x7f5698075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:17.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f569d743640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56981094a0 0x7f5698075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f569d743640 1 -- 172.21.15.29:0/1460210452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56981a85e0 con 0x7f5698109df0 2024-09-21T02:49:17.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f569d743640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698109df0 0x7f5698076190 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f569000eaa0 tx=0x7f569000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f568dffb640 1 -- 172.21.15.29:0/1460210452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5690004d20 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f568dffb640 1 -- 172.21.15.29:0/1460210452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5690005020 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f568dffb640 1 -- 172.21.15.29:0/1460210452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56900116d0 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56981a88d0 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.704+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56981a8d90 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.706+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5698103f70 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.707+0000 7f568dffb640 1 -- 172.21.15.29:0/1460210452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f56900051d0 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.708+0000 7f568dffb640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56800b6510 0x7f56800b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.708+0000 7f569cf42640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56800b6510 0x7f56800b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.709+0000 7f568dffb640 1 -- 172.21.15.29:0/1460210452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f56900e7790 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.709+0000 7f569cf42640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56800b6510 0x7f56800b8a20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5698077160 tx=0x7f569400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.713+0000 7f568dffb640 1 -- 172.21.15.29:0/1460210452 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5690016030 con 0x7f5698109df0 2024-09-21T02:49:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.940+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f569802e640 con 0x7f5698109df0 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.941+0000 7f568dffb640 1 -- 172.21.15.29:0/1460210452 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f56900adca0 con 0x7f5698109df0 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.943+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56800b6510 msgr2=0x7f56800b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.943+0000 7f569f1a6640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56800b6510 0x7f56800b8a20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5698077160 tx=0x7f569400a2d0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.943+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698109df0 msgr2=0x7f5698076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.943+0000 7f569f1a6640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698109df0 0x7f5698076190 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f569000eaa0 tx=0x7f569000eea0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.944+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 shutdown_connections 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.944+0000 7f569f1a6640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f56800b6510 0x7f56800b8a20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.944+0000 7f569f1a6640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698109df0 0x7f5698076190 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.944+0000 7f569f1a6640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f56981094a0 0x7f5698075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.944+0000 7f569f1a6640 1 --2- 172.21.15.29:0/1460210452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698103b60 0x7f5698075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.944+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 >> 172.21.15.29:0/1460210452 conn(0x7f569807b850 msgr2=0x7f56981055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.944+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 shutdown_connections 2024-09-21T02:49:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:16.944+0000 7f569f1a6640 1 -- 172.21.15.29:0/1460210452 wait complete. 2024-09-21T02:49:17.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.20 MB/sec execute 414 sec latency 24256.706 ms 2024-09-21T02:49:17.814 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:17 smithi029 ceph-mon[31006]: pgmap v694: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.5 KiB/s wr, 13 op/s 2024-09-21T02:49:17.814 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:17 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1460210452' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:17.848 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id fc8b81d5 --release --await' 2024-09-21T02:49:17.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id fc8b81d5 --release --await 2024-09-21T02:49:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:17 smithi190 ceph-mon[33871]: pgmap v694: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.5 KiB/s wr, 13 op/s 2024-09-21T02:49:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:17 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1460210452' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:17 smithi066 ceph-mon[33436]: pgmap v694: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.5 KiB/s wr, 13 op/s 2024-09-21T02:49:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:17 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1460210452' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:18.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198651 2.19 MB/sec execute 415 sec latency 25256.764 ms 2024-09-21T02:49:18.113 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:49:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 -- 172.21.15.29:0/2602656449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc1083b0 msgr2=0x7fdebc1087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 --2- 172.21.15.29:0/2602656449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc1083b0 0x7fdebc1087c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009900 tx=0x7fdeac02a410 comp rx=0 tx=0).stop 2024-09-21T02:49:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 -- 172.21.15.29:0/2602656449 shutdown_connections 2024-09-21T02:49:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 --2- 172.21.15.29:0/2602656449 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdebc108d00 0x7fdebc10f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 --2- 172.21.15.29:0/2602656449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc1083b0 0x7fdebc1087c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 --2- 172.21.15.29:0/2602656449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdebc102a70 0x7fdebc102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 -- 172.21.15.29:0/2602656449 >> 172.21.15.29:0/2602656449 conn(0x7fdebc079710 msgr2=0x7fdebc079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 -- 172.21.15.29:0/2602656449 shutdown_connections 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.613+0000 7fdec386b640 1 -- 172.21.15.29:0/2602656449 wait complete. 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.614+0000 7fdec386b640 1 Processor -- start 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.614+0000 7fdec386b640 1 -- start start 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.614+0000 7fdec386b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc102a70 0x7fdebc199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.614+0000 7fdec386b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdebc1083b0 0x7fdebc1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.614+0000 7fdec386b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdebc108d00 0x7fdebc19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.614+0000 7fdec386b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc112870 con 0x7fdebc102a70 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.614+0000 7fdec386b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc1126f0 con 0x7fdebc108d00 2024-09-21T02:49:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.614+0000 7fdec386b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc1129f0 con 0x7fdebc1083b0 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc102a70 0x7fdebc199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc102a70 0x7fdebc199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40554/0 (socket says 172.21.15.29:40554) 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 -- 172.21.15.29:0/2579475373 learned_addr learned my addr 172.21.15.29:0/2579475373 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec0e06640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdebc1083b0 0x7fdebc1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1e08640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdebc108d00 0x7fdebc19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 -- 172.21.15.29:0/2579475373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdebc1083b0 msgr2=0x7fdebc1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdebc1083b0 0x7fdebc1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 -- 172.21.15.29:0/2579475373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdebc108d00 msgr2=0x7fdebc19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdebc108d00 0x7fdebc19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 -- 172.21.15.29:0/2579475373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc19f270 con 0x7fdebc102a70 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec1607640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc102a70 0x7fdebc199300 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009900 tx=0x7fdeb8009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdeb27fc640 1 -- 172.21.15.29:0/2579475373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb800d030 con 0x7fdebc102a70 2024-09-21T02:49:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdeb27fc640 1 -- 172.21.15.29:0/2579475373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeb800d1d0 con 0x7fdebc102a70 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec386b640 1 -- 172.21.15.29:0/2579475373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc19f560 con 0x7fdebc102a70 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdeb27fc640 1 -- 172.21.15.29:0/2579475373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8005280 con 0x7fdebc102a70 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.615+0000 7fdec386b640 1 -- 172.21.15.29:0/2579475373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc06bb80 con 0x7fdebc102a70 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdec9ddf740 1 Processor -- start 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdec9ddf740 1 -- start start 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdeb27fc640 1 -- 172.21.15.29:0/2579475373 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fdeb8005420 con 0x7fdebc102a70 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 0x55c3d0260510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe91660 0x55c3d0260a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0260f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c3d014bdc0 con 0x55c3cfe99840 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c3d01d81f0 con 0x55c3cfeba520 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.618+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c3d01c7620 con 0x55c3cfe91660 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0260f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0260f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40558/0 (socket says 172.21.15.29:40558) 2024-09-21T02:49:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 -- 172.21.15.29:0/3039494928 learned_addr learned my addr 172.21.15.29:0/3039494928 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec1e08640 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 0x55c3d0260510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec1607640 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe91660 0x55c3d0260a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 -- 172.21.15.29:0/3039494928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe91660 msgr2=0x55c3d0260a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe91660 0x55c3d0260a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 -- 172.21.15.29:0/3039494928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 msgr2=0x55c3d0260510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 0x55c3d0260510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 -- 172.21.15.29:0/3039494928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3d018ffd0 con 0x55c3cfe99840 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec1607640 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe91660 0x55c3d0260a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdeb27fc640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdea80b6590 0x7fdea80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec0e06640 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0260f90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fdeac000c00 tx=0x7fdeac032b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c1950b6d3c5f08 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.619+0000 7fdec1e08640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdea80b6590 0x7fdea80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.620+0000 7fdeb27fc640 1 -- 172.21.15.29:0/2579475373 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdeb80e8aa0 con 0x7fdebc102a70 2024-09-21T02:49:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.620+0000 7fdea77fe640 1 -- 172.21.15.29:0/3039494928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac03f070 con 0x55c3cfe99840 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.620+0000 7fdea77fe640 1 -- 172.21.15.29:0/3039494928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeac004030 con 0x55c3cfe99840 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.620+0000 7fdea77fe640 1 -- 172.21.15.29:0/3039494928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac035070 con 0x55c3cfe99840 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.620+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3039494928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 msgr2=0x55c3d0260f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.620+0000 7fdec1e08640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdea80b6590 0x7fdea80b8aa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4005fd0 tx=0x7fdeb4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.620+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0260f90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fdeac000c00 tx=0x7fdeac032b10 comp rx=0 tx=0).stop 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3039494928 shutdown_connections 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0260f90 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe91660 0x55c3d0260a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3039494928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 0x55c3d0260510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3039494928 >> 172.21.15.29:0/3039494928 conn(0x55c3cfeae910 msgr2=0x55c3d025e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3039494928 shutdown_connections 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3039494928 wait complete. 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 -- start start 2024-09-21T02:49:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.621+0000 7fdec9ddf740 1 Processor -- start 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe91660 0x55c3d026baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 0x55c3d026c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 0x55c3d026c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c3d0162300 con 0x55c3cfe91660 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c3d01cb430 con 0x55c3cfeba520 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c3d0028bb0 con 0x55c3cfe99840 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec9ddf740 10 client.0 _ll_register_callbacks cb 0x55c3d017c120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe91660 0x55c3d026baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe91660 0x55c3d026baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40568/0 (socket says 172.21.15.29:40568) 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 -- 172.21.15.29:0/1869166930 learned_addr learned my addr 172.21.15.29:0/1869166930 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec0e06640 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 0x55c3d026c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 -- 172.21.15.29:0/1869166930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 msgr2=0x55c3d026c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 0x55c3d026c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 -- 172.21.15.29:0/1869166930 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 msgr2=0x55c3d026c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1e08640 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 0x55c3d026c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 0x55c3d026c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 -- 172.21.15.29:0/1869166930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3d0151780 con 0x55c3cfe91660 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdec1607640 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe91660 0x55c3d026baf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80abb20 tx=0x7fdeb80aaf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee08590893fcdc41 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.622+0000 7fdea4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdea4ff9640 1 -- 172.21.15.29:0/1869166930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb801e030 con 0x55c3cfe91660 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdea4ff9640 1 -- 172.21.15.29:0/1869166930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdeb80afde0 con 0x55c3cfe91660 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 1 -- 172.21.15.29:0/1869166930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdeac004030 con 0x55c3cfe91660 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdea4ff9640 1 -- 172.21.15.29:0/1869166930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdeb80ae680 con 0x55c3cfe91660 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 10 client.16908 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 1 -- 172.21.15.29:0/1869166930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c3cff25b00 con 0x55c3cfe91660 2024-09-21T02:49:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdea4ff9640 1 -- 172.21.15.29:0/1869166930 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeac004030 con 0x55c3cfe91660 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdea4ff9640 1 -- 172.21.15.29:0/1869166930 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fdeb80ad580 con 0x55c3cfe91660 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 10 client.16908 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 10 client.16908 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 20 client.16908 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3cfe78ea0 0x55c3d027de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 4 client.16908 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.623+0000 7fdec9ddf740 1 -- 172.21.15.29:0/1869166930 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3d01da140 con 0x55c3cfe78ea0 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.624+0000 7fdec0e06640 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3cfe78ea0 0x55c3d027de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.624+0000 7fdea4ff9640 10 client.16908 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.624+0000 7fdec0e06640 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3cfe78ea0 0x55c3d027de60 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53a16130c163878c server_cookie=1a4e906b4e9d1e36 in_seq=0 out_seq=0 2024-09-21T02:49:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdea4ff9640 1 -- 172.21.15.29:0/1869166930 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c3d01da140 con 0x55c3cfe78ea0 2024-09-21T02:49:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdea4ff9640 10 client.16908 handle_command_reply: tid=0 2024-09-21T02:49:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 client.16908 shutdown 2024-09-21T02:49:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 -- 172.21.15.29:0/1869166930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe91660 msgr2=0x55c3d026baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe91660 0x55c3d026baf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80abb20 tx=0x7fdeb80aaf00 comp rx=0 tx=0).stop 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 -- 172.21.15.29:0/1869166930 shutdown_connections 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfeba520 0x55c3d026c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 0x55c3d026c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3cfe78ea0 0x55c3d027de60 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/1869166930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe91660 0x55c3d026baf0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdea4ff9640 0 client.16908 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 -- 172.21.15.29:0/1869166930 >> 172.21.15.29:0/1869166930 conn(0x55c3cfeae910 msgr2=0x55c3d02636a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 -- 172.21.15.29:0/1869166930 shutdown_connections 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 1 -- 172.21.15.29:0/1869166930 wait complete. 2024-09-21T02:49:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.625+0000 7fdec9ddf740 20 client.16908 trim_cache size 0 max 16384 2024-09-21T02:49:18.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec9ddf740 1 Processor -- start 2024-09-21T02:49:18.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec9ddf740 1 -- start start 2024-09-21T02:49:18.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d026d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0268ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfeba520 0x55c3d0269220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc1ae760 con 0x55c3cfe99840 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc1b02a0 con 0x55c3cfe91660 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc1aee30 con 0x55c3cfeba520 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec0e06640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0268ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec0e06640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0268ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40570/0 (socket says 172.21.15.29:40570) 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.643+0000 7fdec0e06640 1 -- 172.21.15.29:0/3962792778 learned_addr learned my addr 172.21.15.29:0/3962792778 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec1e08640 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfeba520 0x55c3d0269220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec1607640 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d026d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec0e06640 1 -- 172.21.15.29:0/3962792778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfeba520 msgr2=0x55c3d0269220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec0e06640 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfeba520 0x55c3d0269220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec0e06640 1 -- 172.21.15.29:0/3962792778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 msgr2=0x55c3d026d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec0e06640 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d026d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec0e06640 1 -- 172.21.15.29:0/3962792778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3cff9c960 con 0x55c3cfe99840 2024-09-21T02:49:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec1607640 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d026d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:49:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec1e08640 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfeba520 0x55c3d0269220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:49:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdec0e06640 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0268ce0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009900 tx=0x7fdeac005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05b25961e98680a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdea6ffd640 1 -- 172.21.15.29:0/3962792778 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac035050 con 0x55c3cfe99840 2024-09-21T02:49:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdea6ffd640 1 -- 172.21.15.29:0/3962792778 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeac005d50 con 0x55c3cfe99840 2024-09-21T02:49:18.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.644+0000 7fdea6ffd640 1 -- 172.21.15.29:0/3962792778 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac03f040 con 0x55c3cfe99840 2024-09-21T02:49:18.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3962792778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 msgr2=0x55c3d0268ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0268ce0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009900 tx=0x7fdeac005a40 comp rx=0 tx=0).stop 2024-09-21T02:49:18.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3962792778 shutdown_connections 2024-09-21T02:49:18.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfeba520 0x55c3d0269220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfe99840 0x55c3d0268ce0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3962792778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d026d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3962792778 >> 172.21.15.29:0/3962792778 conn(0x55c3cfeae910 msgr2=0x55c3d026d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3962792778 shutdown_connections 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3962792778 wait complete. 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.645+0000 7fdec9ddf740 1 -- start start 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec9ddf740 1 Processor -- start 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d02e3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 0x55c3d0272730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec9ddf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfeba520 0x55c3d0272c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc1afbd0 con 0x55c3cfeba520 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc1af500 con 0x55c3cfe91660 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec9ddf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc1ac740 con 0x55c3cfe99840 2024-09-21T02:49:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec9ddf740 10 client.0 _ll_register_callbacks cb 0x7fdeb8009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec1e08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfeba520 0x55c3d0272c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec1e08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfeba520 0x55c3d0272c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40574/0 (socket says 172.21.15.29:40574) 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec1e08640 1 -- 172.21.15.29:0/3523777255 learned_addr learned my addr 172.21.15.29:0/3523777255 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec1607640 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d02e3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.646+0000 7fdec0e06640 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 0x55c3d0272730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec1e08640 1 -- 172.21.15.29:0/3523777255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 msgr2=0x55c3d0272730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec1e08640 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 0x55c3d0272730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec1e08640 1 -- 172.21.15.29:0/3523777255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 msgr2=0x55c3d02e3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec1e08640 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d02e3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec1e08640 1 -- 172.21.15.29:0/3523777255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3d02731b0 con 0x55c3cfeba520 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec1607640 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d02e3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec1e08640 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfeba520 0x55c3d0272c70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4004010 tx=0x7fdeb4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0e54b0317c9404 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdea4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdea4ff9640 1 -- 172.21.15.29:0/3523777255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb402c060 con 0x55c3cfeba520 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdea4ff9640 1 -- 172.21.15.29:0/3523777255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdeb4011810 con 0x55c3cfeba520 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3523777255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdeac005d50 con 0x55c3cfeba520 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdea4ff9640 1 -- 172.21.15.29:0/3523777255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdeb4033050 con 0x55c3cfeba520 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec9ddf740 10 client.16914 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.647+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3523777255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c3d02ed7b0 con 0x55c3cfeba520 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdea4ff9640 1 -- 172.21.15.29:0/3523777255 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeac005d50 con 0x55c3cfeba520 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdea4ff9640 1 -- 172.21.15.29:0/3523777255 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fdeb400ea70 con 0x55c3cfeba520 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdec9ddf740 10 client.16914 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdec9ddf740 10 client.16914 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdec9ddf740 20 client.16914 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3cfe4e250 0x55c3d02effa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdec9ddf740 4 client.16914 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "fc8b81d5", "release": true, "await": true}] 2024-09-21T02:49:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3523777255 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "fc8b81d5", "release": true, "await": true}) -- 0x55c3d01d5bf0 con 0x55c3cfe4e250 2024-09-21T02:49:18.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdec1607640 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3cfe4e250 0x55c3d02effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:18.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdea4ff9640 10 client.16914 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:18.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.648+0000 7fdec1607640 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3cfe4e250 0x55c3d02effa0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4ce11a4969afd4a server_cookie=66b5a7b2a55482b2 in_seq=0 out_seq=0 2024-09-21T02:49:18.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.660+0000 7fdea4ff9640 1 -- 172.21.15.29:0/3523777255 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c3d01d5bf0 con 0x55c3cfe4e250 2024-09-21T02:49:18.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.660+0000 7fdea4ff9640 10 client.16914 handle_command_reply: tid=0 2024-09-21T02:49:18.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.660+0000 7fdec9ddf740 1 client.16914 shutdown 2024-09-21T02:49:18.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3523777255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfeba520 msgr2=0x55c3d0272c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfeba520 0x55c3d0272c70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4004010 tx=0x7fdeb4004ab0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3523777255 shutdown_connections 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3cfe4e250 0x55c3d02effa0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3cfeba520 0x55c3d0272c70 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3cfe99840 0x55c3d0272730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdea4ff9640 0 client.16914 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 --2- 172.21.15.29:0/3523777255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3cfe91660 0x55c3d02e3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3523777255 >> 172.21.15.29:0/3523777255 conn(0x55c3cfeae910 msgr2=0x55c3d0276d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3523777255 shutdown_connections 2024-09-21T02:49:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 1 -- 172.21.15.29:0/3523777255 wait complete. 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.661+0000 7fdec9ddf740 20 client.16914 trim_cache size 0 max 16384 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 40, 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stdout: "fc8b81d5": { 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 40, 2024-09-21T02:49:18.664 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 144.5, 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:18.665 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:18.666 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:18.666 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:18.666 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:18.666 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:49:18.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 -- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdea80b6590 msgr2=0x7fdea80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdea80b6590 0x7fdea80b8aa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4005fd0 tx=0x7fdeb4009500 comp rx=0 tx=0).stop 2024-09-21T02:49:18.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 -- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc102a70 msgr2=0x7fdebc199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc102a70 0x7fdebc199300 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009900 tx=0x7fdeb8009dc0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 -- 172.21.15.29:0/2579475373 shutdown_connections 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdea80b6590 0x7fdea80b8aa0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdebc108d00 0x7fdebc19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdebc1083b0 0x7fdebc1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 --2- 172.21.15.29:0/2579475373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdebc102a70 0x7fdebc199300 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 -- 172.21.15.29:0/2579475373 >> 172.21.15.29:0/2579475373 conn(0x7fdebc079710 msgr2=0x7fdebc106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 -- 172.21.15.29:0/2579475373 shutdown_connections 2024-09-21T02:49:18.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:18.662+0000 7fdec386b640 1 -- 172.21.15.29:0/2579475373 wait complete. 2024-09-21T02:49:19.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198862 2.19 MB/sec execute 416 sec latency 25816.354 ms 2024-09-21T02:49:19.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:19 smithi029 ceph-mon[31006]: pgmap v695: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-09-21T02:49:19.862 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'fc8b81d5', total seconds elapsed: 30.1 2024-09-21T02:49:19.862 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-09-21T02:49:19.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:49:19.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:19 smithi190 ceph-mon[33871]: pgmap v695: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-09-21T02:49:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:19 smithi066 ceph-mon[33436]: pgmap v695: 129 pgs: 129 active+clean; 39 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 14 op/s 2024-09-21T02:49:20.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 199078 2.19 MB/sec execute 417 sec latency 193.656 ms 2024-09-21T02:49:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:49:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 -- 172.21.15.29:0/2328569371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e881026a0 msgr2=0x7f7e88102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 --2- 172.21.15.29:0/2328569371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e881026a0 0x7f7e88102ab0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009900 tx=0x7f7e8402a480 comp rx=0 tx=0).stop 2024-09-21T02:49:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 -- 172.21.15.29:0/2328569371 shutdown_connections 2024-09-21T02:49:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 --2- 172.21.15.29:0/2328569371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e88110250 0x7f7e88112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 --2- 172.21.15.29:0/2328569371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e88102ff0 0x7f7e8810fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 --2- 172.21.15.29:0/2328569371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e881026a0 0x7f7e88102ab0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 -- 172.21.15.29:0/2328569371 >> 172.21.15.29:0/2328569371 conn(0x7f7e880796d0 msgr2=0x7f7e88079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 -- 172.21.15.29:0/2328569371 shutdown_connections 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.687+0000 7f7e900ff640 1 -- 172.21.15.29:0/2328569371 wait complete. 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.688+0000 7f7e900ff640 1 Processor -- start 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.688+0000 7f7e900ff640 1 -- start start 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.688+0000 7f7e900ff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e881026a0 0x7f7e8819c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.688+0000 7f7e900ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88102ff0 0x7f7e8819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.688+0000 7f7e900ff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e88110250 0x7f7e881a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.688+0000 7f7e900ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7e88115b40 con 0x7f7e88102ff0 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.688+0000 7f7e900ff640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881159c0 con 0x7f7e88110250 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.688+0000 7f7e900ff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e88115cc0 con 0x7f7e881026a0 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8d69a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88102ff0 0x7f7e8819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8d69a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88102ff0 0x7f7e8819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40584/0 (socket says 172.21.15.29:40584) 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8d69a640 1 -- 172.21.15.29:0/3256191483 learned_addr learned my addr 172.21.15.29:0/3256191483 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8de9b640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e881026a0 0x7f7e8819c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8e69c640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e88110250 0x7f7e881a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8de9b640 1 -- 172.21.15.29:0/3256191483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e88110250 msgr2=0x7f7e881a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8de9b640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e88110250 0x7f7e881a1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8de9b640 1 -- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88102ff0 msgr2=0x7f7e8819cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8de9b640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88102ff0 0x7f7e8819cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8de9b640 1 -- 172.21.15.29:0/3256191483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e881a2480 con 0x7f7e881026a0 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8d69a640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88102ff0 0x7f7e8819cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e8de9b640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e881026a0 0x7f7e8819c5d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005940 tx=0x7f7e84032f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.689+0000 7f7e7effd640 1 -- 172.21.15.29:0/3256191483 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8403f030 con 0x7f7e881026a0 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.690+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e881a2710 con 0x7f7e881026a0 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.690+0000 7f7e7effd640 1 -- 172.21.15.29:0/3256191483 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e84004450 con 0x7f7e881026a0 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.690+0000 7f7e7effd640 1 -- 172.21.15.29:0/3256191483 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84035070 con 0x7f7e881026a0 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.690+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e881a2c40 con 0x7f7e881026a0 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.691+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e8806c190 con 0x7f7e881026a0 2024-09-21T02:49:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.692+0000 7f7e7effd640 1 -- 172.21.15.29:0/3256191483 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f7e84042030 con 0x7f7e881026a0 2024-09-21T02:49:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.693+0000 7f7e7effd640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e740b6540 0x7f7e740b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.693+0000 7f7e8d69a640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e740b6540 0x7f7e740b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.693+0000 7f7e7effd640 1 -- 172.21.15.29:0/3256191483 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7e84107cc0 con 0x7f7e881026a0 2024-09-21T02:49:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.694+0000 7f7e8d69a640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e740b6540 0x7f7e740b8a50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7e8807a040 tx=0x7f7e78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.696+0000 7f7e7effd640 1 -- 172.21.15.29:0/3256191483 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7e881a2c40 con 0x7f7e881026a0 2024-09-21T02:49:20.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.984+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e8819d700 con 0x7f7e881026a0 2024-09-21T02:49:20.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.985+0000 7f7e7effd640 1 -- 172.21.15.29:0/3256191483 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f7e840ce1d0 con 0x7f7e881026a0 2024-09-21T02:49:20.987 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:49:20.987 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:49:20.987 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e740b6540 msgr2=0x7f7e740b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e740b6540 0x7f7e740b8a50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7e8807a040 tx=0x7f7e78005f90 comp rx=0 tx=0).stop 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e881026a0 msgr2=0x7f7e8819c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e881026a0 0x7f7e8819c5d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005940 tx=0x7f7e84032f90 comp rx=0 tx=0).stop 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 shutdown_connections 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e740b6540 0x7f7e740b8a50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e88110250 0x7f7e881a1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88102ff0 0x7f7e8819cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 --2- 172.21.15.29:0/3256191483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e881026a0 0x7f7e8819c5d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 >> 172.21.15.29:0/3256191483 conn(0x7f7e880796d0 msgr2=0x7f7e88111780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 shutdown_connections 2024-09-21T02:49:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:20.987+0000 7f7e900ff640 1 -- 172.21.15.29:0/3256191483 wait complete. 2024-09-21T02:49:21.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 199512 2.18 MB/sec execute 418 sec latency 18.857 ms 2024-09-21T02:49:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:21 smithi029 ceph-mon[31006]: pgmap v696: 129 pgs: 129 active+clean; 39 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 60 KiB/s wr, 16 op/s 2024-09-21T02:49:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:21 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3256191483' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:21.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:21 smithi190 ceph-mon[33871]: pgmap v696: 129 pgs: 129 active+clean; 39 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 60 KiB/s wr, 16 op/s 2024-09-21T02:49:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:21 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3256191483' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:21.906 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0' 2024-09-21T02:49:21.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0 2024-09-21T02:49:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:21 smithi066 ceph-mon[33436]: pgmap v696: 129 pgs: 129 active+clean; 39 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 60 KiB/s wr, 16 op/s 2024-09-21T02:49:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:21 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3256191483' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:22.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200127 2.19 MB/sec execute 419 sec latency 20.972 ms 2024-09-21T02:49:22.205 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:49:22.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 -- 172.21.15.29:0/2820268012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04105260 msgr2=0x7f6f0410d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2820268012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04105260 0x7f6f0410d980 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009900 tx=0x7f6ef402a410 comp rx=0 tx=0).stop 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 -- 172.21.15.29:0/2820268012 shutdown_connections 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2820268012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6f0410dec0 0x7f6f04110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2820268012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04105260 0x7f6f0410d980 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2820268012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f04104910 0x7f6f04104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 -- 172.21.15.29:0/2820268012 >> 172.21.15.29:0/2820268012 conn(0x7f6f04079830 msgr2=0x7f6f04079c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 -- 172.21.15.29:0/2820268012 shutdown_connections 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.780+0000 7f6f0be45640 1 -- 172.21.15.29:0/2820268012 wait complete. 2024-09-21T02:49:22.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0be45640 1 Processor -- start 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0be45640 1 -- start start 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0be45640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f04104910 0x7f6f0419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0be45640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6f04105260 0x7f6f0419cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0be45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0410dec0 0x7f6f041a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0be45640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04104480 con 0x7f6f0410dec0 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0be45640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04104300 con 0x7f6f04105260 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0be45640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04104600 con 0x7f6f04104910 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0a3e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0410dec0 0x7f6f041a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0a3e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0410dec0 0x7f6f041a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40598/0 (socket says 172.21.15.29:40598) 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f09be1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f04104910 0x7f6f0419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f0a3e2640 1 -- 172.21.15.29:0/2850897539 learned_addr learned my addr 172.21.15.29:0/2850897539 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:22.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.781+0000 7f6f093e0640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6f04105260 0x7f6f0419cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f0a3e2640 1 -- 172.21.15.29:0/2850897539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f04104910 msgr2=0x7f6f0419c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f0a3e2640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f04104910 0x7f6f0419c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f0a3e2640 1 -- 172.21.15.29:0/2850897539 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6f04105260 msgr2=0x7f6f0419cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f0a3e2640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6f04105260 0x7f6f0419cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f0a3e2640 1 -- 172.21.15.29:0/2850897539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f041a2530 con 0x7f6f0410dec0 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f09be1640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f04104910 0x7f6f0419c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f0a3e2640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0410dec0 0x7f6f041a1e00 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00eb10 tx=0x7f6efc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6efaffd640 1 -- 172.21.15.29:0/2850897539 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc004b20 con 0x7f6f0410dec0 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f0be45640 1 -- 172.21.15.29:0/2850897539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f041a2820 con 0x7f6f0410dec0 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6f0be45640 1 -- 172.21.15.29:0/2850897539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f041a8f80 con 0x7f6f0410dec0 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6efaffd640 1 -- 172.21.15.29:0/2850897539 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6efc004cc0 con 0x7f6f0410dec0 2024-09-21T02:49:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.782+0000 7f6efaffd640 1 -- 172.21.15.29:0/2850897539 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc010690 con 0x7f6f0410dec0 2024-09-21T02:49:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.785+0000 7f6f123b9740 1 Processor -- start 2024-09-21T02:49:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.786+0000 7f6efaffd640 1 -- 172.21.15.29:0/2850897539 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f6efc005020 con 0x7f6f0410dec0 2024-09-21T02:49:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.786+0000 7f6f123b9740 1 -- start start 2024-09-21T02:49:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec07c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcac980 0x55e1ec07cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcb4b60 0x55e1ec07d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f123b9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e1ebff2b90 con 0x55e1ebcb4b60 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f123b9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e1ebfe1fc0 con 0x55e1ebcac980 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f123b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e1ebfe5dd0 con 0x55e1ebcd5840 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec07c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec07c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45680/0 (socket says 172.21.15.29:45680) 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f093e0640 1 -- 172.21.15.29:0/1489563922 learned_addr learned my addr 172.21.15.29:0/1489563922 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f093e0640 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcb4b60 0x55e1ec07d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f09be1640 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcac980 0x55e1ec07cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6efaffd640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6ef00b6510 0x7f6ef00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f09be1640 1 -- 172.21.15.29:0/1489563922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 msgr2=0x55e1ec07c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.787+0000 7f6f09be1640 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec07c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6f09be1640 1 -- 172.21.15.29:0/1489563922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcb4b60 msgr2=0x55e1ec07d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6f09be1640 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcb4b60 0x55e1ec07d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6f09be1640 1 -- 172.21.15.29:0/1489563922 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1ebf69060 con 0x55e1ebcac980 2024-09-21T02:49:22.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6f0a3e2640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6ef00b6510 0x7f6ef00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6efaffd640 1 -- 172.21.15.29:0/2850897539 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6efc0e7ba0 con 0x7f6f0410dec0 2024-09-21T02:49:22.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6f09be1640 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcac980 0x55e1ec07cb40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009fd0 tx=0x7f6f000118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f70a797bc5356162 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:22.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6eeffff640 1 -- 172.21.15.29:0/1489563922 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00004030 con 0x55e1ebcac980 2024-09-21T02:49:22.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6eeffff640 1 -- 172.21.15.29:0/1489563922 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f00005020 con 0x55e1ebcac980 2024-09-21T02:49:22.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.788+0000 7f6eeffff640 1 -- 172.21.15.29:0/1489563922 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00004ad0 con 0x55e1ebcac980 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.789+0000 7f6f123b9740 1 -- 172.21.15.29:0/1489563922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcac980 msgr2=0x55e1ec07cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.789+0000 7f6f0a3e2640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6ef00b6510 0x7f6ef00b8a20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0aac20 tx=0x7f6efc00cee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.789+0000 7f6f123b9740 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcac980 0x55e1ec07cb40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009fd0 tx=0x7f6f000118d0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 -- 172.21.15.29:0/1489563922 shutdown_connections 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcb4b60 0x55e1ec07d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcac980 0x55e1ec07cb40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 --2- 172.21.15.29:0/1489563922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec07c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 -- 172.21.15.29:0/1489563922 >> 172.21.15.29:0/1489563922 conn(0x55e1ebcc9990 msgr2=0x55e1ec07a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 -- 172.21.15.29:0/1489563922 shutdown_connections 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 -- 172.21.15.29:0/1489563922 wait complete. 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 -- start start 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.790+0000 7f6f123b9740 1 Processor -- start 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcac980 0x55e1ec0910a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec091680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcd5840 0x55e1ec0966a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f123b9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e1ebe437a0 con 0x55e1ebcd5840 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f123b9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e1ebf870a0 con 0x55e1ebcb4b60 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f123b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e1ec08ade0 con 0x55e1ebcac980 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f123b9740 10 client.0 _ll_register_callbacks cb 0x55e1ebf643c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcac980 0x55e1ec0910a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcd5840 0x55e1ec0966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f09be1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec091680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcac980 0x55e1ec0910a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45688/0 (socket says 172.21.15.29:45688) 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f09be1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec091680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:50958/0 (socket says 172.21.15.29:50958) 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.791+0000 7f6f093e0640 1 -- 172.21.15.29:0/3509917791 learned_addr learned my addr 172.21.15.29:0/3509917791 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f093e0640 1 -- 172.21.15.29:0/3509917791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 msgr2=0x55e1ec091680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f093e0640 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec091680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f093e0640 1 -- 172.21.15.29:0/3509917791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcd5840 msgr2=0x55e1ec0966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f093e0640 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcd5840 0x55e1ec0966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f093e0640 1 -- 172.21.15.29:0/3509917791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1ebf75880 con 0x55e1ebcac980 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f093e0640 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcac980 0x55e1ec0910a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6f0406a890 tx=0x7f6ef402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50e1f05dff50eebc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6eed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6eed7fa640 1 -- 172.21.15.29:0/3509917791 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4035040 con 0x55e1ebcac980 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6eed7fa640 1 -- 172.21.15.29:0/3509917791 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6ef4004050 con 0x55e1ebcac980 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f123b9740 1 -- 172.21.15.29:0/3509917791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f00005020 con 0x55e1ebcac980 2024-09-21T02:49:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6eed7fa640 1 -- 172.21.15.29:0/3509917791 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6ef4004250 con 0x55e1ebcac980 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f123b9740 10 client.26488 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.792+0000 7f6f123b9740 1 -- 172.21.15.29:0/3509917791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1ebd4fb90 con 0x55e1ebcac980 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6eed7fa640 1 -- 172.21.15.29:0/3509917791 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ef40043f0 con 0x55e1ebcac980 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6eed7fa640 1 -- 172.21.15.29:0/3509917791 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6ef403d440 con 0x55e1ebcac980 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6f123b9740 10 client.26488 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6f123b9740 10 client.26488 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6f123b9740 20 client.26488 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6f123b9740 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1ebc941c0 0x55e1ec0a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6f123b9740 4 client.26488 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6f123b9740 1 -- 172.21.15.29:0/3509917791 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6f00005020 con 0x55e1ebc941c0 2024-09-21T02:49:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.793+0000 7f6f09be1640 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1ebc941c0 0x55e1ec0a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.794+0000 7f6f09be1640 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1ebc941c0 0x55e1ec0a28f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=686adca15dcbdce3 server_cookie=eca7649610df9a3e in_seq=0 out_seq=0 2024-09-21T02:49:22.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.794+0000 7f6eed7fa640 10 client.26488 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6eed7fa640 1 -- 172.21.15.29:0/3509917791 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f6f00005020 con 0x55e1ebc941c0 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6eed7fa640 10 client.26488 handle_command_reply: tid=0 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 client.26488 shutdown 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 -- 172.21.15.29:0/3509917791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcac980 msgr2=0x55e1ec0910a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcac980 0x55e1ec0910a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6f0406a890 tx=0x7f6ef402a410 comp rx=0 tx=0).stop 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 -- 172.21.15.29:0/3509917791 shutdown_connections 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1ebc941c0 0x55e1ec0a28f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcd5840 0x55e1ec0966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec091680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6eed7fa640 0 client.26488 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 --2- 172.21.15.29:0/3509917791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcac980 0x55e1ec0910a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 -- 172.21.15.29:0/3509917791 >> 172.21.15.29:0/3509917791 conn(0x55e1ebcc9990 msgr2=0x55e1ec07f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 -- 172.21.15.29:0/3509917791 shutdown_connections 2024-09-21T02:49:22.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.795+0000 7f6f123b9740 1 -- 172.21.15.29:0/3509917791 wait complete. 2024-09-21T02:49:22.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.796+0000 7f6f123b9740 20 client.26488 trim_cache size 0 max 16384 2024-09-21T02:49:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.815+0000 7f6f123b9740 1 Processor -- start 2024-09-21T02:49:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.815+0000 7f6f123b9740 1 -- start start 2024-09-21T02:49:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.815+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcac980 0x55e1ec08e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.815+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec08e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.815+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec0a8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.815+0000 7f6f123b9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041ae810 con 0x55e1ebcac980 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.815+0000 7f6f123b9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041b0350 con 0x55e1ebcb4b60 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f123b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041aeee0 con 0x55e1ebcd5840 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcac980 0x55e1ec08e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcac980 0x55e1ec08e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40628/0 (socket says 172.21.15.29:40628) 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 -- 172.21.15.29:0/2747427084 learned_addr learned my addr 172.21.15.29:0/2747427084 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec0a8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f09be1640 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec08e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 -- 172.21.15.29:0/2747427084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 msgr2=0x55e1ec0a8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec0a8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 -- 172.21.15.29:0/2747427084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 msgr2=0x55e1ec08e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:22.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec08e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 -- 172.21.15.29:0/2747427084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1ec075ad0 con 0x55e1ebcac980 2024-09-21T02:49:22.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f09be1640 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec08e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:49:22.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6f093e0640 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcac980 0x55e1ec08e090 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4001eb0 tx=0x7f6ef4009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8550165e84cf777 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.816+0000 7f6eef7fe640 1 -- 172.21.15.29:0/2747427084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4035040 con 0x55e1ebcac980 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6eef7fe640 1 -- 172.21.15.29:0/2747427084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ef40044b0 con 0x55e1ebcac980 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6eef7fe640 1 -- 172.21.15.29:0/2747427084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef404c7d0 con 0x55e1ebcac980 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6f123b9740 1 -- 172.21.15.29:0/2747427084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcac980 msgr2=0x55e1ec08e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcac980 0x55e1ec08e090 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4001eb0 tx=0x7f6ef4009640 comp rx=0 tx=0).stop 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6f123b9740 1 -- 172.21.15.29:0/2747427084 shutdown_connections 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec0a8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec08e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2747427084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebcac980 0x55e1ec08e090 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.817+0000 7f6f123b9740 1 -- 172.21.15.29:0/2747427084 >> 172.21.15.29:0/2747427084 conn(0x55e1ebcc9990 msgr2=0x55e1ec08b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:22.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.818+0000 7f6f123b9740 1 -- 172.21.15.29:0/2747427084 shutdown_connections 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.818+0000 7f6f123b9740 1 -- 172.21.15.29:0/2747427084 wait complete. 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.818+0000 7f6f123b9740 1 -- start start 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.818+0000 7f6f123b9740 1 Processor -- start 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.818+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec09a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.818+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec09a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.818+0000 7f6f123b9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebc69570 0x55e1ec10d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f123b9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041afc80 con 0x55e1ebc69570 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f123b9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041af5b0 con 0x55e1ebcb4b60 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f123b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041ac810 con 0x55e1ebcd5840 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f123b9740 10 client.0 _ll_register_callbacks cb 0x7f6ef4031a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebc69570 0x55e1ec10d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f093e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebc69570 0x55e1ec10d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40632/0 (socket says 172.21.15.29:40632) 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f09be1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec09a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f093e0640 1 -- 172.21.15.29:0/2937130161 learned_addr learned my addr 172.21.15.29:0/2937130161 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f093e0640 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec09a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f09be1640 1 -- 172.21.15.29:0/2937130161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 msgr2=0x55e1ec09a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f09be1640 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec09a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f09be1640 1 -- 172.21.15.29:0/2937130161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebc69570 msgr2=0x55e1ec10d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f09be1640 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebc69570 0x55e1ec10d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f09be1640 1 -- 172.21.15.29:0/2937130161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1ec10d7e0 con 0x55e1ebcd5840 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6f09be1640 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec09a7d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f00012fd0 tx=0x7f6f00011dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2deccd4b7381acbb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6eed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.819+0000 7f6eed7fa640 1 -- 172.21.15.29:0/2937130161 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f000050b0 con 0x55e1ebcd5840 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6eed7fa640 1 -- 172.21.15.29:0/2937130161 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6f00016040 con 0x55e1ebcd5840 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 1 -- 172.21.15.29:0/2937130161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef40044b0 con 0x55e1ebcd5840 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6eed7fa640 1 -- 172.21.15.29:0/2937130161 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f0003a050 con 0x55e1ebcd5840 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 10 client.26494 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:22.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 1 -- 172.21.15.29:0/2937130161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1ec10e9e0 con 0x55e1ebcd5840 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6eed7fa640 1 -- 172.21.15.29:0/2937130161 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ef40044b0 con 0x55e1ebcd5840 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6eed7fa640 1 -- 172.21.15.29:0/2937130161 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6f0000f660 con 0x55e1ebcd5840 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 10 client.26494 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 10 client.26494 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 20 client.26494 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1ec10f040 0x55e1ec111460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 4 client.26494 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}] 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.820+0000 7f6f123b9740 1 -- 172.21.15.29:0/2937130161 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}) -- 0x55e1ebff4ae0 con 0x55e1ec10f040 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.821+0000 7f6f093e0640 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1ec10f040 0x55e1ec111460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:22.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.821+0000 7f6eed7fa640 10 client.26494 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:22.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.821+0000 7f6f093e0640 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1ec10f040 0x55e1ec111460 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dfd8c48ca16454b server_cookie=ff766421ed7a3001 in_seq=0 out_seq=0 2024-09-21T02:49:22.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.909+0000 7f6eed7fa640 1 -- 172.21.15.29:0/2937130161 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e1ebff4ae0 con 0x55e1ec10f040 2024-09-21T02:49:22.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6eed7fa640 10 client.26494 handle_command_reply: tid=0 2024-09-21T02:49:22.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 client.26494 shutdown 2024-09-21T02:49:22.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 -- 172.21.15.29:0/2937130161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 msgr2=0x55e1ec09a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec09a7d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f00012fd0 tx=0x7f6f00011dd0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 -- 172.21.15.29:0/2937130161 shutdown_connections 2024-09-21T02:49:22.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e1ec10f040 0x55e1ec111460 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1ebc69570 0x55e1ec10d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e1ebcd5840 0x55e1ec09a7d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6eed7fa640 0 client.26494 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 --2- 172.21.15.29:0/2937130161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e1ebcb4b60 0x55e1ec09a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 -- 172.21.15.29:0/2937130161 >> 172.21.15.29:0/2937130161 conn(0x55e1ebcc9990 msgr2=0x55e1ec0922e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 -- 172.21.15.29:0/2937130161 shutdown_connections 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 1 -- 172.21.15.29:0/2937130161 wait complete. 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.910+0000 7f6f123b9740 20 client.26494 trim_cache size 0 max 16384 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 42, 2024-09-21T02:49:22.913 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "c8aeb95": { 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 42, 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 144.5, 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:49:22.914 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:49:22.915 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:49:22.915 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:22.915 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:22.915 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:22.915 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:22.915 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:22.915 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 -- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6ef00b6510 msgr2=0x7f6ef00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6ef00b6510 0x7f6ef00b8a20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0aac20 tx=0x7f6efc00cee0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 -- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0410dec0 msgr2=0x7f6f041a1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0410dec0 0x7f6f041a1e00 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00eb10 tx=0x7f6efc00c2d0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 -- 172.21.15.29:0/2850897539 shutdown_connections 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6ef00b6510 0x7f6ef00b8a20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0410dec0 0x7f6f041a1e00 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6f04105260 0x7f6f0419cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 --2- 172.21.15.29:0/2850897539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f04104910 0x7f6f0419c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 -- 172.21.15.29:0/2850897539 >> 172.21.15.29:0/2850897539 conn(0x7f6f04079830 msgr2=0x7f6f040ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 -- 172.21.15.29:0/2850897539 shutdown_connections 2024-09-21T02:49:22.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:22.911+0000 7f6f0be45640 1 -- 172.21.15.29:0/2850897539 wait complete. 2024-09-21T02:49:23.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.19 MB/sec execute 420 sec latency 176.551 ms 2024-09-21T02:49:23.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:23 smithi029 ceph-mon[31006]: pgmap v697: 129 pgs: 129 active+clean; 39 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 131 KiB/s wr, 24 op/s 2024-09-21T02:49:23.847 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c8aeb95', quiesce took 4.0 seconds. Will release after: 20.5 2024-09-21T02:49:23.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:23 smithi190 ceph-mon[33871]: pgmap v697: 129 pgs: 129 active+clean; 39 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 131 KiB/s wr, 24 op/s 2024-09-21T02:49:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:23 smithi066 ceph-mon[33436]: pgmap v697: 129 pgs: 129 active+clean; 39 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 131 KiB/s wr, 24 op/s 2024-09-21T02:49:24.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.18 MB/sec execute 421 sec latency 1176.630 ms 2024-09-21T02:49:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 -- 172.21.15.29:0/2370482115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103620 msgr2=0x7fc12c103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 --2- 172.21.15.29:0/2370482115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103620 0x7fc12c103a30 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009900 tx=0x7fc11c02a410 comp rx=0 tx=0).stop 2024-09-21T02:49:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 -- 172.21.15.29:0/2370482115 shutdown_connections 2024-09-21T02:49:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 --2- 172.21.15.29:0/2370482115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc12c103fc0 0x7fc12c10a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 --2- 172.21.15.29:0/2370482115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103620 0x7fc12c103a30 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 --2- 172.21.15.29:0/2370482115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc12c1025a0 0x7fc12c102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 -- 172.21.15.29:0/2370482115 >> 172.21.15.29:0/2370482115 conn(0x7fc12c0fdee0 msgr2=0x7fc12c100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 -- 172.21.15.29:0/2370482115 shutdown_connections 2024-09-21T02:49:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.245+0000 7fc133a0a640 1 -- 172.21.15.29:0/2370482115 wait complete. 2024-09-21T02:49:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.246+0000 7fc133a0a640 1 Processor -- start 2024-09-21T02:49:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.246+0000 7fc133a0a640 1 -- start start 2024-09-21T02:49:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc133a0a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc12c1025a0 0x7fc12c1a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc133a0a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc12c103620 0x7fc12c1a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc133a0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103fc0 0x7fc12c1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc133a0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c10cfd0 con 0x7fc12c103fc0 2024-09-21T02:49:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc133a0a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c10ce50 con 0x7fc12c103620 2024-09-21T02:49:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc133a0a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c10d150 con 0x7fc12c1025a0 2024-09-21T02:49:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc131fa7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103fc0 0x7fc12c1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc1317a6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc12c1025a0 0x7fc12c1a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc131fa7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103fc0 0x7fc12c1a16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60026/0 (socket says 172.21.15.29:60026) 2024-09-21T02:49:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc1317a6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc12c1025a0 0x7fc12c1a0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44750/0 (socket says 172.21.15.29:44750) 2024-09-21T02:49:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc131fa7640 1 -- 172.21.15.29:0/4140475121 learned_addr learned my addr 172.21.15.29:0/4140475121 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc130fa5640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc12c103620 0x7fc12c1a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc131fa7640 1 -- 172.21.15.29:0/4140475121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc12c1025a0 msgr2=0x7fc12c1a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc131fa7640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc12c1025a0 0x7fc12c1a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc131fa7640 1 -- 172.21.15.29:0/4140475121 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc12c103620 msgr2=0x7fc12c1a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc131fa7640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc12c103620 0x7fc12c1a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc131fa7640 1 -- 172.21.15.29:0/4140475121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12c1a1ec0 con 0x7fc12c103fc0 2024-09-21T02:49:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.247+0000 7fc130fa5640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc12c103620 0x7fc12c1a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:49:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.248+0000 7fc131fa7640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103fc0 0x7fc12c1a16a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc12400db30 tx=0x7fc12400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.248+0000 7fc1227fc640 1 -- 172.21.15.29:0/4140475121 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc124015070 con 0x7fc12c103fc0 2024-09-21T02:49:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.248+0000 7fc1227fc640 1 -- 172.21.15.29:0/4140475121 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc124019890 con 0x7fc12c103fc0 2024-09-21T02:49:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.248+0000 7fc133a0a640 1 -- 172.21.15.29:0/4140475121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc12c1a2150 con 0x7fc12c103fc0 2024-09-21T02:49:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.248+0000 7fc1227fc640 1 -- 172.21.15.29:0/4140475121 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12400f6b0 con 0x7fc12c103fc0 2024-09-21T02:49:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.248+0000 7fc133a0a640 1 -- 172.21.15.29:0/4140475121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc12c1a88a0 con 0x7fc12c103fc0 2024-09-21T02:49:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.253+0000 7fc1227fc640 1 -- 172.21.15.29:0/4140475121 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc124019a40 con 0x7fc12c103fc0 2024-09-21T02:49:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.253+0000 7fc13aa51740 1 Processor -- start 2024-09-21T02:49:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc13aa51740 1 -- start start 2024-09-21T02:49:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d812e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 0x55c99d8134c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d44a4d0 0x55c99d814010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc13aa51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c99d7856f0 con 0x55c99d44a4d0 2024-09-21T02:49:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc13aa51740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c99d774b20 con 0x55c99d4422f0 2024-09-21T02:49:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc13aa51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c99d778930 con 0x55c99d46b1b0 2024-09-21T02:49:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc130fa5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d812e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc130fa5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d812e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44760/0 (socket says 172.21.15.29:44760) 2024-09-21T02:49:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc130fa5640 1 -- 172.21.15.29:0/1289184631 learned_addr learned my addr 172.21.15.29:0/1289184631 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc130fa5640 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d44a4d0 0x55c99d814010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc1317a6640 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 0x55c99d8134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.254+0000 7fc1227fc640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1180b6660 0x7fc1180b8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc130fa5640 1 -- 172.21.15.29:0/1289184631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 msgr2=0x55c99d8134c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc130fa5640 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 0x55c99d8134c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc130fa5640 1 -- 172.21.15.29:0/1289184631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d44a4d0 msgr2=0x55c99d814010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc130fa5640 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d44a4d0 0x55c99d814010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc130fa5640 1 -- 172.21.15.29:0/1289184631 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c99d73a530 con 0x55c99d46b1b0 2024-09-21T02:49:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc131fa7640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1180b6660 0x7fc1180b8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc1227fc640 1 -- 172.21.15.29:0/4140475121 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc1240e7350 con 0x7fc12c103fc0 2024-09-21T02:49:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc130fa5640 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d812e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc11c02a930 tx=0x7fc11c0058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9306a0b2e4286f19 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc1177fe640 1 -- 172.21.15.29:0/1289184631 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c035050 con 0x55c99d46b1b0 2024-09-21T02:49:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc1177fe640 1 -- 172.21.15.29:0/1289184631 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc11c0044c0 con 0x55c99d46b1b0 2024-09-21T02:49:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc1177fe640 1 -- 172.21.15.29:0/1289184631 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c003c60 con 0x55c99d46b1b0 2024-09-21T02:49:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.255+0000 7fc131fa7640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1180b6660 0x7fc1180b8b70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc1240acf00 tx=0x7fc12407a5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 -- 172.21.15.29:0/1289184631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 msgr2=0x55c99d812e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d812e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc11c02a930 tx=0x7fc11c0058b0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 -- 172.21.15.29:0/1289184631 shutdown_connections 2024-09-21T02:49:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d44a4d0 0x55c99d814010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 0x55c99d8134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 --2- 172.21.15.29:0/1289184631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d812e50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 -- 172.21.15.29:0/1289184631 >> 172.21.15.29:0/1289184631 conn(0x55c99d45f300 msgr2=0x55c99d80e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 -- 172.21.15.29:0/1289184631 shutdown_connections 2024-09-21T02:49:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 -- 172.21.15.29:0/1289184631 wait complete. 2024-09-21T02:49:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.256+0000 7fc13aa51740 1 -- start start 2024-09-21T02:49:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc13aa51740 1 Processor -- start 2024-09-21T02:49:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d81c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d44a4d0 0x55c99d81cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d81d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc13aa51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c99d6acab0 con 0x55c99d4422f0 2024-09-21T02:49:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc13aa51740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c99d71a730 con 0x55c99d44a4d0 2024-09-21T02:49:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc13aa51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c99d814ad0 con 0x55c99d46b1b0 2024-09-21T02:49:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc13aa51740 10 client.0 _ll_register_callbacks cb 0x55c99d76e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc130fa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d81c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc130fa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d81c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60052/0 (socket says 172.21.15.29:60052) 2024-09-21T02:49:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc130fa5640 1 -- 172.21.15.29:0/400296897 learned_addr learned my addr 172.21.15.29:0/400296897 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc130fa5640 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d81d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.257+0000 7fc1317a6640 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d44a4d0 0x55c99d81cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc130fa5640 1 -- 172.21.15.29:0/400296897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 msgr2=0x55c99d81d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc130fa5640 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d81d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc130fa5640 1 -- 172.21.15.29:0/400296897 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d44a4d0 msgr2=0x55c99d81cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc130fa5640 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d44a4d0 0x55c99d81cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc130fa5640 1 -- 172.21.15.29:0/400296897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c99d4d6790 con 0x55c99d4422f0 2024-09-21T02:49:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc130fa5640 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d81c5e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc11c031680 tx=0x7fc11c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ed4644d98f693d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc114ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc114ff9640 1 -- 172.21.15.29:0/400296897 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c041020 con 0x55c99d4422f0 2024-09-21T02:49:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc114ff9640 1 -- 172.21.15.29:0/400296897 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc11c003df0 con 0x55c99d4422f0 2024-09-21T02:49:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc13aa51740 1 -- 172.21.15.29:0/400296897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c99d787640 con 0x55c99d4422f0 2024-09-21T02:49:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc114ff9640 1 -- 172.21.15.29:0/400296897 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc11c05b050 con 0x55c99d4422f0 2024-09-21T02:49:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc13aa51740 10 client.16950 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc13aa51740 1 -- 172.21.15.29:0/400296897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c99d54f620 con 0x55c99d4422f0 2024-09-21T02:49:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.258+0000 7fc114ff9640 1 -- 172.21.15.29:0/400296897 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55c99d787640 con 0x55c99d4422f0 2024-09-21T02:49:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc114ff9640 1 -- 172.21.15.29:0/400296897 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc11c03d980 con 0x55c99d4422f0 2024-09-21T02:49:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc13aa51740 10 client.16950 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc13aa51740 10 client.16950 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:49:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc13aa51740 20 client.16950 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc13aa51740 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c99d429d80 0x55c99d8298f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc13aa51740 4 client.16950 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:49:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc13aa51740 1 -- 172.21.15.29:0/400296897 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c99d7830f0 con 0x55c99d429d80 2024-09-21T02:49:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc1317a6640 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c99d429d80 0x55c99d8298f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc1317a6640 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c99d429d80 0x55c99d8298f0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50cc0dd2c6105178 server_cookie=bfe94ed05cc84bd0 in_seq=0 out_seq=0 2024-09-21T02:49:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.259+0000 7fc114ff9640 10 client.16950 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc114ff9640 1 -- 172.21.15.29:0/400296897 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c99d7830f0 con 0x55c99d429d80 2024-09-21T02:49:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc114ff9640 10 client.16950 handle_command_reply: tid=0 2024-09-21T02:49:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 client.16950 shutdown 2024-09-21T02:49:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 -- 172.21.15.29:0/400296897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 msgr2=0x55c99d81c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d81c5e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc11c031680 tx=0x7fc11c02a410 comp rx=0 tx=0).stop 2024-09-21T02:49:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 -- 172.21.15.29:0/400296897 shutdown_connections 2024-09-21T02:49:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d46b1b0 0x55c99d81d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d44a4d0 0x55c99d81cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c99d429d80 0x55c99d8298f0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 --2- 172.21.15.29:0/400296897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d81c5e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc114ff9640 0 client.16950 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.260+0000 7fc13aa51740 1 -- 172.21.15.29:0/400296897 >> 172.21.15.29:0/400296897 conn(0x55c99d45f300 msgr2=0x55c99d80cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.261+0000 7fc13aa51740 1 -- 172.21.15.29:0/400296897 shutdown_connections 2024-09-21T02:49:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.261+0000 7fc13aa51740 1 -- 172.21.15.29:0/400296897 wait complete. 2024-09-21T02:49:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.261+0000 7fc13aa51740 20 client.16950 trim_cache size 0 max 16384 2024-09-21T02:49:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.275+0000 7fc13aa51740 1 Processor -- start 2024-09-21T02:49:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.275+0000 7fc13aa51740 1 -- start start 2024-09-21T02:49:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 0x55c99d81b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d812df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d46b1b0 0x55c99d813330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc13aa51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c1b2b20 con 0x55c99d46b1b0 2024-09-21T02:49:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc13aa51740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c1b4660 con 0x55c99d4422f0 2024-09-21T02:49:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc13aa51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c1b31f0 con 0x55c99d44a4d0 2024-09-21T02:49:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d46b1b0 0x55c99d813330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d46b1b0 0x55c99d813330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60060/0 (socket says 172.21.15.29:60060) 2024-09-21T02:49:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 -- 172.21.15.29:0/3785477293 learned_addr learned my addr 172.21.15.29:0/3785477293 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc1317a6640 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d812df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 0x55c99d81b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 -- 172.21.15.29:0/3785477293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 msgr2=0x55c99d812df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d812df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 -- 172.21.15.29:0/3785477293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 msgr2=0x55c99d81b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 0x55c99d81b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 -- 172.21.15.29:0/3785477293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c99d81a7f0 con 0x55c99d46b1b0 2024-09-21T02:49:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc1317a6640 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d812df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:49:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc130fa5640 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d46b1b0 0x55c99d813330 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc11c02a930 tx=0x7fc11c0092b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc52d01570a8bf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.276+0000 7fc116ffd640 1 -- 172.21.15.29:0/3785477293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c041070 con 0x55c99d46b1b0 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc116ffd640 1 -- 172.21.15.29:0/3785477293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc11c004280 con 0x55c99d46b1b0 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc116ffd640 1 -- 172.21.15.29:0/3785477293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c035050 con 0x55c99d46b1b0 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 -- 172.21.15.29:0/3785477293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d46b1b0 msgr2=0x55c99d813330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d46b1b0 0x55c99d813330 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc11c02a930 tx=0x7fc11c0092b0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 -- 172.21.15.29:0/3785477293 shutdown_connections 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d46b1b0 0x55c99d813330 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d812df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 --2- 172.21.15.29:0/3785477293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d4422f0 0x55c99d81b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 -- 172.21.15.29:0/3785477293 >> 172.21.15.29:0/3785477293 conn(0x55c99d45f300 msgr2=0x55c99d81ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 -- 172.21.15.29:0/3785477293 shutdown_connections 2024-09-21T02:49:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.277+0000 7fc13aa51740 1 -- 172.21.15.29:0/3785477293 wait complete. 2024-09-21T02:49:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 1 -- start start 2024-09-21T02:49:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 1 Processor -- start 2024-09-21T02:49:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d824030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d8246f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d46b1b0 0x55c99d822ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c1b3f90 con 0x55c99d4422f0 2024-09-21T02:49:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c1b38c0 con 0x55c99d46b1b0 2024-09-21T02:49:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c1b0b00 con 0x55c99d44a4d0 2024-09-21T02:49:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc13aa51740 10 client.0 _ll_register_callbacks cb 0x7fc128005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc130fa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d824030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc130fa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d824030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60074/0 (socket says 172.21.15.29:60074) 2024-09-21T02:49:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.278+0000 7fc130fa5640 1 -- 172.21.15.29:0/2759301354 learned_addr learned my addr 172.21.15.29:0/2759301354 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc1317a6640 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d8246f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc130fa5640 1 -- 172.21.15.29:0/2759301354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 msgr2=0x55c99d8246f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc130fa5640 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d8246f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc130fa5640 1 -- 172.21.15.29:0/2759301354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d46b1b0 msgr2=0x55c99d822ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc130fa5640 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d46b1b0 0x55c99d822ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc130fa5640 1 -- 172.21.15.29:0/2759301354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c99d823610 con 0x55c99d4422f0 2024-09-21T02:49:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc130fa5640 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d824030 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc11c031680 tx=0x7fc11c032f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e864f33ad5a8feec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc114ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc114ff9640 1 -- 172.21.15.29:0/2759301354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c035070 con 0x55c99d4422f0 2024-09-21T02:49:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc114ff9640 1 -- 172.21.15.29:0/2759301354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc11c003df0 con 0x55c99d4422f0 2024-09-21T02:49:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc13aa51740 1 -- 172.21.15.29:0/2759301354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc11c004280 con 0x55c99d4422f0 2024-09-21T02:49:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc114ff9640 1 -- 172.21.15.29:0/2759301354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc11c041040 con 0x55c99d4422f0 2024-09-21T02:49:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc13aa51740 10 client.16956 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc13aa51740 1 -- 172.21.15.29:0/2759301354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c99d8224f0 con 0x55c99d4422f0 2024-09-21T02:49:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.279+0000 7fc114ff9640 1 -- 172.21.15.29:0/2759301354 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc11c004280 con 0x55c99d4422f0 2024-09-21T02:49:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc114ff9640 1 -- 172.21.15.29:0/2759301354 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc11c03d950 con 0x55c99d4422f0 2024-09-21T02:49:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc13aa51740 10 client.16956 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc13aa51740 10 client.16956 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:49:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc13aa51740 20 client.16956 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc13aa51740 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c99d3ff1c0 0x55c99d896460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc13aa51740 4 client.16956 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:49:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc13aa51740 1 -- 172.21.15.29:0/2759301354 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c99d780520 con 0x55c99d3ff1c0 2024-09-21T02:49:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc1317a6640 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c99d3ff1c0 0x55c99d896460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc114ff9640 10 client.16956 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.280+0000 7fc1317a6640 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c99d3ff1c0 0x55c99d896460 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a531275fe76045a server_cookie=48a475f427e745a4 in_seq=0 out_seq=0 2024-09-21T02:49:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc114ff9640 1 -- 172.21.15.29:0/2759301354 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c99d780520 con 0x55c99d3ff1c0 2024-09-21T02:49:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc114ff9640 10 client.16956 handle_command_reply: tid=0 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 client.16956 shutdown 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 -- 172.21.15.29:0/2759301354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 msgr2=0x55c99d824030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d824030 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc11c031680 tx=0x7fc11c032f80 comp rx=0 tx=0).stop 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 -- 172.21.15.29:0/2759301354 shutdown_connections 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c99d46b1b0 0x55c99d822ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c99d44a4d0 0x55c99d8246f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c99d3ff1c0 0x55c99d896460 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 --2- 172.21.15.29:0/2759301354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c99d4422f0 0x55c99d824030 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc114ff9640 0 client.16956 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 -- 172.21.15.29:0/2759301354 >> 172.21.15.29:0/2759301354 conn(0x55c99d45f300 msgr2=0x55c99d81aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 -- 172.21.15.29:0/2759301354 shutdown_connections 2024-09-21T02:49:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 1 -- 172.21.15.29:0/2759301354 wait complete. 2024-09-21T02:49:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.308+0000 7fc13aa51740 20 client.16956 trim_cache size 0 max 16384 2024-09-21T02:49:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:49:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:49:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:49:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 -- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1180b6660 msgr2=0x7fc1180b8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1180b6660 0x7fc1180b8b70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc1240acf00 tx=0x7fc12407a5b0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 -- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103fc0 msgr2=0x7fc12c1a16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103fc0 0x7fc12c1a16a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc12400db30 tx=0x7fc12400b260 comp rx=0 tx=0).stop 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 -- 172.21.15.29:0/4140475121 shutdown_connections 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc1180b6660 0x7fc1180b8b70 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c103fc0 0x7fc12c1a16a0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc12c103620 0x7fc12c1a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 --2- 172.21.15.29:0/4140475121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc12c1025a0 0x7fc12c1a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:24.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 -- 172.21.15.29:0/4140475121 >> 172.21.15.29:0/4140475121 conn(0x7fc12c0fdee0 msgr2=0x7fc12c106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:24.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 -- 172.21.15.29:0/4140475121 shutdown_connections 2024-09-21T02:49:24.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:24.309+0000 7fc133a0a640 1 -- 172.21.15.29:0/4140475121 wait complete. 2024-09-21T02:49:25.097 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.17 MB/sec execute 422 sec latency 2176.728 ms 2024-09-21T02:49:25.812 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:25 smithi066 ceph-mon[33436]: pgmap v698: 129 pgs: 129 active+clean; 39 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 248 KiB/s wr, 49 op/s 2024-09-21T02:49:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:25 smithi190 ceph-mon[33871]: pgmap v698: 129 pgs: 129 active+clean; 39 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 248 KiB/s wr, 49 op/s 2024-09-21T02:49:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:25 smithi029 ceph-mon[31006]: pgmap v698: 129 pgs: 129 active+clean; 39 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 248 KiB/s wr, 49 op/s 2024-09-21T02:49:26.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.17 MB/sec execute 423 sec latency 3176.816 ms 2024-09-21T02:49:27.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.16 MB/sec execute 424 sec latency 4176.875 ms 2024-09-21T02:49:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:27 smithi190 ceph-mon[33871]: pgmap v699: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 348 KiB/s wr, 53 op/s 2024-09-21T02:49:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:27 smithi029 ceph-mon[31006]: pgmap v699: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 348 KiB/s wr, 53 op/s 2024-09-21T02:49:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:27 smithi066 ceph-mon[33436]: pgmap v699: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 348 KiB/s wr, 53 op/s 2024-09-21T02:49:28.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.16 MB/sec execute 425 sec latency 5176.955 ms 2024-09-21T02:49:29.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.15 MB/sec execute 426 sec latency 6177.045 ms 2024-09-21T02:49:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:29 smithi190 ceph-mon[33871]: pgmap v700: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 353 KiB/s wr, 60 op/s 2024-09-21T02:49:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:29 smithi029 ceph-mon[31006]: pgmap v700: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 353 KiB/s wr, 60 op/s 2024-09-21T02:49:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:29 smithi066 ceph-mon[33436]: pgmap v700: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 353 KiB/s wr, 60 op/s 2024-09-21T02:49:30.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.15 MB/sec execute 427 sec latency 7177.174 ms 2024-09-21T02:49:31.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.14 MB/sec execute 428 sec latency 8177.302 ms 2024-09-21T02:49:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:31 smithi190 ceph-mon[33871]: pgmap v701: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 359 KiB/s wr, 64 op/s 2024-09-21T02:49:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:31 smithi029 ceph-mon[31006]: pgmap v701: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 359 KiB/s wr, 64 op/s 2024-09-21T02:49:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:31 smithi066 ceph-mon[33436]: pgmap v701: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 359 KiB/s wr, 64 op/s 2024-09-21T02:49:32.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.14 MB/sec execute 429 sec latency 9177.360 ms 2024-09-21T02:49:33.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.13 MB/sec execute 430 sec latency 10177.500 ms 2024-09-21T02:49:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:33 smithi190 ceph-mon[33871]: pgmap v702: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 303 KiB/s wr, 61 op/s 2024-09-21T02:49:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:33 smithi029 ceph-mon[31006]: pgmap v702: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 303 KiB/s wr, 61 op/s 2024-09-21T02:49:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:33 smithi066 ceph-mon[33436]: pgmap v702: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 303 KiB/s wr, 61 op/s 2024-09-21T02:49:34.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.13 MB/sec execute 431 sec latency 11177.556 ms 2024-09-21T02:49:35.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.12 MB/sec execute 432 sec latency 12177.600 ms 2024-09-21T02:49:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:35 smithi190 ceph-mon[33871]: pgmap v703: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 232 KiB/s wr, 55 op/s 2024-09-21T02:49:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:49:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:35 smithi029 ceph-mon[31006]: pgmap v703: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 232 KiB/s wr, 55 op/s 2024-09-21T02:49:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:49:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:35 smithi066 ceph-mon[33436]: pgmap v703: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 232 KiB/s wr, 55 op/s 2024-09-21T02:49:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:49:36.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.12 MB/sec execute 433 sec latency 13177.640 ms 2024-09-21T02:49:37.098 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.11 MB/sec execute 434 sec latency 14177.752 ms 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:49:37.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:49:37.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:49:37.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:49:37.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 -- 172.21.15.29:0/1261361761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 msgr2=0x7f2df810db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/1261361761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 0x7f2df810db10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009ca0 tx=0x7f2df002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 -- 172.21.15.29:0/1261361761 shutdown_connections 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/1261361761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 0x7f2df810db10 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/1261361761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df8075cf0 0x7f2df8076100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/1261361761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df80770f0 0x7f2df80757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 -- 172.21.15.29:0/1261361761 >> 172.21.15.29:0/1261361761 conn(0x7f2df80ff030 msgr2=0x7f2df81014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 -- 172.21.15.29:0/1261361761 shutdown_connections 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.420+0000 7f2dffa0e640 1 -- 172.21.15.29:0/1261361761 wait complete. 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.421+0000 7f2dffa0e640 1 Processor -- start 2024-09-21T02:49:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.421+0000 7f2dffa0e640 1 -- start start 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.421+0000 7f2dffa0e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df8075cf0 0x7f2df819c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.421+0000 7f2dffa0e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80770f0 0x7f2df819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.421+0000 7f2dffa0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 0x7f2df819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.421+0000 7f2dffa0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8111410 con 0x7f2df810b660 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.421+0000 7f2dffa0e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8111290 con 0x7f2df80770f0 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.421+0000 7f2dffa0e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8111590 con 0x7f2df8075cf0 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 0x7f2df819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 0x7f2df819d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43814/0 (socket says 172.21.15.29:43814) 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 -- 172.21.15.29:0/673641682 learned_addr learned my addr 172.21.15.29:0/673641682 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df8075cf0 0x7f2df819c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80770f0 0x7f2df819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 -- 172.21.15.29:0/673641682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df8075cf0 msgr2=0x7f2df819c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df8075cf0 0x7f2df819c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 -- 172.21.15.29:0/673641682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80770f0 msgr2=0x7f2df819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80770f0 0x7f2df819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 -- 172.21.15.29:0/673641682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df819db00 con 0x7f2df810b660 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 0x7f2df819d2e0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2df0005e30 tx=0x7f2df0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dee7fc640 1 -- 172.21.15.29:0/673641682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0032e90 con 0x7f2df810b660 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dee7fc640 1 -- 172.21.15.29:0/673641682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2df00044d0 con 0x7f2df810b660 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dee7fc640 1 -- 172.21.15.29:0/673641682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0030ce0 con 0x7f2df810b660 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dffa0e640 1 -- 172.21.15.29:0/673641682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df819dd90 con 0x7f2df810b660 2024-09-21T02:49:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.422+0000 7f2dffa0e640 1 -- 172.21.15.29:0/673641682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df81a44e0 con 0x7f2df810b660 2024-09-21T02:49:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2e06a58740 1 Processor -- start 2024-09-21T02:49:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2dee7fc640 1 -- 172.21.15.29:0/673641682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f2df0004040 con 0x7f2df810b660 2024-09-21T02:49:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2e06a58740 1 -- start start 2024-09-21T02:49:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 0x560ac930be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f3b2f0 0x560ac930c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 0x560ac930d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2e06a58740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560ac927e6f0 con 0x560ac8f3b2f0 2024-09-21T02:49:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2e06a58740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560ac926db20 con 0x560ac8f434d0 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.426+0000 7f2e06a58740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560ac9271930 con 0x560ac8f641b0 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f3b2f0 0x560ac930c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f3b2f0 0x560ac930c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43826/0 (socket says 172.21.15.29:43826) 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 -- 172.21.15.29:0/8803713 learned_addr learned my addr 172.21.15.29:0/8803713 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 0x560ac930be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 0x560ac930d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 -- 172.21.15.29:0/8803713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 msgr2=0x560ac930be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 0x560ac930be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 -- 172.21.15.29:0/8803713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 msgr2=0x560ac930d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 0x560ac930d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 -- 172.21.15.29:0/8803713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ac9233530 con 0x560ac8f3b2f0 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dee7fc640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2de40b6590 0x7f2de40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f3b2f0 0x560ac930c4c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009ca0 tx=0x7f2df4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eeea61f1000922d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2de40b6590 0x7f2de40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2dee7fc640 1 -- 172.21.15.29:0/673641682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2df01088c0 con 0x7f2df810b660 2024-09-21T02:49:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2de37fe640 1 -- 172.21.15.29:0/8803713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df401e070 con 0x560ac8f3b2f0 2024-09-21T02:49:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2de37fe640 1 -- 172.21.15.29:0/8803713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2df4019ac0 con 0x560ac8f3b2f0 2024-09-21T02:49:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.427+0000 7f2de37fe640 1 -- 172.21.15.29:0/8803713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df400e890 con 0x560ac8f3b2f0 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 -- 172.21.15.29:0/8803713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f3b2f0 msgr2=0x560ac930c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2de40b6590 0x7f2de40b8aa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2de800ccf0 tx=0x7f2de80046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f3b2f0 0x560ac930c4c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009ca0 tx=0x7f2df4011810 comp rx=0 tx=0).stop 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 -- 172.21.15.29:0/8803713 shutdown_connections 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 0x560ac930d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f3b2f0 0x560ac930c4c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 --2- 172.21.15.29:0/8803713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 0x560ac930be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 -- 172.21.15.29:0/8803713 >> 172.21.15.29:0/8803713 conn(0x560ac8f58300 msgr2=0x560ac93078e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 -- 172.21.15.29:0/8803713 shutdown_connections 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.428+0000 7f2e06a58740 1 -- 172.21.15.29:0/8803713 wait complete. 2024-09-21T02:49:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 1 -- start start 2024-09-21T02:49:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 1 Processor -- start 2024-09-21T02:49:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f3b2f0 0x560ac9315650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 0x560ac9315d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9316810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560ac91a5ab0 con 0x560ac8f641b0 2024-09-21T02:49:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560ac9213730 con 0x560ac8f434d0 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560ac930db30 con 0x560ac8f3b2f0 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.429+0000 7f2e06a58740 10 client.0 _ll_register_callbacks cb 0x560ac92676b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9316810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9316810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43840/0 (socket says 172.21.15.29:43840) 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 -- 172.21.15.29:0/689798310 learned_addr learned my addr 172.21.15.29:0/689798310 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f3b2f0 0x560ac9315650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 0x560ac9315d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 -- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f3b2f0 msgr2=0x560ac9315650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f3b2f0 0x560ac9315650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 -- 172.21.15.29:0/689798310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 msgr2=0x560ac9315d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 0x560ac9315d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 -- 172.21.15.29:0/689798310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ac8fcf790 con 0x560ac8f641b0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f3b2f0 0x560ac9315650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9316810 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f2df00caca0 tx=0x7f2df00ce4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e364500e135e16 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2de0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2de0ff9640 1 -- 172.21.15.29:0/689798310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0004330 con 0x560ac8f641b0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2de0ff9640 1 -- 172.21.15.29:0/689798310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2df00d7c60 con 0x560ac8f641b0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2e06a58740 1 -- 172.21.15.29:0/689798310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2df4019ac0 con 0x560ac8f641b0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.430+0000 7f2de0ff9640 1 -- 172.21.15.29:0/689798310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2df00ce1e0 con 0x560ac8f641b0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2e06a58740 10 client.16968 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2e06a58740 1 -- 172.21.15.29:0/689798310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560ac9048620 con 0x560ac8f641b0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2de0ff9640 1 -- 172.21.15.29:0/689798310 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2df4019ac0 con 0x560ac8f641b0 2024-09-21T02:49:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2de0ff9640 1 -- 172.21.15.29:0/689798310 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2df00ca020 con 0x560ac8f641b0 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2e06a58740 10 client.16968 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2e06a58740 10 client.16968 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2e06a58740 20 client.16968 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2e06a58740 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560ac8f22d80 0x560ac9322da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2e06a58740 4 client.16968 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2e06a58740 1 -- 172.21.15.29:0/689798310 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ac9280640 con 0x560ac8f22d80 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.431+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560ac8f22d80 0x560ac9322da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.432+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560ac8f22d80 0x560ac9322da0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fd3ccb494fb3437 server_cookie=dc66db30d7e8e1bb in_seq=0 out_seq=0 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.432+0000 7f2de0ff9640 10 client.16968 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2de0ff9640 1 -- 172.21.15.29:0/689798310 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560ac9280640 con 0x560ac8f22d80 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2de0ff9640 10 client.16968 handle_command_reply: tid=0 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 client.16968 shutdown 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 -- 172.21.15.29:0/689798310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 msgr2=0x560ac9316810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9316810 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f2df00caca0 tx=0x7f2df00ce4a0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 -- 172.21.15.29:0/689798310 shutdown_connections 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560ac8f22d80 0x560ac9322da0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9316810 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f434d0 0x560ac9315d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 --2- 172.21.15.29:0/689798310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f3b2f0 0x560ac9315650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2de0ff9640 0 client.16968 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 -- 172.21.15.29:0/689798310 >> 172.21.15.29:0/689798310 conn(0x560ac8f58300 msgr2=0x560ac93084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 -- 172.21.15.29:0/689798310 shutdown_connections 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 1 -- 172.21.15.29:0/689798310 wait complete. 2024-09-21T02:49:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.433+0000 7f2e06a58740 20 client.16968 trim_cache size 0 max 16384 2024-09-21T02:49:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2e06a58740 1 Processor -- start 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2e06a58740 1 -- start start 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 0x560ac93142b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f434d0 0x560ac9307220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9307760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2e06a58740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2df81ae790 con 0x560ac8f641b0 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2e06a58740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2df81b02d0 con 0x560ac8f3b2f0 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2e06a58740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2df81aee60 con 0x560ac8f434d0 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2dfdfab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9307760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2dfdfab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9307760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43856/0 (socket says 172.21.15.29:43856) 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2dfdfab640 1 -- 172.21.15.29:0/3123021530 learned_addr learned my addr 172.21.15.29:0/3123021530 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.448+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 0x560ac93142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f434d0 0x560ac9307220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2dfdfab640 1 -- 172.21.15.29:0/3123021530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f434d0 msgr2=0x560ac9307220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f434d0 0x560ac9307220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2dfdfab640 1 -- 172.21.15.29:0/3123021530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 msgr2=0x560ac93142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 0x560ac93142b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2dfdfab640 1 -- 172.21.15.29:0/3123021530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ac93153e0 con 0x560ac8f641b0 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f434d0 0x560ac9307220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9307760 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2df0106090 tx=0x7f2df0003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916a687b7c3bc6e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2de2ffd640 1 -- 172.21.15.29:0/3123021530 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0004330 con 0x560ac8f641b0 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2de2ffd640 1 -- 172.21.15.29:0/3123021530 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2df0036040 con 0x560ac8f641b0 2024-09-21T02:49:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.449+0000 7f2de2ffd640 1 -- 172.21.15.29:0/3123021530 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df00cedd0 con 0x560ac8f641b0 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 -- 172.21.15.29:0/3123021530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 msgr2=0x560ac9307760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9307760 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2df0106090 tx=0x7f2df0003f90 comp rx=0 tx=0).stop 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 -- 172.21.15.29:0/3123021530 shutdown_connections 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f641b0 0x560ac9307760 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f434d0 0x560ac9307220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 --2- 172.21.15.29:0/3123021530 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 0x560ac93142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 -- 172.21.15.29:0/3123021530 >> 172.21.15.29:0/3123021530 conn(0x560ac8f58300 msgr2=0x560ac9313ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 -- 172.21.15.29:0/3123021530 shutdown_connections 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 -- 172.21.15.29:0/3123021530 wait complete. 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 -- start start 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.450+0000 7f2e06a58740 1 Processor -- start 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 0x560ac931a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f434d0 0x560ac931b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2e06a58740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 0x560ac931bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2e06a58740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2df81afc00 con 0x560ac8f434d0 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2e06a58740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2df81af530 con 0x560ac8f3b2f0 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2e06a58740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2df81ac770 con 0x560ac8f641b0 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2e06a58740 10 client.0 _ll_register_callbacks cb 0x7f2df0009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f434d0 0x560ac931b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f434d0 0x560ac931b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43860/0 (socket says 172.21.15.29:43860) 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 -- 172.21.15.29:0/1807370367 learned_addr learned my addr 172.21.15.29:0/1807370367 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 0x560ac931a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfdfab640 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 0x560ac931bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 -- 172.21.15.29:0/1807370367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 msgr2=0x560ac931bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 0x560ac931bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 -- 172.21.15.29:0/1807370367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 msgr2=0x560ac931a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 0x560ac931a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 -- 172.21.15.29:0/1807370367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ac931c2e0 con 0x560ac8f434d0 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2dfcfa9640 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f434d0 0x560ac931b010 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2de80038a0 tx=0x7f2de8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab6b267b61d344e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.451+0000 7f2de0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2de0ff9640 1 -- 172.21.15.29:0/1807370367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de800ce70 con 0x560ac8f434d0 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2de0ff9640 1 -- 172.21.15.29:0/1807370367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2de801e620 con 0x560ac8f434d0 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 1 -- 172.21.15.29:0/1807370367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2df0036040 con 0x560ac8f434d0 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2de0ff9640 1 -- 172.21.15.29:0/1807370367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2de801e820 con 0x560ac8f434d0 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 10 client.16974 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 1 -- 172.21.15.29:0/1807370367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560ac931d640 con 0x560ac8f434d0 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2de0ff9640 1 -- 172.21.15.29:0/1807370367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2df0036040 con 0x560ac8f434d0 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2de0ff9640 1 -- 172.21.15.29:0/1807370367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2de80299e0 con 0x560ac8f434d0 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 10 client.16974 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 10 client.16974 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 20 client.16974 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560ac8ef81c0 0x560ac938fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 4 client.16974 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.452+0000 7f2e06a58740 1 -- 172.21.15.29:0/1807370367 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560ac927c0f0 con 0x560ac8ef81c0 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.453+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560ac8ef81c0 0x560ac938fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.453+0000 7f2de0ff9640 10 client.16974 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.453+0000 7f2dfd7aa640 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560ac8ef81c0 0x560ac938fc20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83d189e4d4241c4e server_cookie=89b81dc9067a9b8 in_seq=0 out_seq=0 2024-09-21T02:49:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.467+0000 7f2de0ff9640 1 -- 172.21.15.29:0/1807370367 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560ac927c0f0 con 0x560ac8ef81c0 2024-09-21T02:49:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.467+0000 7f2de0ff9640 10 client.16974 handle_command_reply: tid=0 2024-09-21T02:49:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.467+0000 7f2e06a58740 1 client.16974 shutdown 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 -- 172.21.15.29:0/1807370367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f434d0 msgr2=0x560ac931b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f434d0 0x560ac931b010 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2de80038a0 tx=0x7f2de8004730 comp rx=0 tx=0).stop 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 -- 172.21.15.29:0/1807370367 shutdown_connections 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ac8f641b0 0x560ac931bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560ac8ef81c0 0x560ac938fc20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ac8f434d0 0x560ac931b010 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2de0ff9640 0 client.16974 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 --2- 172.21.15.29:0/1807370367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560ac8f3b2f0 0x560ac931a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 -- 172.21.15.29:0/1807370367 >> 172.21.15.29:0/1807370367 conn(0x560ac8f58300 msgr2=0x560ac9313770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 -- 172.21.15.29:0/1807370367 shutdown_connections 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 1 -- 172.21.15.29:0/1807370367 wait complete. 2024-09-21T02:49:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.468+0000 7f2e06a58740 20 client.16974 trim_cache size 0 max 16384 2024-09-21T02:49:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:49:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:49:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:49:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:49:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 -- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2de40b6590 msgr2=0x7f2de40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2de40b6590 0x7f2de40b8aa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2de800ccf0 tx=0x7f2de80046c0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 -- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 msgr2=0x7f2df819d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 0x7f2df819d2e0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2df0005e30 tx=0x7f2df0003ba0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 -- 172.21.15.29:0/673641682 shutdown_connections 2024-09-21T02:49:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2de40b6590 0x7f2de40b8aa0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df810b660 0x7f2df819d2e0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80770f0 0x7f2df819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 --2- 172.21.15.29:0/673641682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df8075cf0 0x7f2df819c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 -- 172.21.15.29:0/673641682 >> 172.21.15.29:0/673641682 conn(0x7f2df80ff030 msgr2=0x7f2df8107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 -- 172.21.15.29:0/673641682 shutdown_connections 2024-09-21T02:49:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:37.469+0000 7f2dffa0e640 1 -- 172.21.15.29:0/673641682 wait complete. 2024-09-21T02:49:38.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.11 MB/sec execute 435 sec latency 15177.852 ms 2024-09-21T02:49:38.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:37 smithi190 ceph-mon[33871]: pgmap v704: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 114 KiB/s wr, 28 op/s 2024-09-21T02:49:38.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:37 smithi029 ceph-mon[31006]: pgmap v704: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 114 KiB/s wr, 28 op/s 2024-09-21T02:49:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:37 smithi066 ceph-mon[33436]: pgmap v704: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 114 KiB/s wr, 28 op/s 2024-09-21T02:49:39.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.11 MB/sec execute 436 sec latency 16177.976 ms 2024-09-21T02:49:39.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:38 smithi190 ceph-mon[33871]: pgmap v705: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 22 op/s 2024-09-21T02:49:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:38 smithi029 ceph-mon[31006]: pgmap v705: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 22 op/s 2024-09-21T02:49:39.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:38 smithi066 ceph-mon[33436]: pgmap v705: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 22 op/s 2024-09-21T02:49:40.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.10 MB/sec execute 437 sec latency 17178.027 ms 2024-09-21T02:49:41.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.10 MB/sec execute 438 sec latency 18178.174 ms 2024-09-21T02:49:41.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:41 smithi066 ceph-mon[33436]: pgmap v706: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 18 op/s 2024-09-21T02:49:41.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:41 smithi190 ceph-mon[33871]: pgmap v706: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 18 op/s 2024-09-21T02:49:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:41 smithi029 ceph-mon[31006]: pgmap v706: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 18 op/s 2024-09-21T02:49:42.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.09 MB/sec execute 439 sec latency 19178.306 ms 2024-09-21T02:49:43.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.09 MB/sec execute 440 sec latency 20178.343 ms 2024-09-21T02:49:43.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:43 smithi066 ceph-mon[33436]: pgmap v707: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.1 KiB/s wr, 13 op/s 2024-09-21T02:49:43.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:43 smithi190 ceph-mon[33871]: pgmap v707: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.1 KiB/s wr, 13 op/s 2024-09-21T02:49:43.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:43 smithi029 ceph-mon[31006]: pgmap v707: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.1 KiB/s wr, 13 op/s 2024-09-21T02:49:44.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.08 MB/sec execute 441 sec latency 21178.437 ms 2024-09-21T02:49:44.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:49:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:49:45.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.08 MB/sec execute 442 sec latency 22178.485 ms 2024-09-21T02:49:45.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 -- 172.21.15.29:0/3859296319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c108cf0 msgr2=0x7fa95c10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:45.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 --2- 172.21.15.29:0/3859296319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c108cf0 0x7fa95c10f520 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa954009ca0 tx=0x7fa95402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 -- 172.21.15.29:0/3859296319 shutdown_connections 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 --2- 172.21.15.29:0/3859296319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c108cf0 0x7fa95c10f520 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 --2- 172.21.15.29:0/3859296319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa95c1083a0 0x7fa95c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 --2- 172.21.15.29:0/3859296319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa95c102a60 0x7fa95c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 -- 172.21.15.29:0/3859296319 >> 172.21.15.29:0/3859296319 conn(0x7fa95c079730 msgr2=0x7fa95c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 -- 172.21.15.29:0/3859296319 shutdown_connections 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.140+0000 7fa963242640 1 -- 172.21.15.29:0/3859296319 wait complete. 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.141+0000 7fa963242640 1 Processor -- start 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.141+0000 7fa963242640 1 -- start start 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa963242640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c102a60 0x7fa95c111bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa963242640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa95c1083a0 0x7fa95c112110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa963242640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa95c108cf0 0x7fa95c112650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa963242640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c06bc10 con 0x7fa95c102a60 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa963242640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c06ba90 con 0x7fa95c108cf0 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa963242640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c06bd90 con 0x7fa95c1083a0 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c102a60 0x7fa95c111bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c102a60 0x7fa95c111bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34160/0 (socket says 172.21.15.29:34160) 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 -- 172.21.15.29:0/3166283358 learned_addr learned my addr 172.21.15.29:0/3166283358 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa95bfff640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa95c1083a0 0x7fa95c112110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa9617df640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa95c108cf0 0x7fa95c112650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 -- 172.21.15.29:0/3166283358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa95c1083a0 msgr2=0x7fa95c112110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa95c1083a0 0x7fa95c112110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 -- 172.21.15.29:0/3166283358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa95c108cf0 msgr2=0x7fa95c112650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa95c108cf0 0x7fa95c112650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 -- 172.21.15.29:0/3166283358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c1a8690 con 0x7fa95c102a60 2024-09-21T02:49:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa960fde640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c102a60 0x7fa95c111bd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00d950 tx=0x7fa94c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.142+0000 7fa959ffb640 1 -- 172.21.15.29:0/3166283358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c017030 con 0x7fa95c102a60 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.143+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa95c1a8980 con 0x7fa95c102a60 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.143+0000 7fa959ffb640 1 -- 172.21.15.29:0/3166283358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa94c004020 con 0x7fa95c102a60 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.143+0000 7fa959ffb640 1 -- 172.21.15.29:0/3166283358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c0055d0 con 0x7fa95c102a60 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.143+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa95c1a8f10 con 0x7fa95c102a60 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.144+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa95c102e70 con 0x7fa95c102a60 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.145+0000 7fa959ffb640 1 -- 172.21.15.29:0/3166283358 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fa94c009a30 con 0x7fa95c102a60 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.146+0000 7fa959ffb640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9440b6510 0x7fa9440b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.146+0000 7fa95bfff640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9440b6510 0x7fa9440b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.146+0000 7fa959ffb640 1 -- 172.21.15.29:0/3166283358 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa94c0e70c0 con 0x7fa95c102a60 2024-09-21T02:49:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.147+0000 7fa95bfff640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9440b6510 0x7fa9440b8a20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa95c105dd0 tx=0x7fa95000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.150+0000 7fa959ffb640 1 -- 172.21.15.29:0/3166283358 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa94c0ad570 con 0x7fa95c102a60 2024-09-21T02:49:45.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:45 smithi029 ceph-mon[31006]: pgmap v708: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-09-21T02:49:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.378+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa95c02e640 con 0x7fa95c102a60 2024-09-21T02:49:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.378+0000 7fa959ffb640 1 -- 172.21.15.29:0/3166283358 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7fa94c0b1950 con 0x7fa95c102a60 2024-09-21T02:49:45.380 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:49:45.380 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:49:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:49:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.380+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9440b6510 msgr2=0x7fa9440b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.380+0000 7fa963242640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9440b6510 0x7fa9440b8a20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa95c105dd0 tx=0x7fa95000b390 comp rx=0 tx=0).stop 2024-09-21T02:49:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c102a60 msgr2=0x7fa95c111bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c102a60 0x7fa95c111bd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00d950 tx=0x7fa94c00de10 comp rx=0 tx=0).stop 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 shutdown_connections 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9440b6510 0x7fa9440b8a20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa95c108cf0 0x7fa95c112650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa95c1083a0 0x7fa95c112110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 --2- 172.21.15.29:0/3166283358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa95c102a60 0x7fa95c111bd0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 >> 172.21.15.29:0/3166283358 conn(0x7fa95c079730 msgr2=0x7fa95c104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 shutdown_connections 2024-09-21T02:49:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:45.381+0000 7fa963242640 1 -- 172.21.15.29:0/3166283358 wait complete. 2024-09-21T02:49:45.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:45 smithi066 ceph-mon[33436]: pgmap v708: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-09-21T02:49:45.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:45 smithi190 ceph-mon[33871]: pgmap v708: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-09-21T02:49:46.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200514 2.07 MB/sec execute 443 sec latency 23178.516 ms 2024-09-21T02:49:46.221 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id c8aeb95 --release --await' 2024-09-21T02:49:46.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id c8aeb95 --release --await 2024-09-21T02:49:46.488 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:49:46.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:46 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3166283358' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:46.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:46 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3166283358' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:46.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:46 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3166283358' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:49:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/3020717307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28102000 msgr2=0x7f6a28102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/3020717307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28102000 0x7f6a28102410 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009900 tx=0x7f6a1802a410 comp rx=0 tx=0).stop 2024-09-21T02:49:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/3020717307 shutdown_connections 2024-09-21T02:49:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/3020717307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a28106c30 0x7f6a28110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/3020717307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a281062e0 0x7f6a281066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/3020717307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28102000 0x7f6a28102410 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/3020717307 >> 172.21.15.29:0/3020717307 conn(0x7f6a28079e40 msgr2=0x7f6a2807a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/3020717307 shutdown_connections 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.002+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/3020717307 wait complete. 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2e5bb640 1 Processor -- start 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2e5bb640 1 -- start start 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2e5bb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a28102000 0x7f6a2819c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2e5bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a281062e0 0x7f6a2819cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2e5bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28106c30 0x7f6a281a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2e5bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6a28113980 con 0x7f6a28106c30 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2e5bb640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6a28113800 con 0x7f6a28102000 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2e5bb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6a28113b00 con 0x7f6a281062e0 2024-09-21T02:49:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2cb58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28106c30 0x7f6a281a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2cb58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28106c30 0x7f6a281a1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34168/0 (socket says 172.21.15.29:34168) 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a2cb58640 1 -- 172.21.15.29:0/699747002 learned_addr learned my addr 172.21.15.29:0/699747002 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a27fff640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a28102000 0x7f6a2819c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.003+0000 7f6a277fe640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a281062e0 0x7f6a2819cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a2cb58640 1 -- 172.21.15.29:0/699747002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a281062e0 msgr2=0x7f6a2819cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a281062e0 0x7f6a2819cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a2cb58640 1 -- 172.21.15.29:0/699747002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a28102000 msgr2=0x7f6a2819c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a28102000 0x7f6a2819c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a2cb58640 1 -- 172.21.15.29:0/699747002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a281a2520 con 0x7f6a28106c30 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a27fff640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a28102000 0x7f6a2819c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a277fe640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a281062e0 0x7f6a2819cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:49:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28106c30 0x7f6a281a1df0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000cad0 tx=0x7f6a2000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a257fa640 1 -- 172.21.15.29:0/699747002 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2000ae20 con 0x7f6a28106c30 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/699747002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a281a2810 con 0x7f6a28106c30 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a257fa640 1 -- 172.21.15.29:0/699747002 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a20005020 con 0x7f6a28106c30 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a257fa640 1 -- 172.21.15.29:0/699747002 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2000f6b0 con 0x7f6a28106c30 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.004+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/699747002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a281a8f70 con 0x7f6a28106c30 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a34b2f740 1 Processor -- start 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a257fa640 1 -- 172.21.15.29:0/699747002 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f6a200051d0 con 0x7f6a28106c30 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a34b2f740 1 -- start start 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7efa5520 0x556f7f34b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7ef7c660 0x556f7f34ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 0x556f7f34bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556f7f236dc0 con 0x556f7ef7c660 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556f7f2c31f0 con 0x556f7efa5520 2024-09-21T02:49:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556f7f2b2620 con 0x556f7ef84840 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7ef7c660 0x556f7f34ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7ef7c660 0x556f7f34ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34176/0 (socket says 172.21.15.29:34176) 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.007+0000 7f6a27fff640 1 -- 172.21.15.29:0/71639340 learned_addr learned my addr 172.21.15.29:0/71639340 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a277fe640 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7efa5520 0x556f7f34b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a277fe640 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 0x556f7f34bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a27fff640 1 -- 172.21.15.29:0/71639340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 msgr2=0x556f7f34bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a27fff640 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 0x556f7f34bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a27fff640 1 -- 172.21.15.29:0/71639340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7efa5520 msgr2=0x556f7f34b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a27fff640 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7efa5520 0x556f7f34b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a257fa640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a100b6510 0x7f6a100b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a27fff640 1 -- 172.21.15.29:0/71639340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f7f27afd0 con 0x556f7ef7c660 2024-09-21T02:49:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a277fe640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a100b6510 0x7f6a100b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a257fa640 1 -- 172.21.15.29:0/699747002 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6a200e7a60 con 0x7f6a28106c30 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a27fff640 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7ef7c660 0x556f7f34ba50 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6a18000c00 tx=0x7f6a18032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2814166fdd4a3723 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a0e7fc640 1 -- 172.21.15.29:0/71639340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1803f070 con 0x556f7ef7c660 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a0e7fc640 1 -- 172.21.15.29:0/71639340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a180044c0 con 0x556f7ef7c660 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.008+0000 7f6a0e7fc640 1 -- 172.21.15.29:0/71639340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a18035070 con 0x556f7ef7c660 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 -- 172.21.15.29:0/71639340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7ef7c660 msgr2=0x556f7f34ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a277fe640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a100b6510 0x7f6a100b8a20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c005d00 tx=0x7f6a1c015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7ef7c660 0x556f7f34ba50 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6a18000c00 tx=0x7f6a18032dc0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 -- 172.21.15.29:0/71639340 shutdown_connections 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 0x556f7f34bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7ef7c660 0x556f7f34ba50 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/71639340 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7efa5520 0x556f7f34b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 -- 172.21.15.29:0/71639340 >> 172.21.15.29:0/71639340 conn(0x556f7ef99910 msgr2=0x556f7f349400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 -- 172.21.15.29:0/71639340 shutdown_connections 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.009+0000 7f6a34b2f740 1 -- 172.21.15.29:0/71639340 wait complete. 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 1 -- start start 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 1 Processor -- start 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f356bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f3570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f357670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556f7f24d300 con 0x556f7efa5520 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556f7f2b6430 con 0x556f7ef84840 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556f7f113bb0 con 0x556f7ef7c660 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.010+0000 7f6a34b2f740 10 client.0 _ll_register_callbacks cb 0x556f7f267120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f357670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f357670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34178/0 (socket says 172.21.15.29:34178) 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 -- 172.21.15.29:0/3872105843 learned_addr learned my addr 172.21.15.29:0/3872105843 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a27fff640 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f356bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a277fe640 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f3570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 -- 172.21.15.29:0/3872105843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 msgr2=0x556f7f356bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f356bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 -- 172.21.15.29:0/3872105843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 msgr2=0x556f7f3570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f3570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 -- 172.21.15.29:0/3872105843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f7f23c780 con 0x556f7efa5520 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a277fe640 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f3570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f357670 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6a200ad800 tx=0x7f6a200ad110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ecbbbc667fca52f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a077fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a077fe640 1 -- 172.21.15.29:0/3872105843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2001d070 con 0x556f7efa5520 2024-09-21T02:49:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a077fe640 1 -- 172.21.15.29:0/3872105843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6a200a91c0 con 0x556f7efa5520 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3872105843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f7f2c5140 con 0x556f7efa5520 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a077fe640 1 -- 172.21.15.29:0/3872105843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6a200b2350 con 0x556f7efa5520 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a34b2f740 10 client.16992 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.011+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3872105843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556f7f010b00 con 0x556f7efa5520 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a077fe640 1 -- 172.21.15.29:0/3872105843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x556f7f2c5140 con 0x556f7efa5520 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a077fe640 1 -- 172.21.15.29:0/3872105843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6a200aec30 con 0x556f7efa5520 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a34b2f740 10 client.16992 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a34b2f740 10 client.16992 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a34b2f740 20 client.16992 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556f7ef63ea0 0x556f7f368e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a34b2f740 4 client.16992 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3872105843 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f7f2c0bf0 con 0x556f7ef63ea0 2024-09-21T02:49:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.012+0000 7f6a27fff640 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556f7ef63ea0 0x556f7f368e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.013+0000 7f6a077fe640 10 client.16992 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.013+0000 7f6a27fff640 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556f7ef63ea0 0x556f7f368e40 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=318fee427a879cdc server_cookie=9cafe2114b88b9d3 in_seq=0 out_seq=0 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.013+0000 7f6a077fe640 1 -- 172.21.15.29:0/3872105843 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556f7f2c0bf0 con 0x556f7ef63ea0 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.013+0000 7f6a077fe640 10 client.16992 handle_command_reply: tid=0 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.013+0000 7f6a34b2f740 1 client.16992 shutdown 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3872105843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 msgr2=0x556f7f357670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f357670 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6a200ad800 tx=0x7f6a200ad110 comp rx=0 tx=0).stop 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3872105843 shutdown_connections 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556f7ef63ea0 0x556f7f368e40 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f357670 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f3570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3872105843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f356bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3872105843 >> 172.21.15.29:0/3872105843 conn(0x556f7ef99910 msgr2=0x556f7f34e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a077fe640 0 client.16992 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3872105843 shutdown_connections 2024-09-21T02:49:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3872105843 wait complete. 2024-09-21T02:49:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.014+0000 7f6a34b2f740 20 client.16992 trim_cache size 0 max 16384 2024-09-21T02:49:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.031+0000 7f6a34b2f740 1 Processor -- start 2024-09-21T02:49:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.031+0000 7f6a34b2f740 1 -- start start 2024-09-21T02:49:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.031+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef7c660 0x556f7f34c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.031+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 0x556f7f353da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.031+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f3542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.031+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6a281b2c00 con 0x556f7efa5520 2024-09-21T02:49:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.031+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6a281b4740 con 0x556f7ef7c660 2024-09-21T02:49:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.031+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6a281b32d0 con 0x556f7ef84840 2024-09-21T02:49:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f3542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f3542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34202/0 (socket says 172.21.15.29:34202) 2024-09-21T02:49:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 -- 172.21.15.29:0/3783065616 learned_addr learned my addr 172.21.15.29:0/3783065616 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a277fe640 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 0x556f7f353da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a27fff640 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef7c660 0x556f7f34c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 -- 172.21.15.29:0/3783065616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 msgr2=0x556f7f353da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 0x556f7f353da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 -- 172.21.15.29:0/3783065616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef7c660 msgr2=0x556f7f34c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:47.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef7c660 0x556f7f34c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 -- 172.21.15.29:0/3783065616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f7f34d450 con 0x556f7efa5520 2024-09-21T02:49:47.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f3542e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6a200e5550 tx=0x7f6a200a9790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8d3a5d9bb31237 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:47.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a0dffb640 1 -- 172.21.15.29:0/3783065616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2001d040 con 0x556f7efa5520 2024-09-21T02:49:47.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a0dffb640 1 -- 172.21.15.29:0/3783065616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a200b3890 con 0x556f7efa5520 2024-09-21T02:49:47.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.032+0000 7f6a0dffb640 1 -- 172.21.15.29:0/3783065616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a200aedc0 con 0x556f7efa5520 2024-09-21T02:49:47.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3783065616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 msgr2=0x556f7f3542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f3542e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6a200e5550 tx=0x7f6a200a9790 comp rx=0 tx=0).stop 2024-09-21T02:49:47.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3783065616 shutdown_connections 2024-09-21T02:49:47.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f3542e0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef84840 0x556f7f353da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/3783065616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef7c660 0x556f7f34c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3783065616 >> 172.21.15.29:0/3783065616 conn(0x556f7ef99910 msgr2=0x556f7f3626e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3783065616 shutdown_connections 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.033+0000 7f6a34b2f740 1 -- 172.21.15.29:0/3783065616 wait complete. 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 1 -- start start 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 1 Processor -- start 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f35ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f35b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f35b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6a281b4070 con 0x556f7efa5520 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6a281b39a0 con 0x556f7ef84840 2024-09-21T02:49:47.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6a281b0be0 con 0x556f7ef7c660 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a34b2f740 10 client.0 _ll_register_callbacks cb 0x7f6a20007780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a27fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f35ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a2cb58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f35b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a27fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f35ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41086/0 (socket says 172.21.15.29:41086) 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a27fff640 1 -- 172.21.15.29:0/2901193084 learned_addr learned my addr 172.21.15.29:0/2901193084 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.034+0000 7f6a277fe640 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f35b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a2cb58640 1 -- 172.21.15.29:0/2901193084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 msgr2=0x556f7f35ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f35ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a2cb58640 1 -- 172.21.15.29:0/2901193084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 msgr2=0x556f7f35b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f35b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a2cb58640 1 -- 172.21.15.29:0/2901193084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f7f087960 con 0x556f7efa5520 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a277fe640 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f35b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:49:47.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a2cb58640 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f35b7f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6a20004ff0 tx=0x7f6a20005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e093af807f2ce6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a077fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a077fe640 1 -- 172.21.15.29:0/2901193084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2001d070 con 0x556f7efa5520 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a077fe640 1 -- 172.21.15.29:0/2901193084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6a200ac540 con 0x556f7efa5520 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a34b2f740 1 -- 172.21.15.29:0/2901193084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f7f2be020 con 0x556f7efa5520 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a077fe640 1 -- 172.21.15.29:0/2901193084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6a200b34e0 con 0x556f7efa5520 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a34b2f740 10 client.16998 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.035+0000 7f6a34b2f740 1 -- 172.21.15.29:0/2901193084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556f7f3d38f0 con 0x556f7efa5520 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a077fe640 1 -- 172.21.15.29:0/2901193084 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x556f7f2be020 con 0x556f7efa5520 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a077fe640 1 -- 172.21.15.29:0/2901193084 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6a200d4020 con 0x556f7efa5520 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a34b2f740 10 client.16998 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a34b2f740 10 client.16998 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a34b2f740 20 client.16998 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556f7ef39250 0x556f7f3d5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:47.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a34b2f740 4 client.16998 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c8aeb95", "release": true, "await": true}] 2024-09-21T02:49:47.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a34b2f740 1 -- 172.21.15.29:0/2901193084 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c8aeb95", "release": true, "await": true}) -- 0x556f7f233ec0 con 0x556f7ef39250 2024-09-21T02:49:47.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a27fff640 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556f7ef39250 0x556f7f3d5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:47.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a27fff640 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556f7ef39250 0x556f7f3d5dd0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de271dc43b8d9caa server_cookie=796d1db96fc248eb in_seq=0 out_seq=0 2024-09-21T02:49:47.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.036+0000 7f6a077fe640 10 client.16998 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:47.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.046+0000 7f6a077fe640 1 -- 172.21.15.29:0/2901193084 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x556f7f233ec0 con 0x556f7ef39250 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.046+0000 7f6a077fe640 10 client.16998 handle_command_reply: tid=0 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 client.16998 shutdown 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 -- 172.21.15.29:0/2901193084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 msgr2=0x556f7f35b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f35b7f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6a20004ff0 tx=0x7f6a20005b20 comp rx=0 tx=0).stop 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 -- 172.21.15.29:0/2901193084 shutdown_connections 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556f7ef39250 0x556f7f3d5dd0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f7efa5520 0x556f7f35b7f0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556f7ef84840 0x556f7f35b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 --2- 172.21.15.29:0/2901193084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f7ef7c660 0x556f7f35ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a077fe640 0 client.16998 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:47.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 -- 172.21.15.29:0/2901193084 >> 172.21.15.29:0/2901193084 conn(0x556f7ef99910 msgr2=0x556f7f355430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:47.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 -- 172.21.15.29:0/2901193084 shutdown_connections 2024-09-21T02:49:47.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 1 -- 172.21.15.29:0/2901193084 wait complete. 2024-09-21T02:49:47.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.047+0000 7f6a34b2f740 20 client.16998 trim_cache size 0 max 16384 2024-09-21T02:49:47.050 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:49:47.050 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:49:47.050 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:49:47.050 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 44, 2024-09-21T02:49:47.050 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "c8aeb95": { 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 44, 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 144.5, 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:49:47.051 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:49:47.052 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:49:47.052 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:49:47.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:47.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:47.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:47.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:47.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:49:47.052 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a100b6510 msgr2=0x7f6a100b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a100b6510 0x7f6a100b8a20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c005d00 tx=0x7f6a1c015040 comp rx=0 tx=0).stop 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28106c30 msgr2=0x7f6a281a1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28106c30 0x7f6a281a1df0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000cad0 tx=0x7f6a2000cf90 comp rx=0 tx=0).stop 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/699747002 shutdown_connections 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6a100b6510 0x7f6a100b8a20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a28106c30 0x7f6a281a1df0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a281062e0 0x7f6a2819cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 --2- 172.21.15.29:0/699747002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a28102000 0x7f6a2819c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/699747002 >> 172.21.15.29:0/699747002 conn(0x7f6a28079e40 msgr2=0x7f6a28107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/699747002 shutdown_connections 2024-09-21T02:49:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:49:47.048+0000 7f6a2e5bb640 1 -- 172.21.15.29:0/699747002 wait complete. 2024-09-21T02:49:47.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200548 2.07 MB/sec execute 444 sec latency 24130.658 ms 2024-09-21T02:49:47.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:47 smithi029 ceph-mon[31006]: pgmap v709: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-09-21T02:49:47.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:47 smithi066 ceph-mon[33436]: pgmap v709: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-09-21T02:49:47.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:47 smithi190 ceph-mon[33871]: pgmap v709: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-09-21T02:49:48.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 201111 2.07 MB/sec execute 445 sec latency 70.280 ms 2024-09-21T02:49:48.764 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c8aeb95', total seconds elapsed: 28.9 2024-09-21T02:49:49.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 201927 2.08 MB/sec execute 446 sec latency 82.174 ms 2024-09-21T02:49:49.509 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:49 smithi066 ceph-mon[33436]: pgmap v710: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-09-21T02:49:49.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:49 smithi190 ceph-mon[33871]: pgmap v710: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-09-21T02:49:49.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:49 smithi029 ceph-mon[31006]: pgmap v710: 129 pgs: 129 active+clean; 38 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-09-21T02:49:50.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 202416 2.08 MB/sec execute 447 sec latency 17.898 ms 2024-09-21T02:49:50.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.575+0000 7f53907e4640 1 -- 172.21.15.29:0/2028160531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53881036a0 msgr2=0x7f5388103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.575+0000 7f53907e4640 1 --2- 172.21.15.29:0/2028160531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53881036a0 0x7f5388103b10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5384009900 tx=0x7f538402a410 comp rx=0 tx=0).stop 2024-09-21T02:49:50.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.575+0000 7f53907e4640 1 -- 172.21.15.29:0/2028160531 shutdown_connections 2024-09-21T02:49:50.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.575+0000 7f53907e4640 1 --2- 172.21.15.29:0/2028160531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53881050c0 0x7f538810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.575+0000 7f53907e4640 1 --2- 172.21.15.29:0/2028160531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5388104720 0x7f5388104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.575+0000 7f53907e4640 1 --2- 172.21.15.29:0/2028160531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53881036a0 0x7f5388103b10 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.575+0000 7f53907e4640 1 -- 172.21.15.29:0/2028160531 >> 172.21.15.29:0/2028160531 conn(0x7f53880fefe0 msgr2=0x7f5388101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:50.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.575+0000 7f53907e4640 1 -- 172.21.15.29:0/2028160531 shutdown_connections 2024-09-21T02:49:50.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.576+0000 7f53907e4640 1 -- 172.21.15.29:0/2028160531 wait complete. 2024-09-21T02:49:50.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.576+0000 7f53907e4640 1 Processor -- start 2024-09-21T02:49:50.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.576+0000 7f53907e4640 1 -- start start 2024-09-21T02:49:50.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f53907e4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53881036a0 0x7f5388199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f53907e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388104720 0x7f538819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f53907e4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53881050c0 0x7f538819b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f53907e4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f538810f190 con 0x7f5388104720 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f53907e4640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f538810f010 con 0x7f53881050c0 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f53907e4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f538810f310 con 0x7f53881036a0 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538dd7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388104720 0x7f538819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538e580640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53881036a0 0x7f5388199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538dd7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388104720 0x7f538819a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34234/0 (socket says 172.21.15.29:34234) 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538dd7f640 1 -- 172.21.15.29:0/4200204737 learned_addr learned my addr 172.21.15.29:0/4200204737 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538ed81640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53881050c0 0x7f538819b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538e580640 1 -- 172.21.15.29:0/4200204737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53881050c0 msgr2=0x7f538819b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538e580640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53881050c0 0x7f538819b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538e580640 1 -- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388104720 msgr2=0x7f538819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538e580640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388104720 0x7f538819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538e580640 1 -- 172.21.15.29:0/4200204737 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538819b890 con 0x7f53881036a0 2024-09-21T02:49:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538dd7f640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388104720 0x7f538819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:49:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.577+0000 7f538e580640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53881036a0 0x7f5388199e50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5384005760 tx=0x7f5384004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.578+0000 7f537f7fe640 1 -- 172.21.15.29:0/4200204737 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538403f070 con 0x7f53881036a0 2024-09-21T02:49:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.578+0000 7f53907e4640 1 -- 172.21.15.29:0/4200204737 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538819bb20 con 0x7f53881036a0 2024-09-21T02:49:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.578+0000 7f537f7fe640 1 -- 172.21.15.29:0/4200204737 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5384032e20 con 0x7f53881036a0 2024-09-21T02:49:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.578+0000 7f537f7fe640 1 -- 172.21.15.29:0/4200204737 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384035070 con 0x7f53881036a0 2024-09-21T02:49:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.578+0000 7f53907e4640 1 -- 172.21.15.29:0/4200204737 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538819be30 con 0x7f53881036a0 2024-09-21T02:49:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.580+0000 7f5397833740 1 Processor -- start 2024-09-21T02:49:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.580+0000 7f5397833740 1 -- start start 2024-09-21T02:49:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f5397833740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e3540260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f5397833740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 0x55e7e35408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f5397833740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3541420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f5397833740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e7e34ae6f0 con 0x55e7e31734d0 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f5397833740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e7e349db20 con 0x55e7e316b2f0 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f5397833740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e7e34a1930 con 0x55e7e31941b0 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f537f7fe640 1 -- 172.21.15.29:0/4200204737 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f53840037f0 con 0x7f53881036a0 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e3540260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e3540260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41106/0 (socket says 172.21.15.29:41106) 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 -- 172.21.15.29:0/2017214706 learned_addr learned my addr 172.21.15.29:0/2017214706 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538dd7f640 1 --2- 172.21.15.29:0/2017214706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3541420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 -- 172.21.15.29:0/2017214706 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 msgr2=0x55e7e35408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 --2- 172.21.15.29:0/2017214706 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 0x55e7e35408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 -- 172.21.15.29:0/2017214706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 msgr2=0x55e7e3541420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 --2- 172.21.15.29:0/2017214706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3541420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 -- 172.21.15.29:0/2017214706 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7e3463530 con 0x55e7e31941b0 2024-09-21T02:49:50.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.581+0000 7f538ed81640 1 --2- 172.21.15.29:0/2017214706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e3540260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5380009fd0 tx=0x7f53800118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81b5b172f8553330 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:50.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.582+0000 7f537cff9640 1 -- 172.21.15.29:0/2017214706 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380004d20 con 0x55e7e31941b0 2024-09-21T02:49:50.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.582+0000 7f537cff9640 1 -- 172.21.15.29:0/2017214706 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5380004050 con 0x55e7e31941b0 2024-09-21T02:49:50.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.582+0000 7f537cff9640 1 -- 172.21.15.29:0/2017214706 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380005220 con 0x55e7e31941b0 2024-09-21T02:49:50.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.582+0000 7f537f7fe640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53740b6510 0x7f53740b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.582+0000 7f538e580640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53740b6510 0x7f53740b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.582+0000 7f537f7fe640 1 -- 172.21.15.29:0/4200204737 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5384107d00 con 0x7f53881036a0 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 -- 172.21.15.29:0/2017214706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 msgr2=0x55e7e3540260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f538e580640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53740b6510 0x7f53740b8a20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f53840cdb50 tx=0x7f5384097660 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 --2- 172.21.15.29:0/2017214706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e3540260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5380009fd0 tx=0x7f53800118d0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 -- 172.21.15.29:0/2017214706 shutdown_connections 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 --2- 172.21.15.29:0/2017214706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3541420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 --2- 172.21.15.29:0/2017214706 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 0x55e7e35408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 --2- 172.21.15.29:0/2017214706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e3540260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 -- 172.21.15.29:0/2017214706 >> 172.21.15.29:0/2017214706 conn(0x55e7e3188300 msgr2=0x55e7e35378e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 -- 172.21.15.29:0/2017214706 shutdown_connections 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 -- 172.21.15.29:0/2017214706 wait complete. 2024-09-21T02:49:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 -- start start 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.583+0000 7f5397833740 1 Processor -- start 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5397833740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 0x55e7e3549a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5397833740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e354a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5397833740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e354ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5397833740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e7e33d5ab0 con 0x55e7e31734d0 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5397833740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e7e3443730 con 0x55e7e316b2f0 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5397833740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e7e3541f40 con 0x55e7e31941b0 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5397833740 10 client.0 _ll_register_callbacks cb 0x55e7e34976b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e354a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e354a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34254/0 (socket says 172.21.15.29:34254) 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 -- 172.21.15.29:0/1185476871 learned_addr learned my addr 172.21.15.29:0/1185476871 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538ed81640 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e354ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538ed81640 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 0x55e7e3549a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 -- 172.21.15.29:0/1185476871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 msgr2=0x55e7e354ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e354ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 -- 172.21.15.29:0/1185476871 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 msgr2=0x55e7e3549a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 0x55e7e3549a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 -- 172.21.15.29:0/1185476871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7e31ff790 con 0x55e7e31734d0 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f538dd7f640 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e354a100 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5378010f10 tx=0x7f537800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dd382b63b0da3b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5371ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5371ffb640 1 -- 172.21.15.29:0/1185476871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537801e060 con 0x55e7e31734d0 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5371ffb640 1 -- 172.21.15.29:0/1185476871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5378019410 con 0x55e7e31734d0 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.584+0000 7f5397833740 1 -- 172.21.15.29:0/1185476871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5380004050 con 0x55e7e31734d0 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5371ffb640 1 -- 172.21.15.29:0/1185476871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5378025050 con 0x55e7e31734d0 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5397833740 10 client.17010 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5397833740 1 -- 172.21.15.29:0/1185476871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e7e3278620 con 0x55e7e31734d0 2024-09-21T02:49:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5371ffb640 1 -- 172.21.15.29:0/1185476871 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5380004050 con 0x55e7e31734d0 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5371ffb640 1 -- 172.21.15.29:0/1185476871 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f5378018390 con 0x55e7e31734d0 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5397833740 10 client.17010 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5397833740 10 client.17010 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5397833740 20 client.17010 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5397833740 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e7e3152d80 0x55e7e35571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5397833740 4 client.17010 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f5397833740 1 -- 172.21.15.29:0/1185476871 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7e34b0640 con 0x55e7e3152d80 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.585+0000 7f538ed81640 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e7e3152d80 0x55e7e35571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.586+0000 7f5371ffb640 10 client.17010 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.586+0000 7f538ed81640 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e7e3152d80 0x55e7e35571d0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d409dedc5803985 server_cookie=8b579f1a9dc0e7a2 in_seq=0 out_seq=0 2024-09-21T02:49:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.586+0000 7f5371ffb640 1 -- 172.21.15.29:0/1185476871 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e7e34b0640 con 0x55e7e3152d80 2024-09-21T02:49:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.586+0000 7f5371ffb640 10 client.17010 handle_command_reply: tid=0 2024-09-21T02:49:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 client.17010 shutdown 2024-09-21T02:49:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 -- 172.21.15.29:0/1185476871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 msgr2=0x55e7e354a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e354a100 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5378010f10 tx=0x7f537800f7f0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 -- 172.21.15.29:0/1185476871 shutdown_connections 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31941b0 0x55e7e354ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e7e3152d80 0x55e7e35571d0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e354a100 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 --2- 172.21.15.29:0/1185476871 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e316b2f0 0x55e7e3549a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 -- 172.21.15.29:0/1185476871 >> 172.21.15.29:0/1185476871 conn(0x55e7e3188300 msgr2=0x55e7e3535ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5371ffb640 0 client.17010 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 -- 172.21.15.29:0/1185476871 shutdown_connections 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 1 -- 172.21.15.29:0/1185476871 wait complete. 2024-09-21T02:49:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.587+0000 7f5397833740 20 client.17010 trim_cache size 0 max 16384 2024-09-21T02:49:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f5397833740 1 Processor -- start 2024-09-21T02:49:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f5397833740 1 -- start start 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f5397833740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e316b2f0 0x55e7e3541260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f5397833740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31734d0 0x55e7e35325c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f5397833740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3547f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f5397833740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53881aa2b0 con 0x55e7e316b2f0 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f5397833740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f53881abdf0 con 0x55e7e31941b0 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f5397833740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53881aa980 con 0x55e7e31734d0 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f538ed81640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e316b2f0 0x55e7e3541260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.603+0000 7f538dd7f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31734d0 0x55e7e35325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538ed81640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e316b2f0 0x55e7e3541260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34270/0 (socket says 172.21.15.29:34270) 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538ed81640 1 -- 172.21.15.29:0/571803906 learned_addr learned my addr 172.21.15.29:0/571803906 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538ed81640 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3547f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538dd7f640 1 -- 172.21.15.29:0/571803906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 msgr2=0x55e7e3547f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538dd7f640 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3547f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538dd7f640 1 -- 172.21.15.29:0/571803906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e316b2f0 msgr2=0x55e7e3541260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538dd7f640 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e316b2f0 0x55e7e3541260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538dd7f640 1 -- 172.21.15.29:0/571803906 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7e3551f10 con 0x55e7e31734d0 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538ed81640 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e316b2f0 0x55e7e3541260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f538dd7f640 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31734d0 0x55e7e35325c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5378018920 tx=0x7f5378018580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ab4671a68e9b063 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f5373fff640 1 -- 172.21.15.29:0/571803906 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537802e070 con 0x55e7e31734d0 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f5373fff640 1 -- 172.21.15.29:0/571803906 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53780054c0 con 0x55e7e31734d0 2024-09-21T02:49:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.604+0000 7f5373fff640 1 -- 172.21.15.29:0/571803906 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53780048c0 con 0x55e7e31734d0 2024-09-21T02:49:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 -- 172.21.15.29:0/571803906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31734d0 msgr2=0x55e7e35325c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31734d0 0x55e7e35325c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5378018920 tx=0x7f5378018580 comp rx=0 tx=0).stop 2024-09-21T02:49:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 -- 172.21.15.29:0/571803906 shutdown_connections 2024-09-21T02:49:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3547f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e31734d0 0x55e7e35325c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 --2- 172.21.15.29:0/571803906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e316b2f0 0x55e7e3541260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 -- 172.21.15.29:0/571803906 >> 172.21.15.29:0/571803906 conn(0x55e7e3188300 msgr2=0x55e7e3548c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 -- 172.21.15.29:0/571803906 shutdown_connections 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.605+0000 7f5397833740 1 -- 172.21.15.29:0/571803906 wait complete. 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 1 -- start start 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 1 Processor -- start 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e316b2f0 0x55e7e35518f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3550650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3550d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53881ab720 con 0x55e7e31734d0 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f53881ab050 con 0x55e7e31941b0 2024-09-21T02:49:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53881a8290 con 0x55e7e316b2f0 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f5397833740 10 client.0 _ll_register_callbacks cb 0x7f5378010b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f538dd7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3550650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f538dd7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3550650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34276/0 (socket says 172.21.15.29:34276) 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f538dd7f640 1 -- 172.21.15.29:0/2392936422 learned_addr learned my addr 172.21.15.29:0/2392936422 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f538ed81640 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3550d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.606+0000 7f538ed81640 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e316b2f0 0x55e7e35518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f538ed81640 1 -- 172.21.15.29:0/2392936422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e316b2f0 msgr2=0x55e7e35518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f538ed81640 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e316b2f0 0x55e7e35518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f538ed81640 1 -- 172.21.15.29:0/2392936422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 msgr2=0x55e7e3550650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f538ed81640 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3550650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f538ed81640 1 -- 172.21.15.29:0/2392936422 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7e354f230 con 0x55e7e31941b0 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f538dd7f640 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3550650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:49:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f538ed81640 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3550d60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5380014040 tx=0x7f53800118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=347ce2f8143f3cd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f5371ffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f5371ffb640 1 -- 172.21.15.29:0/2392936422 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380003a30 con 0x55e7e31941b0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f5397833740 1 -- 172.21.15.29:0/2392936422 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53780054c0 con 0x55e7e31941b0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f5371ffb640 1 -- 172.21.15.29:0/2392936422 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5380003bd0 con 0x55e7e31941b0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f5371ffb640 1 -- 172.21.15.29:0/2392936422 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5380003dd0 con 0x55e7e31941b0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f5397833740 10 client.26420 fetch_fsmap learned FSMap version 31 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.607+0000 7f5397833740 1 -- 172.21.15.29:0/2392936422 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e7e35c1e20 con 0x55e7e31941b0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5371ffb640 1 -- 172.21.15.29:0/2392936422 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53780054c0 con 0x55e7e31941b0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5371ffb640 1 -- 172.21.15.29:0/2392936422 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f5380025040 con 0x55e7e31941b0 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5397833740 10 client.26420 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5397833740 10 client.26420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5397833740 20 client.26420 populate_metadata read hostname 'smithi029' 2024-09-21T02:49:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5397833740 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e7e31281c0 0x55e7e35c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:49:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5397833740 4 client.26420 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:49:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5397833740 1 -- 172.21.15.29:0/2392936422 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7e34ac0f0 con 0x55e7e31281c0 2024-09-21T02:49:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f538dd7f640 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e7e31281c0 0x55e7e35c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:49:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f538dd7f640 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e7e31281c0 0x55e7e35c45d0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d74e713560b2c99 server_cookie=1c37eb1345754cbc in_seq=0 out_seq=0 2024-09-21T02:49:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.608+0000 7f5371ffb640 10 client.26420 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.645+0000 7f5371ffb640 1 -- 172.21.15.29:0/2392936422 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e7e34ac0f0 con 0x55e7e31281c0 2024-09-21T02:49:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.645+0000 7f5371ffb640 10 client.26420 handle_command_reply: tid=0 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.645+0000 7f5397833740 1 client.26420 shutdown 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 -- 172.21.15.29:0/2392936422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 msgr2=0x55e7e3550d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3550d60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5380014040 tx=0x7f53800118d0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 -- 172.21.15.29:0/2392936422 shutdown_connections 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e7e31941b0 0x55e7e3550d60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e7e31281c0 0x55e7e35c45d0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7e31734d0 0x55e7e3550650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 --2- 172.21.15.29:0/2392936422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7e316b2f0 0x55e7e35518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5371ffb640 0 client.26420 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 -- 172.21.15.29:0/2392936422 >> 172.21.15.29:0/2392936422 conn(0x55e7e3188300 msgr2=0x55e7e35325c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 -- 172.21.15.29:0/2392936422 shutdown_connections 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 1 -- 172.21.15.29:0/2392936422 wait complete. 2024-09-21T02:49:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f5397833740 20 client.26420 trim_cache size 0 max 16384 2024-09-21T02:49:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:49:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:49:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:49:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 -- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53740b6510 msgr2=0x7f53740b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53740b6510 0x7f53740b8a20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f53840cdb50 tx=0x7f5384097660 comp rx=0 tx=0).stop 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 -- 172.21.15.29:0/4200204737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53881036a0 msgr2=0x7f5388199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53881036a0 0x7f5388199e50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5384005760 tx=0x7f5384004270 comp rx=0 tx=0).stop 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 -- 172.21.15.29:0/4200204737 shutdown_connections 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53740b6510 0x7f53740b8a20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53881050c0 0x7f538819b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388104720 0x7f538819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 --2- 172.21.15.29:0/4200204737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53881036a0 0x7f5388199e50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:49:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.646+0000 7f53907e4640 1 -- 172.21.15.29:0/4200204737 >> 172.21.15.29:0/4200204737 conn(0x7f53880fefe0 msgr2=0x7f5388107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:49:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.647+0000 7f53907e4640 1 -- 172.21.15.29:0/4200204737 shutdown_connections 2024-09-21T02:49:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:49:50.647+0000 7f53907e4640 1 -- 172.21.15.29:0/4200204737 wait complete. 2024-09-21T02:49:51.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 202647 2.08 MB/sec execute 448 sec latency 43.693 ms 2024-09-21T02:49:51.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:51 smithi190 ceph-mon[33871]: pgmap v711: 129 pgs: 129 active+clean; 39 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 237 KiB/s wr, 21 op/s 2024-09-21T02:49:51.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:51 smithi029 ceph-mon[31006]: pgmap v711: 129 pgs: 129 active+clean; 39 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 237 KiB/s wr, 21 op/s 2024-09-21T02:49:51.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:51 smithi066 ceph-mon[33436]: pgmap v711: 129 pgs: 129 active+clean; 39 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 237 KiB/s wr, 21 op/s 2024-09-21T02:49:52.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 203166 2.08 MB/sec execute 449 sec latency 17.388 ms 2024-09-21T02:49:53.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 203802 2.08 MB/sec execute 450 sec latency 19.992 ms 2024-09-21T02:49:53.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:53 smithi190 ceph-mon[33871]: pgmap v712: 129 pgs: 129 active+clean; 40 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 380 KiB/s wr, 33 op/s 2024-09-21T02:49:53.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:53 smithi029 ceph-mon[31006]: pgmap v712: 129 pgs: 129 active+clean; 40 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 380 KiB/s wr, 33 op/s 2024-09-21T02:49:53.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:53 smithi066 ceph-mon[33436]: pgmap v712: 129 pgs: 129 active+clean; 40 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 380 KiB/s wr, 33 op/s 2024-09-21T02:49:54.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 204486 2.08 MB/sec execute 451 sec latency 18.020 ms 2024-09-21T02:49:55.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 205210 2.09 MB/sec execute 452 sec latency 18.464 ms 2024-09-21T02:49:55.562 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:55 smithi066 ceph-mon[33436]: pgmap v713: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 635 KiB/s wr, 70 op/s 2024-09-21T02:49:55.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:55 smithi190 ceph-mon[33871]: pgmap v713: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 635 KiB/s wr, 70 op/s 2024-09-21T02:49:55.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:55 smithi029 ceph-mon[31006]: pgmap v713: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 635 KiB/s wr, 70 op/s 2024-09-21T02:49:56.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 205850 2.10 MB/sec execute 453 sec latency 17.298 ms 2024-09-21T02:49:57.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 206158 2.09 MB/sec execute 454 sec latency 13.219 ms 2024-09-21T02:49:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:57 smithi190 ceph-mon[33871]: pgmap v714: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 971 KiB/s wr, 85 op/s 2024-09-21T02:49:57.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:57 smithi029 ceph-mon[31006]: pgmap v714: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 971 KiB/s wr, 85 op/s 2024-09-21T02:49:57.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:57 smithi066 ceph-mon[33436]: pgmap v714: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 971 KiB/s wr, 85 op/s 2024-09-21T02:49:58.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 206668 2.09 MB/sec execute 455 sec latency 18.589 ms 2024-09-21T02:49:59.099 INFO:tasks.workunit.client.0.smithi029.stdout: 1 207320 2.10 MB/sec execute 456 sec latency 15.809 ms 2024-09-21T02:49:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:49:59 smithi190 ceph-mon[33871]: pgmap v715: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-09-21T02:49:59.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:49:59 smithi029 ceph-mon[31006]: pgmap v715: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-09-21T02:49:59.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:49:59 smithi066 ceph-mon[33436]: pgmap v715: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-09-21T02:50:00.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 207989 2.09 MB/sec execute 457 sec latency 17.221 ms 2024-09-21T02:50:00.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:00 smithi190 ceph-mon[33871]: overall HEALTH_OK 2024-09-21T02:50:00.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:00 smithi029 ceph-mon[31006]: overall HEALTH_OK 2024-09-21T02:50:00.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:00 smithi066 ceph-mon[33436]: overall HEALTH_OK 2024-09-21T02:50:01.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 208721 2.10 MB/sec execute 458 sec latency 17.411 ms 2024-09-21T02:50:01.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:01 smithi190 ceph-mon[33871]: pgmap v716: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-09-21T02:50:01.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:01 smithi029 ceph-mon[31006]: pgmap v716: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-09-21T02:50:01.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:01 smithi066 ceph-mon[33436]: pgmap v716: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-09-21T02:50:02.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 209364 2.11 MB/sec execute 459 sec latency 16.082 ms 2024-09-21T02:50:03.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 209687 2.11 MB/sec execute 460 sec latency 18.098 ms 2024-09-21T02:50:03.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:03 smithi190 ceph-mon[33871]: pgmap v717: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:50:03.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:03 smithi029 ceph-mon[31006]: pgmap v717: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:50:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 -- 172.21.15.29:0/1550726571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103fc0 msgr2=0x7f037410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 --2- 172.21.15.29:0/1550726571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103fc0 0x7f037410a960 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f036c009c90 tx=0x7f036c02a860 comp rx=0 tx=0).stop 2024-09-21T02:50:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 -- 172.21.15.29:0/1550726571 shutdown_connections 2024-09-21T02:50:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 --2- 172.21.15.29:0/1550726571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103fc0 0x7f037410a960 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 --2- 172.21.15.29:0/1550726571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0374103620 0x7f0374103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 --2- 172.21.15.29:0/1550726571 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03741025a0 0x7f0374102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 -- 172.21.15.29:0/1550726571 >> 172.21.15.29:0/1550726571 conn(0x7f03740fdee0 msgr2=0x7f0374100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 -- 172.21.15.29:0/1550726571 shutdown_connections 2024-09-21T02:50:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 -- 172.21.15.29:0/1550726571 wait complete. 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.752+0000 7f037b1ad640 1 Processor -- start 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f037b1ad640 1 -- start start 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f037b1ad640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03741025a0 0x7f0374199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f037b1ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103620 0x7f037419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f037b1ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0374103fc0 0x7f037419b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f037b1ad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f037410e0b0 con 0x7f0374103620 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f037b1ad640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f037410df30 con 0x7f03741025a0 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f037b1ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f037410e230 con 0x7f0374103fc0 2024-09-21T02:50:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0373fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103620 0x7f037419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0373fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103620 0x7f037419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57744/0 (socket says 172.21.15.29:57744) 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0373fff640 1 -- 172.21.15.29:0/4257975265 learned_addr learned my addr 172.21.15.29:0/4257975265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0378f49640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03741025a0 0x7f0374199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f037974a640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0374103fc0 0x7f037419b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0373fff640 1 -- 172.21.15.29:0/4257975265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0374103fc0 msgr2=0x7f037419b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0373fff640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0374103fc0 0x7f037419b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0373fff640 1 -- 172.21.15.29:0/4257975265 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03741025a0 msgr2=0x7f0374199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0373fff640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03741025a0 0x7f0374199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0373fff640 1 -- 172.21.15.29:0/4257975265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037419b890 con 0x7f0374103620 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.753+0000 7f0378f49640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03741025a0 0x7f0374199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.754+0000 7f0373fff640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103620 0x7f037419a530 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f036000e9f0 tx=0x7f036000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.754+0000 7f0371ffb640 1 -- 172.21.15.29:0/4257975265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036000cd80 con 0x7f0374103620 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.754+0000 7f037b1ad640 1 -- 172.21.15.29:0/4257975265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037419bb80 con 0x7f0374103620 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.754+0000 7f0371ffb640 1 -- 172.21.15.29:0/4257975265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03600054b0 con 0x7f0374103620 2024-09-21T02:50:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.754+0000 7f0371ffb640 1 -- 172.21.15.29:0/4257975265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360010430 con 0x7f0374103620 2024-09-21T02:50:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.754+0000 7f037b1ad640 1 -- 172.21.15.29:0/4257975265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741a2270 con 0x7f0374103620 2024-09-21T02:50:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.757+0000 7f03821fc740 1 Processor -- start 2024-09-21T02:50:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.757+0000 7f03821fc740 1 -- start start 2024-09-21T02:50:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.757+0000 7f0371ffb640 1 -- 172.21.15.29:0/4257975265 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f03600046c0 con 0x7f0374103620 2024-09-21T02:50:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.757+0000 7f03821fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f741e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.757+0000 7f03821fc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3712f0 0x56286f7424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.757+0000 7f03821fc740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3794d0 0x56286f743010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.757+0000 7f03821fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56286f6b46f0 con 0x56286f39a1b0 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f03821fc740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56286f6a3b20 con 0x56286f3794d0 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f03821fc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56286f6a7930 con 0x56286f3712f0 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f037974a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f741e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0378f49640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3712f0 0x56286f7424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f037974a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f741e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57746/0 (socket says 172.21.15.29:57746) 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f037974a640 1 -- 172.21.15.29:0/3491906667 learned_addr learned my addr 172.21.15.29:0/3491906667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0373fff640 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3794d0 0x56286f743010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0378f49640 1 -- 172.21.15.29:0/3491906667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3794d0 msgr2=0x56286f743010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0378f49640 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3794d0 0x56286f743010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0378f49640 1 -- 172.21.15.29:0/3491906667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 msgr2=0x56286f741e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0378f49640 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f741e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0378f49640 1 -- 172.21.15.29:0/3491906667 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56286f669530 con 0x56286f3712f0 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0373fff640 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3794d0 0x56286f743010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f037974a640 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f741e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.758+0000 7f0378f49640 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3712f0 0x56286f7424c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0368009f50 tx=0x7f0368013c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3e9536c6725270 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.759+0000 7f035effd640 1 -- 172.21.15.29:0/3491906667 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036800e9d0 con 0x56286f3712f0 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.759+0000 7f0371ffb640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03640b6510 0x7f03640b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.759+0000 7f035effd640 1 -- 172.21.15.29:0/3491906667 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0368005020 con 0x56286f3712f0 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.759+0000 7f035effd640 1 -- 172.21.15.29:0/3491906667 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368004b40 con 0x56286f3712f0 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.759+0000 7f037974a640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03640b6510 0x7f03640b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.759+0000 7f0371ffb640 1 -- 172.21.15.29:0/4257975265 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f03600e7a20 con 0x7f0374103620 2024-09-21T02:50:03.763 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:03 smithi066 ceph-mon[33436]: pgmap v717: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 -- 172.21.15.29:0/3491906667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3712f0 msgr2=0x56286f7424c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f037974a640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03640b6510 0x7f03640b8a20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f036c02ad80 tx=0x7f036c031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3712f0 0x56286f7424c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0368009f50 tx=0x7f0368013c60 comp rx=0 tx=0).stop 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 -- 172.21.15.29:0/3491906667 shutdown_connections 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3794d0 0x56286f743010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3712f0 0x56286f7424c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 --2- 172.21.15.29:0/3491906667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f741e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 -- 172.21.15.29:0/3491906667 >> 172.21.15.29:0/3491906667 conn(0x56286f38e300 msgr2=0x56286f73d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 -- 172.21.15.29:0/3491906667 shutdown_connections 2024-09-21T02:50:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 -- 172.21.15.29:0/3491906667 wait complete. 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 -- start start 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.760+0000 7f03821fc740 1 Processor -- start 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f03821fc740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f74b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f03821fc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f74bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f03821fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f74c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f03821fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56286f5dbab0 con 0x56286f39a1b0 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f03821fc740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56286f649730 con 0x56286f3712f0 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f03821fc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56286f743b30 con 0x56286f3794d0 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f03821fc740 10 client.0 _ll_register_callbacks cb 0x56286f69d6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f037974a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f74c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f037974a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f74c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57760/0 (socket says 172.21.15.29:57760) 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f037974a640 1 -- 172.21.15.29:0/4089415199 learned_addr learned my addr 172.21.15.29:0/4089415199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f0378f49640 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f74b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.761+0000 7f0373fff640 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f74bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0378f49640 1 -- 172.21.15.29:0/4089415199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 msgr2=0x56286f74bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0378f49640 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f74bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0378f49640 1 -- 172.21.15.29:0/4089415199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 msgr2=0x56286f74c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0378f49640 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f74c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0378f49640 1 -- 172.21.15.29:0/4089415199 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56286f405790 con 0x56286f3712f0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f037974a640 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f74c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0378f49640 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f74b590 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0368000c00 tx=0x7f036800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99b5b3be6982d9c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0353fff640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0353fff640 1 -- 172.21.15.29:0/4089415199 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368004260 con 0x56286f3712f0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f03821fc740 1 -- 172.21.15.29:0/4089415199 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0368005020 con 0x56286f3712f0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0353fff640 1 -- 172.21.15.29:0/4089415199 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0368004400 con 0x56286f3712f0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f0353fff640 1 -- 172.21.15.29:0/4089415199 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0368034050 con 0x56286f3712f0 2024-09-21T02:50:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f03821fc740 10 client.26432 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.762+0000 7f03821fc740 1 -- 172.21.15.29:0/4089415199 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56286f47e620 con 0x56286f3712f0 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f0353fff640 1 -- 172.21.15.29:0/4089415199 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0368005020 con 0x56286f3712f0 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f0353fff640 1 -- 172.21.15.29:0/4089415199 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f0368003730 con 0x56286f3712f0 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f03821fc740 10 client.26432 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f03821fc740 10 client.26432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f03821fc740 20 client.26432 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f03821fc740 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56286f358d80 0x56286f7589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f03821fc740 4 client.26432 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f03821fc740 1 -- 172.21.15.29:0/4089415199 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56286f6b6640 con 0x56286f358d80 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f0373fff640 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56286f358d80 0x56286f7589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f0353fff640 10 client.26432 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.763+0000 7f0373fff640 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56286f358d80 0x56286f7589b0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9982d2bc51e0268 server_cookie=1df6224f1cb4f9de in_seq=0 out_seq=0 2024-09-21T02:50:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.764+0000 7f0353fff640 1 -- 172.21.15.29:0/4089415199 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56286f6b6640 con 0x56286f358d80 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.764+0000 7f0353fff640 10 client.26432 handle_command_reply: tid=0 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.764+0000 7f03821fc740 1 client.26432 shutdown 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 -- 172.21.15.29:0/4089415199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 msgr2=0x56286f74b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f74b590 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0368000c00 tx=0x7f036800ec80 comp rx=0 tx=0).stop 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 -- 172.21.15.29:0/4089415199 shutdown_connections 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56286f358d80 0x56286f7589b0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f39a1b0 0x56286f74c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f74bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f0353fff640 0 client.26432 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 --2- 172.21.15.29:0/4089415199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f74b590 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 -- 172.21.15.29:0/4089415199 >> 172.21.15.29:0/4089415199 conn(0x56286f38e300 msgr2=0x56286f73bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 -- 172.21.15.29:0/4089415199 shutdown_connections 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 1 -- 172.21.15.29:0/4089415199 wait complete. 2024-09-21T02:50:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.765+0000 7f03821fc740 20 client.26432 trim_cache size 0 max 16384 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.783+0000 7f03821fc740 1 Processor -- start 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.783+0000 7f03821fc740 1 -- start start 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.783+0000 7f03821fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f358d80 0x56286f74a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.783+0000 7f03821fc740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f741df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.783+0000 7f03821fc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f742330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.783+0000 7f03821fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03741aa390 con 0x56286f358d80 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.783+0000 7f03821fc740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f03741abed0 con 0x56286f3712f0 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.783+0000 7f03821fc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f03741aaa60 con 0x56286f3794d0 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f0378f49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f358d80 0x56286f74a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f0378f49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f358d80 0x56286f74a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57772/0 (socket says 172.21.15.29:57772) 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f037974a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f742330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f0378f49640 1 -- 172.21.15.29:0/15547002 learned_addr learned my addr 172.21.15.29:0/15547002 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f0373fff640 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f741df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f037974a640 1 -- 172.21.15.29:0/15547002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 msgr2=0x56286f741df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f037974a640 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f741df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f037974a640 1 -- 172.21.15.29:0/15547002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f358d80 msgr2=0x56286f74a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f037974a640 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f358d80 0x56286f74a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f037974a640 1 -- 172.21.15.29:0/15547002 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56286f7390a0 con 0x56286f3794d0 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f0373fff640 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f741df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f0378f49640 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f358d80 0x56286f74a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f037974a640 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f742330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f036c009c60 tx=0x7f036c040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a151a9836b5bbb87 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f035e7fc640 1 -- 172.21.15.29:0/15547002 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c031c00 con 0x56286f3794d0 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f035e7fc640 1 -- 172.21.15.29:0/15547002 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f036c036040 con 0x56286f3794d0 2024-09-21T02:50:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.784+0000 7f035e7fc640 1 -- 172.21.15.29:0/15547002 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c004630 con 0x56286f3794d0 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 -- 172.21.15.29:0/15547002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 msgr2=0x56286f742330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f742330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f036c009c60 tx=0x7f036c040000 comp rx=0 tx=0).stop 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 -- 172.21.15.29:0/15547002 shutdown_connections 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f3794d0 0x56286f742330 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f741df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 --2- 172.21.15.29:0/15547002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f358d80 0x56286f74a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 -- 172.21.15.29:0/15547002 >> 172.21.15.29:0/15547002 conn(0x56286f38e300 msgr2=0x56286f742ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 -- 172.21.15.29:0/15547002 shutdown_connections 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 -- 172.21.15.29:0/15547002 wait complete. 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 -- start start 2024-09-21T02:50:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.786+0000 7f03821fc740 1 Processor -- start 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f03821fc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f358d80 0x56286f750520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f03821fc740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f750be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f03821fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f3794d0 0x56286f7526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f03821fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ab800 con 0x56286f3794d0 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f03821fc740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ab130 con 0x56286f3712f0 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f03821fc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f03741a8370 con 0x56286f358d80 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f03821fc740 10 client.0 _ll_register_callbacks cb 0x7f0368005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f037974a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f3794d0 0x56286f7526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0378f49640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f358d80 0x56286f750520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0378f49640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f358d80 0x56286f750520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:46308/0 (socket says 172.21.15.29:46308) 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0378f49640 1 -- 172.21.15.29:0/1406449161 learned_addr learned my addr 172.21.15.29:0/1406449161 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0373fff640 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f750be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0378f49640 1 -- 172.21.15.29:0/1406449161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 msgr2=0x56286f750be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0378f49640 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f750be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0378f49640 1 -- 172.21.15.29:0/1406449161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f3794d0 msgr2=0x56286f7526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0378f49640 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f3794d0 0x56286f7526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.787+0000 7f0378f49640 1 -- 172.21.15.29:0/1406449161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56286f752e40 con 0x56286f358d80 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f037974a640 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f3794d0 0x56286f7526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f0378f49640 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f358d80 0x56286f750520 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0368003bd0 tx=0x7f0368003fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37362f3e2fd75f80 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f0353fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f0353fff640 1 -- 172.21.15.29:0/1406449161 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368026460 con 0x56286f358d80 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f0353fff640 1 -- 172.21.15.29:0/1406449161 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f03680035f0 con 0x56286f358d80 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f03821fc740 1 -- 172.21.15.29:0/1406449161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f036c036040 con 0x56286f358d80 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f0353fff640 1 -- 172.21.15.29:0/1406449161 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0368047050 con 0x56286f358d80 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f03821fc740 10 client.26581 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f03821fc740 1 -- 172.21.15.29:0/1406449161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56286f752080 con 0x56286f358d80 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.788+0000 7f0353fff640 1 -- 172.21.15.29:0/1406449161 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f036c036040 con 0x56286f358d80 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f0353fff640 1 -- 172.21.15.29:0/1406449161 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f0368040060 con 0x56286f358d80 2024-09-21T02:50:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f03821fc740 10 client.26581 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f03821fc740 10 client.26581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f03821fc740 20 client.26581 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f03821fc740 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56286f39a1b0 0x56286f7c5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f03821fc740 4 client.26581 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:50:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f03821fc740 1 -- 172.21.15.29:0/1406449161 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56286f6b20f0 con 0x56286f39a1b0 2024-09-21T02:50:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f0373fff640 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56286f39a1b0 0x56286f7c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f0353fff640 10 client.26581 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.789+0000 7f0373fff640 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56286f39a1b0 0x56286f7c5830 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b029f932e6a9e3a2 server_cookie=e2846ddfbe84db97 in_seq=0 out_seq=0 2024-09-21T02:50:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f0353fff640 1 -- 172.21.15.29:0/1406449161 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56286f6b20f0 con 0x56286f39a1b0 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f0353fff640 10 client.26581 handle_command_reply: tid=0 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 client.26581 shutdown 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 -- 172.21.15.29:0/1406449161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f358d80 msgr2=0x56286f750520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f358d80 0x56286f750520 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0368003bd0 tx=0x7f0368003fd0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 -- 172.21.15.29:0/1406449161 shutdown_connections 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56286f39a1b0 0x56286f7c5830 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56286f3794d0 0x56286f7526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56286f3712f0 0x56286f750be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f0353fff640 0 client.26581 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 --2- 172.21.15.29:0/1406449161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56286f358d80 0x56286f750520 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 -- 172.21.15.29:0/1406449161 >> 172.21.15.29:0/1406449161 conn(0x56286f38e300 msgr2=0x56286f745ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 -- 172.21.15.29:0/1406449161 shutdown_connections 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 1 -- 172.21.15.29:0/1406449161 wait complete. 2024-09-21T02:50:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.835+0000 7f03821fc740 20 client.26581 trim_cache size 0 max 16384 2024-09-21T02:50:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:50:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:50:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:50:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 -- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03640b6510 msgr2=0x7f03640b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03640b6510 0x7f03640b8a20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f036c02ad80 tx=0x7f036c031000 comp rx=0 tx=0).stop 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 -- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103620 msgr2=0x7f037419a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103620 0x7f037419a530 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f036000e9f0 tx=0x7f036000edf0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 -- 172.21.15.29:0/4257975265 shutdown_connections 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f03640b6510 0x7f03640b8a20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0374103fc0 0x7f037419b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0374103620 0x7f037419a530 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 --2- 172.21.15.29:0/4257975265 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03741025a0 0x7f0374199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.836+0000 7f037b1ad640 1 -- 172.21.15.29:0/4257975265 >> 172.21.15.29:0/4257975265 conn(0x7f03740fdee0 msgr2=0x7f03740ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.837+0000 7f037b1ad640 1 -- 172.21.15.29:0/4257975265 shutdown_connections 2024-09-21T02:50:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:03.837+0000 7f037b1ad640 1 -- 172.21.15.29:0/4257975265 wait complete. 2024-09-21T02:50:04.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 210102 2.10 MB/sec execute 461 sec latency 42.809 ms 2024-09-21T02:50:05.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 210750 2.11 MB/sec execute 462 sec latency 14.557 ms 2024-09-21T02:50:05.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:05 smithi190 ceph-mon[33871]: pgmap v718: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:50:05.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:05 smithi029 ceph-mon[31006]: pgmap v718: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:50:05.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:05 smithi066 ceph-mon[33436]: pgmap v718: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:50:06.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 211378 2.11 MB/sec execute 463 sec latency 17.346 ms 2024-09-21T02:50:07.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 212151 2.11 MB/sec execute 464 sec latency 17.594 ms 2024-09-21T02:50:07.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:07 smithi190 ceph-mon[33871]: pgmap v719: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-09-21T02:50:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:07 smithi029 ceph-mon[31006]: pgmap v719: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-09-21T02:50:07.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:07 smithi066 ceph-mon[33436]: pgmap v719: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-09-21T02:50:08.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 212668 2.12 MB/sec execute 465 sec latency 26.683 ms 2024-09-21T02:50:09.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 213180 2.12 MB/sec execute 466 sec latency 16.589 ms 2024-09-21T02:50:09.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:09 smithi190 ceph-mon[33871]: pgmap v720: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-09-21T02:50:09.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:09 smithi029 ceph-mon[31006]: pgmap v720: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-09-21T02:50:09.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:09 smithi066 ceph-mon[33436]: pgmap v720: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-09-21T02:50:10.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 213542 2.12 MB/sec execute 467 sec latency 18.174 ms 2024-09-21T02:50:11.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 214224 2.12 MB/sec execute 468 sec latency 14.782 ms 2024-09-21T02:50:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:11 smithi190 ceph-mon[33871]: pgmap v721: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.5 MiB/s wr, 122 op/s 2024-09-21T02:50:11.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:11 smithi029 ceph-mon[31006]: pgmap v721: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.5 MiB/s wr, 122 op/s 2024-09-21T02:50:11.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:11 smithi066 ceph-mon[33436]: pgmap v721: 129 pgs: 129 active+clean; 40 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.5 MiB/s wr, 122 op/s 2024-09-21T02:50:12.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 214855 2.12 MB/sec execute 469 sec latency 14.828 ms 2024-09-21T02:50:13.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 215474 2.12 MB/sec execute 470 sec latency 19.942 ms 2024-09-21T02:50:13.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:13 smithi190 ceph-mon[33871]: pgmap v722: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.6 MiB/s wr, 150 op/s 2024-09-21T02:50:13.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:13 smithi029 ceph-mon[31006]: pgmap v722: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.6 MiB/s wr, 150 op/s 2024-09-21T02:50:13.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:13 smithi066 ceph-mon[33436]: pgmap v722: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.6 MiB/s wr, 150 op/s 2024-09-21T02:50:14.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 216205 2.13 MB/sec execute 471 sec latency 16.236 ms 2024-09-21T02:50:15.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 216727 2.14 MB/sec execute 472 sec latency 15.849 ms 2024-09-21T02:50:15.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:15 smithi190 ceph-mon[33871]: pgmap v723: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:50:15.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:15 smithi029 ceph-mon[31006]: pgmap v723: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:50:15.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:15 smithi066 ceph-mon[33436]: pgmap v723: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:50:16.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 217025 2.13 MB/sec execute 473 sec latency 16.534 ms 2024-09-21T02:50:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1064577732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 msgr2=0x7f17a410b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1064577732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 0x7f17a410b910 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f179c009ca0 tx=0x7f179c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1064577732 shutdown_connections 2024-09-21T02:50:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1064577732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 0x7f17a410b910 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1064577732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17a4106b10 0x7f17a4108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1064577732 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17a4101a80 0x7f17a4101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1064577732 >> 172.21.15.29:0/1064577732 conn(0x7f17a40fd680 msgr2=0x7f17a40ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1064577732 shutdown_connections 2024-09-21T02:50:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.948+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1064577732 wait complete. 2024-09-21T02:50:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17ac1e6640 1 Processor -- start 2024-09-21T02:50:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17ac1e6640 1 -- start start 2024-09-21T02:50:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17ac1e6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17a4101a80 0x7f17a4104150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17ac1e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17a4106b10 0x7f17a4102a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17ac1e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 0x7f17a4102f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17ac1e6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17a410f210 con 0x7f17a4109470 2024-09-21T02:50:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17ac1e6640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f17a410f090 con 0x7f17a4101a80 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17ac1e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17a410f390 con 0x7f17a4106b10 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9781640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17a4106b10 0x7f17a4102a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17aa783640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 0x7f17a4102f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9781640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17a4106b10 0x7f17a4102a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55140/0 (socket says 172.21.15.29:55140) 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9781640 1 -- 172.21.15.29:0/1981486965 learned_addr learned my addr 172.21.15.29:0/1981486965 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9f82640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17a4101a80 0x7f17a4104150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9781640 1 -- 172.21.15.29:0/1981486965 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17a4101a80 msgr2=0x7f17a4104150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9781640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17a4101a80 0x7f17a4104150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9781640 1 -- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 msgr2=0x7f17a4102f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9781640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 0x7f17a4102f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9781640 1 -- 172.21.15.29:0/1981486965 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a4103680 con 0x7f17a4106b10 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17a9f82640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17a4101a80 0x7f17a4104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:50:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.949+0000 7f17aa783640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 0x7f17a4102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.950+0000 7f17a9781640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17a4106b10 0x7f17a4102a10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f179400e9f0 tx=0x7f179400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.950+0000 7f179affd640 1 -- 172.21.15.29:0/1981486965 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179400cd80 con 0x7f17a4106b10 2024-09-21T02:50:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.950+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1981486965 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a41a3ee0 con 0x7f17a4106b10 2024-09-21T02:50:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.950+0000 7f179affd640 1 -- 172.21.15.29:0/1981486965 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17940054b0 con 0x7f17a4106b10 2024-09-21T02:50:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.950+0000 7f179affd640 1 -- 172.21.15.29:0/1981486965 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794010430 con 0x7f17a4106b10 2024-09-21T02:50:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.950+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1981486965 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a41a4290 con 0x7f17a4106b10 2024-09-21T02:50:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.952+0000 7f17b322e740 1 Processor -- start 2024-09-21T02:50:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17b322e740 1 -- start start 2024-09-21T02:50:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17b322e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eae141b0 0x5555eb1bbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17b322e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1bc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17b322e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadf34d0 0x5555eb1bd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17b322e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5555eb12e6f0 con 0x5555eae141b0 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17b322e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5555eb11db20 con 0x5555eadeb2f0 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17b322e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5555eb121930 con 0x5555eadf34d0 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17aa783640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eae141b0 0x5555eb1bbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17aa783640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eae141b0 0x5555eb1bbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57644/0 (socket says 172.21.15.29:57644) 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17aa783640 1 -- 172.21.15.29:0/2243697746 learned_addr learned my addr 172.21.15.29:0/2243697746 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f17a9f82640 1 --2- 172.21.15.29:0/2243697746 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1bc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.953+0000 7f179affd640 1 -- 172.21.15.29:0/1981486965 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f17940046c0 con 0x7f17a4106b10 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f17aa783640 1 -- 172.21.15.29:0/2243697746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadf34d0 msgr2=0x5555eb1bd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f17aa783640 1 --2- 172.21.15.29:0/2243697746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadf34d0 0x5555eb1bd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f17aa783640 1 -- 172.21.15.29:0/2243697746 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 msgr2=0x5555eb1bc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f17aa783640 1 --2- 172.21.15.29:0/2243697746 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1bc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f17aa783640 1 -- 172.21.15.29:0/2243697746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555eb0e3530 con 0x5555eae141b0 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f17aa783640 1 --2- 172.21.15.29:0/2243697746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eae141b0 0x5555eb1bbe50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f179c035000 tx=0x7f179c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46022e4fda555fe3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f178ffff640 1 -- 172.21.15.29:0/2243697746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c031d40 con 0x5555eae141b0 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f178ffff640 1 -- 172.21.15.29:0/2243697746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f179c036030 con 0x5555eae141b0 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f178ffff640 1 -- 172.21.15.29:0/2243697746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c0047a0 con 0x5555eae141b0 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.954+0000 7f179affd640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17900b6510 0x7f17900b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17a9f82640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17900b6510 0x7f17900b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f179affd640 1 -- 172.21.15.29:0/1981486965 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f17940e7ad0 con 0x7f17a4106b10 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17a9f82640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17900b6510 0x7f17900b8a20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009db0 tx=0x7f17a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17b322e740 1 -- 172.21.15.29:0/2243697746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eae141b0 msgr2=0x5555eb1bbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17b322e740 1 --2- 172.21.15.29:0/2243697746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eae141b0 0x5555eb1bbe50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f179c035000 tx=0x7f179c009800 comp rx=0 tx=0).stop 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17b322e740 1 -- 172.21.15.29:0/2243697746 shutdown_connections 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17b322e740 1 --2- 172.21.15.29:0/2243697746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadf34d0 0x5555eb1bd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17b322e740 1 --2- 172.21.15.29:0/2243697746 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1bc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17b322e740 1 --2- 172.21.15.29:0/2243697746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eae141b0 0x5555eb1bbe50 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17b322e740 1 -- 172.21.15.29:0/2243697746 >> 172.21.15.29:0/2243697746 conn(0x5555eae08300 msgr2=0x5555eb1b78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.955+0000 7f17b322e740 1 -- 172.21.15.29:0/2243697746 shutdown_connections 2024-09-21T02:50:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.956+0000 7f17b322e740 1 -- 172.21.15.29:0/2243697746 wait complete. 2024-09-21T02:50:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.956+0000 7f17b322e740 1 -- start start 2024-09-21T02:50:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.956+0000 7f17b322e740 1 Processor -- start 2024-09-21T02:50:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.956+0000 7f17b322e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadeb2f0 0x5555eb1c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.956+0000 7f17b322e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17b322e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eae141b0 0x5555eb1c6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17b322e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5555eb055ab0 con 0x5555eadf34d0 2024-09-21T02:50:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17b322e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5555eb0c3730 con 0x5555eae141b0 2024-09-21T02:50:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17b322e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5555eb1bd950 con 0x5555eadeb2f0 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17b322e740 10 client.0 _ll_register_callbacks cb 0x5555eb1176b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadeb2f0 0x5555eb1c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9781640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadeb2f0 0x5555eb1c54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55158/0 (socket says 172.21.15.29:55158) 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 -- 172.21.15.29:0/4215384307 learned_addr learned my addr 172.21.15.29:0/4215384307 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17aa783640 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eae141b0 0x5555eb1c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 -- 172.21.15.29:0/4215384307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eae141b0 msgr2=0x5555eb1c6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eae141b0 0x5555eb1c6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 -- 172.21.15.29:0/4215384307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 msgr2=0x5555eb1c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 -- 172.21.15.29:0/4215384307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555eae7f790 con 0x5555eadeb2f0 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f17a9f82640 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadeb2f0 0x5555eb1c54d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f17a0005f60 tx=0x7f17a0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8116044b9492b3a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.957+0000 7f178d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f178d7fa640 1 -- 172.21.15.29:0/4215384307 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0011b50 con 0x5555eadeb2f0 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f178d7fa640 1 -- 172.21.15.29:0/4215384307 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f17a0018030 con 0x5555eadeb2f0 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 1 -- 172.21.15.29:0/4215384307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f179c036030 con 0x5555eadeb2f0 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f178d7fa640 1 -- 172.21.15.29:0/4215384307 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f17a0033050 con 0x5555eadeb2f0 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 10 client.26590 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 1 -- 172.21.15.29:0/4215384307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5555eaef8620 con 0x5555eadeb2f0 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f178d7fa640 1 -- 172.21.15.29:0/4215384307 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f179c036030 con 0x5555eadeb2f0 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f178d7fa640 1 -- 172.21.15.29:0/4215384307 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f17a0028c40 con 0x5555eadeb2f0 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 10 client.26590 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 10 client.26590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 20 client.26590 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5555eadd2d80 0x5555eb1d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 4 client.26590 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.958+0000 7f17b322e740 1 -- 172.21.15.29:0/4215384307 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555eb130640 con 0x5555eadd2d80 2024-09-21T02:50:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.959+0000 7f17a9781640 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5555eadd2d80 0x5555eb1d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.959+0000 7f178d7fa640 10 client.26590 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.959+0000 7f17a9781640 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5555eadd2d80 0x5555eb1d2ce0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4206308440d11fa server_cookie=15f260c05f92d14f in_seq=0 out_seq=0 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f178d7fa640 1 -- 172.21.15.29:0/4215384307 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5555eb130640 con 0x5555eadd2d80 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f178d7fa640 10 client.26590 handle_command_reply: tid=0 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 client.26590 shutdown 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 -- 172.21.15.29:0/4215384307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadeb2f0 msgr2=0x5555eb1c54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadeb2f0 0x5555eb1c54d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f17a0005f60 tx=0x7f17a0004730 comp rx=0 tx=0).stop 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 -- 172.21.15.29:0/4215384307 shutdown_connections 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eae141b0 0x5555eb1c6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5555eadd2d80 0x5555eb1d2ce0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 --2- 172.21.15.29:0/4215384307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eadeb2f0 0x5555eb1c54d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f178d7fa640 0 client.26590 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 -- 172.21.15.29:0/4215384307 >> 172.21.15.29:0/4215384307 conn(0x5555eae08300 msgr2=0x5555eb1b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 -- 172.21.15.29:0/4215384307 shutdown_connections 2024-09-21T02:50:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 1 -- 172.21.15.29:0/4215384307 wait complete. 2024-09-21T02:50:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.960+0000 7f17b322e740 20 client.26590 trim_cache size 0 max 16384 2024-09-21T02:50:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.977+0000 7f17b322e740 1 Processor -- start 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.977+0000 7f17b322e740 1 -- start start 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.977+0000 7f17b322e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.977+0000 7f17b322e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.977+0000 7f17b322e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.977+0000 7f17b322e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41ae6c0 con 0x5555eadf34d0 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.977+0000 7f17b322e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41b0200 con 0x5555eadeb2f0 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.977+0000 7f17b322e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41aed90 con 0x5555eae141b0 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57664/0 (socket says 172.21.15.29:57664) 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 -- 172.21.15.29:0/905764123 learned_addr learned my addr 172.21.15.29:0/905764123 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17aa783640 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9f82640 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 -- 172.21.15.29:0/905764123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 msgr2=0x5555eb1bffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1bffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 -- 172.21.15.29:0/905764123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 msgr2=0x5555eb1c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 -- 172.21.15.29:0/905764123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555eb1b7900 con 0x5555eadf34d0 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9f82640 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1c3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17aa783640 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f17a9781640 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c3fa0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f17940e51f0 tx=0x7f17940e7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e60a8e15e77940 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f178f7fe640 1 -- 172.21.15.29:0/905764123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940a9f30 con 0x5555eadf34d0 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f178f7fe640 1 -- 172.21.15.29:0/905764123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17940ad3f0 con 0x5555eadf34d0 2024-09-21T02:50:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.978+0000 7f178f7fe640 1 -- 172.21.15.29:0/905764123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940acb40 con 0x5555eadf34d0 2024-09-21T02:50:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 -- 172.21.15.29:0/905764123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 msgr2=0x5555eb1c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c3fa0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f17940e51f0 tx=0x7f17940e7040 comp rx=0 tx=0).stop 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 -- 172.21.15.29:0/905764123 shutdown_connections 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadf34d0 0x5555eb1c3fa0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 --2- 172.21.15.29:0/905764123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadeb2f0 0x5555eb1c3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 -- 172.21.15.29:0/905764123 >> 172.21.15.29:0/905764123 conn(0x5555eae08300 msgr2=0x5555eb1bcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 -- 172.21.15.29:0/905764123 shutdown_connections 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.979+0000 7f17b322e740 1 -- 172.21.15.29:0/905764123 wait complete. 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 1 -- start start 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 1 Processor -- start 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadeb2f0 0x5555eb1ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadf34d0 0x5555eb1cab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41afb30 con 0x5555eadeb2f0 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41af460 con 0x5555eadf34d0 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41ac6a0 con 0x5555eae141b0 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17b322e740 10 client.0 _ll_register_callbacks cb 0x7f17a0009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17a9f82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadeb2f0 0x5555eb1ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17a9f82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadeb2f0 0x5555eb1ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57678/0 (socket says 172.21.15.29:57678) 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17a9f82640 1 -- 172.21.15.29:0/363456279 learned_addr learned my addr 172.21.15.29:0/363456279 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17aa783640 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.980+0000 7f17a9781640 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadf34d0 0x5555eb1cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17a9f82640 1 -- 172.21.15.29:0/363456279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 msgr2=0x5555eb1cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17a9f82640 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17a9f82640 1 -- 172.21.15.29:0/363456279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadf34d0 msgr2=0x5555eb1cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17a9f82640 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadf34d0 0x5555eb1cab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17a9f82640 1 -- 172.21.15.29:0/363456279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555eb1cbe30 con 0x5555eadeb2f0 2024-09-21T02:50:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17a9f82640 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadeb2f0 0x5555eb1ca4a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f17a0004700 tx=0x7f17a001d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f03390bfd2868f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f178d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f178d7fa640 1 -- 172.21.15.29:0/363456279 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0011d60 con 0x5555eadeb2f0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f178d7fa640 1 -- 172.21.15.29:0/363456279 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f17a0004030 con 0x5555eadeb2f0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17b322e740 1 -- 172.21.15.29:0/363456279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17940ad3f0 con 0x5555eadeb2f0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f178d7fa640 1 -- 172.21.15.29:0/363456279 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f17a0004230 con 0x5555eadeb2f0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17b322e740 10 client.17052 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f17b322e740 1 -- 172.21.15.29:0/363456279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5555eb1cd130 con 0x5555eadeb2f0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.981+0000 7f178d7fa640 1 -- 172.21.15.29:0/363456279 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17940ad3f0 con 0x5555eadeb2f0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f178d7fa640 1 -- 172.21.15.29:0/363456279 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f17a0003880 con 0x5555eadeb2f0 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f17b322e740 10 client.17052 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f17b322e740 10 client.17052 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f17b322e740 20 client.17052 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f17b322e740 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5555eada81c0 0x5555eb23f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f17b322e740 4 client.17052 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:50:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f17b322e740 1 -- 172.21.15.29:0/363456279 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5555eb12c0f0 con 0x5555eada81c0 2024-09-21T02:50:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f17a9781640 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5555eada81c0 0x5555eb23f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f178d7fa640 10 client.17052 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:16.982+0000 7f17a9781640 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5555eada81c0 0x5555eb23f6e0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c19b644bfb41390d server_cookie=e01f6528d9dbb5e in_seq=0 out_seq=0 2024-09-21T02:50:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.031+0000 7f178d7fa640 1 -- 172.21.15.29:0/363456279 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5555eb12c0f0 con 0x5555eada81c0 2024-09-21T02:50:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.031+0000 7f178d7fa640 10 client.17052 handle_command_reply: tid=0 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 client.17052 shutdown 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 -- 172.21.15.29:0/363456279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadeb2f0 msgr2=0x5555eb1ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadeb2f0 0x5555eb1ca4a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f17a0004700 tx=0x7f17a001d810 comp rx=0 tx=0).stop 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 -- 172.21.15.29:0/363456279 shutdown_connections 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555eae141b0 0x5555eb1cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5555eadf34d0 0x5555eb1cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5555eada81c0 0x5555eb23f6e0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 --2- 172.21.15.29:0/363456279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555eadeb2f0 0x5555eb1ca4a0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f178d7fa640 0 client.17052 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 -- 172.21.15.29:0/363456279 >> 172.21.15.29:0/363456279 conn(0x5555eae08300 msgr2=0x5555eb1c3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 -- 172.21.15.29:0/363456279 shutdown_connections 2024-09-21T02:50:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 1 -- 172.21.15.29:0/363456279 wait complete. 2024-09-21T02:50:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.032+0000 7f17b322e740 20 client.17052 trim_cache size 0 max 16384 2024-09-21T02:50:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:50:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:50:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:50:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17900b6510 msgr2=0x7f17900b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17900b6510 0x7f17900b8a20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009db0 tx=0x7f17a0005f90 comp rx=0 tx=0).stop 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1981486965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17a4106b10 msgr2=0x7f17a4102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17a4106b10 0x7f17a4102a10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f179400e9f0 tx=0x7f179400edf0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1981486965 shutdown_connections 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17900b6510 0x7f17900b8a20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17a4109470 0x7f17a4102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17a4106b10 0x7f17a4102a10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 --2- 172.21.15.29:0/1981486965 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17a4101a80 0x7f17a4104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1981486965 >> 172.21.15.29:0/1981486965 conn(0x7f17a40fd680 msgr2=0x7f17a4107940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1981486965 shutdown_connections 2024-09-21T02:50:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:17.033+0000 7f17ac1e6640 1 -- 172.21.15.29:0/1981486965 wait complete. 2024-09-21T02:50:17.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 217506 2.13 MB/sec execute 474 sec latency 172.883 ms 2024-09-21T02:50:17.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:17 smithi190 ceph-mon[33871]: pgmap v724: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 1.7 MiB/s wr, 141 op/s 2024-09-21T02:50:17.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:17 smithi029 ceph-mon[31006]: pgmap v724: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 1.7 MiB/s wr, 141 op/s 2024-09-21T02:50:17.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:17 smithi066 ceph-mon[33436]: pgmap v724: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 1.7 MiB/s wr, 141 op/s 2024-09-21T02:50:18.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 218129 2.13 MB/sec execute 475 sec latency 17.015 ms 2024-09-21T02:50:19.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 218786 2.13 MB/sec execute 476 sec latency 30.312 ms 2024-09-21T02:50:19.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:19 smithi190 ceph-mon[33871]: pgmap v725: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:50:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:19 smithi029 ceph-mon[31006]: pgmap v725: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:50:19.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:19 smithi066 ceph-mon[33436]: pgmap v725: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-09-21T02:50:20.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 219685 2.14 MB/sec execute 477 sec latency 13.795 ms 2024-09-21T02:50:21.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 220166 2.15 MB/sec execute 478 sec latency 35.203 ms 2024-09-21T02:50:21.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:21 smithi190 ceph-mon[33871]: pgmap v726: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-09-21T02:50:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:21 smithi029 ceph-mon[31006]: pgmap v726: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-09-21T02:50:21.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:21 smithi066 ceph-mon[33436]: pgmap v726: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-09-21T02:50:22.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 220407 2.14 MB/sec execute 479 sec latency 15.838 ms 2024-09-21T02:50:23.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 220903 2.14 MB/sec execute 480 sec latency 27.337 ms 2024-09-21T02:50:23.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:23 smithi190 ceph-mon[33871]: pgmap v727: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 145 op/s 2024-09-21T02:50:23.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:23 smithi029 ceph-mon[31006]: pgmap v727: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 145 op/s 2024-09-21T02:50:23.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:23 smithi066 ceph-mon[33436]: pgmap v727: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.6 MiB/s wr, 145 op/s 2024-09-21T02:50:24.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 221524 2.15 MB/sec execute 481 sec latency 16.783 ms 2024-09-21T02:50:25.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 222144 2.14 MB/sec execute 482 sec latency 22.972 ms 2024-09-21T02:50:25.562 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:25 smithi066 ceph-mon[33436]: pgmap v728: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-09-21T02:50:25.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:25 smithi190 ceph-mon[33871]: pgmap v728: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-09-21T02:50:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:25 smithi029 ceph-mon[31006]: pgmap v728: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-09-21T02:50:26.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 222893 2.15 MB/sec execute 483 sec latency 17.463 ms 2024-09-21T02:50:27.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 223517 2.16 MB/sec execute 484 sec latency 17.123 ms 2024-09-21T02:50:27.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:27 smithi190 ceph-mon[33871]: pgmap v729: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:50:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:27 smithi029 ceph-mon[31006]: pgmap v729: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:50:27.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:27 smithi066 ceph-mon[33436]: pgmap v729: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-09-21T02:50:28.100 INFO:tasks.workunit.client.0.smithi029.stdout: 1 223892 2.16 MB/sec execute 485 sec latency 16.643 ms 2024-09-21T02:50:29.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 224269 2.15 MB/sec execute 486 sec latency 17.702 ms 2024-09-21T02:50:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:29 smithi190 ceph-mon[33871]: pgmap v730: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-09-21T02:50:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:29 smithi029 ceph-mon[31006]: pgmap v730: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-09-21T02:50:29.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:29 smithi066 ceph-mon[33436]: pgmap v730: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-09-21T02:50:30.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 224917 2.16 MB/sec execute 487 sec latency 34.044 ms 2024-09-21T02:50:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3003276102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103620 msgr2=0x7f6e78103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3003276102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103620 0x7f6e78103a30 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009900 tx=0x7f6e6802a410 comp rx=0 tx=0).stop 2024-09-21T02:50:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3003276102 shutdown_connections 2024-09-21T02:50:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3003276102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e78103fc0 0x7f6e7810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3003276102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103620 0x7f6e78103a30 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3003276102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e781025a0 0x7f6e78102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3003276102 >> 172.21.15.29:0/3003276102 conn(0x7f6e780fdee0 msgr2=0x7f6e78100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3003276102 shutdown_connections 2024-09-21T02:50:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.141+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3003276102 wait complete. 2024-09-21T02:50:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7f6f9640 1 Processor -- start 2024-09-21T02:50:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7f6f9640 1 -- start start 2024-09-21T02:50:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7f6f9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e781025a0 0x7f6e7806b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7f6f9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e78103620 0x7f6e7819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7f6f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103fc0 0x7f6e781a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7f6f9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810e0b0 con 0x7f6e78103fc0 2024-09-21T02:50:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7f6f9640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810df30 con 0x7f6e781025a0 2024-09-21T02:50:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7f6f9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810e230 con 0x7f6e78103620 2024-09-21T02:50:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7dc96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103fc0 0x7f6e781a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7d495640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e781025a0 0x7f6e7806b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e78103620 0x7f6e7819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7dc96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103fc0 0x7f6e781a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60912/0 (socket says 172.21.15.29:60912) 2024-09-21T02:50:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7dc96640 1 -- 172.21.15.29:0/3489853370 learned_addr learned my addr 172.21.15.29:0/3489853370 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e78103620 0x7f6e7819ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58872/0 (socket says 172.21.15.29:58872) 2024-09-21T02:50:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7d495640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e781025a0 0x7f6e7806b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:47426/0 (socket says 172.21.15.29:47426) 2024-09-21T02:50:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7dc96640 1 -- 172.21.15.29:0/3489853370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e78103620 msgr2=0x7f6e7819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7dc96640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e78103620 0x7f6e7819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7dc96640 1 -- 172.21.15.29:0/3489853370 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e781025a0 msgr2=0x7f6e7806b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.142+0000 7f6e7dc96640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e781025a0 0x7f6e7806b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.143+0000 7f6e7dc96640 1 -- 172.21.15.29:0/3489853370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e781a1950 con 0x7f6e78103fc0 2024-09-21T02:50:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.143+0000 7f6e7d495640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e781025a0 0x7f6e7806b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.143+0000 7f6e7dc96640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103fc0 0x7f6e781a1150 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000eb40 tx=0x7f6e7000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.143+0000 7f6e6e7fc640 1 -- 172.21.15.29:0/3489853370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70010430 con 0x7f6e78103fc0 2024-09-21T02:50:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.143+0000 7f6e6e7fc640 1 -- 172.21.15.29:0/3489853370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e70010a60 con 0x7f6e78103fc0 2024-09-21T02:50:30.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.143+0000 7f6e6e7fc640 1 -- 172.21.15.29:0/3489853370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70018ac0 con 0x7f6e78103fc0 2024-09-21T02:50:30.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.143+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3489853370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e781a1be0 con 0x7f6e78103fc0 2024-09-21T02:50:30.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.143+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3489853370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e781a2180 con 0x7f6e78103fc0 2024-09-21T02:50:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.146+0000 7f6e86666740 1 Processor -- start 2024-09-21T02:50:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e6e7fc640 1 -- 172.21.15.29:0/3489853370 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f6e70010c10 con 0x7f6e78103fc0 2024-09-21T02:50:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e86666740 1 -- start start 2024-09-21T02:50:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e86666740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64541b0 0x55b8a67fbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e86666740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a67fc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e86666740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 0x55b8a67fd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e86666740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b8a676e6f0 con 0x55b8a64334d0 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e86666740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b8a675db20 con 0x55b8a64541b0 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e86666740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b8a6761930 con 0x55b8a642b2f0 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64541b0 0x55b8a67fbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e7d495640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a67fc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64541b0 0x55b8a67fbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:47432/0 (socket says 172.21.15.29:47432) 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e7cc94640 1 -- 172.21.15.29:0/3954850675 learned_addr learned my addr 172.21.15.29:0/3954850675 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e7d495640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a67fc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58888/0 (socket says 172.21.15.29:58888) 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.147+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 0x55b8a67fd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e7cc94640 1 -- 172.21.15.29:0/3954850675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 msgr2=0x55b8a67fc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a67fc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e7cc94640 1 -- 172.21.15.29:0/3954850675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 msgr2=0x55b8a67fd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 0x55b8a67fd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e6e7fc640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e640b6590 0x7f6e640b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e7cc94640 1 -- 172.21.15.29:0/3954850675 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8a6723530 con 0x55b8a64541b0 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e7d495640 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a67fc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e7dc96640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e640b6590 0x7f6e640b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64541b0 0x55b8a67fbe50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6e68031460 tx=0x7f6e68005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4ffcf54990de8f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e6e7fc640 1 -- 172.21.15.29:0/3489853370 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6e700e7bb0 con 0x7f6e78103fc0 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e637fe640 1 -- 172.21.15.29:0/3954850675 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68035040 con 0x55b8a64541b0 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e637fe640 1 -- 172.21.15.29:0/3954850675 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e680044c0 con 0x55b8a64541b0 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.148+0000 7f6e637fe640 1 -- 172.21.15.29:0/3954850675 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68003d60 con 0x55b8a64541b0 2024-09-21T02:50:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 -- 172.21.15.29:0/3954850675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64541b0 msgr2=0x55b8a67fbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e7dc96640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e640b6590 0x7f6e640b8aa0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000eb10 tx=0x7f6e7007ada0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64541b0 0x55b8a67fbe50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6e68031460 tx=0x7f6e68005fd0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 -- 172.21.15.29:0/3954850675 shutdown_connections 2024-09-21T02:50:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 0x55b8a67fd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a67fc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 --2- 172.21.15.29:0/3954850675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64541b0 0x55b8a67fbe50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 -- 172.21.15.29:0/3954850675 >> 172.21.15.29:0/3954850675 conn(0x55b8a6448300 msgr2=0x55b8a67f78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 -- 172.21.15.29:0/3954850675 shutdown_connections 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.149+0000 7f6e86666740 1 -- 172.21.15.29:0/3954850675 wait complete. 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 1 -- start start 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 1 Processor -- start 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6805560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6805c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64541b0 0x55b8a6806790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b8a6695ab0 con 0x55b8a64541b0 2024-09-21T02:50:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b8a6703730 con 0x55b8a64334d0 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b8a67fda70 con 0x55b8a642b2f0 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e86666740 10 client.0 _ll_register_callbacks cb 0x55b8a67576b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64541b0 0x55b8a6806790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e7d495640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6805c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64541b0 0x55b8a6806790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60938/0 (socket says 172.21.15.29:60938) 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e7cc94640 1 -- 172.21.15.29:0/2635783610 learned_addr learned my addr 172.21.15.29:0/2635783610 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.150+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6805560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e7d495640 1 -- 172.21.15.29:0/2635783610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 msgr2=0x55b8a6805560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6805560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e7d495640 1 -- 172.21.15.29:0/2635783610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64541b0 msgr2=0x55b8a6806790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64541b0 0x55b8a6806790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e7d495640 1 -- 172.21.15.29:0/2635783610 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8a64bf790 con 0x55b8a64334d0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64541b0 0x55b8a6806790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6805c20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6e74012d20 tx=0x7f6e7400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=159d8cd55e9c4e0a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2635783610 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74012110 con 0x55b8a64334d0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2635783610 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6e74004050 con 0x55b8a64334d0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e86666740 1 -- 172.21.15.29:0/2635783610 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b8a6770640 con 0x55b8a64334d0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2635783610 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6e740122b0 con 0x55b8a64334d0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.151+0000 7f6e86666740 10 client.26465 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e86666740 1 -- 172.21.15.29:0/2635783610 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b8a6538620 con 0x55b8a64334d0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2635783610 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55b8a6770640 con 0x55b8a64334d0 2024-09-21T02:50:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2635783610 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6e74010430 con 0x55b8a64334d0 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e86666740 10 client.26465 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e86666740 10 client.26465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e86666740 20 client.26465 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e86666740 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b8a6412d80 0x55b8a6812dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e86666740 4 client.26465 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.152+0000 7f6e86666740 1 -- 172.21.15.29:0/2635783610 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8a676c0f0 con 0x55b8a6412d80 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.153+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b8a6412d80 0x55b8a6812dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.153+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b8a6412d80 0x55b8a6812dc0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be8ee7c2b5ea9806 server_cookie=bd6de6d1fee4fe33 in_seq=0 out_seq=0 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.153+0000 7f6e60ff9640 10 client.26465 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.154+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2635783610 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b8a676c0f0 con 0x55b8a6412d80 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.154+0000 7f6e60ff9640 10 client.26465 handle_command_reply: tid=0 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.154+0000 7f6e86666740 1 client.26465 shutdown 2024-09-21T02:50:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.154+0000 7f6e86666740 1 -- 172.21.15.29:0/2635783610 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 msgr2=0x55b8a6805c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.154+0000 7f6e86666740 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6805c20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6e74012d20 tx=0x7f6e7400f790 comp rx=0 tx=0).stop 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 1 -- 172.21.15.29:0/2635783610 shutdown_connections 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b8a6412d80 0x55b8a6812dc0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64541b0 0x55b8a6806790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e60ff9640 0 client.26465 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6805c20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 1 --2- 172.21.15.29:0/2635783610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6805560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 1 -- 172.21.15.29:0/2635783610 >> 172.21.15.29:0/2635783610 conn(0x55b8a6448300 msgr2=0x55b8a67f5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 1 -- 172.21.15.29:0/2635783610 shutdown_connections 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 1 -- 172.21.15.29:0/2635783610 wait complete. 2024-09-21T02:50:30.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.155+0000 7f6e86666740 20 client.26465 trim_cache size 0 max 16384 2024-09-21T02:50:30.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e86666740 1 Processor -- start 2024-09-21T02:50:30.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e86666740 1 -- start start 2024-09-21T02:50:30.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e86666740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a6412d80 0x55b8a6803af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e86666740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a642b2f0 0x55b8a6804030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e86666740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 0x55b8a6805560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e86666740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781ac4f0 con 0x55b8a64334d0 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e86666740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781ae030 con 0x55b8a642b2f0 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e86666740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781acbc0 con 0x55b8a6412d80 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e7d495640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a642b2f0 0x55b8a6804030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 0x55b8a6805560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e7d495640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a642b2f0 0x55b8a6804030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:47444/0 (socket says 172.21.15.29:47444) 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.172+0000 7f6e7d495640 1 -- 172.21.15.29:0/2822619399 learned_addr learned my addr 172.21.15.29:0/2822619399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2822619399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a6412d80 0x55b8a6803af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e7d495640 1 -- 172.21.15.29:0/2822619399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a6412d80 msgr2=0x55b8a6803af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2822619399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a6412d80 0x55b8a6803af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e7d495640 1 -- 172.21.15.29:0/2822619399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 msgr2=0x55b8a6805560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:30.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2822619399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 0x55b8a6805560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e7d495640 1 -- 172.21.15.29:0/2822619399 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8a680da30 con 0x55b8a642b2f0 2024-09-21T02:50:30.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2822619399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a642b2f0 0x55b8a6804030 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6e74001eb0 tx=0x7f6e74012970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c3e24d31702da1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e62ffd640 1 -- 172.21.15.29:0/2822619399 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7402e070 con 0x55b8a642b2f0 2024-09-21T02:50:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e62ffd640 1 -- 172.21.15.29:0/2822619399 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e74005570 con 0x55b8a642b2f0 2024-09-21T02:50:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.173+0000 7f6e62ffd640 1 -- 172.21.15.29:0/2822619399 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74010e60 con 0x55b8a642b2f0 2024-09-21T02:50:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.174+0000 7f6e86666740 1 -- 172.21.15.29:0/2822619399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a642b2f0 msgr2=0x55b8a6804030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.174+0000 7f6e86666740 1 --2- 172.21.15.29:0/2822619399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a642b2f0 0x55b8a6804030 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6e74001eb0 tx=0x7f6e74012970 comp rx=0 tx=0).stop 2024-09-21T02:50:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 -- 172.21.15.29:0/2822619399 shutdown_connections 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 --2- 172.21.15.29:0/2822619399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a64334d0 0x55b8a6805560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 --2- 172.21.15.29:0/2822619399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a642b2f0 0x55b8a6804030 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 --2- 172.21.15.29:0/2822619399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a6412d80 0x55b8a6803af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 -- 172.21.15.29:0/2822619399 >> 172.21.15.29:0/2822619399 conn(0x55b8a6448300 msgr2=0x55b8a67f25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 -- 172.21.15.29:0/2822619399 shutdown_connections 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 -- 172.21.15.29:0/2822619399 wait complete. 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 -- start start 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 Processor -- start 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a6412d80 0x55b8a68084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6808b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6809700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781ad960 con 0x55b8a6412d80 2024-09-21T02:50:30.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781ad290 con 0x55b8a64334d0 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781aa4d0 con 0x55b8a642b2f0 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.175+0000 7f6e86666740 10 client.0 _ll_register_callbacks cb 0x7f6e74009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a6412d80 0x55b8a68084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a6412d80 0x55b8a68084d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60954/0 (socket says 172.21.15.29:60954) 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 -- 172.21.15.29:0/2991283150 learned_addr learned my addr 172.21.15.29:0/2991283150 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7d495640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6808b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6809700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 -- 172.21.15.29:0/2991283150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 msgr2=0x55b8a6808b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6808b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 -- 172.21.15.29:0/2991283150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 msgr2=0x55b8a6809700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6809700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 -- 172.21.15.29:0/2991283150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8a6809e60 con 0x55b8a6412d80 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6808b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:30.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e7cc94640 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a6412d80 0x55b8a68084d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6e680319b0 tx=0x7f6e68005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22ddfad7abb349c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2991283150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68035040 con 0x55b8a6412d80 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2991283150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6e68050080 con 0x55b8a6412d80 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e86666740 1 -- 172.21.15.29:0/2991283150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e74005570 con 0x55b8a6412d80 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2991283150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6e68050280 con 0x55b8a6412d80 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e86666740 10 client.17070 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.176+0000 7f6e86666740 1 -- 172.21.15.29:0/2991283150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b8a680d160 con 0x55b8a6412d80 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2991283150 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e74005570 con 0x55b8a6412d80 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2991283150 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6e6803d4e0 con 0x55b8a6412d80 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e86666740 10 client.17070 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e86666740 10 client.17070 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e86666740 20 client.17070 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:30.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e86666740 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b8a64541b0 0x55b8a687f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e86666740 4 client.17070 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:50:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e86666740 1 -- 172.21.15.29:0/2991283150 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8a6769520 con 0x55b8a64541b0 2024-09-21T02:50:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.177+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b8a64541b0 0x55b8a687f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.178+0000 7f6e60ff9640 10 client.17070 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.178+0000 7f6e7d495640 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b8a64541b0 0x55b8a687f790 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acc528b5d4d5a505 server_cookie=fb5cb36c98a0787b in_seq=0 out_seq=0 2024-09-21T02:50:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e60ff9640 1 -- 172.21.15.29:0/2991283150 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8a6769520 con 0x55b8a64541b0 2024-09-21T02:50:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e60ff9640 10 client.17070 handle_command_reply: tid=0 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 client.17070 shutdown 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 -- 172.21.15.29:0/2991283150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a6412d80 msgr2=0x55b8a68084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a6412d80 0x55b8a68084d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6e680319b0 tx=0x7f6e68005fd0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 -- 172.21.15.29:0/2991283150 shutdown_connections 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b8a64334d0 0x55b8a6809700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b8a642b2f0 0x55b8a6808b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b8a64541b0 0x55b8a687f790 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e60ff9640 0 client.17070 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 --2- 172.21.15.29:0/2991283150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8a6412d80 0x55b8a68084d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.230+0000 7f6e86666740 1 -- 172.21.15.29:0/2991283150 >> 172.21.15.29:0/2991283150 conn(0x55b8a6448300 msgr2=0x55b8a6803960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e86666740 1 -- 172.21.15.29:0/2991283150 shutdown_connections 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e86666740 1 -- 172.21.15.29:0/2991283150 wait complete. 2024-09-21T02:50:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e86666740 20 client.17070 trim_cache size 0 max 16384 2024-09-21T02:50:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:50:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:50:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:50:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e640b6590 msgr2=0x7f6e640b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e640b6590 0x7f6e640b8aa0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000eb10 tx=0x7f6e7007ada0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103fc0 msgr2=0x7f6e781a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103fc0 0x7f6e781a1150 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000eb40 tx=0x7f6e7000c2d0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3489853370 shutdown_connections 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e640b6590 0x7f6e640b8aa0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103fc0 0x7f6e781a1150 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e78103620 0x7f6e7819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 --2- 172.21.15.29:0/3489853370 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e781025a0 0x7f6e7806b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.231+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3489853370 >> 172.21.15.29:0/3489853370 conn(0x7f6e780fdee0 msgr2=0x7f6e78106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.232+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3489853370 shutdown_connections 2024-09-21T02:50:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:30.232+0000 7f6e7f6f9640 1 -- 172.21.15.29:0/3489853370 wait complete. 2024-09-21T02:50:31.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 225494 2.16 MB/sec execute 488 sec latency 45.615 ms 2024-09-21T02:50:31.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:31 smithi190 ceph-mon[33871]: pgmap v731: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-09-21T02:50:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:31 smithi029 ceph-mon[31006]: pgmap v731: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-09-21T02:50:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:31 smithi066 ceph-mon[33436]: pgmap v731: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-09-21T02:50:32.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 226094 2.16 MB/sec execute 489 sec latency 17.727 ms 2024-09-21T02:50:33.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 226834 2.16 MB/sec execute 490 sec latency 14.164 ms 2024-09-21T02:50:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:33 smithi190 ceph-mon[33871]: pgmap v732: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.5 MiB/s wr, 140 op/s 2024-09-21T02:50:33.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:33 smithi029 ceph-mon[31006]: pgmap v732: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.5 MiB/s wr, 140 op/s 2024-09-21T02:50:33.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:33 smithi066 ceph-mon[33436]: pgmap v732: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.5 MiB/s wr, 140 op/s 2024-09-21T02:50:34.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 227342 2.17 MB/sec execute 491 sec latency 13.673 ms 2024-09-21T02:50:35.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 227644 2.17 MB/sec execute 492 sec latency 20.291 ms 2024-09-21T02:50:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:35 smithi190 ceph-mon[33871]: pgmap v733: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:50:35.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:35 smithi029 ceph-mon[31006]: pgmap v733: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:50:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:35 smithi066 ceph-mon[33436]: pgmap v733: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-09-21T02:50:36.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 228241 2.17 MB/sec execute 493 sec latency 23.479 ms 2024-09-21T02:50:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:36 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:50:36.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:36 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:50:36.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:36 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:50:37.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 228900 2.17 MB/sec execute 494 sec latency 22.663 ms 2024-09-21T02:50:37.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:37 smithi190 ceph-mon[33871]: pgmap v734: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.7 MiB/s wr, 144 op/s 2024-09-21T02:50:37.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:50:37.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:50:37.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:50:37.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:37 smithi029 ceph-mon[31006]: pgmap v734: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.7 MiB/s wr, 144 op/s 2024-09-21T02:50:37.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:50:37.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:50:37.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:50:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:37 smithi066 ceph-mon[33436]: pgmap v734: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.7 MiB/s wr, 144 op/s 2024-09-21T02:50:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:50:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:50:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:50:38.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 229496 2.17 MB/sec execute 495 sec latency 15.281 ms 2024-09-21T02:50:39.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 230360 2.18 MB/sec execute 496 sec latency 14.224 ms 2024-09-21T02:50:39.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:39 smithi190 ceph-mon[33871]: pgmap v735: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-09-21T02:50:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:39 smithi029 ceph-mon[31006]: pgmap v735: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-09-21T02:50:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:39 smithi066 ceph-mon[33436]: pgmap v735: 129 pgs: 129 active+clean; 43 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-09-21T02:50:40.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 230848 2.18 MB/sec execute 497 sec latency 17.889 ms 2024-09-21T02:50:41.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 231102 2.18 MB/sec execute 498 sec latency 16.947 ms 2024-09-21T02:50:41.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:41 smithi190 ceph-mon[33871]: pgmap v736: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:50:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:41 smithi029 ceph-mon[31006]: pgmap v736: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:50:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:41 smithi066 ceph-mon[33436]: pgmap v736: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:50:42.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 231701 2.18 MB/sec execute 499 sec latency 27.691 ms 2024-09-21T02:50:43.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 232379 2.18 MB/sec execute 500 sec latency 18.452 ms 2024-09-21T02:50:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 -- 172.21.15.29:0/3802813928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ac1025c0 msgr2=0x7f60ac102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 --2- 172.21.15.29:0/3802813928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ac1025c0 0x7f60ac102a30 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f609c009900 tx=0x7f609c02a410 comp rx=0 tx=0).stop 2024-09-21T02:50:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 -- 172.21.15.29:0/3802813928 shutdown_connections 2024-09-21T02:50:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 --2- 172.21.15.29:0/3802813928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60ac103fe0 0x7f60ac10a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 --2- 172.21.15.29:0/3802813928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60ac103640 0x7f60ac103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 --2- 172.21.15.29:0/3802813928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ac1025c0 0x7f60ac102a30 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 -- 172.21.15.29:0/3802813928 >> 172.21.15.29:0/3802813928 conn(0x7f60ac0fdf00 msgr2=0x7f60ac1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 -- 172.21.15.29:0/3802813928 shutdown_connections 2024-09-21T02:50:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.352+0000 7f60b18a9640 1 -- 172.21.15.29:0/3802813928 wait complete. 2024-09-21T02:50:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.353+0000 7f60b18a9640 1 Processor -- start 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.353+0000 7f60b18a9640 1 -- start start 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60b18a9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60ac1025c0 0x7f60ac19c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60b18a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ac103640 0x7f60ac19c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60b18a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60ac103fe0 0x7f60ac19d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60b18a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac10e090 con 0x7f60ac103640 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60b18a9640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac10df10 con 0x7f60ac1025c0 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60b18a9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac10e210 con 0x7f60ac103fe0 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60ac1025c0 0x7f60ac19c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ac103640 0x7f60ac19c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60ac1025c0 0x7f60ac19c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:43130/0 (socket says 172.21.15.29:43130) 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 -- 172.21.15.29:0/2104791798 learned_addr learned my addr 172.21.15.29:0/2104791798 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60ac103fe0 0x7f60ac19d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 -- 172.21.15.29:0/2104791798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60ac103fe0 msgr2=0x7f60ac19d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60ac103fe0 0x7f60ac19d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 -- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ac103640 msgr2=0x7f60ac19c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ac103640 0x7f60ac19c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 -- 172.21.15.29:0/2104791798 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac19dab0 con 0x7f60ac1025c0 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.354+0000 7f60aaffd640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60ac1025c0 0x7f60ac19c070 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f609c009230 tx=0x7f609c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.355+0000 7f60b08a7640 1 -- 172.21.15.29:0/2104791798 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c03f030 con 0x7f60ac1025c0 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.355+0000 7f60b18a9640 1 -- 172.21.15.29:0/2104791798 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ac19dd40 con 0x7f60ac1025c0 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.355+0000 7f60b18a9640 1 -- 172.21.15.29:0/2104791798 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ac1a4490 con 0x7f60ac1025c0 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.355+0000 7f60b08a7640 1 -- 172.21.15.29:0/2104791798 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f609c004440 con 0x7f60ac1025c0 2024-09-21T02:50:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.355+0000 7f60b08a7640 1 -- 172.21.15.29:0/2104791798 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c035070 con 0x7f60ac1025c0 2024-09-21T02:50:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b08a7640 1 -- 172.21.15.29:0/2104791798 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f609c042030 con 0x7f60ac1025c0 2024-09-21T02:50:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b8819740 1 Processor -- start 2024-09-21T02:50:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b8819740 1 -- start start 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b8819740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238e41b0 0x55c323c8be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b8819740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238bb2f0 0x55c323c8c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b8819740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238c34d0 0x55c323c8d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b8819740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c323bfe6f0 con 0x55c3238bb2f0 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b8819740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c323bedb20 con 0x55c3238e41b0 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60b8819740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c323bf1930 con 0x55c3238c34d0 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238bb2f0 0x55c323c8c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238e41b0 0x55c323c8be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238e41b0 0x55c323c8be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:43146/0 (socket says 172.21.15.29:43146) 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60ab7fe640 1 -- 172.21.15.29:0/3134225140 learned_addr learned my addr 172.21.15.29:0/3134225140 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238bb2f0 0x55c323c8c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52872/0 (socket says 172.21.15.29:52872) 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.358+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238c34d0 0x55c323c8d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60ab7fe640 1 -- 172.21.15.29:0/3134225140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238c34d0 msgr2=0x55c323c8d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238c34d0 0x55c323c8d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60ab7fe640 1 -- 172.21.15.29:0/3134225140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238bb2f0 msgr2=0x55c323c8c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238bb2f0 0x55c323c8c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60ab7fe640 1 -- 172.21.15.29:0/3134225140 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c323bb3530 con 0x55c3238e41b0 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60aaffd640 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238bb2f0 0x55c323c8c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60b08a7640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60940b6510 0x7f60940b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238e41b0 0x55c323c8be50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009fd0 tx=0x7f60a40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a151d46f815f2965 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60aaffd640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60940b6510 0x7f60940b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f60b08a7640 1 -- 172.21.15.29:0/2104791798 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f609c107b90 con 0x7f60ac1025c0 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f6091ffb640 1 -- 172.21.15.29:0/3134225140 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a4004d20 con 0x55c3238e41b0 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f6091ffb640 1 -- 172.21.15.29:0/3134225140 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60a4004050 con 0x55c3238e41b0 2024-09-21T02:50:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.359+0000 7f6091ffb640 1 -- 172.21.15.29:0/3134225140 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a4005260 con 0x55c3238e41b0 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 -- 172.21.15.29:0/3134225140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238e41b0 msgr2=0x55c323c8be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60aaffd640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60940b6510 0x7f60940b8a20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f609c005760 tx=0x7f609c10b670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238e41b0 0x55c323c8be50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009fd0 tx=0x7f60a40118d0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 -- 172.21.15.29:0/3134225140 shutdown_connections 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238c34d0 0x55c323c8d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238bb2f0 0x55c323c8c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 --2- 172.21.15.29:0/3134225140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238e41b0 0x55c323c8be50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 -- 172.21.15.29:0/3134225140 >> 172.21.15.29:0/3134225140 conn(0x55c3238d8300 msgr2=0x55c323c878e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 -- 172.21.15.29:0/3134225140 shutdown_connections 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.360+0000 7f60b8819740 1 -- 172.21.15.29:0/3134225140 wait complete. 2024-09-21T02:50:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 1 -- start start 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 1 Processor -- start 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238bb2f0 0x55c323c955f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238c34d0 0x55c323c95cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238e41b0 0x55c323c96820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c323b25ab0 con 0x55c3238e41b0 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c323b93730 con 0x55c3238c34d0 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c323c8db30 con 0x55c3238bb2f0 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60b8819740 10 client.0 _ll_register_callbacks cb 0x55c323be76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238bb2f0 0x55c323c955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238e41b0 0x55c323c96820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238e41b0 0x55c323c96820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52880/0 (socket says 172.21.15.29:52880) 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60ab7fe640 1 -- 172.21.15.29:0/3022354093 learned_addr learned my addr 172.21.15.29:0/3022354093 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.361+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238c34d0 0x55c323c95cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60ab7fe640 1 -- 172.21.15.29:0/3022354093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238bb2f0 msgr2=0x55c323c955f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238bb2f0 0x55c323c955f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60ab7fe640 1 -- 172.21.15.29:0/3022354093 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238c34d0 msgr2=0x55c323c95cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238c34d0 0x55c323c95cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60ab7fe640 1 -- 172.21.15.29:0/3022354093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c32394f790 con 0x55c3238e41b0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238e41b0 0x55c323c96820 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f60a401b040 tx=0x7f60a4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0586421950a952 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f608affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f608affd640 1 -- 172.21.15.29:0/3022354093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a402d060 con 0x55c3238e41b0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f608affd640 1 -- 172.21.15.29:0/3022354093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f60a400f4e0 con 0x55c3238e41b0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60b8819740 1 -- 172.21.15.29:0/3022354093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60a4004050 con 0x55c3238e41b0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f608affd640 1 -- 172.21.15.29:0/3022354093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f60a4009c50 con 0x55c3238e41b0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60b8819740 10 client.17082 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.362+0000 7f60b8819740 1 -- 172.21.15.29:0/3022354093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c3239c8620 con 0x55c3238e41b0 2024-09-21T02:50:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.363+0000 7f608affd640 1 -- 172.21.15.29:0/3022354093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60a4004050 con 0x55c3238e41b0 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.363+0000 7f608affd640 1 -- 172.21.15.29:0/3022354093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f60a4025040 con 0x55c3238e41b0 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.363+0000 7f60b8819740 10 client.17082 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.363+0000 7f60b8819740 10 client.17082 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.363+0000 7f60b8819740 20 client.17082 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.363+0000 7f60b8819740 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3238a2d80 0x55c323ca2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.363+0000 7f60b8819740 4 client.17082 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.363+0000 7f60b8819740 1 -- 172.21.15.29:0/3022354093 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c323c00640 con 0x55c3238a2d80 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.364+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3238a2d80 0x55c323ca2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.364+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3238a2d80 0x55c323ca2ac0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7650a0fff5b34b9b server_cookie=ceecc2967806a7bf in_seq=0 out_seq=0 2024-09-21T02:50:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.364+0000 7f608affd640 10 client.17082 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f608affd640 1 -- 172.21.15.29:0/3022354093 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c323c00640 con 0x55c3238a2d80 2024-09-21T02:50:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f608affd640 10 client.17082 handle_command_reply: tid=0 2024-09-21T02:50:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 client.17082 shutdown 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 -- 172.21.15.29:0/3022354093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238e41b0 msgr2=0x55c323c96820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238e41b0 0x55c323c96820 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f60a401b040 tx=0x7f60a4005fd0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 -- 172.21.15.29:0/3022354093 shutdown_connections 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3238a2d80 0x55c323ca2ac0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238e41b0 0x55c323c96820 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238c34d0 0x55c323c95cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 --2- 172.21.15.29:0/3022354093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238bb2f0 0x55c323c955f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f60b8819740 1 -- 172.21.15.29:0/3022354093 >> 172.21.15.29:0/3022354093 conn(0x55c3238d8300 msgr2=0x55c323c8cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.365+0000 7f608affd640 0 client.17082 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.366+0000 7f60b8819740 1 -- 172.21.15.29:0/3022354093 shutdown_connections 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.366+0000 7f60b8819740 1 -- 172.21.15.29:0/3022354093 wait complete. 2024-09-21T02:50:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.366+0000 7f60b8819740 20 client.17082 trim_cache size 0 max 16384 2024-09-21T02:50:43.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60b8819740 1 Processor -- start 2024-09-21T02:50:43.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60b8819740 1 -- start start 2024-09-21T02:50:43.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60b8819740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 0x55c323c94f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60b8819740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c870f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60b8819740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c87630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60b8819740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1ae7c0 con 0x55c3238c34d0 2024-09-21T02:50:43.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60b8819740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1b0300 con 0x55c3238bb2f0 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60b8819740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1aee90 con 0x55c3238e41b0 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c870f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c87630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c870f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52890/0 (socket says 172.21.15.29:52890) 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c87630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38806/0 (socket says 172.21.15.29:38806) 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60aa7fc640 1 -- 172.21.15.29:0/2446106853 learned_addr learned my addr 172.21.15.29:0/2446106853 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 0x55c323c94f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60aa7fc640 1 -- 172.21.15.29:0/2446106853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 msgr2=0x55c323c87630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c87630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60aa7fc640 1 -- 172.21.15.29:0/2446106853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 msgr2=0x55c323c94f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 0x55c323c94f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60aa7fc640 1 -- 172.21.15.29:0/2446106853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c323c82f50 con 0x55c3238c34d0 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.382+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c87630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.383+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c870f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f60a0005850 tx=0x7f60a0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283738a461f3399a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.383+0000 7f60917fa640 1 -- 172.21.15.29:0/2446106853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0004310 con 0x55c3238c34d0 2024-09-21T02:50:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.383+0000 7f60917fa640 1 -- 172.21.15.29:0/2446106853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60a00044b0 con 0x55c3238c34d0 2024-09-21T02:50:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.383+0000 7f60917fa640 1 -- 172.21.15.29:0/2446106853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a00035f0 con 0x55c3238c34d0 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 -- 172.21.15.29:0/2446106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 msgr2=0x55c323c870f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c870f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f60a0005850 tx=0x7f60a0005f90 comp rx=0 tx=0).stop 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 -- 172.21.15.29:0/2446106853 shutdown_connections 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c87630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c870f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 --2- 172.21.15.29:0/2446106853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 0x55c323c94f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 -- 172.21.15.29:0/2446106853 >> 172.21.15.29:0/2446106853 conn(0x55c3238d8300 msgr2=0x55c323c947f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 -- 172.21.15.29:0/2446106853 shutdown_connections 2024-09-21T02:50:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 -- 172.21.15.29:0/2446106853 wait complete. 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 -- start start 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.384+0000 7f60b8819740 1 Processor -- start 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60b8819740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 0x55c323c988a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60b8819740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c98f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60b8819740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c99ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60b8819740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1afc30 con 0x55c3238c34d0 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60b8819740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1af560 con 0x55c3238bb2f0 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60b8819740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1ac7a0 con 0x55c3238e41b0 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60b8819740 10 client.0 _ll_register_callbacks cb 0x7f609c10b540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c98f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c98f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52902/0 (socket says 172.21.15.29:52902) 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60aa7fc640 1 -- 172.21.15.29:0/1695861656 learned_addr learned my addr 172.21.15.29:0/1695861656 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c99ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 0x55c323c988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60aa7fc640 1 -- 172.21.15.29:0/1695861656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 msgr2=0x55c323c99ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c99ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60aa7fc640 1 -- 172.21.15.29:0/1695861656 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 msgr2=0x55c323c988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 0x55c323c988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.385+0000 7f60aa7fc640 1 -- 172.21.15.29:0/1695861656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c323c9a230 con 0x55c3238c34d0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60aa7fc640 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c98f60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f60a0004330 tx=0x7f60a0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28106c027c052c9e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f608affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f608affd640 1 -- 172.21.15.29:0/1695861656 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a00035f0 con 0x55c3238c34d0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 1 -- 172.21.15.29:0/1695861656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60a00044b0 con 0x55c3238c34d0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f608affd640 1 -- 172.21.15.29:0/1695861656 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f60a0003790 con 0x55c3238c34d0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f608affd640 1 -- 172.21.15.29:0/1695861656 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f60a0003990 con 0x55c3238c34d0 2024-09-21T02:50:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 10 client.17088 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 1 -- 172.21.15.29:0/1695861656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c323c9d5d0 con 0x55c3238c34d0 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f608affd640 1 -- 172.21.15.29:0/1695861656 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60a00044b0 con 0x55c3238c34d0 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f608affd640 1 -- 172.21.15.29:0/1695861656 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f60a0005350 con 0x55c3238c34d0 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 10 client.17088 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 10 client.17088 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 20 client.17088 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3238781c0 0x55c323d0f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 4 client.17088 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.386+0000 7f60b8819740 1 -- 172.21.15.29:0/1695861656 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c323bfc0f0 con 0x55c3238781c0 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.387+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3238781c0 0x55c323d0f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.387+0000 7f60ab7fe640 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3238781c0 0x55c323d0f750 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2725d249abc64e0 server_cookie=c2f46faa6fcf9d9a in_seq=0 out_seq=0 2024-09-21T02:50:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.387+0000 7f608affd640 10 client.17088 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f608affd640 1 -- 172.21.15.29:0/1695861656 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c323bfc0f0 con 0x55c3238781c0 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f608affd640 10 client.17088 handle_command_reply: tid=0 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 client.17088 shutdown 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 -- 172.21.15.29:0/1695861656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 msgr2=0x55c323c98f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c98f60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f60a0004330 tx=0x7f60a0004360 comp rx=0 tx=0).stop 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 -- 172.21.15.29:0/1695861656 shutdown_connections 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3238e41b0 0x55c323c99ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3238781c0 0x55c323d0f750 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3238c34d0 0x55c323c98f60 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f608affd640 0 client.17088 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 --2- 172.21.15.29:0/1695861656 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3238bb2f0 0x55c323c988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 -- 172.21.15.29:0/1695861656 >> 172.21.15.29:0/1695861656 conn(0x55c3238d8300 msgr2=0x55c323c93700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 -- 172.21.15.29:0/1695861656 shutdown_connections 2024-09-21T02:50:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 1 -- 172.21.15.29:0/1695861656 wait complete. 2024-09-21T02:50:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.427+0000 7f60b8819740 20 client.17088 trim_cache size 0 max 16384 2024-09-21T02:50:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:50:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:50:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:50:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:50:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 -- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60940b6510 msgr2=0x7f60940b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60940b6510 0x7f60940b8a20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f609c005760 tx=0x7f609c10b670 comp rx=0 tx=0).stop 2024-09-21T02:50:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 -- 172.21.15.29:0/2104791798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60ac1025c0 msgr2=0x7f60ac19c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60ac1025c0 0x7f60ac19c070 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f609c009230 tx=0x7f609c004040 comp rx=0 tx=0).stop 2024-09-21T02:50:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 -- 172.21.15.29:0/2104791798 shutdown_connections 2024-09-21T02:50:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f60940b6510 0x7f60940b8a20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60ac103fe0 0x7f60ac19d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ac103640 0x7f60ac19c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 --2- 172.21.15.29:0/2104791798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60ac1025c0 0x7f60ac19c070 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 -- 172.21.15.29:0/2104791798 >> 172.21.15.29:0/2104791798 conn(0x7f60ac0fdf00 msgr2=0x7f60ac0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 -- 172.21.15.29:0/2104791798 shutdown_connections 2024-09-21T02:50:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:43.428+0000 7f60b18a9640 1 -- 172.21.15.29:0/2104791798 wait complete. 2024-09-21T02:50:43.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:43 smithi190 ceph-mon[33871]: pgmap v737: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-09-21T02:50:43.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:43 smithi029 ceph-mon[31006]: pgmap v737: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-09-21T02:50:43.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:43 smithi066 ceph-mon[33436]: pgmap v737: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-09-21T02:50:44.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 232982 2.18 MB/sec execute 501 sec latency 18.495 ms 2024-09-21T02:50:45.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 233884 2.19 MB/sec execute 502 sec latency 15.493 ms 2024-09-21T02:50:45.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:45 smithi190 ceph-mon[33871]: pgmap v738: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:50:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:45 smithi029 ceph-mon[31006]: pgmap v738: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:50:45.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:45 smithi066 ceph-mon[33436]: pgmap v738: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-09-21T02:50:46.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 234369 2.19 MB/sec execute 503 sec latency 16.868 ms 2024-09-21T02:50:47.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 234609 2.19 MB/sec execute 504 sec latency 13.839 ms 2024-09-21T02:50:47.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:47 smithi190 ceph-mon[33871]: pgmap v739: 129 pgs: 129 active+clean; 41 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:50:47.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:47 smithi029 ceph-mon[31006]: pgmap v739: 129 pgs: 129 active+clean; 41 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:50:47.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:47 smithi066 ceph-mon[33436]: pgmap v739: 129 pgs: 129 active+clean; 41 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-09-21T02:50:48.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 235149 2.19 MB/sec execute 505 sec latency 16.274 ms 2024-09-21T02:50:49.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 235794 2.19 MB/sec execute 506 sec latency 16.296 ms 2024-09-21T02:50:49.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:49 smithi190 ceph-mon[33871]: pgmap v740: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:50:49.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:49 smithi029 ceph-mon[31006]: pgmap v740: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:50:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:49 smithi066 ceph-mon[33436]: pgmap v740: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-09-21T02:50:50.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 236473 2.19 MB/sec execute 507 sec latency 20.804 ms 2024-09-21T02:50:51.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 237320 2.20 MB/sec execute 508 sec latency 15.971 ms 2024-09-21T02:50:51.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:51 smithi190 ceph-mon[33871]: pgmap v741: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-09-21T02:50:51.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:51 smithi029 ceph-mon[31006]: pgmap v741: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-09-21T02:50:51.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:51 smithi066 ceph-mon[33436]: pgmap v741: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-09-21T02:50:52.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 237822 2.20 MB/sec execute 509 sec latency 15.989 ms 2024-09-21T02:50:53.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 238120 2.20 MB/sec execute 510 sec latency 14.831 ms 2024-09-21T02:50:53.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:53 smithi190 ceph-mon[33871]: pgmap v742: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-09-21T02:50:53.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:53 smithi029 ceph-mon[31006]: pgmap v742: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-09-21T02:50:53.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:53 smithi066 ceph-mon[33436]: pgmap v742: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-09-21T02:50:54.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 238530 2.20 MB/sec execute 511 sec latency 74.195 ms 2024-09-21T02:50:55.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 239192 2.20 MB/sec execute 512 sec latency 16.984 ms 2024-09-21T02:50:55.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:55 smithi190 ceph-mon[33871]: pgmap v743: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-21T02:50:55.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:55 smithi029 ceph-mon[31006]: pgmap v743: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-21T02:50:55.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:55 smithi066 ceph-mon[33436]: pgmap v743: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-21T02:50:56.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 239797 2.20 MB/sec execute 513 sec latency 18.896 ms 2024-09-21T02:50:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.540+0000 7f70bdd20640 1 -- 172.21.15.29:0/3389805687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 msgr2=0x7f70b8103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.540+0000 7f70bdd20640 1 --2- 172.21.15.29:0/3389805687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 0x7f70b8103a30 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f70ac00d910 tx=0x7f70ac02c380 comp rx=0 tx=0).stop 2024-09-21T02:50:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.540+0000 7f70bdd20640 1 -- 172.21.15.29:0/3389805687 shutdown_connections 2024-09-21T02:50:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.540+0000 7f70bdd20640 1 --2- 172.21.15.29:0/3389805687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70b8103fc0 0x7f70b810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.540+0000 7f70bdd20640 1 --2- 172.21.15.29:0/3389805687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 0x7f70b8103a30 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.540+0000 7f70bdd20640 1 --2- 172.21.15.29:0/3389805687 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70b81025a0 0x7f70b8102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.540+0000 7f70bdd20640 1 -- 172.21.15.29:0/3389805687 >> 172.21.15.29:0/3389805687 conn(0x7f70b80fdee0 msgr2=0x7f70b8100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.541+0000 7f70bdd20640 1 -- 172.21.15.29:0/3389805687 shutdown_connections 2024-09-21T02:50:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.541+0000 7f70bdd20640 1 -- 172.21.15.29:0/3389805687 wait complete. 2024-09-21T02:50:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.541+0000 7f70bdd20640 1 Processor -- start 2024-09-21T02:50:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.541+0000 7f70bdd20640 1 -- start start 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70bdd20640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70b81025a0 0x7f70b819c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70bdd20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 0x7f70b819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70bdd20640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70b8103fc0 0x7f70b819d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70bdd20640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70b810cfd0 con 0x7f70b8103620 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70bdd20640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f70b810ce50 con 0x7f70b8103fc0 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70bdd20640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f70b810d150 con 0x7f70b81025a0 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 0x7f70b819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 0x7f70b819c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60780/0 (socket says 172.21.15.29:60780) 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70b81025a0 0x7f70b819c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 -- 172.21.15.29:0/80262193 learned_addr learned my addr 172.21.15.29:0/80262193 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b7fff640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70b8103fc0 0x7f70b819d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 -- 172.21.15.29:0/80262193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70b81025a0 msgr2=0x7f70b819c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70b81025a0 0x7f70b819c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 -- 172.21.15.29:0/80262193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70b8103fc0 msgr2=0x7f70b819d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70b8103fc0 0x7f70b819d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 -- 172.21.15.29:0/80262193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b819dae0 con 0x7f70b8103620 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b77fe640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70b81025a0 0x7f70b819c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.542+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 0x7f70b819c710 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f70ac000c00 tx=0x7f70ac0050f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.543+0000 7f70b4ff9640 1 -- 172.21.15.29:0/80262193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac02cd80 con 0x7f70b8103620 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.543+0000 7f70bdd20640 1 -- 172.21.15.29:0/80262193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b819dd70 con 0x7f70b8103620 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.543+0000 7f70b4ff9640 1 -- 172.21.15.29:0/80262193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70ac041070 con 0x7f70b8103620 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.543+0000 7f70b4ff9640 1 -- 172.21.15.29:0/80262193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac034420 con 0x7f70b8103620 2024-09-21T02:50:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.543+0000 7f70bdd20640 1 -- 172.21.15.29:0/80262193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b81a42e0 con 0x7f70b8103620 2024-09-21T02:50:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70c4c8f740 1 Processor -- start 2024-09-21T02:50:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70c4c8f740 1 -- start start 2024-09-21T02:50:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2374e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d23754c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2376010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c6d22e76f0 con 0x55c6d1fcd1b0 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c6d22d6b20 con 0x55c6d1fa42f0 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c6d22da930 con 0x55c6d1fac4d0 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70b4ff9640 1 -- 172.21.15.29:0/80262193 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f70ac00c050 con 0x7f70b8103620 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2374e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d23754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2374e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60794/0 (socket says 172.21.15.29:60794) 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d23754c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:59800/0 (socket says 172.21.15.29:59800) 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70b7fff640 1 -- 172.21.15.29:0/2548980143 learned_addr learned my addr 172.21.15.29:0/2548980143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.546+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2376010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b7fff640 1 -- 172.21.15.29:0/2548980143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 msgr2=0x55c6d2376010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b7fff640 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2376010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b7fff640 1 -- 172.21.15.29:0/2548980143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 msgr2=0x55c6d23754c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b7fff640 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d23754c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b7fff640 1 -- 172.21.15.29:0/2548980143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d229c530 con 0x55c6d1fcd1b0 2024-09-21T02:50:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b77fe640 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d23754c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b7fff640 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2374e50 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009d40 tx=0x7f70b00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d59830429ed47b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f709dffb640 1 -- 172.21.15.29:0/2548980143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b0012650 con 0x55c6d1fcd1b0 2024-09-21T02:50:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b4ff9640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70a00b6590 0x7f70a00b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f709dffb640 1 -- 172.21.15.29:0/2548980143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70b0009e30 con 0x55c6d1fcd1b0 2024-09-21T02:50:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f709dffb640 1 -- 172.21.15.29:0/2548980143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b000f9e0 con 0x55c6d1fcd1b0 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.547+0000 7f70b77fe640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70a00b6590 0x7f70a00b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70b4ff9640 1 -- 172.21.15.29:0/80262193 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f70ac1071f0 con 0x7f70b8103620 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2548980143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 msgr2=0x55c6d2374e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2374e50 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009d40 tx=0x7f70b00117b0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70b77fe640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70a00b6590 0x7f70a00b8aa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009fd0 tx=0x7f70a8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2548980143 shutdown_connections 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2376010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d23754c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2548980143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2374e50 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2548980143 >> 172.21.15.29:0/2548980143 conn(0x55c6d1fc1300 msgr2=0x55c6d23708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2548980143 shutdown_connections 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.548+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2548980143 wait complete. 2024-09-21T02:50:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 1 -- start start 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 1 Processor -- start 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d237e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 0x55c6d237f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c6d220eab0 con 0x55c6d1fa42f0 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c6d227c730 con 0x55c6d1fcd1b0 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c6d2376b30 con 0x55c6d1fac4d0 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70c4c8f740 10 client.0 _ll_register_callbacks cb 0x55c6d22d06b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d237e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.549+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d237e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60802/0 (socket says 172.21.15.29:60802) 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b77fe640 1 -- 172.21.15.29:0/2427884275 learned_addr learned my addr 172.21.15.29:0/2427884275 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b7fff640 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 0x55c6d237f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2427884275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 msgr2=0x55c6d237f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 0x55c6d237f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2427884275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 msgr2=0x55c6d237e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d237e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b6ffd640 1 -- 172.21.15.29:0/2427884275 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6d2038790 con 0x55c6d1fac4d0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237ecd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f70ac0ccca0 tx=0x7f70ac10af90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5fd459335d35f01 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f7096ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f7096ffd640 1 -- 172.21.15.29:0/2427884275 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0ccd70 con 0x55c6d1fac4d0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2427884275 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70b0009e30 con 0x55c6d1fac4d0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f7096ffd640 1 -- 172.21.15.29:0/2427884275 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f70ac0c9210 con 0x55c6d1fac4d0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f7096ffd640 1 -- 172.21.15.29:0/2427884275 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f70ac0cc710 con 0x55c6d1fac4d0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70c4c8f740 10 client.26644 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.550+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2427884275 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c6d20b1620 con 0x55c6d1fac4d0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f7096ffd640 1 -- 172.21.15.29:0/2427884275 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70b0009e30 con 0x55c6d1fac4d0 2024-09-21T02:50:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f7096ffd640 1 -- 172.21.15.29:0/2427884275 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f70ac0c88f0 con 0x55c6d1fac4d0 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f70c4c8f740 10 client.26644 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f70c4c8f740 10 client.26644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f70c4c8f740 20 client.26644 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c6d1f8bd80 0x55c6d238bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f70c4c8f740 4 client.26644 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2427884275 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6d22e9640 con 0x55c6d1f8bd80 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.551+0000 7f70b77fe640 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c6d1f8bd80 0x55c6d238bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.552+0000 7f7096ffd640 10 client.26644 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.552+0000 7f70b77fe640 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c6d1f8bd80 0x55c6d238bc30 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39ad250dc6039e82 server_cookie=45a453d3b866c3ab in_seq=0 out_seq=0 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f7096ffd640 1 -- 172.21.15.29:0/2427884275 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c6d22e9640 con 0x55c6d1f8bd80 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f7096ffd640 10 client.26644 handle_command_reply: tid=0 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 client.26644 shutdown 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 msgr2=0x55c6d237ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237ecd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f70ac0ccca0 tx=0x7f70ac10af90 comp rx=0 tx=0).stop 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2427884275 shutdown_connections 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 0x55c6d237f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237ecd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c6d1f8bd80 0x55c6d238bc30 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/2427884275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d237e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f7096ffd640 0 client.26644 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2427884275 >> 172.21.15.29:0/2427884275 conn(0x55c6d1fc1300 msgr2=0x55c6d236eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2427884275 shutdown_connections 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 1 -- 172.21.15.29:0/2427884275 wait complete. 2024-09-21T02:50:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.553+0000 7f70c4c8f740 20 client.26644 trim_cache size 0 max 16384 2024-09-21T02:50:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70c4c8f740 1 Processor -- start 2024-09-21T02:50:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70c4c8f740 1 -- start start 2024-09-21T02:50:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d236b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 0x55c6d237d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70b81ae700 con 0x55c6d1fa42f0 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f70b81b0240 con 0x55c6d1fcd1b0 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f70b81aedd0 con 0x55c6d1fac4d0 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d236b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d236b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60810/0 (socket says 172.21.15.29:60810) 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70b77fe640 1 -- 172.21.15.29:0/1233498291 learned_addr learned my addr 172.21.15.29:0/1233498291 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.569+0000 7f70b7fff640 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 0x55c6d237d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f70b77fe640 1 -- 172.21.15.29:0/1233498291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 msgr2=0x55c6d237cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f70b77fe640 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f70b77fe640 1 -- 172.21.15.29:0/1233498291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 msgr2=0x55c6d237d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f70b77fe640 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 0x55c6d237d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f70b77fe640 1 -- 172.21.15.29:0/1233498291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d237e3a0 con 0x55c6d1fa42f0 2024-09-21T02:50:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f70b77fe640 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d236b5c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x55c6d238c370 tx=0x7f70a801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27aad50be2676575 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f709d7fa640 1 -- 172.21.15.29:0/1233498291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a801e8c0 con 0x55c6d1fa42f0 2024-09-21T02:50:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f709d7fa640 1 -- 172.21.15.29:0/1233498291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70a802a420 con 0x55c6d1fa42f0 2024-09-21T02:50:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.570+0000 7f709d7fa640 1 -- 172.21.15.29:0/1233498291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a80316b0 con 0x55c6d1fa42f0 2024-09-21T02:50:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1233498291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 msgr2=0x55c6d236b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d236b5c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x55c6d238c370 tx=0x7f70a801a000 comp rx=0 tx=0).stop 2024-09-21T02:50:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1233498291 shutdown_connections 2024-09-21T02:50:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fcd1b0 0x55c6d237d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d237cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1233498291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fa42f0 0x55c6d236b5c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1233498291 >> 172.21.15.29:0/1233498291 conn(0x55c6d1fc1300 msgr2=0x55c6d2375ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1233498291 shutdown_connections 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1233498291 wait complete. 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.571+0000 7f70c4c8f740 1 -- start start 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70c4c8f740 1 Processor -- start 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d2383340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2383a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70c4c8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2384570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70b81afb70 con 0x55c6d1fcd1b0 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f70b81af4a0 con 0x55c6d1fa42f0 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70c4c8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f70b81ac6e0 con 0x55c6d1fac4d0 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70c4c8f740 10 client.0 _ll_register_callbacks cb 0x7f70ac02cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2384570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2383a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2384570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60824/0 (socket says 172.21.15.29:60824) 2024-09-21T02:50:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b7fff640 1 -- 172.21.15.29:0/1761046637 learned_addr learned my addr 172.21.15.29:0/1761046637 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2383a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38454/0 (socket says 172.21.15.29:38454) 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b77fe640 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d2383340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1761046637 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 msgr2=0x55c6d2383340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d2383340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1761046637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 msgr2=0x55c6d2384570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.572+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2384570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f70b6ffd640 1 -- 172.21.15.29:0/1761046637 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6d2384cd0 con 0x55c6d1fac4d0 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f70b7fff640 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2384570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f70b6ffd640 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2383a00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f70ac02cfd0 tx=0x7f70ac009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=907e73e8f9e01c1c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f7096ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f7096ffd640 1 -- 172.21.15.29:0/1761046637 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0f4040 con 0x55c6d1fac4d0 2024-09-21T02:50:56.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f7096ffd640 1 -- 172.21.15.29:0/1761046637 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f70ac10a390 con 0x55c6d1fac4d0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1761046637 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70a802a420 con 0x55c6d1fac4d0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f7096ffd640 1 -- 172.21.15.29:0/1761046637 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f70ac0cc8f0 con 0x55c6d1fac4d0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f70c4c8f740 10 client.26650 fetch_fsmap learned FSMap version 31 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1761046637 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c6d2386030 con 0x55c6d1fac4d0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.573+0000 7f7096ffd640 1 -- 172.21.15.29:0/1761046637 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70a802a420 con 0x55c6d1fac4d0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f7096ffd640 1 -- 172.21.15.29:0/1761046637 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f70ac0b8170 con 0x55c6d1fac4d0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f70c4c8f740 10 client.26650 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f70c4c8f740 10 client.26650 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f70c4c8f740 20 client.26650 populate_metadata read hostname 'smithi029' 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c6d1f611c0 0x55c6d23f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f70c4c8f740 4 client.26650 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1761046637 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c6d22e50f0 con 0x55c6d1f611c0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f70b77fe640 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c6d1f611c0 0x55c6d23f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f70b77fe640 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c6d1f611c0 0x55c6d23f86c0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c61def2b160d7f23 server_cookie=22dc2fd3c01fb4b1 in_seq=0 out_seq=0 2024-09-21T02:50:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.574+0000 7f7096ffd640 10 client.26650 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f7096ffd640 1 -- 172.21.15.29:0/1761046637 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c6d22e50f0 con 0x55c6d1f611c0 2024-09-21T02:50:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f7096ffd640 10 client.26650 handle_command_reply: tid=0 2024-09-21T02:50:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 client.26650 shutdown 2024-09-21T02:50:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1761046637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 msgr2=0x55c6d2383a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2383a00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f70ac02cfd0 tx=0x7f70ac009260 comp rx=0 tx=0).stop 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1761046637 shutdown_connections 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c6d1f611c0 0x55c6d23f86c0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c6d1fcd1b0 0x55c6d2384570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6d1fac4d0 0x55c6d2383a00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f7096ffd640 0 client.26650 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 --2- 172.21.15.29:0/1761046637 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c6d1fa42f0 0x55c6d2383340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.619+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1761046637 >> 172.21.15.29:0/1761046637 conn(0x55c6d1fc1300 msgr2=0x55c6d236b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1761046637 shutdown_connections 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70c4c8f740 1 -- 172.21.15.29:0/1761046637 wait complete. 2024-09-21T02:50:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70c4c8f740 20 client.26650 trim_cache size 0 max 16384 2024-09-21T02:50:56.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:50:56.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:50:56.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:50:56.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:50:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 -- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70a00b6590 msgr2=0x7f70a00b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70a00b6590 0x7f70a00b8aa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009fd0 tx=0x7f70a8009500 comp rx=0 tx=0).stop 2024-09-21T02:50:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 -- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 msgr2=0x7f70b819c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:50:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 0x7f70b819c710 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f70ac000c00 tx=0x7f70ac0050f0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 -- 172.21.15.29:0/80262193 shutdown_connections 2024-09-21T02:50:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f70a00b6590 0x7f70a00b8aa0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70b8103fc0 0x7f70b819d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b8103620 0x7f70b819c710 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 --2- 172.21.15.29:0/80262193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70b81025a0 0x7f70b819c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:50:56.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.620+0000 7f70bdd20640 1 -- 172.21.15.29:0/80262193 >> 172.21.15.29:0/80262193 conn(0x7f70b80fdee0 msgr2=0x7f70b8106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:50:56.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.621+0000 7f70bdd20640 1 -- 172.21.15.29:0/80262193 shutdown_connections 2024-09-21T02:50:56.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:50:56.621+0000 7f70bdd20640 1 -- 172.21.15.29:0/80262193 wait complete. 2024-09-21T02:50:57.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 240455 2.21 MB/sec execute 514 sec latency 21.222 ms 2024-09-21T02:50:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:57 smithi190 ceph-mon[33871]: pgmap v744: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.6 MiB/s wr, 126 op/s 2024-09-21T02:50:57.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:57 smithi029 ceph-mon[31006]: pgmap v744: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.6 MiB/s wr, 126 op/s 2024-09-21T02:50:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:57 smithi066 ceph-mon[33436]: pgmap v744: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.6 MiB/s wr, 126 op/s 2024-09-21T02:50:58.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 241015 2.21 MB/sec execute 515 sec latency 208.747 ms 2024-09-21T02:50:59.101 INFO:tasks.workunit.client.0.smithi029.stdout: 1 241528 2.21 MB/sec execute 516 sec latency 14.240 ms 2024-09-21T02:50:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:50:59 smithi190 ceph-mon[33871]: pgmap v745: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:50:59.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:50:59 smithi029 ceph-mon[31006]: pgmap v745: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:50:59.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:50:59 smithi066 ceph-mon[33436]: pgmap v745: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-09-21T02:51:00.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 241824 2.21 MB/sec execute 517 sec latency 14.963 ms 2024-09-21T02:51:01.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 242393 2.21 MB/sec execute 518 sec latency 36.884 ms 2024-09-21T02:51:01.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:01 smithi190 ceph-mon[33871]: pgmap v746: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-09-21T02:51:01.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:01 smithi029 ceph-mon[31006]: pgmap v746: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-09-21T02:51:01.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:01 smithi066 ceph-mon[33436]: pgmap v746: 129 pgs: 129 active+clean; 45 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-09-21T02:51:02.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 243053 2.21 MB/sec execute 519 sec latency 14.777 ms 2024-09-21T02:51:03.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 243673 2.21 MB/sec execute 520 sec latency 17.871 ms 2024-09-21T02:51:03.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:03 smithi190 ceph-mon[33871]: pgmap v747: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:51:03.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:03 smithi029 ceph-mon[31006]: pgmap v747: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:51:03.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:03 smithi066 ceph-mon[33436]: pgmap v747: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-09-21T02:51:04.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 244531 2.22 MB/sec execute 521 sec latency 14.285 ms 2024-09-21T02:51:05.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 245012 2.22 MB/sec execute 522 sec latency 18.241 ms 2024-09-21T02:51:05.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:05 smithi190 ceph-mon[33871]: pgmap v748: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-09-21T02:51:05.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:05 smithi029 ceph-mon[31006]: pgmap v748: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-09-21T02:51:05.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:05 smithi066 ceph-mon[33436]: pgmap v748: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-09-21T02:51:06.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 245254 2.22 MB/sec execute 523 sec latency 13.667 ms 2024-09-21T02:51:07.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 245790 2.22 MB/sec execute 524 sec latency 18.743 ms 2024-09-21T02:51:07.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:07 smithi190 ceph-mon[33871]: pgmap v749: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.6 MiB/s wr, 110 op/s 2024-09-21T02:51:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:07 smithi029 ceph-mon[31006]: pgmap v749: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.6 MiB/s wr, 110 op/s 2024-09-21T02:51:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:07 smithi066 ceph-mon[33436]: pgmap v749: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.6 MiB/s wr, 110 op/s 2024-09-21T02:51:08.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 246436 2.22 MB/sec execute 525 sec latency 14.375 ms 2024-09-21T02:51:09.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 247122 2.22 MB/sec execute 526 sec latency 22.428 ms 2024-09-21T02:51:09.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:09 smithi190 ceph-mon[33871]: pgmap v750: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:51:09.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:09 smithi029 ceph-mon[31006]: pgmap v750: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:51:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 -- 172.21.15.29:0/2612074106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103fc0 msgr2=0x7f274410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 --2- 172.21.15.29:0/2612074106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103fc0 0x7f274410a960 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f273c009c90 tx=0x7f273c02a860 comp rx=0 tx=0).stop 2024-09-21T02:51:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 -- 172.21.15.29:0/2612074106 shutdown_connections 2024-09-21T02:51:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 --2- 172.21.15.29:0/2612074106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103fc0 0x7f274410a960 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 --2- 172.21.15.29:0/2612074106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2744103620 0x7f2744103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 --2- 172.21.15.29:0/2612074106 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27441025a0 0x7f2744102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 -- 172.21.15.29:0/2612074106 >> 172.21.15.29:0/2612074106 conn(0x7f27440fdee0 msgr2=0x7f2744100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 -- 172.21.15.29:0/2612074106 shutdown_connections 2024-09-21T02:51:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.728+0000 7f2748e4d640 1 -- 172.21.15.29:0/2612074106 wait complete. 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2748e4d640 1 Processor -- start 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2748e4d640 1 -- start start 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2748e4d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27441025a0 0x7f2744106100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2748e4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103620 0x7f2744104a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2748e4d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2744103fc0 0x7f2744104f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2748e4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f274410e0b0 con 0x7f2744103620 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2748e4d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f274410df30 con 0x7f27441025a0 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2748e4d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f274410e230 con 0x7f2744103fc0 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f273bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103620 0x7f2744104a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f273bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103620 0x7f2744104a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45908/0 (socket says 172.21.15.29:45908) 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f273bfff640 1 -- 172.21.15.29:0/321639008 learned_addr learned my addr 172.21.15.29:0/321639008 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f2742d9d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2744103fc0 0x7f2744104f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f274259c640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27441025a0 0x7f2744106100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f273bfff640 1 -- 172.21.15.29:0/321639008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2744103fc0 msgr2=0x7f2744104f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f273bfff640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2744103fc0 0x7f2744104f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f273bfff640 1 -- 172.21.15.29:0/321639008 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27441025a0 msgr2=0x7f2744106100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f273bfff640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27441025a0 0x7f2744106100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.729+0000 7f273bfff640 1 -- 172.21.15.29:0/321639008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2744105610 con 0x7f2744103620 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.730+0000 7f273bfff640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103620 0x7f2744104a30 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f272c00e9f0 tx=0x7f272c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.730+0000 7f273b7fe640 1 -- 172.21.15.29:0/321639008 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c00cd80 con 0x7f2744103620 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.730+0000 7f2748e4d640 1 -- 172.21.15.29:0/321639008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27441a3f40 con 0x7f2744103620 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.730+0000 7f273b7fe640 1 -- 172.21.15.29:0/321639008 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f272c0054b0 con 0x7f2744103620 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.730+0000 7f273b7fe640 1 -- 172.21.15.29:0/321639008 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c010430 con 0x7f2744103620 2024-09-21T02:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.730+0000 7f2748e4d640 1 -- 172.21.15.29:0/321639008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27441a42f0 con 0x7f2744103620 2024-09-21T02:51:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274fdc4740 1 Processor -- start 2024-09-21T02:51:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274fdc4740 1 -- start start 2024-09-21T02:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f273b7fe640 1 -- 172.21.15.29:0/321639008 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f272c0046c0 con 0x7f2744103620 2024-09-21T02:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2aade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2aae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2aaf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274fdc4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5583d2a206f0 con 0x5583d26dd2f0 2024-09-21T02:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274fdc4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5583d2a0fb20 con 0x5583d27061b0 2024-09-21T02:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274fdc4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5583d2a13930 con 0x5583d26e54d0 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f273bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2aaf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f2742d9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2aade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f274259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2aae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f2742d9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2aade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:47470/0 (socket says 172.21.15.29:47470) 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.733+0000 7f273bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2aaf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39450/0 (socket says 172.21.15.29:39450) 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f2742d9d640 1 -- 172.21.15.29:0/2104342086 learned_addr learned my addr 172.21.15.29:0/2104342086 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f274259c640 1 -- 172.21.15.29:0/2104342086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 msgr2=0x5583d2aaf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f274259c640 1 --2- 172.21.15.29:0/2104342086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2aaf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f274259c640 1 -- 172.21.15.29:0/2104342086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 msgr2=0x5583d2aade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f274259c640 1 --2- 172.21.15.29:0/2104342086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2aade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f274259c640 1 -- 172.21.15.29:0/2104342086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583d29d5530 con 0x5583d26dd2f0 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f2742d9d640 1 --2- 172.21.15.29:0/2104342086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2aade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f274259c640 1 --2- 172.21.15.29:0/2104342086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2aae4c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f2734009c20 tx=0x7f2734011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8618a54e5fb907a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f273b7fe640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27300b6510 0x7f27300b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f2738ff9640 1 -- 172.21.15.29:0/2104342086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734004050 con 0x5583d26dd2f0 2024-09-21T02:51:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f2742d9d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27300b6510 0x7f27300b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.734+0000 7f2738ff9640 1 -- 172.21.15.29:0/2104342086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2734005020 con 0x5583d26dd2f0 2024-09-21T02:51:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.735+0000 7f2738ff9640 1 -- 172.21.15.29:0/2104342086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734004ba0 con 0x5583d26dd2f0 2024-09-21T02:51:09.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.735+0000 7f273b7fe640 1 -- 172.21.15.29:0/321639008 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f272c0e7a20 con 0x7f2744103620 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.735+0000 7f274fdc4740 1 -- 172.21.15.29:0/2104342086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 msgr2=0x5583d2aae4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.735+0000 7f2742d9d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27300b6510 0x7f27300b8a20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f273c02ad80 tx=0x7f273c031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.735+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2104342086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2aae4c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f2734009c20 tx=0x7f2734011810 comp rx=0 tx=0).stop 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 -- 172.21.15.29:0/2104342086 shutdown_connections 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2104342086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2aaf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2104342086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2aae4c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2104342086 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2aade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 -- 172.21.15.29:0/2104342086 >> 172.21.15.29:0/2104342086 conn(0x5583d26fa300 msgr2=0x5583d2aa98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 -- 172.21.15.29:0/2104342086 shutdown_connections 2024-09-21T02:51:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 -- 172.21.15.29:0/2104342086 wait complete. 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 -- start start 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 Processor -- start 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2ab75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 0x5583d2ab7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d27061b0 0x5583d2ab8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5583d2947ab0 con 0x5583d26dd2f0 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5583d29b5730 con 0x5583d26e54d0 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.736+0000 7f274fdc4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5583d2aafb30 con 0x5583d27061b0 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2ab75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274fdc4740 10 client.0 _ll_register_callbacks cb 0x5583d2a096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2ab75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45918/0 (socket says 172.21.15.29:45918) 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 -- 172.21.15.29:0/2983687323 learned_addr learned my addr 172.21.15.29:0/2983687323 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f2742d9d640 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d27061b0 0x5583d2ab8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f273bfff640 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 0x5583d2ab7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 -- 172.21.15.29:0/2983687323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d27061b0 msgr2=0x5583d2ab8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d27061b0 0x5583d2ab8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 -- 172.21.15.29:0/2983687323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 msgr2=0x5583d2ab7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 0x5583d2ab7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 -- 172.21.15.29:0/2983687323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583d2771790 con 0x5583d26dd2f0 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274259c640 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2ab75d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f273401b040 tx=0x7f2734004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274e21305791d37d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f2729ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f2729ffb640 1 -- 172.21.15.29:0/2983687323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734029070 con 0x5583d26dd2f0 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f274fdc4740 1 -- 172.21.15.29:0/2983687323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2734005020 con 0x5583d26dd2f0 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.737+0000 7f2729ffb640 1 -- 172.21.15.29:0/2983687323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2734004920 con 0x5583d26dd2f0 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f2729ffb640 1 -- 172.21.15.29:0/2983687323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2734035050 con 0x5583d26dd2f0 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f274fdc4740 10 client.17118 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f274fdc4740 1 -- 172.21.15.29:0/2983687323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5583d27ea620 con 0x5583d26dd2f0 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f2729ffb640 1 -- 172.21.15.29:0/2983687323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2734005020 con 0x5583d26dd2f0 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f2729ffb640 1 -- 172.21.15.29:0/2983687323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2734003cd0 con 0x5583d26dd2f0 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f274fdc4740 10 client.17118 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f274fdc4740 10 client.17118 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f274fdc4740 20 client.17118 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5583d26c4d80 0x5583d2ac4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f274fdc4740 4 client.17118 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f274fdc4740 1 -- 172.21.15.29:0/2983687323 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583d2a22640 con 0x5583d26c4d80 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.738+0000 7f273bfff640 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5583d26c4d80 0x5583d2ac4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.739+0000 7f2729ffb640 10 client.17118 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.739+0000 7f273bfff640 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5583d26c4d80 0x5583d2ac4a90 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc6311bae6b9439b server_cookie=4640351429b9e270 in_seq=0 out_seq=0 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.739+0000 7f2729ffb640 1 -- 172.21.15.29:0/2983687323 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5583d2a22640 con 0x5583d26c4d80 2024-09-21T02:51:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.739+0000 7f2729ffb640 10 client.17118 handle_command_reply: tid=0 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.739+0000 7f274fdc4740 1 client.17118 shutdown 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 -- 172.21.15.29:0/2983687323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 msgr2=0x5583d2ab75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2ab75d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f273401b040 tx=0x7f2734004280 comp rx=0 tx=0).stop 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 -- 172.21.15.29:0/2983687323 shutdown_connections 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d27061b0 0x5583d2ab8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 0x5583d2ab7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5583d26c4d80 0x5583d2ac4a90 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2983687323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2ab75d0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 -- 172.21.15.29:0/2983687323 >> 172.21.15.29:0/2983687323 conn(0x5583d26fa300 msgr2=0x5583d2aa7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f2729ffb640 0 client.17118 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 -- 172.21.15.29:0/2983687323 shutdown_connections 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 1 -- 172.21.15.29:0/2983687323 wait complete. 2024-09-21T02:51:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.740+0000 7f274fdc4740 20 client.17118 trim_cache size 0 max 16384 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.755+0000 7f274fdc4740 1 Processor -- start 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.755+0000 7f274fdc4740 1 -- start start 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.755+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26dd2f0 0x5583d2ac4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.755+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 0x5583d2aa45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.755+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d27061b0 0x5583d2aa8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.755+0000 7f274fdc4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27441ae780 con 0x5583d27061b0 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.755+0000 7f274fdc4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f27441b02c0 con 0x5583d26e54d0 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.755+0000 7f274fdc4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f27441aee50 con 0x5583d26dd2f0 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d27061b0 0x5583d2aa8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d27061b0 0x5583d2aa8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45928/0 (socket says 172.21.15.29:45928) 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 -- 172.21.15.29:0/3153790096 learned_addr learned my addr 172.21.15.29:0/3153790096 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f274259c640 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26dd2f0 0x5583d2ac4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f273bfff640 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 0x5583d2aa45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 -- 172.21.15.29:0/3153790096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26dd2f0 msgr2=0x5583d2ac4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26dd2f0 0x5583d2ac4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 -- 172.21.15.29:0/3153790096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 msgr2=0x5583d2aa45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 0x5583d2aa45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 -- 172.21.15.29:0/3153790096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583d26c4fb0 con 0x5583d27061b0 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f2742d9d640 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d27061b0 0x5583d2aa8610 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f273c035000 tx=0x7f273c040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895eae66a89f3594 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f272bfff640 1 -- 172.21.15.29:0/3153790096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c031c00 con 0x5583d27061b0 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f272bfff640 1 -- 172.21.15.29:0/3153790096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f273c036040 con 0x5583d27061b0 2024-09-21T02:51:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.756+0000 7f272bfff640 1 -- 172.21.15.29:0/3153790096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c0045f0 con 0x5583d27061b0 2024-09-21T02:51:09.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:09 smithi066 ceph-mon[33436]: pgmap v750: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:51:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 -- 172.21.15.29:0/3153790096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d27061b0 msgr2=0x5583d2aa8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d27061b0 0x5583d2aa8610 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f273c035000 tx=0x7f273c040000 comp rx=0 tx=0).stop 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 -- 172.21.15.29:0/3153790096 shutdown_connections 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d27061b0 0x5583d2aa8610 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d26e54d0 0x5583d2aa45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 --2- 172.21.15.29:0/3153790096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26dd2f0 0x5583d2ac4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 -- 172.21.15.29:0/3153790096 >> 172.21.15.29:0/3153790096 conn(0x5583d26fa300 msgr2=0x5583d2ab67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 -- 172.21.15.29:0/3153790096 shutdown_connections 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.757+0000 7f274fdc4740 1 -- 172.21.15.29:0/3153790096 wait complete. 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 1 -- start start 2024-09-21T02:51:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 1 Processor -- start 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2abbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2abc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2abcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27441afbf0 con 0x5583d26dd2f0 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f27441af520 con 0x5583d27061b0 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f27441ac760 con 0x5583d26e54d0 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274fdc4740 10 client.0 _ll_register_callbacks cb 0x7f2734005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2abbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2abbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45936/0 (socket says 172.21.15.29:45936) 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f274259c640 1 -- 172.21.15.29:0/2817340135 learned_addr learned my addr 172.21.15.29:0/2817340135 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.758+0000 7f2742d9d640 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2abcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f273bfff640 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2abc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f274259c640 1 -- 172.21.15.29:0/2817340135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 msgr2=0x5583d2abc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f274259c640 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2abc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f274259c640 1 -- 172.21.15.29:0/2817340135 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 msgr2=0x5583d2abcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f274259c640 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2abcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f274259c640 1 -- 172.21.15.29:0/2817340135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583d2abd490 con 0x5583d26dd2f0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f274259c640 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2abbb00 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f27340235f0 tx=0x7f27340236e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c0d9f525481834 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f2729ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f2729ffb640 1 -- 172.21.15.29:0/2817340135 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734029040 con 0x5583d26dd2f0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f2729ffb640 1 -- 172.21.15.29:0/2817340135 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f2734023b10 con 0x5583d26dd2f0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.759+0000 7f274fdc4740 1 -- 172.21.15.29:0/2817340135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f273c036040 con 0x5583d26dd2f0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f2729ffb640 1 -- 172.21.15.29:0/2817340135 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2734042050 con 0x5583d26dd2f0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f274fdc4740 10 client.17124 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f274fdc4740 1 -- 172.21.15.29:0/2817340135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5583d2abe7b0 con 0x5583d26dd2f0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f2729ffb640 1 -- 172.21.15.29:0/2817340135 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f273c036040 con 0x5583d26dd2f0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f2729ffb640 1 -- 172.21.15.29:0/2817340135 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2734004c30 con 0x5583d26dd2f0 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f274fdc4740 10 client.17124 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f274fdc4740 10 client.17124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:51:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f274fdc4740 20 client.17124 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5583d269a1c0 0x5583d2b314a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f274fdc4740 4 client.17124 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:51:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f274fdc4740 1 -- 172.21.15.29:0/2817340135 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5583d2a1e0f0 con 0x5583d269a1c0 2024-09-21T02:51:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.760+0000 7f273bfff640 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5583d269a1c0 0x5583d2b314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.761+0000 7f2729ffb640 10 client.17124 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.761+0000 7f273bfff640 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5583d269a1c0 0x5583d2b314a0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6ad984a231eabf6 server_cookie=f93114a3a5c3b527 in_seq=0 out_seq=0 2024-09-21T02:51:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f2729ffb640 1 -- 172.21.15.29:0/2817340135 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5583d2a1e0f0 con 0x5583d269a1c0 2024-09-21T02:51:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f2729ffb640 10 client.17124 handle_command_reply: tid=0 2024-09-21T02:51:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 client.17124 shutdown 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 -- 172.21.15.29:0/2817340135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 msgr2=0x5583d2abbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2abbb00 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f27340235f0 tx=0x7f27340236e0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 -- 172.21.15.29:0/2817340135 shutdown_connections 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5583d27061b0 0x5583d2abcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5583d26e54d0 0x5583d2abc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5583d269a1c0 0x5583d2b314a0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 --2- 172.21.15.29:0/2817340135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583d26dd2f0 0x5583d2abbb00 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f2729ffb640 0 client.17124 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.807+0000 7f274fdc4740 1 -- 172.21.15.29:0/2817340135 >> 172.21.15.29:0/2817340135 conn(0x5583d26fa300 msgr2=0x5583d2aa8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f274fdc4740 1 -- 172.21.15.29:0/2817340135 shutdown_connections 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f274fdc4740 1 -- 172.21.15.29:0/2817340135 wait complete. 2024-09-21T02:51:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f274fdc4740 20 client.17124 trim_cache size 0 max 16384 2024-09-21T02:51:09.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:51:09.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:51:09.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:51:09.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 -- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27300b6510 msgr2=0x7f27300b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27300b6510 0x7f27300b8a20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f273c02ad80 tx=0x7f273c031000 comp rx=0 tx=0).stop 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 -- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103620 msgr2=0x7f2744104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103620 0x7f2744104a30 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f272c00e9f0 tx=0x7f272c00edf0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 -- 172.21.15.29:0/321639008 shutdown_connections 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f27300b6510 0x7f27300b8a20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2744103fc0 0x7f2744104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2744103620 0x7f2744104a30 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 --2- 172.21.15.29:0/321639008 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f27441025a0 0x7f2744106100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 -- 172.21.15.29:0/321639008 >> 172.21.15.29:0/321639008 conn(0x7f27440fdee0 msgr2=0x7f27440ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 -- 172.21.15.29:0/321639008 shutdown_connections 2024-09-21T02:51:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:09.808+0000 7f2748e4d640 1 -- 172.21.15.29:0/321639008 wait complete. 2024-09-21T02:51:10.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 247816 2.23 MB/sec execute 527 sec latency 26.487 ms 2024-09-21T02:51:11.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 248446 2.23 MB/sec execute 528 sec latency 16.345 ms 2024-09-21T02:51:11.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:11 smithi190 ceph-mon[33871]: pgmap v751: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-09-21T02:51:11.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:11 smithi029 ceph-mon[31006]: pgmap v751: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-09-21T02:51:11.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:11 smithi066 ceph-mon[33436]: pgmap v751: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-09-21T02:51:12.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 248744 2.23 MB/sec execute 529 sec latency 16.124 ms 2024-09-21T02:51:13.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 249220 2.23 MB/sec execute 530 sec latency 17.771 ms 2024-09-21T02:51:13.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:13 smithi190 ceph-mon[33871]: pgmap v752: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-09-21T02:51:13.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:13 smithi029 ceph-mon[31006]: pgmap v752: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-09-21T02:51:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:13 smithi066 ceph-mon[33436]: pgmap v752: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-09-21T02:51:14.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 249866 2.23 MB/sec execute 531 sec latency 13.716 ms 2024-09-21T02:51:15.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 250529 2.23 MB/sec execute 532 sec latency 15.265 ms 2024-09-21T02:51:15.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:15 smithi029 ceph-mon[31006]: pgmap v753: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-09-21T02:51:15.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:15 smithi066 ceph-mon[33436]: pgmap v753: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-09-21T02:51:15.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:15 smithi190 ceph-mon[33871]: pgmap v753: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-09-21T02:51:16.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 251299 2.24 MB/sec execute 533 sec latency 18.233 ms 2024-09-21T02:51:17.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 251912 2.24 MB/sec execute 534 sec latency 22.874 ms 2024-09-21T02:51:17.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:17 smithi029 ceph-mon[31006]: pgmap v754: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-09-21T02:51:17.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:17 smithi066 ceph-mon[33436]: pgmap v754: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-09-21T02:51:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:17 smithi190 ceph-mon[33871]: pgmap v754: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-09-21T02:51:18.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 252265 2.24 MB/sec execute 535 sec latency 15.771 ms 2024-09-21T02:51:19.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 252703 2.24 MB/sec execute 536 sec latency 15.567 ms 2024-09-21T02:51:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:19 smithi029 ceph-mon[31006]: pgmap v755: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-09-21T02:51:19.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:19 smithi066 ceph-mon[33436]: pgmap v755: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-09-21T02:51:19.765 INFO:tasks.quiescer.fs.[cephfs]:Total duration (35.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:51:19.765 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.8 2024-09-21T02:51:19.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:51:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:19 smithi190 ceph-mon[33871]: pgmap v755: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-09-21T02:51:20.057 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:51:20.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 253334 2.24 MB/sec execute 537 sec latency 16.640 ms 2024-09-21T02:51:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 -- 172.21.15.29:0/3711805427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4109e10 msgr2=0x7f6bf4110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 --2- 172.21.15.29:0/3711805427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4109e10 0x7f6bf4110640 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009ca0 tx=0x7f6bec02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 -- 172.21.15.29:0/3711805427 shutdown_connections 2024-09-21T02:51:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 --2- 172.21.15.29:0/3711805427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4109e10 0x7f6bf4110640 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 --2- 172.21.15.29:0/3711805427 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bf41094c0 0x7f6bf41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 --2- 172.21.15.29:0/3711805427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf4103b80 0x7f6bf4103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 -- 172.21.15.29:0/3711805427 >> 172.21.15.29:0/3711805427 conn(0x7f6bf40ff610 msgr2=0x7f6bf4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 -- 172.21.15.29:0/3711805427 shutdown_connections 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.619+0000 7f6bf9947640 1 -- 172.21.15.29:0/3711805427 wait complete. 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.620+0000 7f6bf9947640 1 Processor -- start 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.620+0000 7f6bf9947640 1 -- start start 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf9947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4103b80 0x7f6bf419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf9947640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bf41094c0 0x7f6bf419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf9947640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf4109e10 0x7f6bf419fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf9947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf41139d0 con 0x7f6bf4103b80 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf9947640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf4113850 con 0x7f6bf41094c0 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf9947640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf4113b50 con 0x7f6bf4109e10 2024-09-21T02:51:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4103b80 0x7f6bf419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bf41094c0 0x7f6bf419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4103b80 0x7f6bf419a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37002/0 (socket says 172.21.15.29:37002) 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 -- 172.21.15.29:0/2324430385 learned_addr learned my addr 172.21.15.29:0/2324430385 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf37fe640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf4109e10 0x7f6bf419fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 -- 172.21.15.29:0/2324430385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf4109e10 msgr2=0x7f6bf419fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf4109e10 0x7f6bf419fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 -- 172.21.15.29:0/2324430385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bf41094c0 msgr2=0x7f6bf419a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bf41094c0 0x7f6bf419a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 -- 172.21.15.29:0/2324430385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf41a02f0 con 0x7f6bf4103b80 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bebfff640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bf41094c0 0x7f6bf419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf2ffd640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4103b80 0x7f6bf419a380 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6be400e9f0 tx=0x7f6be400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.621+0000 7f6bf8945640 1 -- 172.21.15.29:0/2324430385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be400cd80 con 0x7f6bf4103b80 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.622+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf41a05e0 con 0x7f6bf4103b80 2024-09-21T02:51:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.622+0000 7f6bf8945640 1 -- 172.21.15.29:0/2324430385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6be40108c0 con 0x7f6bf4103b80 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.622+0000 7f6bf8945640 1 -- 172.21.15.29:0/2324430385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be40187c0 con 0x7f6bf4103b80 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.622+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf41a6d40 con 0x7f6bf4103b80 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.623+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf4104e00 con 0x7f6bf4103b80 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.625+0000 7f6bf8945640 1 -- 172.21.15.29:0/2324430385 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f6be4010430 con 0x7f6bf4103b80 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.625+0000 7f6bf8945640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6be00b6510 0x7f6be00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.625+0000 7f6bebfff640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6be00b6510 0x7f6be00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.625+0000 7f6bf8945640 1 -- 172.21.15.29:0/2324430385 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6be40e7a50 con 0x7f6bf4103b80 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.626+0000 7f6bebfff640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6be00b6510 0x7f6be00b8a20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4102e60 tx=0x7f6bdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.631+0000 7f6bf8945640 1 -- 172.21.15.29:0/2324430385 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6be40b22e0 con 0x7f6bf4103b80 2024-09-21T02:51:20.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.895+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bf4103f90 con 0x7f6bf4103b80 2024-09-21T02:51:20.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.896+0000 7f6bf8945640 1 -- 172.21.15.29:0/2324430385 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f6be40b7220 con 0x7f6bf4103b80 2024-09-21T02:51:20.898 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:51:20.898 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:51:20.898 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:51:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.898+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6be00b6510 msgr2=0x7f6be00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.898+0000 7f6bf9947640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6be00b6510 0x7f6be00b8a20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4102e60 tx=0x7f6bdc005f90 comp rx=0 tx=0).stop 2024-09-21T02:51:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.898+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4103b80 msgr2=0x7f6bf419a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.898+0000 7f6bf9947640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4103b80 0x7f6bf419a380 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6be400e9f0 tx=0x7f6be400eeb0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.898+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 shutdown_connections 2024-09-21T02:51:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.898+0000 7f6bf9947640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6be00b6510 0x7f6be00b8a20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.898+0000 7f6bf9947640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf4109e10 0x7f6bf419fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.898+0000 7f6bf9947640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bf41094c0 0x7f6bf419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.899+0000 7f6bf9947640 1 --2- 172.21.15.29:0/2324430385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf4103b80 0x7f6bf419a380 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.899+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 >> 172.21.15.29:0/2324430385 conn(0x7f6bf40ff610 msgr2=0x7f6bf4106ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.899+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 shutdown_connections 2024-09-21T02:51:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:20.899+0000 7f6bf9947640 1 -- 172.21.15.29:0/2324430385 wait complete. 2024-09-21T02:51:21.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 253874 2.24 MB/sec execute 538 sec latency 117.470 ms 2024-09-21T02:51:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:21 smithi029 ceph-mon[31006]: pgmap v756: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T02:51:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:21 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2324430385' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:21.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:21 smithi190 ceph-mon[33871]: pgmap v756: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T02:51:21.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:21 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2324430385' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:21.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:21 smithi066 ceph-mon[33436]: pgmap v756: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T02:51:21.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:21 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2324430385' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:22.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 254382 2.24 MB/sec execute 539 sec latency 146.454 ms 2024-09-21T02:51:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 -- 172.21.15.29:0/1707296604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 msgr2=0x7fc2a0103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 --2- 172.21.15.29:0/1707296604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 0x7fc2a0103a40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc294009900 tx=0x7fc29402a410 comp rx=0 tx=0).stop 2024-09-21T02:51:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 -- 172.21.15.29:0/1707296604 shutdown_connections 2024-09-21T02:51:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 --2- 172.21.15.29:0/1707296604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2a0103fd0 0x7fc2a010a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 --2- 172.21.15.29:0/1707296604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 0x7fc2a0103a40 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 --2- 172.21.15.29:0/1707296604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2a01025b0 0x7fc2a0102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 -- 172.21.15.29:0/1707296604 >> 172.21.15.29:0/1707296604 conn(0x7fc2a00fdf30 msgr2=0x7fc2a01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 -- 172.21.15.29:0/1707296604 shutdown_connections 2024-09-21T02:51:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.916+0000 7fc2a5720640 1 -- 172.21.15.29:0/1707296604 wait complete. 2024-09-21T02:51:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.917+0000 7fc2a5720640 1 Processor -- start 2024-09-21T02:51:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.917+0000 7fc2a5720640 1 -- start start 2024-09-21T02:51:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc2a5720640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2a01025b0 0x7fc2a0197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc2a5720640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 0x7fc2a0198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc2a5720640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2a0103fd0 0x7fc2a0198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc2a5720640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a010cfe0 con 0x7fc2a0103630 2024-09-21T02:51:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc2a5720640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a010ce60 con 0x7fc2a0103fd0 2024-09-21T02:51:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc2a5720640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a010d160 con 0x7fc2a01025b0 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 0x7fc2a0198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2a01025b0 0x7fc2a0197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 0x7fc2a0198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37018/0 (socket says 172.21.15.29:37018) 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2a01025b0 0x7fc2a0197cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53018/0 (socket says 172.21.15.29:53018) 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 -- 172.21.15.29:0/4277471401 learned_addr learned my addr 172.21.15.29:0/4277471401 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29f7fe640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2a0103fd0 0x7fc2a0198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 -- 172.21.15.29:0/4277471401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2a01025b0 msgr2=0x7fc2a0197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2a01025b0 0x7fc2a0197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 -- 172.21.15.29:0/4277471401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2a0103fd0 msgr2=0x7fc2a0198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2a0103fd0 0x7fc2a0198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 -- 172.21.15.29:0/4277471401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a01996f0 con 0x7fc2a0103630 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29effd640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2a01025b0 0x7fc2a0197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.918+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 0x7fc2a0198320 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc294009230 tx=0x7fc294004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.919+0000 7fc287fff640 1 -- 172.21.15.29:0/4277471401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29403f030 con 0x7fc2a0103630 2024-09-21T02:51:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.919+0000 7fc287fff640 1 -- 172.21.15.29:0/4277471401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc294004440 con 0x7fc2a0103630 2024-09-21T02:51:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.919+0000 7fc2a5720640 1 -- 172.21.15.29:0/4277471401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a0199980 con 0x7fc2a0103630 2024-09-21T02:51:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.919+0000 7fc287fff640 1 -- 172.21.15.29:0/4277471401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc294035070 con 0x7fc2a0103630 2024-09-21T02:51:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.919+0000 7fc2a5720640 1 -- 172.21.15.29:0/4277471401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a01a00d0 con 0x7fc2a0103630 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc2ac68a740 1 Processor -- start 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc287fff640 1 -- 172.21.15.29:0/4277471401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc294042030 con 0x7fc2a0103630 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc2ac68a740 1 -- start start 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450af010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2450206f0 con 0x55a244d061b0 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a24500fb20 con 0x55a244cdd2f0 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a245013930 con 0x55a244ce54d0 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc29f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc29f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37030/0 (socket says 172.21.15.29:37030) 2024-09-21T02:51:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc29f7fe640 1 -- 172.21.15.29:0/2581140463 learned_addr learned my addr 172.21.15.29:0/2581140463 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.922+0000 7fc29effd640 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc29effd640 1 -- 172.21.15.29:0/2581140463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 msgr2=0x55a2450af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc29effd640 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc29effd640 1 -- 172.21.15.29:0/2581140463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 msgr2=0x55a2450ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc29effd640 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc29effd640 1 -- 172.21.15.29:0/2581140463 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a244fd5530 con 0x55a244cdd2f0 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc29f7fe640 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc29effd640 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450ae4c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc290009fd0 tx=0x7fc2900118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2604fad38a04959 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc287fff640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2880b6590 0x7fc2880b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc2857fa640 1 -- 172.21.15.29:0/2581140463 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc290004310 con 0x55a244cdd2f0 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc29f7fe640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2880b6590 0x7fc2880b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc2857fa640 1 -- 172.21.15.29:0/2581140463 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2900044b0 con 0x55a244cdd2f0 2024-09-21T02:51:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc2857fa640 1 -- 172.21.15.29:0/2581140463 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2900035f0 con 0x55a244cdd2f0 2024-09-21T02:51:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.923+0000 7fc287fff640 1 -- 172.21.15.29:0/4277471401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc294107a60 con 0x7fc2a0103630 2024-09-21T02:51:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 -- 172.21.15.29:0/2581140463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 msgr2=0x55a2450ae4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450ae4c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc290009fd0 tx=0x7fc2900118d0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc29f7fe640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2880b6590 0x7fc2880b8aa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc29800b420 tx=0x7fc29800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 -- 172.21.15.29:0/2581140463 shutdown_connections 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450ae4c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/2581140463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 -- 172.21.15.29:0/2581140463 >> 172.21.15.29:0/2581140463 conn(0x55a244cfa300 msgr2=0x55a2450a98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 -- 172.21.15.29:0/2581140463 shutdown_connections 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.924+0000 7fc2ac68a740 1 -- 172.21.15.29:0/2581140463 wait complete. 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 1 -- start start 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 1 Processor -- start 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450b8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a244f47ab0 con 0x55a244d061b0 2024-09-21T02:51:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a244fb5730 con 0x55a244cdd2f0 2024-09-21T02:51:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a2450afb30 con 0x55a244ce54d0 2024-09-21T02:51:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc2ac68a740 10 client.0 _ll_register_callbacks cb 0x55a2450096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29effd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29effd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450b7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:56432/0 (socket says 172.21.15.29:56432) 2024-09-21T02:51:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29effd640 1 -- 172.21.15.29:0/1378942865 learned_addr learned my addr 172.21.15.29:0/1378942865 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29effd640 1 -- 172.21.15.29:0/1378942865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 msgr2=0x55a2450b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29effd640 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450b7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29effd640 1 -- 172.21.15.29:0/1378942865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 msgr2=0x55a2450b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.925+0000 7fc29effd640 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc29effd640 1 -- 172.21.15.29:0/1378942865 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a244d71790 con 0x55a244cdd2f0 2024-09-21T02:51:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc29f7fe640 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc29effd640 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450b7600 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc29001b040 tx=0x7fc2900043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4dc754d97c45e34 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc282ff5640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc282ff5640 1 -- 172.21.15.29:0/1378942865 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc290029070 con 0x55a244cdd2f0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc282ff5640 1 -- 172.21.15.29:0/1378942865 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc290003b80 con 0x55a244cdd2f0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc2ac68a740 1 -- 172.21.15.29:0/1378942865 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2900044b0 con 0x55a244cdd2f0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc282ff5640 1 -- 172.21.15.29:0/1378942865 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc290035050 con 0x55a244cdd2f0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc2ac68a740 10 client.26543 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc2ac68a740 1 -- 172.21.15.29:0/1378942865 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a244dea620 con 0x55a244cdd2f0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.926+0000 7fc282ff5640 1 -- 172.21.15.29:0/1378942865 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2900044b0 con 0x55a244cdd2f0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc282ff5640 1 -- 172.21.15.29:0/1378942865 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc290023430 con 0x55a244cdd2f0 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc2ac68a740 10 client.26543 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc2ac68a740 10 client.26543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc2ac68a740 20 client.26543 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a244cc4d80 0x55a2450c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc2ac68a740 4 client.26543 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc2ac68a740 1 -- 172.21.15.29:0/1378942865 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a245022640 con 0x55a244cc4d80 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a244cc4d80 0x55a2450c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc282ff5640 10 client.26543 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.927+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a244cc4d80 0x55a2450c4b50 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8417ad5ccc90c546 server_cookie=9fe605946d5b09fc in_seq=0 out_seq=0 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc282ff5640 1 -- 172.21.15.29:0/1378942865 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a245022640 con 0x55a244cc4d80 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc282ff5640 10 client.26543 handle_command_reply: tid=0 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 client.26543 shutdown 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 -- 172.21.15.29:0/1378942865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 msgr2=0x55a2450b7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450b7600 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc29001b040 tx=0x7fc2900043b0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 -- 172.21.15.29:0/1378942865 shutdown_connections 2024-09-21T02:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a244cc4d80 0x55a2450c4b50 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/1378942865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450b7600 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc2ac68a740 1 -- 172.21.15.29:0/1378942865 >> 172.21.15.29:0/1378942865 conn(0x55a244cfa300 msgr2=0x55a2450a7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.928+0000 7fc282ff5640 0 client.26543 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.929+0000 7fc2ac68a740 1 -- 172.21.15.29:0/1378942865 shutdown_connections 2024-09-21T02:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.929+0000 7fc2ac68a740 1 -- 172.21.15.29:0/1378942865 wait complete. 2024-09-21T02:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.929+0000 7fc2ac68a740 20 client.26543 trim_cache size 0 max 16384 2024-09-21T02:51:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc2ac68a740 1 Processor -- start 2024-09-21T02:51:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc2ac68a740 1 -- start start 2024-09-21T02:51:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450bf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450a8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a01aa2d0 con 0x55a244d061b0 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a01abe10 con 0x55a244cdd2f0 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a01aa9a0 con 0x55a244ce54d0 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450bf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450bf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53066/0 (socket says 172.21.15.29:53066) 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29e7fc640 1 -- 172.21.15.29:0/565293461 learned_addr learned my addr 172.21.15.29:0/565293461 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29effd640 1 --2- 172.21.15.29:0/565293461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29e7fc640 1 -- 172.21.15.29:0/565293461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 msgr2=0x55a2450c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/565293461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29e7fc640 1 -- 172.21.15.29:0/565293461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 msgr2=0x55a2450a8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/565293461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450a8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.945+0000 7fc29e7fc640 1 -- 172.21.15.29:0/565293461 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2450b5780 con 0x55a244ce54d0 2024-09-21T02:51:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.946+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/565293461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450bf710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc2940cd8b0 tx=0x7fc2941050b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46cbebd450a0f67f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.946+0000 7fc284ff9640 1 -- 172.21.15.29:0/565293461 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc294049040 con 0x55a244ce54d0 2024-09-21T02:51:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.946+0000 7fc284ff9640 1 -- 172.21.15.29:0/565293461 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2940cd430 con 0x55a244ce54d0 2024-09-21T02:51:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.946+0000 7fc284ff9640 1 -- 172.21.15.29:0/565293461 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2940d2350 con 0x55a244ce54d0 2024-09-21T02:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 -- 172.21.15.29:0/565293461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 msgr2=0x55a2450bf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/565293461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450bf710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc2940cd8b0 tx=0x7fc2941050b0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 -- 172.21.15.29:0/565293461 shutdown_connections 2024-09-21T02:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/565293461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244d061b0 0x55a2450a8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/565293461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244ce54d0 0x55a2450bf710 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/565293461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244cdd2f0 0x55a2450c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 -- 172.21.15.29:0/565293461 >> 172.21.15.29:0/565293461 conn(0x55a244cfa300 msgr2=0x55a2450aec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 -- 172.21.15.29:0/565293461 shutdown_connections 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 -- 172.21.15.29:0/565293461 wait complete. 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 -- start start 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 Processor -- start 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.947+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244cdd2f0 0x55a2450bc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244ce54d0 0x55a2450bc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc2ac68a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244d061b0 0x55a2450bd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a01ab740 con 0x55a244cdd2f0 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a01ab070 con 0x55a244ce54d0 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc2ac68a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a01a82b0 con 0x55a244d061b0 2024-09-21T02:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc2ac68a740 10 client.0 _ll_register_callbacks cb 0x7fc29800bd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244cdd2f0 0x55a2450bc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244cdd2f0 0x55a2450bc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37060/0 (socket says 172.21.15.29:37060) 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 -- 172.21.15.29:0/858071376 learned_addr learned my addr 172.21.15.29:0/858071376 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29f7fe640 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244d061b0 0x55a2450bd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244ce54d0 0x55a2450bc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 -- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244d061b0 msgr2=0x55a2450bd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244d061b0 0x55a2450bd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 -- 172.21.15.29:0/858071376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244ce54d0 msgr2=0x55a2450bc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244ce54d0 0x55a2450bc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 -- 172.21.15.29:0/858071376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2450bdad0 con 0x55a244cdd2f0 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29f7fe640 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244d061b0 0x55a2450bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc29effd640 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244cdd2f0 0x55a2450bc1b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc2900239a0 tx=0x7fc290005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b3ad578fa74962 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc282ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc282ff5640 1 -- 172.21.15.29:0/858071376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29003e060 con 0x55a244cdd2f0 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc282ff5640 1 -- 172.21.15.29:0/858071376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc290023630 con 0x55a244cdd2f0 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.948+0000 7fc2ac68a740 1 -- 172.21.15.29:0/858071376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2940cd430 con 0x55a244cdd2f0 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc282ff5640 1 -- 172.21.15.29:0/858071376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc290045050 con 0x55a244cdd2f0 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc2ac68a740 10 client.17148 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc2ac68a740 1 -- 172.21.15.29:0/858071376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a2450bedf0 con 0x55a244cdd2f0 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc282ff5640 1 -- 172.21.15.29:0/858071376 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2940cd430 con 0x55a244cdd2f0 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc282ff5640 1 -- 172.21.15.29:0/858071376 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fc29000f4e0 con 0x55a244cdd2f0 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc2ac68a740 10 client.17148 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc2ac68a740 10 client.17148 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc2ac68a740 20 client.17148 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a244c9a1c0 0x55a245131520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc2ac68a740 4 client.17148 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:51:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc2ac68a740 1 -- 172.21.15.29:0/858071376 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a24501e0f0 con 0x55a244c9a1c0 2024-09-21T02:51:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.949+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a244c9a1c0 0x55a245131520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.950+0000 7fc282ff5640 10 client.17148 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:22.950+0000 7fc29e7fc640 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a244c9a1c0 0x55a245131520 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0c738654b75a47c server_cookie=ce36d4ccaac18ca0 in_seq=0 out_seq=0 2024-09-21T02:51:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc282ff5640 1 -- 172.21.15.29:0/858071376 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a24501e0f0 con 0x55a244c9a1c0 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc282ff5640 10 client.17148 handle_command_reply: tid=0 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 client.17148 shutdown 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 -- 172.21.15.29:0/858071376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244cdd2f0 msgr2=0x55a2450bc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244cdd2f0 0x55a2450bc1b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc2900239a0 tx=0x7fc290005970 comp rx=0 tx=0).stop 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 -- 172.21.15.29:0/858071376 shutdown_connections 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a244d061b0 0x55a2450bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a244ce54d0 0x55a2450bc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a244c9a1c0 0x55a245131520 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 --2- 172.21.15.29:0/858071376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a244cdd2f0 0x55a2450bc1b0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc282ff5640 0 client.17148 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 -- 172.21.15.29:0/858071376 >> 172.21.15.29:0/858071376 conn(0x55a244cfa300 msgr2=0x55a2450a8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 -- 172.21.15.29:0/858071376 shutdown_connections 2024-09-21T02:51:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 1 -- 172.21.15.29:0/858071376 wait complete. 2024-09-21T02:51:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.055+0000 7fc2ac68a740 20 client.17148 trim_cache size 0 max 16384 2024-09-21T02:51:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:51:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:51:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:51:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:51:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 -- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2880b6590 msgr2=0x7fc2880b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2880b6590 0x7fc2880b8aa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc29800b420 tx=0x7fc29800d040 comp rx=0 tx=0).stop 2024-09-21T02:51:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 -- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 msgr2=0x7fc2a0198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 0x7fc2a0198320 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc294009230 tx=0x7fc294004040 comp rx=0 tx=0).stop 2024-09-21T02:51:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 -- 172.21.15.29:0/4277471401 shutdown_connections 2024-09-21T02:51:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2880b6590 0x7fc2880b8aa0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2a0103fd0 0x7fc2a0198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2a0103630 0x7fc2a0198320 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 --2- 172.21.15.29:0/4277471401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2a01025b0 0x7fc2a0197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 -- 172.21.15.29:0/4277471401 >> 172.21.15.29:0/4277471401 conn(0x7fc2a00fdf30 msgr2=0x7fc2a0106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 -- 172.21.15.29:0/4277471401 shutdown_connections 2024-09-21T02:51:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:23.056+0000 7fc2a5720640 1 -- 172.21.15.29:0/4277471401 wait complete. 2024-09-21T02:51:23.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255173 2.25 MB/sec execute 540 sec latency 53.381 ms 2024-09-21T02:51:23.378 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0' 2024-09-21T02:51:23.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0 2024-09-21T02:51:23.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:23 smithi029 ceph-mon[31006]: pgmap v757: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T02:51:23.660 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:51:23.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:23 smithi066 ceph-mon[33436]: pgmap v757: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T02:51:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:23 smithi190 ceph-mon[33871]: pgmap v757: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T02:51:24.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255651 2.25 MB/sec execute 541 sec latency 26.662 ms 2024-09-21T02:51:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.195+0000 7fe3c8236640 1 -- 172.21.15.29:0/2461911255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c01094c0 msgr2=0x7fe3c01098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.195+0000 7fe3c8236640 1 --2- 172.21.15.29:0/2461911255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c01094c0 0x7fe3c01098d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009860 tx=0x7fe3b002a340 comp rx=0 tx=0).stop 2024-09-21T02:51:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.195+0000 7fe3c8236640 1 -- 172.21.15.29:0/2461911255 shutdown_connections 2024-09-21T02:51:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.195+0000 7fe3c8236640 1 --2- 172.21.15.29:0/2461911255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0109e10 0x7fe3c0110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.195+0000 7fe3c8236640 1 --2- 172.21.15.29:0/2461911255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c01094c0 0x7fe3c01098d0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.195+0000 7fe3c8236640 1 --2- 172.21.15.29:0/2461911255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3c0103b80 0x7fe3c0103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.195+0000 7fe3c8236640 1 -- 172.21.15.29:0/2461911255 >> 172.21.15.29:0/2461911255 conn(0x7fe3c00ff5d0 msgr2=0x7fe3c0101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.196+0000 7fe3c8236640 1 -- 172.21.15.29:0/2461911255 shutdown_connections 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.196+0000 7fe3c8236640 1 -- 172.21.15.29:0/2461911255 wait complete. 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.196+0000 7fe3c8236640 1 Processor -- start 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.196+0000 7fe3c8236640 1 -- start start 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c8236640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3c0103b80 0x7fe3c0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c8236640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c01094c0 0x7fe3c0075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c8236640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c0109e10 0x7fe3c0076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c8236640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0113980 con 0x7fe3c0109e10 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c8236640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0113800 con 0x7fe3c0103b80 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c8236640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0113b00 con 0x7fe3c01094c0 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c67d3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c0109e10 0x7fe3c0076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c57d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c01094c0 0x7fe3c0075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c67d3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c0109e10 0x7fe3c0076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57318/0 (socket says 172.21.15.29:57318) 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c67d3640 1 -- 172.21.15.29:0/1537112489 learned_addr learned my addr 172.21.15.29:0/1537112489 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3c0103b80 0x7fe3c0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c57d1640 1 -- 172.21.15.29:0/1537112489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3c0103b80 msgr2=0x7fe3c0075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3c0103b80 0x7fe3c0075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c57d1640 1 -- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c0109e10 msgr2=0x7fe3c0076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c0109e10 0x7fe3c0076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c57d1640 1 -- 172.21.15.29:0/1537112489 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c01a86b0 con 0x7fe3c01094c0 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c67d3640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c0109e10 0x7fe3c0076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.197+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c01094c0 0x7fe3c0075c50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe3c006c700 tx=0x7fe3b002ae40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.198+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/1537112489 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0040020 con 0x7fe3c01094c0 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.198+0000 7fe3c8236640 1 -- 172.21.15.29:0/1537112489 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c01a8940 con 0x7fe3c01094c0 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.198+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/1537112489 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3b00044d0 con 0x7fe3c01094c0 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.198+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/1537112489 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0037040 con 0x7fe3c01094c0 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.198+0000 7fe3c8236640 1 -- 172.21.15.29:0/1537112489 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c01a8df0 con 0x7fe3c01094c0 2024-09-21T02:51:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.200+0000 7fe3ce7aa740 1 Processor -- start 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3ce7aa740 1 -- start start 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/1537112489 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fe3b0043020 con 0x7fe3c01094c0 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4787840 0x55c0c4b2e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c475e980 0x55c0c4b2eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b2f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4aa4b90 con 0x55c0c4787840 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4a93fc0 con 0x55c0c475e980 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4a97dd0 con 0x55c0c4766b60 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3c5fd2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c475e980 0x55c0c4b2eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3c5fd2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c475e980 0x55c0c4b2eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:40952/0 (socket says 172.21.15.29:40952) 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3c5fd2640 1 -- 172.21.15.29:0/1202157845 learned_addr learned my addr 172.21.15.29:0/1202157845 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.201+0000 7fe3c67d3640 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4787840 0x55c0c4b2e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b2f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c5fd2640 1 -- 172.21.15.29:0/1202157845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 msgr2=0x55c0c4b2f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b2f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c5fd2640 1 -- 172.21.15.29:0/1202157845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4787840 msgr2=0x55c0c4b2e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4787840 0x55c0c4b2e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c5fd2640 1 -- 172.21.15.29:0/1202157845 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c4a1b060 con 0x55c0c475e980 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c67d3640 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4787840 0x55c0c4b2e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3b6ffd640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe3ac0b6510 0x7fe3ac0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c475e980 0x55c0c4b2eb40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009c80 tx=0x7fe3bc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27d0b6204f87b163 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3c67d3640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe3ac0b6510 0x7fe3ac0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3abfff640 1 -- 172.21.15.29:0/1202157845 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc004220 con 0x55c0c475e980 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.202+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/1537112489 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe3b01074c0 con 0x7fe3c01094c0 2024-09-21T02:51:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3abfff640 1 -- 172.21.15.29:0/1202157845 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3bc009e30 con 0x55c0c475e980 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3abfff640 1 -- 172.21.15.29:0/1202157845 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc019690 con 0x55c0c475e980 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1202157845 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c475e980 msgr2=0x55c0c4b2eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c475e980 0x55c0c4b2eb40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009c80 tx=0x7fe3bc011810 comp rx=0 tx=0).stop 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3c67d3640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe3ac0b6510 0x7fe3ac0b8a20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80049d0 tx=0x7fe3b8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1202157845 shutdown_connections 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b2f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c475e980 0x55c0c4b2eb40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.203+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/1202157845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4787840 0x55c0c4b2e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.204+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1202157845 >> 172.21.15.29:0/1202157845 conn(0x55c0c477b990 msgr2=0x55c0c4b2c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.204+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1202157845 shutdown_connections 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.204+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1202157845 wait complete. 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.204+0000 7fe3ce7aa740 1 -- start start 2024-09-21T02:51:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.204+0000 7fe3ce7aa740 1 Processor -- start 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b3a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b3a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4b46120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c0c48f57a0 con 0x55c0c475e980 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4a390a0 con 0x55c0c4787840 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4b3ce30 con 0x55c0c4766b60 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3ce7aa740 10 client.0 _ll_register_callbacks cb 0x55c0c4a163c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3c57d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b3a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3c57d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b3a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39340/0 (socket says 172.21.15.29:39340) 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3c57d1640 1 -- 172.21.15.29:0/2195770256 learned_addr learned my addr 172.21.15.29:0/2195770256 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3c67d3640 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4b46120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.205+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b3a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3c57d1640 1 -- 172.21.15.29:0/2195770256 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 msgr2=0x55c0c4b46120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4b46120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3c57d1640 1 -- 172.21.15.29:0/2195770256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 msgr2=0x55c0c4b3a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b3a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3c57d1640 1 -- 172.21.15.29:0/2195770256 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c4a27880 con 0x55c0c4766b60 2024-09-21T02:51:24.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b3a750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00c98a0 tx=0x7fe3b0030970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3e3fa1cc8e63b29 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:24.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:51:24.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2195770256 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b00c9170 con 0x55c0c4766b60 2024-09-21T02:51:24.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2195770256 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe3b0004040 con 0x55c0c4766b60 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2195770256 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3bc009e30 con 0x55c0c4766b60 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2195770256 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe3b00cca40 con 0x55c0c4766b60 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3ce7aa740 10 client.26704 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.206+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2195770256 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0c4801b90 con 0x55c0c4766b60 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2195770256 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3bc009e30 con 0x55c0c4766b60 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2195770256 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fe3b00c8c20 con 0x55c0c4766b60 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3ce7aa740 10 client.26704 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3ce7aa740 10 client.26704 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3ce7aa740 20 client.26704 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c0c47461c0 0x55c0c4b4c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3ce7aa740 4 client.26704 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2195770256 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0c4aa6ae0 con 0x55c0c47461c0 2024-09-21T02:51:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.207+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c0c47461c0 0x55c0c4b4c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.208+0000 7fe3a97fa640 10 client.26704 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.208+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c0c47461c0 0x55c0c4b4c0e0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfa42d6fe8b84424 server_cookie=f10ab008252ec3b2 in_seq=0 out_seq=0 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.208+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2195770256 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c0c4aa6ae0 con 0x55c0c47461c0 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.208+0000 7fe3a97fa640 10 client.26704 handle_command_reply: tid=0 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.208+0000 7fe3ce7aa740 1 client.26704 shutdown 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2195770256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 msgr2=0x55c0c4b3a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b3a750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00c98a0 tx=0x7fe3b0030970 comp rx=0 tx=0).stop 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2195770256 shutdown_connections 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4b46120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4766b60 0x55c0c4b3a750 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c0c47461c0 0x55c0c4b4c0e0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2195770256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b3a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3a97fa640 0 client.26704 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2195770256 >> 172.21.15.29:0/2195770256 conn(0x55c0c477b990 msgr2=0x55c0c4b2f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2195770256 shutdown_connections 2024-09-21T02:51:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2195770256 wait complete. 2024-09-21T02:51:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.209+0000 7fe3ce7aa740 20 client.26704 trim_cache size 0 max 16384 2024-09-21T02:51:24.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.229+0000 7fe3ce7aa740 1 Processor -- start 2024-09-21T02:51:24.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.229+0000 7fe3ce7aa740 1 -- start start 2024-09-21T02:51:24.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b2fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4766b60 0x55c0c4b38890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4787840 0x55c0c4b3bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b2ad0 con 0x55c0c475e980 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b4610 con 0x55c0c4766b60 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b31a0 con 0x55c0c4787840 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b2fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c67d3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4787840 0x55c0c4b3bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b2fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57340/0 (socket says 172.21.15.29:57340) 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 -- 172.21.15.29:0/1040100693 learned_addr learned my addr 172.21.15.29:0/1040100693 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/1040100693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4766b60 0x55c0c4b38890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 -- 172.21.15.29:0/1040100693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4787840 msgr2=0x55c0c4b3bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/1040100693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4787840 0x55c0c4b3bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 -- 172.21.15.29:0/1040100693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4766b60 msgr2=0x55c0c4b38890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:24.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/1040100693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4766b60 0x55c0c4b38890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 -- 172.21.15.29:0/1040100693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c4b3b2d0 con 0x55c0c475e980 2024-09-21T02:51:24.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.230+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/1040100693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b2fc50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009530 tx=0x7fe3bc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5b664ba1c79775 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:24.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.231+0000 7fe3ab7fe640 1 -- 172.21.15.29:0/1040100693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc003a10 con 0x55c0c475e980 2024-09-21T02:51:24.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.231+0000 7fe3ab7fe640 1 -- 172.21.15.29:0/1040100693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3bc030070 con 0x55c0c475e980 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.231+0000 7fe3ab7fe640 1 -- 172.21.15.29:0/1040100693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc005370 con 0x55c0c475e980 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1040100693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 msgr2=0x55c0c4b2fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/1040100693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b2fc50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009530 tx=0x7fe3bc011810 comp rx=0 tx=0).stop 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1040100693 shutdown_connections 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/1040100693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c4787840 0x55c0c4b3bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/1040100693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4766b60 0x55c0c4b38890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/1040100693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c475e980 0x55c0c4b2fc50 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1040100693 >> 172.21.15.29:0/1040100693 conn(0x55c0c477b990 msgr2=0x55c0c4b33710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1040100693 shutdown_connections 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/1040100693 wait complete. 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 -- start start 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.232+0000 7fe3ce7aa740 1 Processor -- start 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c475e980 0x55c0c4b3f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4766b60 0x55c0c4b3fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3ce7aa740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4bb6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b3f40 con 0x55c0c4766b60 2024-09-21T02:51:24.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b3870 con 0x55c0c4787840 2024-09-21T02:51:24.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3ce7aa740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b0ab0 con 0x55c0c475e980 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3c57d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4766b60 0x55c0c4b3fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3ce7aa740 10 client.0 _ll_register_callbacks cb 0x7fe3b0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3c57d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4766b60 0x55c0c4b3fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57350/0 (socket says 172.21.15.29:57350) 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3c57d1640 1 -- 172.21.15.29:0/2062239470 learned_addr learned my addr 172.21.15.29:0/2062239470 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3c67d3640 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4bb6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.233+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c475e980 0x55c0c4b3f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3c57d1640 1 -- 172.21.15.29:0/2062239470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c475e980 msgr2=0x55c0c4b3f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c475e980 0x55c0c4b3f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3c57d1640 1 -- 172.21.15.29:0/2062239470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 msgr2=0x55c0c4bb6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4bb6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3c57d1640 1 -- 172.21.15.29:0/2062239470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c4bb6ff0 con 0x55c0c4766b60 2024-09-21T02:51:24.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3c67d3640 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4bb6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3c57d1640 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4766b60 0x55c0c4b3fc90 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00095f0 tx=0x7fe3b0107430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9374747037419cd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2062239470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b00e7070 con 0x55c0c4766b60 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2062239470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe3b0004240 con 0x55c0c4766b60 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2062239470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3bc030070 con 0x55c0c4766b60 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2062239470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe3b00f4040 con 0x55c0c4766b60 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3ce7aa740 10 client.17166 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.234+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2062239470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0c4bb81e0 con 0x55c0c4766b60 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2062239470 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3bc030070 con 0x55c0c4766b60 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2062239470 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fe3b00cbfa0 con 0x55c0c4766b60 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3ce7aa740 10 client.17166 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3ce7aa740 10 client.17166 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3ce7aa740 20 client.17166 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c0c471b570 0x55c0c4bba720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:24.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3ce7aa740 4 client.17166 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}] 2024-09-21T02:51:24.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2062239470 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}) -- 0x55c0c4aa2590 con 0x55c0c471b570 2024-09-21T02:51:24.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.235+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c0c471b570 0x55c0c4bba720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:24.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.236+0000 7fe3a97fa640 10 client.17166 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:24.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.236+0000 7fe3c5fd2640 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c0c471b570 0x55c0c4bba720 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9ee8aee300fb4e3 server_cookie=1447da8c219b26ec in_seq=0 out_seq=0 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3a97fa640 1 -- 172.21.15.29:0/2062239470 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c0c4aa2590 con 0x55c0c471b570 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3a97fa640 10 client.17166 handle_command_reply: tid=0 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 client.17166 shutdown 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2062239470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4766b60 msgr2=0x55c0c4b3fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4766b60 0x55c0c4b3fc90 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00095f0 tx=0x7fe3b0107430 comp rx=0 tx=0).stop 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2062239470 shutdown_connections 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c0c4787840 0x55c0c4bb6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c0c471b570 0x55c0c4bba720 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0c4766b60 0x55c0c4b3fc90 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 --2- 172.21.15.29:0/2062239470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c0c475e980 0x55c0c4b3f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3a97fa640 0 client.17166 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2062239470 >> 172.21.15.29:0/2062239470 conn(0x55c0c477b990 msgr2=0x55c0c4b3bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2062239470 shutdown_connections 2024-09-21T02:51:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 1 -- 172.21.15.29:0/2062239470 wait complete. 2024-09-21T02:51:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.387+0000 7fe3ce7aa740 20 client.17166 trim_cache size 0 max 16384 2024-09-21T02:51:24.390 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:51:24.390 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 46, 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "1de7a249": { 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 46, 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:51:24.391 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 137.8, 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:24.392 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:51:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 -- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe3ac0b6510 msgr2=0x7fe3ac0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe3ac0b6510 0x7fe3ac0b8a20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80049d0 tx=0x7fe3b8009500 comp rx=0 tx=0).stop 2024-09-21T02:51:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 -- 172.21.15.29:0/1537112489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c01094c0 msgr2=0x7fe3c0075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c01094c0 0x7fe3c0075c50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe3c006c700 tx=0x7fe3b002ae40 comp rx=0 tx=0).stop 2024-09-21T02:51:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 -- 172.21.15.29:0/1537112489 shutdown_connections 2024-09-21T02:51:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe3ac0b6510 0x7fe3ac0b8a20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c0109e10 0x7fe3c0076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c01094c0 0x7fe3c0075c50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 --2- 172.21.15.29:0/1537112489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3c0103b80 0x7fe3c0075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 -- 172.21.15.29:0/1537112489 >> 172.21.15.29:0/1537112489 conn(0x7fe3c00ff5d0 msgr2=0x7fe3c01019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 -- 172.21.15.29:0/1537112489 shutdown_connections 2024-09-21T02:51:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:24.388+0000 7fe3c8236640 1 -- 172.21.15.29:0/1537112489 wait complete. 2024-09-21T02:51:25.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.25 MB/sec execute 542 sec latency 719.315 ms 2024-09-21T02:51:25.167 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1de7a249', quiesce took 5.4 seconds. Will release after: 12.4 2024-09-21T02:51:25.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:25 smithi066 ceph-mon[33436]: pgmap v758: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T02:51:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:25 smithi190 ceph-mon[33871]: pgmap v758: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T02:51:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:25 smithi029 ceph-mon[31006]: pgmap v758: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T02:51:26.102 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.24 MB/sec execute 543 sec latency 1719.407 ms 2024-09-21T02:51:27.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.24 MB/sec execute 544 sec latency 2719.517 ms 2024-09-21T02:51:27.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:27 smithi066 ceph-mon[33436]: pgmap v759: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T02:51:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:27 smithi190 ceph-mon[33871]: pgmap v759: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T02:51:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:27 smithi029 ceph-mon[31006]: pgmap v759: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T02:51:28.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.24 MB/sec execute 545 sec latency 3719.590 ms 2024-09-21T02:51:29.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.23 MB/sec execute 546 sec latency 4719.658 ms 2024-09-21T02:51:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:29 smithi066 ceph-mon[33436]: pgmap v760: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-09-21T02:51:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:29 smithi190 ceph-mon[33871]: pgmap v760: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-09-21T02:51:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:29 smithi029 ceph-mon[31006]: pgmap v760: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-09-21T02:51:30.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.23 MB/sec execute 547 sec latency 5719.739 ms 2024-09-21T02:51:31.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.22 MB/sec execute 548 sec latency 6719.893 ms 2024-09-21T02:51:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:31 smithi066 ceph-mon[33436]: pgmap v761: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-09-21T02:51:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:31 smithi190 ceph-mon[33871]: pgmap v761: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-09-21T02:51:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:31 smithi029 ceph-mon[31006]: pgmap v761: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-09-21T02:51:32.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.22 MB/sec execute 549 sec latency 7720.006 ms 2024-09-21T02:51:33.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.21 MB/sec execute 550 sec latency 8720.064 ms 2024-09-21T02:51:33.684 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:33 smithi029 ceph-mon[31006]: pgmap v762: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 832 KiB/s wr, 85 op/s 2024-09-21T02:51:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:33 smithi066 ceph-mon[33436]: pgmap v762: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 832 KiB/s wr, 85 op/s 2024-09-21T02:51:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:33 smithi190 ceph-mon[33871]: pgmap v762: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 832 KiB/s wr, 85 op/s 2024-09-21T02:51:34.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.21 MB/sec execute 551 sec latency 9720.138 ms 2024-09-21T02:51:35.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.21 MB/sec execute 552 sec latency 10720.210 ms 2024-09-21T02:51:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:35 smithi066 ceph-mon[33436]: pgmap v763: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 653 KiB/s wr, 83 op/s 2024-09-21T02:51:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:35 smithi190 ceph-mon[33871]: pgmap v763: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 653 KiB/s wr, 83 op/s 2024-09-21T02:51:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:35 smithi029 ceph-mon[31006]: pgmap v763: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 653 KiB/s wr, 83 op/s 2024-09-21T02:51:36.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.20 MB/sec execute 553 sec latency 11720.276 ms 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 -- 172.21.15.29:0/3487742125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 msgr2=0x7f3b68103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 --2- 172.21.15.29:0/3487742125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 0x7f3b68103a60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009900 tx=0x7f3b5c02a3f0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 -- 172.21.15.29:0/3487742125 shutdown_connections 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 --2- 172.21.15.29:0/3487742125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b68103ff0 0x7f3b6810a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 --2- 172.21.15.29:0/3487742125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 0x7f3b68103a60 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 --2- 172.21.15.29:0/3487742125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b681025d0 0x7f3b68102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 -- 172.21.15.29:0/3487742125 >> 172.21.15.29:0/3487742125 conn(0x7f3b680fdf30 msgr2=0x7f3b681003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 -- 172.21.15.29:0/3487742125 shutdown_connections 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.170+0000 7f3b6d627640 1 -- 172.21.15.29:0/3487742125 wait complete. 2024-09-21T02:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b6d627640 1 Processor -- start 2024-09-21T02:51:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b6d627640 1 -- start start 2024-09-21T02:51:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b6d627640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b681025d0 0x7f3b6806c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b6d627640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 0x7f3b681a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b6d627640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b68103ff0 0x7f3b681a3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b6d627640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6810e0b0 con 0x7f3b68103650 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b6d627640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6810df30 con 0x7f3b68103ff0 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b6d627640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6810e230 con 0x7f3b681025d0 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 0x7f3b681a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 0x7f3b681a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39608/0 (socket says 172.21.15.29:39608) 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b667fc640 1 -- 172.21.15.29:0/2252650820 learned_addr learned my addr 172.21.15.29:0/2252650820 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b677fe640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b68103ff0 0x7f3b681a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.171+0000 7f3b66ffd640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b681025d0 0x7f3b6806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b667fc640 1 -- 172.21.15.29:0/2252650820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b681025d0 msgr2=0x7f3b6806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b667fc640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b681025d0 0x7f3b6806c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b667fc640 1 -- 172.21.15.29:0/2252650820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b68103ff0 msgr2=0x7f3b681a3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b667fc640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b68103ff0 0x7f3b681a3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b667fc640 1 -- 172.21.15.29:0/2252650820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b681a3ab0 con 0x7f3b68103650 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b667fc640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 0x7f3b681a0fd0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009a20 tx=0x7f3b5c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2252650820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c03f030 con 0x7f3b68103650 2024-09-21T02:51:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b6d627640 1 -- 172.21.15.29:0/2252650820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b681a3d40 con 0x7f3b68103650 2024-09-21T02:51:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2252650820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b5c003a80 con 0x7f3b68103650 2024-09-21T02:51:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2252650820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c035070 con 0x7f3b68103650 2024-09-21T02:51:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.172+0000 7f3b6d627640 1 -- 172.21.15.29:0/2252650820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b681a4180 con 0x7f3b68103650 2024-09-21T02:51:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b7466d740 1 Processor -- start 2024-09-21T02:51:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b7466d740 1 -- start start 2024-09-21T02:51:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2252650820 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f3b5c042030 con 0x7f3b68103650 2024-09-21T02:51:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f03e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 0x559ce5f044c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f05010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b7466d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559ce5e766f0 con 0x559ce5b3b4d0 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b7466d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559ce5e65b20 con 0x559ce5b332f0 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b7466d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559ce5e69930 con 0x559ce5b5c1b0 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.175+0000 7f3b667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f05010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f05010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39610/0 (socket says 172.21.15.29:39610) 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b667fc640 1 -- 172.21.15.29:0/3815401042 learned_addr learned my addr 172.21.15.29:0/3815401042 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b677fe640 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f03e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b66ffd640 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 0x559ce5f044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b667fc640 1 -- 172.21.15.29:0/3815401042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 msgr2=0x559ce5f03e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b667fc640 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f03e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b667fc640 1 -- 172.21.15.29:0/3815401042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 msgr2=0x559ce5f044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:36.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b667fc640 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 0x559ce5f044c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b667fc640 1 -- 172.21.15.29:0/3815401042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ce5e2b530 con 0x559ce5b3b4d0 2024-09-21T02:51:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b4ffff640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3b500b6510 0x7f3b500b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b667fc640 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f05010 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c10bfc0 tx=0x7f3b5c1101a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e008c1253ef667 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b677fe640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3b500b6510 0x7f3b500b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/3815401042 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0ca310 con 0x559ce5b3b4d0 2024-09-21T02:51:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/3815401042 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b5c115070 con 0x559ce5b3b4d0 2024-09-21T02:51:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2252650820 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3b5c107eb0 con 0x7f3b68103650 2024-09-21T02:51:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.176+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/3815401042 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0e94f0 con 0x559ce5b3b4d0 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 -- 172.21.15.29:0/3815401042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 msgr2=0x559ce5f05010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f05010 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c10bfc0 tx=0x7f3b5c1101a0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b677fe640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3b500b6510 0x7f3b500b8a20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3b60007f80 tx=0x7f3b60007520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 -- 172.21.15.29:0/3815401042 shutdown_connections 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f05010 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 0x559ce5f044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3815401042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f03e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 -- 172.21.15.29:0/3815401042 >> 172.21.15.29:0/3815401042 conn(0x559ce5b50300 msgr2=0x559ce5eff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 -- 172.21.15.29:0/3815401042 shutdown_connections 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.177+0000 7f3b7466d740 1 -- 172.21.15.29:0/3815401042 wait complete. 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 1 -- start start 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 1 Processor -- start 2024-09-21T02:51:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b332f0 0x559ce5f0d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b3b4d0 0x559ce5f0d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f0e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559ce5d9dab0 con 0x559ce5b332f0 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559ce5e0b730 con 0x559ce5b3b4d0 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559ce5f058f0 con 0x559ce5b5c1b0 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b66ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b332f0 0x559ce5f0d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b7466d740 10 client.0 _ll_register_callbacks cb 0x559ce5e5f6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b66ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b332f0 0x559ce5f0d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39618/0 (socket says 172.21.15.29:39618) 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b66ffd640 1 -- 172.21.15.29:0/1237834577 learned_addr learned my addr 172.21.15.29:0/1237834577 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.178+0000 7f3b667fc640 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b3b4d0 0x559ce5f0d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b677fe640 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f0e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b66ffd640 1 -- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 msgr2=0x559ce5f0e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b66ffd640 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f0e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b66ffd640 1 -- 172.21.15.29:0/1237834577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b3b4d0 msgr2=0x559ce5f0d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b66ffd640 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b3b4d0 0x559ce5f0d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b66ffd640 1 -- 172.21.15.29:0/1237834577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ce5bc7790 con 0x559ce5b332f0 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b677fe640 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f0e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b66ffd640 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b332f0 0x559ce5f0d2f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800bec0 tx=0x7f3b58012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7f2199bef4a72c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b4aff5640 1 -- 172.21.15.29:0/1237834577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b580108a0 con 0x559ce5b332f0 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b4aff5640 1 -- 172.21.15.29:0/1237834577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3b58010a40 con 0x559ce5b332f0 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b7466d740 1 -- 172.21.15.29:0/1237834577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b5c115070 con 0x559ce5b332f0 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b4aff5640 1 -- 172.21.15.29:0/1237834577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3b58010c40 con 0x559ce5b332f0 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b7466d740 10 client.17178 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.179+0000 7f3b7466d740 1 -- 172.21.15.29:0/1237834577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559ce5c40620 con 0x559ce5b332f0 2024-09-21T02:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b4aff5640 1 -- 172.21.15.29:0/1237834577 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b5c115070 con 0x559ce5b332f0 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b4aff5640 1 -- 172.21.15.29:0/1237834577 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f3b58019070 con 0x559ce5b332f0 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b7466d740 10 client.17178 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b7466d740 10 client.17178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b7466d740 20 client.17178 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b7466d740 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559ce5b1ad80 0x559ce5f1ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b7466d740 4 client.17178 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b7466d740 1 -- 172.21.15.29:0/1237834577 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ce5e78640 con 0x559ce5b1ad80 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.180+0000 7f3b667fc640 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559ce5b1ad80 0x559ce5f1ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.181+0000 7f3b4aff5640 10 client.17178 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.181+0000 7f3b667fc640 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559ce5b1ad80 0x559ce5f1ab90 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e24885b50565ff91 server_cookie=b1812519e553cac8 in_seq=0 out_seq=0 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b4aff5640 1 -- 172.21.15.29:0/1237834577 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559ce5e78640 con 0x559ce5b1ad80 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b4aff5640 10 client.17178 handle_command_reply: tid=0 2024-09-21T02:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 client.17178 shutdown 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 -- 172.21.15.29:0/1237834577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b332f0 msgr2=0x559ce5f0d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b332f0 0x559ce5f0d2f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800bec0 tx=0x7f3b58012820 comp rx=0 tx=0).stop 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 -- 172.21.15.29:0/1237834577 shutdown_connections 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f0e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b3b4d0 0x559ce5f0d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559ce5b1ad80 0x559ce5f1ab90 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 --2- 172.21.15.29:0/1237834577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b332f0 0x559ce5f0d2f0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b4aff5640 0 client.17178 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 -- 172.21.15.29:0/1237834577 >> 172.21.15.29:0/1237834577 conn(0x559ce5b50300 msgr2=0x559ce5f043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 -- 172.21.15.29:0/1237834577 shutdown_connections 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 1 -- 172.21.15.29:0/1237834577 wait complete. 2024-09-21T02:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.182+0000 7f3b7466d740 20 client.17178 trim_cache size 0 max 16384 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b7466d740 1 Processor -- start 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b7466d740 1 -- start start 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b332f0 0x559ce5f04500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f15400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b5c1b0 0x559ce5f05de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b7466d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681ae6d0 con 0x559ce5b3b4d0 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b7466d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681b0210 con 0x559ce5b5c1b0 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b7466d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681aeda0 con 0x559ce5b332f0 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f15400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f15400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39622/0 (socket says 172.21.15.29:39622) 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.197+0000 7f3b667fc640 1 -- 172.21.15.29:0/2083150421 learned_addr learned my addr 172.21.15.29:0/2083150421 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b667fc640 1 -- 172.21.15.29:0/2083150421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b332f0 msgr2=0x559ce5f04500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b667fc640 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b332f0 0x559ce5f04500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b677fe640 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b5c1b0 0x559ce5f05de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b667fc640 1 -- 172.21.15.29:0/2083150421 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b5c1b0 msgr2=0x559ce5f05de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b667fc640 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b5c1b0 0x559ce5f05de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b667fc640 1 -- 172.21.15.29:0/2083150421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ce5f08b60 con 0x559ce5b3b4d0 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b677fe640 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b5c1b0 0x559ce5f05de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b667fc640 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f15400 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0e9870 tx=0x7f3b5c10b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8474461c497eeb80 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b4cff9640 1 -- 172.21.15.29:0/2083150421 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c049040 con 0x559ce5b3b4d0 2024-09-21T02:51:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b4cff9640 1 -- 172.21.15.29:0/2083150421 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b5c10b760 con 0x559ce5b3b4d0 2024-09-21T02:51:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.198+0000 7f3b4cff9640 1 -- 172.21.15.29:0/2083150421 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0e96d0 con 0x559ce5b3b4d0 2024-09-21T02:51:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 -- 172.21.15.29:0/2083150421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 msgr2=0x559ce5f15400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f15400 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0e9870 tx=0x7f3b5c10b990 comp rx=0 tx=0).stop 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 -- 172.21.15.29:0/2083150421 shutdown_connections 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b5c1b0 0x559ce5f05de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f15400 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 --2- 172.21.15.29:0/2083150421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b332f0 0x559ce5f04500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 -- 172.21.15.29:0/2083150421 >> 172.21.15.29:0/2083150421 conn(0x559ce5b50300 msgr2=0x559ce5f03df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 -- 172.21.15.29:0/2083150421 shutdown_connections 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 -- 172.21.15.29:0/2083150421 wait complete. 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 -- start start 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.199+0000 7f3b7466d740 1 Processor -- start 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 0x559ce5f122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f12970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b7466d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f134e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b7466d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681afb40 con 0x559ce5b3b4d0 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b7466d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681af470 con 0x559ce5b332f0 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b7466d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681ac6b0 con 0x559ce5b5c1b0 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b7466d740 10 client.0 _ll_register_callbacks cb 0x7f3b5800b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f12970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f12970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39634/0 (socket says 172.21.15.29:39634) 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 -- 172.21.15.29:0/3139512862 learned_addr learned my addr 172.21.15.29:0/3139512862 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b66ffd640 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 0x559ce5f122b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b677fe640 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 -- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 msgr2=0x559ce5f134e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f134e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 -- 172.21.15.29:0/3139512862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 msgr2=0x559ce5f122b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 0x559ce5f122b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 -- 172.21.15.29:0/3139512862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ce5f13c40 con 0x559ce5b3b4d0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b677fe640 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f134e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b667fc640 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f12970 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c005760 tx=0x7f3b5c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f24b22c3fc862353 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.200+0000 7f3b4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b4aff5640 1 -- 172.21.15.29:0/3139512862 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0e98a0 con 0x559ce5b3b4d0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b4aff5640 1 -- 172.21.15.29:0/3139512862 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3b5c10bac0 con 0x559ce5b3b4d0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 1 -- 172.21.15.29:0/3139512862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b5c10b760 con 0x559ce5b3b4d0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b4aff5640 1 -- 172.21.15.29:0/3139512862 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3b5c0cd430 con 0x559ce5b3b4d0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 10 client.17184 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 1 -- 172.21.15.29:0/3139512862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559ce5f14ed0 con 0x559ce5b3b4d0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b4aff5640 1 -- 172.21.15.29:0/3139512862 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b5c10b760 con 0x559ce5b3b4d0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b4aff5640 1 -- 172.21.15.29:0/3139512862 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f3b5c03bc70 con 0x559ce5b3b4d0 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 10 client.17184 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 10 client.17184 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 20 client.17184 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559ce5af01c0 0x559ce5f875a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 4 client.17184 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.201+0000 7f3b7466d740 1 -- 172.21.15.29:0/3139512862 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ce5e740f0 con 0x559ce5af01c0 2024-09-21T02:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.202+0000 7f3b66ffd640 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559ce5af01c0 0x559ce5f875a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.202+0000 7f3b66ffd640 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559ce5af01c0 0x559ce5f875a0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f84957e3cd30e4b9 server_cookie=89fa309669316814 in_seq=0 out_seq=0 2024-09-21T02:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.202+0000 7f3b4aff5640 10 client.17184 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.233+0000 7f3b4aff5640 1 -- 172.21.15.29:0/3139512862 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ce5e740f0 con 0x559ce5af01c0 2024-09-21T02:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.233+0000 7f3b4aff5640 10 client.17184 handle_command_reply: tid=0 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.233+0000 7f3b7466d740 1 client.17184 shutdown 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 -- 172.21.15.29:0/3139512862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 msgr2=0x559ce5f12970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f12970 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c005760 tx=0x7f3b5c009680 comp rx=0 tx=0).stop 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 -- 172.21.15.29:0/3139512862 shutdown_connections 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ce5b5c1b0 0x559ce5f134e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x559ce5af01c0 0x559ce5f875a0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ce5b3b4d0 0x559ce5f12970 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 --2- 172.21.15.29:0/3139512862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x559ce5b332f0 0x559ce5f122b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b4aff5640 0 client.17184 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 -- 172.21.15.29:0/3139512862 >> 172.21.15.29:0/3139512862 conn(0x559ce5b50300 msgr2=0x559ce5efdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 -- 172.21.15.29:0/3139512862 shutdown_connections 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 1 -- 172.21.15.29:0/3139512862 wait complete. 2024-09-21T02:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b7466d740 20 client.17184 trim_cache size 0 max 16384 2024-09-21T02:51:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:51:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:51:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:51:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:51:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b6d627640 1 -- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3b500b6510 msgr2=0x7f3b500b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b6d627640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3b500b6510 0x7f3b500b8a20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3b60007f80 tx=0x7f3b60007520 comp rx=0 tx=0).stop 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b6d627640 1 -- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 msgr2=0x7f3b681a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.234+0000 7f3b6d627640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 0x7f3b681a0fd0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009a20 tx=0x7f3b5c004270 comp rx=0 tx=0).stop 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.235+0000 7f3b6d627640 1 -- 172.21.15.29:0/2252650820 shutdown_connections 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.235+0000 7f3b6d627640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3b500b6510 0x7f3b500b8a20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.235+0000 7f3b6d627640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3b68103ff0 0x7f3b681a3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.235+0000 7f3b6d627640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b68103650 0x7f3b681a0fd0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.235+0000 7f3b6d627640 1 --2- 172.21.15.29:0/2252650820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b681025d0 0x7f3b6806c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.235+0000 7f3b6d627640 1 -- 172.21.15.29:0/2252650820 >> 172.21.15.29:0/2252650820 conn(0x7f3b680fdf30 msgr2=0x7f3b68106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.235+0000 7f3b6d627640 1 -- 172.21.15.29:0/2252650820 shutdown_connections 2024-09-21T02:51:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:36.235+0000 7f3b6d627640 1 -- 172.21.15.29:0/2252650820 wait complete. 2024-09-21T02:51:37.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.20 MB/sec execute 554 sec latency 12720.325 ms 2024-09-21T02:51:37.568 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:51:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:37 smithi066 ceph-mon[33436]: pgmap v764: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 381 KiB/s wr, 46 op/s 2024-09-21T02:51:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:51:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:51:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:51:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:51:37.831 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:51:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:37 smithi190 ceph-mon[33871]: pgmap v764: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 381 KiB/s wr, 46 op/s 2024-09-21T02:51:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:51:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:51:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:51:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:51:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:37 smithi029 ceph-mon[31006]: pgmap v764: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 381 KiB/s wr, 46 op/s 2024-09-21T02:51:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:51:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:51:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:51:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:51:38.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.19 MB/sec execute 555 sec latency 13720.399 ms 2024-09-21T02:51:38.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 -- 172.21.15.29:0/3929710043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd120109df0 msgr2=0x7fd120110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 --2- 172.21.15.29:0/3929710043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd120109df0 0x7fd120110620 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd118009ca0 tx=0x7fd11802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 -- 172.21.15.29:0/3929710043 shutdown_connections 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 --2- 172.21.15.29:0/3929710043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd120109df0 0x7fd120110620 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 --2- 172.21.15.29:0/3929710043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1201094a0 0x7fd1201098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 --2- 172.21.15.29:0/3929710043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd120103b60 0x7fd120103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 -- 172.21.15.29:0/3929710043 >> 172.21.15.29:0/3929710043 conn(0x7fd12007b850 msgr2=0x7fd12007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 -- 172.21.15.29:0/3929710043 shutdown_connections 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.364+0000 7fd124c52640 1 -- 172.21.15.29:0/3929710043 wait complete. 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd124c52640 1 Processor -- start 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd124c52640 1 -- start start 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd124c52640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd120103b60 0x7fd12019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd124c52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1201094a0 0x7fd12019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd124c52640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd120109df0 0x7fd12019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd124c52640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201139b0 con 0x7fd1201094a0 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd124c52640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd120113830 con 0x7fd120103b60 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd124c52640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd120113b30 con 0x7fd120109df0 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd11dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1201094a0 0x7fd12019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd11dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1201094a0 0x7fd12019a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39642/0 (socket says 172.21.15.29:39642) 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd11dd9b640 1 -- 172.21.15.29:0/3110540216 learned_addr learned my addr 172.21.15.29:0/3110540216 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd11ed9d640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd120109df0 0x7fd12019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.365+0000 7fd11e59c640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd120103b60 0x7fd12019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd11dd9b640 1 -- 172.21.15.29:0/3110540216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd120109df0 msgr2=0x7fd12019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd11dd9b640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd120109df0 0x7fd12019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd11dd9b640 1 -- 172.21.15.29:0/3110540216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd120103b60 msgr2=0x7fd12019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd11dd9b640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd120103b60 0x7fd12019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd11dd9b640 1 -- 172.21.15.29:0/3110540216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1201a0310 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd11dd9b640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1201094a0 0x7fd12019a940 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd11400d740 tx=0x7fd11400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd10f7fe640 1 -- 172.21.15.29:0/3110540216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd114017070 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd10f7fe640 1 -- 172.21.15.29:0/3110540216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd11400de50 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1201a0600 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd10f7fe640 1 -- 172.21.15.29:0/3110540216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11400f620 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.366+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1201a6d60 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.367+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd120103f70 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.369+0000 7fd10f7fe640 1 -- 172.21.15.29:0/3110540216 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd11401a070 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.369+0000 7fd10f7fe640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1080b6510 0x7fd1080b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.370+0000 7fd11e59c640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1080b6510 0x7fd1080b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.370+0000 7fd10f7fe640 1 -- 172.21.15.29:0/3110540216 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd1140e6f10 con 0x7fd1201094a0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.370+0000 7fd11e59c640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1080b6510 0x7fd1080b8a20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd110009bd0 tx=0x7fd110005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.373+0000 7fd10f7fe640 1 -- 172.21.15.29:0/3110540216 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd1140b1800 con 0x7fd1201094a0 2024-09-21T02:51:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.606+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd12019b4d0 con 0x7fd1201094a0 2024-09-21T02:51:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.607+0000 7fd10f7fe640 1 -- 172.21.15.29:0/3110540216 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7fd12019b4d0 con 0x7fd1201094a0 2024-09-21T02:51:38.610 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:51:38.610 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:51:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.609+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1080b6510 msgr2=0x7fd1080b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.609+0000 7fd124c52640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1080b6510 0x7fd1080b8a20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd110009bd0 tx=0x7fd110005f90 comp rx=0 tx=0).stop 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.609+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1201094a0 msgr2=0x7fd12019a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.609+0000 7fd124c52640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1201094a0 0x7fd12019a940 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd11400d740 tx=0x7fd11400dc00 comp rx=0 tx=0).stop 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.610+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 shutdown_connections 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.610+0000 7fd124c52640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd1080b6510 0x7fd1080b8a20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.610+0000 7fd124c52640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd120109df0 0x7fd12019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.610+0000 7fd124c52640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1201094a0 0x7fd12019a940 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.610+0000 7fd124c52640 1 --2- 172.21.15.29:0/3110540216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd120103b60 0x7fd12019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.610+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 >> 172.21.15.29:0/3110540216 conn(0x7fd12007b850 msgr2=0x7fd1201054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.610+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 shutdown_connections 2024-09-21T02:51:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:38.610+0000 7fd124c52640 1 -- 172.21.15.29:0/3110540216 wait complete. 2024-09-21T02:51:39.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.19 MB/sec execute 556 sec latency 14720.449 ms 2024-09-21T02:51:39.388 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 1de7a249 --release --await' 2024-09-21T02:51:39.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 1de7a249 --release --await 2024-09-21T02:51:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:39 smithi029 ceph-mon[31006]: pgmap v765: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 64 KiB/s wr, 40 op/s 2024-09-21T02:51:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:39 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3110540216' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:51:39.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:39 smithi066 ceph-mon[33436]: pgmap v765: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 64 KiB/s wr, 40 op/s 2024-09-21T02:51:39.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:39 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3110540216' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:39 smithi190 ceph-mon[33871]: pgmap v765: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 64 KiB/s wr, 40 op/s 2024-09-21T02:51:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:39 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3110540216' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:40.103 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255680 2.19 MB/sec execute 557 sec latency 15720.488 ms 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.186+0000 7fba1330a640 1 -- 172.21.15.29:0/3672500371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 msgr2=0x7fba0c10d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.186+0000 7fba1330a640 1 --2- 172.21.15.29:0/3672500371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 0x7fba0c10d930 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009900 tx=0x7fb9fc02a410 comp rx=0 tx=0).stop 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.186+0000 7fba1330a640 1 -- 172.21.15.29:0/3672500371 shutdown_connections 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.186+0000 7fba1330a640 1 --2- 172.21.15.29:0/3672500371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba0c10de70 0x7fba0c110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.186+0000 7fba1330a640 1 --2- 172.21.15.29:0/3672500371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 0x7fba0c10d930 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.186+0000 7fba1330a640 1 --2- 172.21.15.29:0/3672500371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba0c1048c0 0x7fba0c104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.186+0000 7fba1330a640 1 -- 172.21.15.29:0/3672500371 >> 172.21.15.29:0/3672500371 conn(0x7fba0c079760 msgr2=0x7fba0c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.187+0000 7fba1330a640 1 -- 172.21.15.29:0/3672500371 shutdown_connections 2024-09-21T02:51:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.187+0000 7fba1330a640 1 -- 172.21.15.29:0/3672500371 wait complete. 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.187+0000 7fba1330a640 1 Processor -- start 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.187+0000 7fba1330a640 1 -- start start 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.187+0000 7fba1330a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba0c1048c0 0x7fba0c19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba1330a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 0x7fba0c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba1330a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba0c10de70 0x7fba0c19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba1330a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c113960 con 0x7fba0c1052a0 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba1330a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1137e0 con 0x7fba0c1048c0 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba1330a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c113ae0 con 0x7fba0c10de70 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 0x7fba0c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 0x7fba0c19a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39662/0 (socket says 172.21.15.29:39662) 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 -- 172.21.15.29:0/3282559216 learned_addr learned my addr 172.21.15.29:0/3282559216 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba110a6640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba0c1048c0 0x7fba0c19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba118a7640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba0c10de70 0x7fba0c19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 -- 172.21.15.29:0/3282559216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba0c10de70 msgr2=0x7fba0c19fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba0c10de70 0x7fba0c19fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 -- 172.21.15.29:0/3282559216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba0c1048c0 msgr2=0x7fba0c19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba0c1048c0 0x7fba0c19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 -- 172.21.15.29:0/3282559216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c1a0300 con 0x7fba0c1052a0 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba118a7640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba0c10de70 0x7fba0c19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba108a5640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 0x7fba0c19a9f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc005940 tx=0x7fb9fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba027fc640 1 -- 172.21.15.29:0/3282559216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc03f030 con 0x7fba0c1052a0 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba027fc640 1 -- 172.21.15.29:0/3282559216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9fc003c90 con 0x7fba0c1052a0 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba027fc640 1 -- 172.21.15.29:0/3282559216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc035070 con 0x7fba0c1052a0 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.188+0000 7fba1330a640 1 -- 172.21.15.29:0/3282559216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c1a0590 con 0x7fba0c1052a0 2024-09-21T02:51:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.189+0000 7fba1330a640 1 -- 172.21.15.29:0/3282559216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c1a0a70 con 0x7fba0c1052a0 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.191+0000 7fba1987e740 1 Processor -- start 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.191+0000 7fba1987e740 1 -- start start 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.191+0000 7fba027fc640 1 -- 172.21.15.29:0/3282559216 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb9fc042030 con 0x7fba0c1052a0 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba1987e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aac4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba1987e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aac4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba1987e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6fd840 0x56300aac4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba1987e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56300a9afdc0 con 0x56300a71e520 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba1987e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56300aa3c1f0 con 0x56300a6f5660 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba1987e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56300aa2b620 con 0x56300a6fd840 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aac4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aac4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39672/0 (socket says 172.21.15.29:39672) 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 -- 172.21.15.29:0/3193214092 learned_addr learned my addr 172.21.15.29:0/3193214092 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:40.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba108a5640 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6fd840 0x56300aac4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba110a6640 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aac4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 -- 172.21.15.29:0/3193214092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6fd840 msgr2=0x56300aac4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6fd840 0x56300aac4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 -- 172.21.15.29:0/3193214092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 msgr2=0x56300aac4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aac4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba027fc640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb9f80b6540 0x7fb9f80b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 -- 172.21.15.29:0/3193214092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56300a9f3fd0 con 0x56300a71e520 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba110a6640 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aac4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba027fc640 1 -- 172.21.15.29:0/3282559216 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb9fc107c20 con 0x7fba0c1052a0 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.192+0000 7fba118a7640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb9f80b6540 0x7fb9f80b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.193+0000 7fba118a7640 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aac4510 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fba0400ae80 tx=0x7fba0400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947b51fd869ccfa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.193+0000 7fb9f77fe640 1 -- 172.21.15.29:0/3193214092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04010ba0 con 0x56300a71e520 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.193+0000 7fb9f77fe640 1 -- 172.21.15.29:0/3193214092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba0401cdc0 con 0x56300a71e520 2024-09-21T02:51:40.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.193+0000 7fb9f77fe640 1 -- 172.21.15.29:0/3193214092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0400fe40 con 0x56300a71e520 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 -- 172.21.15.29:0/3193214092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 msgr2=0x56300aac4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba118a7640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb9f80b6540 0x7fb9f80b8a50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba0401d000 tx=0x7fba04020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aac4510 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fba0400ae80 tx=0x7fba0400e720 comp rx=0 tx=0).stop 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 -- 172.21.15.29:0/3193214092 shutdown_connections 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6fd840 0x56300aac4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aac4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 --2- 172.21.15.29:0/3193214092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aac4510 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 -- 172.21.15.29:0/3193214092 >> 172.21.15.29:0/3193214092 conn(0x56300a712910 msgr2=0x56300aac2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 -- 172.21.15.29:0/3193214092 shutdown_connections 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 -- 172.21.15.29:0/3193214092 wait complete. 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.194+0000 7fba1987e740 1 -- start start 2024-09-21T02:51:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba1987e740 1 Processor -- start 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba1987e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6f5660 0x56300aacfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba1987e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 0x56300aad00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba1987e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aadbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba1987e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56300a9c6300 con 0x56300a71e520 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba1987e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56300aa2f430 con 0x56300a6fd840 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba1987e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56300a88cbb0 con 0x56300a6f5660 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba1987e740 10 client.0 _ll_register_callbacks cb 0x56300a9e0120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba118a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aadbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba108a5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 0x56300aad00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba118a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aadbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39678/0 (socket says 172.21.15.29:39678) 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.195+0000 7fba118a7640 1 -- 172.21.15.29:0/2673415204 learned_addr learned my addr 172.21.15.29:0/2673415204 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba110a6640 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6f5660 0x56300aacfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba118a7640 1 -- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6f5660 msgr2=0x56300aacfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba118a7640 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6f5660 0x56300aacfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba118a7640 1 -- 172.21.15.29:0/2673415204 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 msgr2=0x56300aad00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba118a7640 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 0x56300aad00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba118a7640 1 -- 172.21.15.29:0/2673415204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56300a9b5780 con 0x56300a71e520 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba110a6640 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6f5660 0x56300aacfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba118a7640 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aadbe90 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fba0400a960 tx=0x7fba0400fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb5d846fa3e9147 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fb9f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/2673415204 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04034020 con 0x56300a71e520 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/2673415204 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fba0400f3c0 con 0x56300a71e520 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba1987e740 1 -- 172.21.15.29:0/2673415204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba0401cdc0 con 0x56300a71e520 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/2673415204 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fba0403f050 con 0x56300a71e520 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba1987e740 10 client.17202 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.196+0000 7fba1987e740 1 -- 172.21.15.29:0/2673415204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56300a789b00 con 0x56300a71e520 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/2673415204 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba0401cdc0 con 0x56300a71e520 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/2673415204 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fba04026760 con 0x56300a71e520 2024-09-21T02:51:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fba1987e740 10 client.17202 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fba1987e740 10 client.17202 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fba1987e740 20 client.17202 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fba1987e740 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56300a6dcea0 0x56300aae1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fba1987e740 4 client.17202 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fba1987e740 1 -- 172.21.15.29:0/2673415204 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56300aa3e140 con 0x56300a6dcea0 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.197+0000 7fba110a6640 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56300a6dcea0 0x56300aae1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.198+0000 7fba110a6640 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56300a6dcea0 0x56300aae1d90 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e48800b1a7aec45c server_cookie=f4124cfa6ab23c67 in_seq=0 out_seq=0 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.198+0000 7fb9f4ff9640 10 client.17202 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/2673415204 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56300aa3e140 con 0x56300a6dcea0 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fb9f4ff9640 10 client.17202 handle_command_reply: tid=0 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 client.17202 shutdown 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 -- 172.21.15.29:0/2673415204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 msgr2=0x56300aadbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aadbe90 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fba0400a960 tx=0x7fba0400fef0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 -- 172.21.15.29:0/2673415204 shutdown_connections 2024-09-21T02:51:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56300a6dcea0 0x56300aae1d90 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a71e520 0x56300aadbe90 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 0x56300aad00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 --2- 172.21.15.29:0/2673415204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a6f5660 0x56300aacfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 -- 172.21.15.29:0/2673415204 >> 172.21.15.29:0/2673415204 conn(0x56300a712910 msgr2=0x56300aac76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:40.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fb9f4ff9640 0 client.17202 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:40.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 -- 172.21.15.29:0/2673415204 shutdown_connections 2024-09-21T02:51:40.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 1 -- 172.21.15.29:0/2673415204 wait complete. 2024-09-21T02:51:40.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.199+0000 7fba1987e740 20 client.17202 trim_cache size 0 max 16384 2024-09-21T02:51:40.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.216+0000 7fba1987e740 1 Processor -- start 2024-09-21T02:51:40.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba1987e740 1 -- start start 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba1987e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6f5660 0x56300aadbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba1987e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 0x56300aac5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba1987e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 0x56300aad17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba1987e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1b09f0 con 0x56300a6f5660 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba1987e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1b2530 con 0x56300a6fd840 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba1987e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1b10c0 con 0x56300a71e520 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba110a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6f5660 0x56300aadbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba110a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6f5660 0x56300aadbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39680/0 (socket says 172.21.15.29:39680) 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba110a6640 1 -- 172.21.15.29:0/623378485 learned_addr learned my addr 172.21.15.29:0/623378485 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba108a5640 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 0x56300aac5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba118a7640 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 0x56300aad17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba110a6640 1 -- 172.21.15.29:0/623378485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 msgr2=0x56300aad17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba110a6640 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 0x56300aad17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba110a6640 1 -- 172.21.15.29:0/623378485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 msgr2=0x56300aac5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba110a6640 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 0x56300aac5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.217+0000 7fba110a6640 1 -- 172.21.15.29:0/623378485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56300a800960 con 0x56300a6f5660 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.218+0000 7fba110a6640 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6f5660 0x56300aadbd90 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x56300aac3190 tx=0x7fba08005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfce7230961ee71a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.218+0000 7fb9f6ffd640 1 -- 172.21.15.29:0/623378485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0801f070 con 0x56300a6f5660 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.218+0000 7fb9f6ffd640 1 -- 172.21.15.29:0/623378485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba08004250 con 0x56300a6f5660 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.218+0000 7fb9f6ffd640 1 -- 172.21.15.29:0/623378485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0800f6b0 con 0x56300a6f5660 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.218+0000 7fba1987e740 1 -- 172.21.15.29:0/623378485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6f5660 msgr2=0x56300aadbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.218+0000 7fba1987e740 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6f5660 0x56300aadbd90 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x56300aac3190 tx=0x7fba08005f90 comp rx=0 tx=0).stop 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 -- 172.21.15.29:0/623378485 shutdown_connections 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 0x56300aad17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6fd840 0x56300aac5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 --2- 172.21.15.29:0/623378485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6f5660 0x56300aadbd90 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 -- 172.21.15.29:0/623378485 >> 172.21.15.29:0/623378485 conn(0x56300a712910 msgr2=0x56300aad10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 -- 172.21.15.29:0/623378485 shutdown_connections 2024-09-21T02:51:40.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 -- 172.21.15.29:0/623378485 wait complete. 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 -- start start 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.219+0000 7fba1987e740 1 Processor -- start 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba1987e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aad64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba1987e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6fd840 0x56300aad69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba1987e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 0x56300ab4b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba1987e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1b1e60 con 0x56300a6fd840 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba1987e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1b1790 con 0x56300a6f5660 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba1987e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1ae9d0 con 0x56300a71e520 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba1987e740 10 client.0 _ll_register_callbacks cb 0x7fba04014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6fd840 0x56300aad69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6fd840 0x56300aad69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39692/0 (socket says 172.21.15.29:39692) 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 -- 172.21.15.29:0/1681192218 learned_addr learned my addr 172.21.15.29:0/1681192218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba110a6640 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aad64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba118a7640 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 0x56300ab4b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 -- 172.21.15.29:0/1681192218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 msgr2=0x56300ab4b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 0x56300ab4b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 -- 172.21.15.29:0/1681192218 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 msgr2=0x56300aad64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aad64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 -- 172.21.15.29:0/1681192218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56300ab4bbe0 con 0x56300a6fd840 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fba108a5640 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6fd840 0x56300aad69e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc1053a0 tx=0x7fb9fc10b2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68955ae0a8e9d51 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.220+0000 7fb9f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/1681192218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc049070 con 0x56300a6fd840 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/1681192218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb9fc0d71c0 con 0x56300a6fd840 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 1 -- 172.21.15.29:0/1681192218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba08004250 con 0x56300a6fd840 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/1681192218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb9fc0c98d0 con 0x56300a6fd840 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 10 client.17208 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 1 -- 172.21.15.29:0/1681192218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56300ab4ce70 con 0x56300a6fd840 2024-09-21T02:51:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/1681192218 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba08004250 con 0x56300a6fd840 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/1681192218 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb9fc003800 con 0x56300a6fd840 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 10 client.17208 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 10 client.17208 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 20 client.17208 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56300a6b2250 0x56300ab4f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 4 client.17208 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1de7a249", "release": true, "await": true}] 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.221+0000 7fba1987e740 1 -- 172.21.15.29:0/1681192218 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1de7a249", "release": true, "await": true}) -- 0x56300aa39bf0 con 0x56300a6b2250 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.222+0000 7fba110a6640 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56300a6b2250 0x56300ab4f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.222+0000 7fba110a6640 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56300a6b2250 0x56300ab4f630 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b1ce310be8cca62 server_cookie=f2ab3a7a446f61f7 in_seq=0 out_seq=0 2024-09-21T02:51:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.222+0000 7fb9f4ff9640 10 client.17208 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fb9f4ff9640 1 -- 172.21.15.29:0/1681192218 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56300aa39bf0 con 0x56300a6b2250 2024-09-21T02:51:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fb9f4ff9640 10 client.17208 handle_command_reply: tid=0 2024-09-21T02:51:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 client.17208 shutdown 2024-09-21T02:51:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 -- 172.21.15.29:0/1681192218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6fd840 msgr2=0x56300aad69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6fd840 0x56300aad69e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc1053a0 tx=0x7fb9fc10b2f0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 -- 172.21.15.29:0/1681192218 shutdown_connections 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56300a71e520 0x56300ab4b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56300a6b2250 0x56300ab4f630 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fb9f4ff9640 0 client.17208 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56300a6fd840 0x56300aad69e0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 --2- 172.21.15.29:0/1681192218 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56300a6f5660 0x56300aad64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.232+0000 7fba1987e740 1 -- 172.21.15.29:0/1681192218 >> 172.21.15.29:0/1681192218 conn(0x56300a712910 msgr2=0x56300aad0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.233+0000 7fba1987e740 1 -- 172.21.15.29:0/1681192218 shutdown_connections 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.233+0000 7fba1987e740 1 -- 172.21.15.29:0/1681192218 wait complete. 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.233+0000 7fba1987e740 20 client.17208 trim_cache size 0 max 16384 2024-09-21T02:51:40.236 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 48, 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "1de7a249": { 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 48, 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 137.8, 2024-09-21T02:51:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:40.238 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.233+0000 7fba1330a640 1 -- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb9f80b6540 msgr2=0x7fb9f80b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.233+0000 7fba1330a640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb9f80b6540 0x7fb9f80b8a50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba0401d000 tx=0x7fba04020000 comp rx=0 tx=0).stop 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.233+0000 7fba1330a640 1 -- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 msgr2=0x7fba0c19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.233+0000 7fba1330a640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 0x7fba0c19a9f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc005940 tx=0x7fb9fc0042f0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.234+0000 7fba1330a640 1 -- 172.21.15.29:0/3282559216 shutdown_connections 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.234+0000 7fba1330a640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb9f80b6540 0x7fb9f80b8a50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.234+0000 7fba1330a640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba0c10de70 0x7fba0c19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.234+0000 7fba1330a640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba0c1052a0 0x7fba0c19a9f0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.234+0000 7fba1330a640 1 --2- 172.21.15.29:0/3282559216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba0c1048c0 0x7fba0c19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.234+0000 7fba1330a640 1 -- 172.21.15.29:0/3282559216 >> 172.21.15.29:0/3282559216 conn(0x7fba0c079760 msgr2=0x7fba0c0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.234+0000 7fba1330a640 1 -- 172.21.15.29:0/3282559216 shutdown_connections 2024-09-21T02:51:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:40.234+0000 7fba1330a640 1 -- 172.21.15.29:0/3282559216 wait complete. 2024-09-21T02:51:41.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 255887 2.18 MB/sec execute 558 sec latency 15853.600 ms 2024-09-21T02:51:41.169 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1de7a249', total seconds elapsed: 21.4 2024-09-21T02:51:41.169 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.8 2024-09-21T02:51:41.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:51:41.452 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:51:41.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:41 smithi029 ceph-mon[31006]: pgmap v766: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 20 op/s 2024-09-21T02:51:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:41 smithi066 ceph-mon[33436]: pgmap v766: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 20 op/s 2024-09-21T02:51:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:41 smithi190 ceph-mon[33871]: pgmap v766: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 20 op/s 2024-09-21T02:51:41.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 -- 172.21.15.29:0/423141511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc102230 msgr2=0x7fc4bc102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:41.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 --2- 172.21.15.29:0/423141511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc102230 0x7fc4bc102640 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00a970 tx=0x7fc4ac02b670 comp rx=0 tx=0).stop 2024-09-21T02:51:41.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 -- 172.21.15.29:0/423141511 shutdown_connections 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 --2- 172.21.15.29:0/423141511 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4bc1084c0 0x7fc4bc10ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 --2- 172.21.15.29:0/423141511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4bc107b70 0x7fc4bc107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 --2- 172.21.15.29:0/423141511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc102230 0x7fc4bc102640 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 -- 172.21.15.29:0/423141511 >> 172.21.15.29:0/423141511 conn(0x7fc4bc079e10 msgr2=0x7fc4bc07a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 -- 172.21.15.29:0/423141511 shutdown_connections 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.966+0000 7fc4c1859640 1 -- 172.21.15.29:0/423141511 wait complete. 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4c1859640 1 Processor -- start 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4c1859640 1 -- start start 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4c1859640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4bc102230 0x7fc4bc19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4c1859640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4bc107b70 0x7fc4bc19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4c1859640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc1084c0 0x7fc4bc19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4c1859640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1117b0 con 0x7fc4bc102230 2024-09-21T02:51:41.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4c1859640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc111630 con 0x7fc4bc107b70 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4c1859640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc111930 con 0x7fc4bc1084c0 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4baffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4bc102230 0x7fc4bc19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4baffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4bc102230 0x7fc4bc19a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39714/0 (socket says 172.21.15.29:39714) 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4baffd640 1 -- 172.21.15.29:0/2046123059 learned_addr learned my addr 172.21.15.29:0/2046123059 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4bb7fe640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc1084c0 0x7fc4bc19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.967+0000 7fc4ba7fc640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4bc107b70 0x7fc4bc19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4bb7fe640 1 -- 172.21.15.29:0/2046123059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4bc107b70 msgr2=0x7fc4bc19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4bb7fe640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4bc107b70 0x7fc4bc19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4bb7fe640 1 -- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4bc102230 msgr2=0x7fc4bc19a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4bb7fe640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4bc102230 0x7fc4bc19a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4bb7fe640 1 -- 172.21.15.29:0/2046123059 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4bc1a0390 con 0x7fc4bc1084c0 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4baffd640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4bc102230 0x7fc4bc19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4bb7fe640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc1084c0 0x7fc4bc19fc60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400e940 tx=0x7fc4b400ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:41.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4c0857640 1 -- 172.21.15.29:0/2046123059 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4004d20 con 0x7fc4bc1084c0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4c0857640 1 -- 172.21.15.29:0/2046123059 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4b4005020 con 0x7fc4bc1084c0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4bc1a0680 con 0x7fc4bc1084c0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4c0857640 1 -- 172.21.15.29:0/2046123059 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40116d0 con 0x7fc4bc1084c0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.968+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4bc1a6de0 con 0x7fc4bc1084c0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.969+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4bc102640 con 0x7fc4bc1084c0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.971+0000 7fc4c0857640 1 -- 172.21.15.29:0/2046123059 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc4b400ca60 con 0x7fc4bc1084c0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.972+0000 7fc4c0857640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4a40b6590 0x7fc4a40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.972+0000 7fc4baffd640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4a40b6590 0x7fc4a40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.972+0000 7fc4c0857640 1 -- 172.21.15.29:0/2046123059 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc4b40e7ad0 con 0x7fc4bc1084c0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.973+0000 7fc4baffd640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4a40b6590 0x7fc4a40b8aa0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac005eb0 tx=0x7fc4ac00a5a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:41.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:41.975+0000 7fc4c0857640 1 -- 172.21.15.29:0/2046123059 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc4b4016030 con 0x7fc4bc1084c0 2024-09-21T02:51:42.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 256372 2.18 MB/sec execute 559 sec latency 20.715 ms 2024-09-21T02:51:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.240+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4bc19b550 con 0x7fc4bc1084c0 2024-09-21T02:51:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.240+0000 7fc4c0857640 1 -- 172.21.15.29:0/2046123059 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7fc4bc19b550 con 0x7fc4bc1084c0 2024-09-21T02:51:42.243 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:51:42.243 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:51:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4a40b6590 msgr2=0x7fc4a40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4a40b6590 0x7fc4a40b8aa0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac005eb0 tx=0x7fc4ac00a5a0 comp rx=0 tx=0).stop 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc1084c0 msgr2=0x7fc4bc19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc1084c0 0x7fc4bc19fc60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400e940 tx=0x7fc4b400ed40 comp rx=0 tx=0).stop 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 shutdown_connections 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4a40b6590 0x7fc4a40b8aa0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc1084c0 0x7fc4bc19fc60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4bc107b70 0x7fc4bc19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 --2- 172.21.15.29:0/2046123059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4bc102230 0x7fc4bc19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 >> 172.21.15.29:0/2046123059 conn(0x7fc4bc079e10 msgr2=0x7fc4bc105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.243+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 shutdown_connections 2024-09-21T02:51:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:42.244+0000 7fc4c1859640 1 -- 172.21.15.29:0/2046123059 wait complete. 2024-09-21T02:51:42.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:42 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2046123059' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:42.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:42 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2046123059' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:42.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:42 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2046123059' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257007 2.18 MB/sec execute 560 sec latency 18.078 ms 2024-09-21T02:51:43.108 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0' 2024-09-21T02:51:43.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0 2024-09-21T02:51:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:51:43.703 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:43 smithi029 ceph-mon[31006]: pgmap v767: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 18 op/s 2024-09-21T02:51:43.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:43 smithi066 ceph-mon[33436]: pgmap v767: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 18 op/s 2024-09-21T02:51:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:43 smithi190 ceph-mon[33871]: pgmap v767: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 18 op/s 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 -- 172.21.15.29:0/2480560273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c1094d0 msgr2=0x7fe11c1098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 --2- 172.21.15.29:0/2480560273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c1094d0 0x7fe11c1098e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe104009900 tx=0x7fe10402a410 comp rx=0 tx=0).stop 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 -- 172.21.15.29:0/2480560273 shutdown_connections 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 --2- 172.21.15.29:0/2480560273 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe11c109e20 0x7fe11c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 --2- 172.21.15.29:0/2480560273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c1094d0 0x7fe11c1098e0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 --2- 172.21.15.29:0/2480560273 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c103b90 0x7fe11c103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 -- 172.21.15.29:0/2480560273 >> 172.21.15.29:0/2480560273 conn(0x7fe11c0ff5e0 msgr2=0x7fe11c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 -- 172.21.15.29:0/2480560273 shutdown_connections 2024-09-21T02:51:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.977+0000 7fe1222f8640 1 -- 172.21.15.29:0/2480560273 wait complete. 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.978+0000 7fe1222f8640 1 Processor -- start 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.978+0000 7fe1222f8640 1 -- start start 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe1222f8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c103b90 0x7fe11c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe1222f8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe11c1094d0 0x7fe11c19a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe1222f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c109e20 0x7fe11c19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe1222f8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c113990 con 0x7fe11c109e20 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe1222f8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c113810 con 0x7fe11c103b90 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe1222f8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c113b10 con 0x7fe11c1094d0 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe11bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c103b90 0x7fe11c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe113fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe11c1094d0 0x7fe11c19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe11bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c103b90 0x7fe11c19a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42936/0 (socket says 172.21.15.29:42936) 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe113fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe11c1094d0 0x7fe11c19a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50428/0 (socket says 172.21.15.29:50428) 2024-09-21T02:51:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe11bfff640 1 -- 172.21.15.29:0/4279303639 learned_addr learned my addr 172.21.15.29:0/4279303639 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe120895640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c109e20 0x7fe11c19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe113fff640 1 -- 172.21.15.29:0/4279303639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c103b90 msgr2=0x7fe11c19a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe113fff640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c103b90 0x7fe11c19a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe113fff640 1 -- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c109e20 msgr2=0x7fe11c19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe113fff640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c109e20 0x7fe11c19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe113fff640 1 -- 172.21.15.29:0/4279303639 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11c1a02f0 con 0x7fe11c1094d0 2024-09-21T02:51:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.979+0000 7fe120895640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c109e20 0x7fe11c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.980+0000 7fe113fff640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe11c1094d0 0x7fe11c19a920 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe10402a930 tx=0x7fe1040319a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.980+0000 7fe119ffb640 1 -- 172.21.15.29:0/4279303639 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104035050 con 0x7fe11c1094d0 2024-09-21T02:51:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.980+0000 7fe119ffb640 1 -- 172.21.15.29:0/4279303639 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1040043d0 con 0x7fe11c1094d0 2024-09-21T02:51:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.980+0000 7fe119ffb640 1 -- 172.21.15.29:0/4279303639 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104003ae0 con 0x7fe11c1094d0 2024-09-21T02:51:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.981+0000 7fe1222f8640 1 -- 172.21.15.29:0/4279303639 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11c1a0580 con 0x7fe11c1094d0 2024-09-21T02:51:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.981+0000 7fe1222f8640 1 -- 172.21.15.29:0/4279303639 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11c1a0920 con 0x7fe11c1094d0 2024-09-21T02:51:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.984+0000 7fe119ffb640 1 -- 172.21.15.29:0/4279303639 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fe10403acb0 con 0x7fe11c1094d0 2024-09-21T02:51:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.984+0000 7fe12886c740 1 Processor -- start 2024-09-21T02:51:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.984+0000 7fe12886c740 1 -- start start 2024-09-21T02:51:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe12886c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817d3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe12886c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817d3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe12886c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817d4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe12886c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564681749b90 con 0x56468140bb60 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe12886c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564681738fc0 con 0x564681403980 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe12886c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56468173cdd0 con 0x56468142c840 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe113fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817d4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe120895640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817d3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe113fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817d4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39744/0 (socket says 172.21.15.29:39744) 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe113fff640 1 -- 172.21.15.29:0/1215763399 learned_addr learned my addr 172.21.15.29:0/1215763399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe11bfff640 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817d3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe119ffb640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1080b6590 0x7fe1080b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe120895640 1 -- 172.21.15.29:0/1215763399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 msgr2=0x5646817d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe120895640 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817d3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe120895640 1 -- 172.21.15.29:0/1215763399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 msgr2=0x5646817d4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:43.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe120895640 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817d4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe120895640 1 -- 172.21.15.29:0/1215763399 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646816c0060 con 0x56468142c840 2024-09-21T02:51:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe113fff640 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817d4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:51:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.985+0000 7fe119ffb640 1 -- 172.21.15.29:0/4279303639 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe10410b370 con 0x7fe11c1094d0 2024-09-21T02:51:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.986+0000 7fe120895640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1080b6590 0x7fe1080b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.986+0000 7fe120895640 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817d3600 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe114005fd0 tx=0x7fe114011d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e079ca6635ca6be0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.986+0000 7fe120895640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1080b6590 0x7fe1080b8aa0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe114023000 tx=0x7fe114024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.986+0000 7fe1127fc640 1 -- 172.21.15.29:0/1215763399 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114017070 con 0x56468142c840 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.986+0000 7fe1127fc640 1 -- 172.21.15.29:0/1215763399 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe11401c430 con 0x56468142c840 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.986+0000 7fe1127fc640 1 -- 172.21.15.29:0/1215763399 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11400e610 con 0x56468142c840 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 -- 172.21.15.29:0/1215763399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 msgr2=0x5646817d3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817d3600 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe114005fd0 tx=0x7fe114011d50 comp rx=0 tx=0).stop 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 -- 172.21.15.29:0/1215763399 shutdown_connections 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817d4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817d3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 --2- 172.21.15.29:0/1215763399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817d3600 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 -- 172.21.15.29:0/1215763399 >> 172.21.15.29:0/1215763399 conn(0x564681420990 msgr2=0x5646817d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 -- 172.21.15.29:0/1215763399 shutdown_connections 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.987+0000 7fe12886c740 1 -- 172.21.15.29:0/1215763399 wait complete. 2024-09-21T02:51:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe12886c740 1 -- start start 2024-09-21T02:51:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe12886c740 1 Processor -- start 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe12886c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817df1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe12886c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817df770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe12886c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe12886c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56468159a7a0 con 0x56468140bb60 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe12886c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5646816de0a0 con 0x564681403980 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe12886c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5646817e1de0 con 0x56468142c840 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.988+0000 7fe113fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817df770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe120895640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817df1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817df1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42956/0 (socket says 172.21.15.29:42956) 2024-09-21T02:51:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe120895640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817eb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50448/0 (socket says 172.21.15.29:50448) 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 -- 172.21.15.29:0/602781064 learned_addr learned my addr 172.21.15.29:0/602781064 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe12886c740 10 client.0 _ll_register_callbacks cb 0x5646816bb3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 -- 172.21.15.29:0/602781064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 msgr2=0x5646817eb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817eb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 -- 172.21.15.29:0/602781064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 msgr2=0x5646817df770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817df770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 -- 172.21.15.29:0/602781064 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646816cc880 con 0x564681403980 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe11bfff640 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817df1f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00bc20 tx=0x7fe10c0039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=634190d5be12cc45 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe0fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/602781064 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00be90 con 0x564681403980 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe12886c740 1 -- 172.21.15.29:0/602781064 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe11401c430 con 0x564681403980 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.989+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/602781064 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe10c011650 con 0x564681403980 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/602781064 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe10c011850 con 0x564681403980 2024-09-21T02:51:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe12886c740 10 client.26618 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe12886c740 1 -- 172.21.15.29:0/602781064 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5646814a6b90 con 0x564681403980 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/602781064 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe11401c430 con 0x564681403980 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/602781064 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fe10c01bad0 con 0x564681403980 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe12886c740 10 client.26618 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe12886c740 10 client.26618 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe12886c740 20 client.26618 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe12886c740 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5646813eb1c0 0x5646817f11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe12886c740 4 client.26618 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe12886c740 1 -- 172.21.15.29:0/602781064 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56468174bae0 con 0x5646813eb1c0 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.990+0000 7fe113fff640 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5646813eb1c0 0x5646817f11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.991+0000 7fe0fb7fe640 10 client.26618 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.991+0000 7fe113fff640 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5646813eb1c0 0x5646817f11b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=396b19c76727485d server_cookie=74f3ae02807ede6 in_seq=0 out_seq=0 2024-09-21T02:51:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.992+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/602781064 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56468174bae0 con 0x5646813eb1c0 2024-09-21T02:51:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.992+0000 7fe0fb7fe640 10 client.26618 handle_command_reply: tid=0 2024-09-21T02:51:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.992+0000 7fe12886c740 1 client.26618 shutdown 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.992+0000 7fe12886c740 1 -- 172.21.15.29:0/602781064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 msgr2=0x5646817df1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.992+0000 7fe12886c740 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817df1f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00bc20 tx=0x7fe10c0039c0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 1 -- 172.21.15.29:0/602781064 shutdown_connections 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56468142c840 0x5646817eb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5646813eb1c0 0x5646817f11b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817df770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 1 --2- 172.21.15.29:0/602781064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564681403980 0x5646817df1f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe0fb7fe640 0 client.26618 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 1 -- 172.21.15.29:0/602781064 >> 172.21.15.29:0/602781064 conn(0x564681420990 msgr2=0x5646817d6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 1 -- 172.21.15.29:0/602781064 shutdown_connections 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 1 -- 172.21.15.29:0/602781064 wait complete. 2024-09-21T02:51:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:43.993+0000 7fe12886c740 20 client.26618 trim_cache size 0 max 16384 2024-09-21T02:51:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.019+0000 7fe12886c740 1 Processor -- start 2024-09-21T02:51:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.019+0000 7fe12886c740 1 -- start start 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.019+0000 7fe12886c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817d6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe12886c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817e0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe12886c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468142c840 0x5646817d4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe12886c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1ae730 con 0x56468140bb60 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe12886c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1b0270 con 0x56468142c840 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe12886c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1aee00 con 0x564681403980 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817e0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe11bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817d6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817e0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39754/0 (socket says 172.21.15.29:39754) 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 -- 172.21.15.29:0/641578641 learned_addr learned my addr 172.21.15.29:0/641578641 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe120895640 1 --2- 172.21.15.29:0/641578641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468142c840 0x5646817d4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 -- 172.21.15.29:0/641578641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 msgr2=0x5646817d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 --2- 172.21.15.29:0/641578641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817d6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 -- 172.21.15.29:0/641578641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468142c840 msgr2=0x5646817d4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 --2- 172.21.15.29:0/641578641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468142c840 0x5646817d4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 -- 172.21.15.29:0/641578641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646817d5b20 con 0x56468140bb60 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.020+0000 7fe113fff640 1 --2- 172.21.15.29:0/641578641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817e0ed0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe1040b8010 tx=0x7fe1040b8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e662700ef86d86 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.021+0000 7fe111ffb640 1 -- 172.21.15.29:0/641578641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1040cd020 con 0x56468140bb60 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.021+0000 7fe111ffb640 1 -- 172.21.15.29:0/641578641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1040d0c90 con 0x56468140bb60 2024-09-21T02:51:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.021+0000 7fe111ffb640 1 -- 172.21.15.29:0/641578641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1040d08f0 con 0x56468140bb60 2024-09-21T02:51:44.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 -- 172.21.15.29:0/641578641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 msgr2=0x5646817e0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:44.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 --2- 172.21.15.29:0/641578641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817e0ed0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe1040b8010 tx=0x7fe1040b8040 comp rx=0 tx=0).stop 2024-09-21T02:51:44.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 -- 172.21.15.29:0/641578641 shutdown_connections 2024-09-21T02:51:44.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 --2- 172.21.15.29:0/641578641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468142c840 0x5646817d4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 --2- 172.21.15.29:0/641578641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468140bb60 0x5646817e0ed0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 --2- 172.21.15.29:0/641578641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817d6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 -- 172.21.15.29:0/641578641 >> 172.21.15.29:0/641578641 conn(0x564681420990 msgr2=0x5646817d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 -- 172.21.15.29:0/641578641 shutdown_connections 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 -- 172.21.15.29:0/641578641 wait complete. 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 -- start start 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.022+0000 7fe12886c740 1 Processor -- start 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe12886c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817e4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe12886c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468140bb60 0x5646817e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe12886c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468142c840 0x564681855bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe12886c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1afba0 con 0x56468142c840 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe12886c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1af4d0 con 0x56468140bb60 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe12886c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1ac710 con 0x564681403980 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe12886c740 10 client.0 _ll_register_callbacks cb 0x7fe10c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe11bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe113fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468140bb60 0x5646817e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe11bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817e4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50462/0 (socket says 172.21.15.29:50462) 2024-09-21T02:51:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe113fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468140bb60 0x5646817e4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42966/0 (socket says 172.21.15.29:42966) 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe11bfff640 1 -- 172.21.15.29:0/2604091551 learned_addr learned my addr 172.21.15.29:0/2604091551 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe120895640 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468142c840 0x564681855bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe113fff640 1 -- 172.21.15.29:0/2604091551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 msgr2=0x5646817e4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe113fff640 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817e4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe113fff640 1 -- 172.21.15.29:0/2604091551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468142c840 msgr2=0x564681855bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe113fff640 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468142c840 0x564681855bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.023+0000 7fe113fff640 1 -- 172.21.15.29:0/2604091551 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564681856230 con 0x56468140bb60 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe113fff640 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468140bb60 0x5646817e4f00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1040ce3f0 tx=0x7fe1040d08f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b68b81a9a7b04194 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe0fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/2604091551 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1040d0ad0 con 0x56468140bb60 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/2604091551 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe1040cd3f0 con 0x56468140bb60 2024-09-21T02:51:44.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe12886c740 1 -- 172.21.15.29:0/2604091551 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1040d0c90 con 0x56468140bb60 2024-09-21T02:51:44.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/2604091551 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe1040d0f50 con 0x56468140bb60 2024-09-21T02:51:44.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe12886c740 10 client.26624 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:44.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe12886c740 1 -- 172.21.15.29:0/2604091551 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56468185d510 con 0x56468140bb60 2024-09-21T02:51:44.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.024+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/2604091551 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1040d5c60 con 0x56468140bb60 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/2604091551 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fe1040da710 con 0x56468140bb60 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe12886c740 10 client.26624 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe12886c740 10 client.26624 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe12886c740 20 client.26624 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe12886c740 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5646813c0570 0x56468185fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe12886c740 4 client.26624 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}] 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe12886c740 1 -- 172.21.15.29:0/2604091551 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}) -- 0x7fe1040d0c90 con 0x5646813c0570 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe11bfff640 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5646813c0570 0x56468185fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe0fb7fe640 10 client.26624 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:44.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.025+0000 7fe11bfff640 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5646813c0570 0x56468185fb60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d43530f20f6f661d server_cookie=79b927966ec93a1b in_seq=0 out_seq=0 2024-09-21T02:51:44.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257564 2.18 MB/sec execute 561 sec latency 71.697 ms 2024-09-21T02:51:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.120+0000 7fe0fb7fe640 1 -- 172.21.15.29:0/2604091551 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7fe1040d0c90 con 0x5646813c0570 2024-09-21T02:51:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.120+0000 7fe0fb7fe640 10 client.26624 handle_command_reply: tid=0 2024-09-21T02:51:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.120+0000 7fe12886c740 1 client.26624 shutdown 2024-09-21T02:51:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 -- 172.21.15.29:0/2604091551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468140bb60 msgr2=0x5646817e4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468140bb60 0x5646817e4f00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1040ce3f0 tx=0x7fe1040d08f0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 -- 172.21.15.29:0/2604091551 shutdown_connections 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5646813c0570 0x56468185fb60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56468142c840 0x564681855bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56468140bb60 0x5646817e4f00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe0fb7fe640 0 client.26624 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 --2- 172.21.15.29:0/2604091551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564681403980 0x5646817e4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 -- 172.21.15.29:0/2604091551 >> 172.21.15.29:0/2604091551 conn(0x564681420990 msgr2=0x5646817d4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 -- 172.21.15.29:0/2604091551 shutdown_connections 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 1 -- 172.21.15.29:0/2604091551 wait complete. 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.121+0000 7fe12886c740 20 client.26624 trim_cache size 0 max 16384 2024-09-21T02:51:44.124 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 50, 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "294917d3": { 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 50, 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 137.8, 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:51:44.125 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:51:44.126 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 -- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1080b6590 msgr2=0x7fe1080b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1080b6590 0x7fe1080b8aa0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe114023000 tx=0x7fe114024000 comp rx=0 tx=0).stop 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 -- 172.21.15.29:0/4279303639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe11c1094d0 msgr2=0x7fe11c19a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe11c1094d0 0x7fe11c19a920 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe10402a930 tx=0x7fe1040319a0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 -- 172.21.15.29:0/4279303639 shutdown_connections 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe1080b6590 0x7fe1080b8aa0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe11c109e20 0x7fe11c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe11c1094d0 0x7fe11c19a920 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 --2- 172.21.15.29:0/4279303639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c103b90 0x7fe11c19a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 -- 172.21.15.29:0/4279303639 >> 172.21.15.29:0/4279303639 conn(0x7fe11c0ff5e0 msgr2=0x7fe11c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 -- 172.21.15.29:0/4279303639 shutdown_connections 2024-09-21T02:51:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:44.122+0000 7fe1222f8640 1 -- 172.21.15.29:0/4279303639 wait complete. 2024-09-21T02:51:44.910 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '294917d3', quiesce took 3.7 seconds. Will release after: 14.100000000000001 2024-09-21T02:51:45.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.18 MB/sec execute 562 sec latency 897.224 ms 2024-09-21T02:51:45.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:45 smithi066 ceph-mon[33436]: pgmap v768: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 113 KiB/s wr, 44 op/s 2024-09-21T02:51:45.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:45 smithi190 ceph-mon[33871]: pgmap v768: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 113 KiB/s wr, 44 op/s 2024-09-21T02:51:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:45 smithi029 ceph-mon[31006]: pgmap v768: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 113 KiB/s wr, 44 op/s 2024-09-21T02:51:46.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.17 MB/sec execute 563 sec latency 1897.362 ms 2024-09-21T02:51:47.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.17 MB/sec execute 564 sec latency 2897.482 ms 2024-09-21T02:51:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:47 smithi066 ceph-mon[33436]: pgmap v769: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 226 KiB/s wr, 46 op/s 2024-09-21T02:51:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:47 smithi190 ceph-mon[33871]: pgmap v769: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 226 KiB/s wr, 46 op/s 2024-09-21T02:51:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:47 smithi029 ceph-mon[31006]: pgmap v769: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 226 KiB/s wr, 46 op/s 2024-09-21T02:51:48.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.17 MB/sec execute 565 sec latency 3897.530 ms 2024-09-21T02:51:49.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.16 MB/sec execute 566 sec latency 4897.620 ms 2024-09-21T02:51:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 -- 172.21.15.29:0/1358106259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01068a0 msgr2=0x7f97f0108cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/1358106259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01068a0 0x7f97f0108cc0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009900 tx=0x7f97e402a410 comp rx=0 tx=0).stop 2024-09-21T02:51:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 -- 172.21.15.29:0/1358106259 shutdown_connections 2024-09-21T02:51:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/1358106259 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f01093e0 0x7f97f010b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/1358106259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01068a0 0x7f97f0108cc0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/1358106259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f006b690 0x7f97f0106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 -- 172.21.15.29:0/1358106259 >> 172.21.15.29:0/1358106259 conn(0x7f97f00fdee0 msgr2=0x7f97f0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 -- 172.21.15.29:0/1358106259 shutdown_connections 2024-09-21T02:51:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.349+0000 7f97f6fa8640 1 -- 172.21.15.29:0/1358106259 wait complete. 2024-09-21T02:51:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f6fa8640 1 Processor -- start 2024-09-21T02:51:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f6fa8640 1 -- start start 2024-09-21T02:51:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f6fa8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f006b690 0x7f97f0199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f6fa8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f01068a0 0x7f97f019a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f6fa8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01093e0 0x7f97f019b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f6fa8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010f1f0 con 0x7f97f01093e0 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f6fa8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010f070 con 0x7f97f01068a0 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f6fa8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010f370 con 0x7f97f006b690 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f5545640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01093e0 0x7f97f019b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f5545640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01093e0 0x7f97f019b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47698/0 (socket says 172.21.15.29:47698) 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.350+0000 7f97f5545640 1 -- 172.21.15.29:0/2351768830 learned_addr learned my addr 172.21.15.29:0/2351768830 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f4d44640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f006b690 0x7f97f0199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97effff640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f01068a0 0x7f97f019a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f5545640 1 -- 172.21.15.29:0/2351768830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f006b690 msgr2=0x7f97f0199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f5545640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f006b690 0x7f97f0199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f5545640 1 -- 172.21.15.29:0/2351768830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f01068a0 msgr2=0x7f97f019a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f5545640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f01068a0 0x7f97f019a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f5545640 1 -- 172.21.15.29:0/2351768830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f019b8b0 con 0x7f97f01093e0 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97effff640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f01068a0 0x7f97f019a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f4d44640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f006b690 0x7f97f0199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f5545640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01093e0 0x7f97f019b120 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f97e800ccb0 tx=0x7f97e800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97edffb640 1 -- 172.21.15.29:0/2351768830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8004b90 con 0x7f97f01093e0 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97edffb640 1 -- 172.21.15.29:0/2351768830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97e8004d30 con 0x7f97f01093e0 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97f6fa8640 1 -- 172.21.15.29:0/2351768830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f019bba0 con 0x7f97f01093e0 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.351+0000 7f97edffb640 1 -- 172.21.15.29:0/2351768830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8005290 con 0x7f97f01093e0 2024-09-21T02:51:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.352+0000 7f97f6fa8640 1 -- 172.21.15.29:0/2351768830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f01a2320 con 0x7f97f01093e0 2024-09-21T02:51:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.354+0000 7f97edffb640 1 -- 172.21.15.29:0/2351768830 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f97e800f450 con 0x7f97f01093e0 2024-09-21T02:51:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.354+0000 7f97fdff2740 1 Processor -- start 2024-09-21T02:51:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.354+0000 7f97fdff2740 1 -- start start 2024-09-21T02:51:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x56402653ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x56402653d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x56402653e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97fdff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5640264af6f0 con 0x5640261951b0 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97fdff2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56402649eb20 con 0x56402616c2f0 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97fdff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5640264a2930 con 0x5640261744d0 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x56402653ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x56402653ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47700/0 (socket says 172.21.15.29:47700) 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 -- 172.21.15.29:0/630066766 learned_addr learned my addr 172.21.15.29:0/630066766 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97f4d44640 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x56402653d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x56402653e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 -- 172.21.15.29:0/630066766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 msgr2=0x56402653e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x56402653e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 -- 172.21.15.29:0/630066766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 msgr2=0x56402653d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x56402653d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97effff640 1 -- 172.21.15.29:0/630066766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564026464530 con 0x5640261951b0 2024-09-21T02:51:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97f4d44640 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x56402653d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.355+0000 7f97edffb640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f97d80b6590 0x7f97d80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.356+0000 7f97f4d44640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f97d80b6590 0x7f97d80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.356+0000 7f97effff640 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x56402653ce50 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f97e40096c0 tx=0x7f97e4009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bfa48f5a9b05e0e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.356+0000 7f97edffb640 1 -- 172.21.15.29:0/2351768830 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f97e80e7700 con 0x7f97f01093e0 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.356+0000 7f97d6ffd640 1 -- 172.21.15.29:0/630066766 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4046070 con 0x5640261951b0 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.356+0000 7f97d6ffd640 1 -- 172.21.15.29:0/630066766 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97e4003d00 con 0x5640261951b0 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.356+0000 7f97d6ffd640 1 -- 172.21.15.29:0/630066766 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4035070 con 0x5640261951b0 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 -- 172.21.15.29:0/630066766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 msgr2=0x56402653ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97f4d44640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f97d80b6590 0x7f97d80b8aa0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009dc0 tx=0x7f97e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x56402653ce50 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f97e40096c0 tx=0x7f97e4009560 comp rx=0 tx=0).stop 2024-09-21T02:51:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 -- 172.21.15.29:0/630066766 shutdown_connections 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x56402653e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x56402653d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 --2- 172.21.15.29:0/630066766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x56402653ce50 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 -- 172.21.15.29:0/630066766 >> 172.21.15.29:0/630066766 conn(0x564026189300 msgr2=0x5640265388e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 -- 172.21.15.29:0/630066766 shutdown_connections 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 -- 172.21.15.29:0/630066766 wait complete. 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 -- start start 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.357+0000 7f97fdff2740 1 Processor -- start 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x5640265465c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x564026546c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x5640265477f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97fdff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5640263d6ab0 con 0x5640261951b0 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97fdff2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564026444730 con 0x56402616c2f0 2024-09-21T02:51:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97fdff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56402653ead0 con 0x5640261744d0 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97fdff2740 10 client.0 _ll_register_callbacks cb 0x5640264986b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f5545640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x5640265477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f5545640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x5640265477f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47710/0 (socket says 172.21.15.29:47710) 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f5545640 1 -- 172.21.15.29:0/2965815766 learned_addr learned my addr 172.21.15.29:0/2965815766 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f4d44640 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x5640265465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97effff640 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x564026546c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f5545640 1 -- 172.21.15.29:0/2965815766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 msgr2=0x564026546c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f5545640 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x564026546c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f5545640 1 -- 172.21.15.29:0/2965815766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 msgr2=0x5640265465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f5545640 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x5640265465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.358+0000 7f97f5545640 1 -- 172.21.15.29:0/2965815766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564026200790 con 0x5640261951b0 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97f5545640 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x5640265477f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f97e80ad4a0 tx=0x7f97e80acc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d3e87bcd3b07f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97cffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97cffff640 1 -- 172.21.15.29:0/2965815766 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e801d040 con 0x5640261951b0 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97cffff640 1 -- 172.21.15.29:0/2965815766 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f97e80ac770 con 0x5640261951b0 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 1 -- 172.21.15.29:0/2965815766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97e4003d00 con 0x5640261951b0 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97cffff640 1 -- 172.21.15.29:0/2965815766 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f97e80b1ff0 con 0x5640261951b0 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 10 client.17244 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 1 -- 172.21.15.29:0/2965815766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564026279620 con 0x5640261951b0 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97cffff640 1 -- 172.21.15.29:0/2965815766 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97e4003d00 con 0x5640261951b0 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97cffff640 1 -- 172.21.15.29:0/2965815766 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f97e80ac3e0 con 0x5640261951b0 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 10 client.17244 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 10 client.17244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 20 client.17244 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564026153d80 0x564026553d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 4 client.17244 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:51:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.359+0000 7f97fdff2740 1 -- 172.21.15.29:0/2965815766 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640264b1640 con 0x564026153d80 2024-09-21T02:51:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.360+0000 7f97f4d44640 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564026153d80 0x564026553d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.360+0000 7f97cffff640 10 client.17244 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.360+0000 7f97f4d44640 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564026153d80 0x564026553d70 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24e062117b0ec8e server_cookie=1ee5e57912e1e2f8 in_seq=0 out_seq=0 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97cffff640 1 -- 172.21.15.29:0/2965815766 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5640264b1640 con 0x564026153d80 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97cffff640 10 client.17244 handle_command_reply: tid=0 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 client.17244 shutdown 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 -- 172.21.15.29:0/2965815766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 msgr2=0x5640265477f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x5640265477f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f97e80ad4a0 tx=0x7f97e80acc80 comp rx=0 tx=0).stop 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 -- 172.21.15.29:0/2965815766 shutdown_connections 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564026153d80 0x564026553d70 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261951b0 0x5640265477f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x564026546c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97cffff640 0 client.17244 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 --2- 172.21.15.29:0/2965815766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56402616c2f0 0x5640265465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.361+0000 7f97fdff2740 1 -- 172.21.15.29:0/2965815766 >> 172.21.15.29:0/2965815766 conn(0x564026189300 msgr2=0x564026536ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.362+0000 7f97fdff2740 1 -- 172.21.15.29:0/2965815766 shutdown_connections 2024-09-21T02:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.362+0000 7f97fdff2740 1 -- 172.21.15.29:0/2965815766 wait complete. 2024-09-21T02:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.362+0000 7f97fdff2740 20 client.17244 trim_cache size 0 max 16384 2024-09-21T02:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.376+0000 7f97fdff2740 1 Processor -- start 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.376+0000 7f97fdff2740 1 -- start start 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56402616c2f0 0x564026544b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261744d0 0x564026545090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402653cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97fdff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01ac390 con 0x5640261744d0 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97fdff2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01aded0 con 0x5640261951b0 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97fdff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01aca60 con 0x56402616c2f0 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261744d0 0x564026545090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261744d0 0x564026545090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47724/0 (socket says 172.21.15.29:47724) 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 -- 172.21.15.29:0/1897271850 learned_addr learned my addr 172.21.15.29:0/1897271850 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97f4d44640 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56402616c2f0 0x564026544b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97f5545640 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402653cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 -- 172.21.15.29:0/1897271850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56402616c2f0 msgr2=0x564026544b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56402616c2f0 0x564026544b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 -- 172.21.15.29:0/1897271850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 msgr2=0x56402653cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402653cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 -- 172.21.15.29:0/1897271850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640265447d0 con 0x5640261744d0 2024-09-21T02:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97f5545640 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402653cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97f4d44640 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56402616c2f0 0x564026544b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.377+0000 7f97effff640 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261744d0 0x564026545090 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f97e4005930 tx=0x7f97e4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79fdea0c3263b9e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97d67fc640 1 -- 172.21.15.29:0/1897271850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e404f070 con 0x5640261744d0 2024-09-21T02:51:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97d67fc640 1 -- 172.21.15.29:0/1897271850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97e4003d00 con 0x5640261744d0 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97d67fc640 1 -- 172.21.15.29:0/1897271850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4035030 con 0x5640261744d0 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97fdff2740 1 -- 172.21.15.29:0/1897271850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261744d0 msgr2=0x564026545090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97fdff2740 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261744d0 0x564026545090 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f97e4005930 tx=0x7f97e4005fd0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97fdff2740 1 -- 172.21.15.29:0/1897271850 shutdown_connections 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97fdff2740 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402653cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97fdff2740 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640261744d0 0x564026545090 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97fdff2740 1 --2- 172.21.15.29:0/1897271850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56402616c2f0 0x564026544b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97fdff2740 1 -- 172.21.15.29:0/1897271850 >> 172.21.15.29:0/1897271850 conn(0x564026189300 msgr2=0x5640265457c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.378+0000 7f97fdff2740 1 -- 172.21.15.29:0/1897271850 shutdown_connections 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 -- 172.21.15.29:0/1897271850 wait complete. 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 -- start start 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 Processor -- start 2024-09-21T02:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56402616c2f0 0x56402654a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x56402654ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402654b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01ad800 con 0x56402616c2f0 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01ad130 con 0x5640261951b0 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01aa370 con 0x5640261744d0 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97fdff2740 10 client.0 _ll_register_callbacks cb 0x7f97e8004a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97f4d44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56402616c2f0 0x56402654a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97f4d44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56402616c2f0 0x56402654a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47734/0 (socket says 172.21.15.29:47734) 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.379+0000 7f97f4d44640 1 -- 172.21.15.29:0/3721138501 learned_addr learned my addr 172.21.15.29:0/3721138501 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97f5545640 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402654b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97effff640 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x56402654ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97f4d44640 1 -- 172.21.15.29:0/3721138501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 msgr2=0x56402654ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97f4d44640 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x56402654ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97f4d44640 1 -- 172.21.15.29:0/3721138501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 msgr2=0x56402654b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97f4d44640 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402654b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97f4d44640 1 -- 172.21.15.29:0/3721138501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56402654bd10 con 0x56402616c2f0 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97f5545640 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402654b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97f4d44640 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56402616c2f0 0x56402654a6d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x564026554470 tx=0x7f97e0015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77821b8c1ffdea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97cffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97cffff640 1 -- 172.21.15.29:0/3721138501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0005020 con 0x56402616c2f0 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97cffff640 1 -- 172.21.15.29:0/3721138501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f97e002d040 con 0x56402616c2f0 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97fdff2740 1 -- 172.21.15.29:0/3721138501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97e4003d00 con 0x56402616c2f0 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97cffff640 1 -- 172.21.15.29:0/3721138501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f97e003e050 con 0x56402616c2f0 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97fdff2740 10 client.17250 fetch_fsmap learned FSMap version 31 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.380+0000 7f97fdff2740 1 -- 172.21.15.29:0/3721138501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5640265bee80 con 0x56402616c2f0 2024-09-21T02:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97cffff640 1 -- 172.21.15.29:0/3721138501 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97e4003d00 con 0x56402616c2f0 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97cffff640 1 -- 172.21.15.29:0/3721138501 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f97e0004c20 con 0x56402616c2f0 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97fdff2740 10 client.17250 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97fdff2740 10 client.17250 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97fdff2740 20 client.17250 populate_metadata read hostname 'smithi029' 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97fdff2740 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640261291c0 0x5640265c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97fdff2740 4 client.17250 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97fdff2740 1 -- 172.21.15.29:0/3721138501 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640264ad0f0 con 0x5640261291c0 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97effff640 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640261291c0 0x5640265c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97effff640 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640261291c0 0x5640265c1850 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97c9890701629843 server_cookie=889c3ee8715d8061 in_seq=0 out_seq=0 2024-09-21T02:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.381+0000 7f97cffff640 10 client.17250 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.410+0000 7f97cffff640 1 -- 172.21.15.29:0/3721138501 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640264ad0f0 con 0x5640261291c0 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.410+0000 7f97cffff640 10 client.17250 handle_command_reply: tid=0 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.410+0000 7f97fdff2740 1 client.17250 shutdown 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 -- 172.21.15.29:0/3721138501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56402616c2f0 msgr2=0x56402654a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56402616c2f0 0x56402654a6d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x564026554470 tx=0x7f97e0015040 comp rx=0 tx=0).stop 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 -- 172.21.15.29:0/3721138501 shutdown_connections 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640261951b0 0x56402654b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640261744d0 0x56402654ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640261291c0 0x5640265c1850 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 --2- 172.21.15.29:0/3721138501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56402616c2f0 0x56402654a6d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97cffff640 0 client.17250 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 -- 172.21.15.29:0/3721138501 >> 172.21.15.29:0/3721138501 conn(0x564026189300 msgr2=0x564026544af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 -- 172.21.15.29:0/3721138501 shutdown_connections 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 1 -- 172.21.15.29:0/3721138501 wait complete. 2024-09-21T02:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.411+0000 7f97fdff2740 20 client.17250 trim_cache size 0 max 16384 2024-09-21T02:51:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:51:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:51:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:51:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:51:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 -- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f97d80b6590 msgr2=0x7f97d80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f97d80b6590 0x7f97d80b8aa0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009dc0 tx=0x7f97e0005f90 comp rx=0 tx=0).stop 2024-09-21T02:51:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 -- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01093e0 msgr2=0x7f97f019b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01093e0 0x7f97f019b120 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f97e800ccb0 tx=0x7f97e800a620 comp rx=0 tx=0).stop 2024-09-21T02:51:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 -- 172.21.15.29:0/2351768830 shutdown_connections 2024-09-21T02:51:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f97d80b6590 0x7f97d80b8aa0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97f01093e0 0x7f97f019b120 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f01068a0 0x7f97f019a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 --2- 172.21.15.29:0/2351768830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f006b690 0x7f97f0199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 -- 172.21.15.29:0/2351768830 >> 172.21.15.29:0/2351768830 conn(0x7f97f00fdee0 msgr2=0x7f97f0104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 -- 172.21.15.29:0/2351768830 shutdown_connections 2024-09-21T02:51:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:51:49.412+0000 7f97f6fa8640 1 -- 172.21.15.29:0/2351768830 wait complete. 2024-09-21T02:51:49.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:49 smithi066 ceph-mon[33436]: pgmap v770: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 274 KiB/s wr, 62 op/s 2024-09-21T02:51:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:49 smithi190 ceph-mon[33871]: pgmap v770: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 274 KiB/s wr, 62 op/s 2024-09-21T02:51:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:49 smithi029 ceph-mon[31006]: pgmap v770: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 274 KiB/s wr, 62 op/s 2024-09-21T02:51:50.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.16 MB/sec execute 567 sec latency 5897.733 ms 2024-09-21T02:51:51.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.16 MB/sec execute 568 sec latency 6897.879 ms 2024-09-21T02:51:51.687 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:51 smithi190 ceph-mon[33871]: pgmap v771: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 272 KiB/s wr, 58 op/s 2024-09-21T02:51:51.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:51 smithi066 ceph-mon[33436]: pgmap v771: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 272 KiB/s wr, 58 op/s 2024-09-21T02:51:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:51 smithi029 ceph-mon[31006]: pgmap v771: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 272 KiB/s wr, 58 op/s 2024-09-21T02:51:52.104 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.15 MB/sec execute 569 sec latency 7897.968 ms 2024-09-21T02:51:53.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.15 MB/sec execute 570 sec latency 8898.090 ms 2024-09-21T02:51:53.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:53 smithi066 ceph-mon[33436]: pgmap v772: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 271 KiB/s wr, 58 op/s 2024-09-21T02:51:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:53 smithi190 ceph-mon[33871]: pgmap v772: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 271 KiB/s wr, 58 op/s 2024-09-21T02:51:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:53 smithi029 ceph-mon[31006]: pgmap v772: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 271 KiB/s wr, 58 op/s 2024-09-21T02:51:54.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.14 MB/sec execute 571 sec latency 9898.198 ms 2024-09-21T02:51:55.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.14 MB/sec execute 572 sec latency 10898.327 ms 2024-09-21T02:51:55.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:55 smithi066 ceph-mon[33436]: pgmap v773: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 276 KiB/s wr, 63 op/s 2024-09-21T02:51:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:55 smithi190 ceph-mon[33871]: pgmap v773: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 276 KiB/s wr, 63 op/s 2024-09-21T02:51:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:55 smithi029 ceph-mon[31006]: pgmap v773: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 276 KiB/s wr, 63 op/s 2024-09-21T02:51:56.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.14 MB/sec execute 573 sec latency 11898.444 ms 2024-09-21T02:51:57.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.13 MB/sec execute 574 sec latency 12898.563 ms 2024-09-21T02:51:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:57 smithi066 ceph-mon[33436]: pgmap v774: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 173 KiB/s wr, 39 op/s 2024-09-21T02:51:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:57 smithi190 ceph-mon[33871]: pgmap v774: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 173 KiB/s wr, 39 op/s 2024-09-21T02:51:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:57 smithi029 ceph-mon[31006]: pgmap v774: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 173 KiB/s wr, 39 op/s 2024-09-21T02:51:58.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.13 MB/sec execute 575 sec latency 13898.623 ms 2024-09-21T02:51:59.010 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:51:59.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.13 MB/sec execute 576 sec latency 14898.676 ms 2024-09-21T02:51:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:51:59.676 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:51:59 smithi029 ceph-mon[31006]: pgmap v775: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 58 KiB/s wr, 36 op/s 2024-09-21T02:51:59.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:51:59 smithi066 ceph-mon[33436]: pgmap v775: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 58 KiB/s wr, 36 op/s 2024-09-21T02:51:59.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 -- 172.21.15.29:0/3461453176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 msgr2=0x7f319c103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:59.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 --2- 172.21.15.29:0/3461453176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 0x7f319c103f90 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f318c009900 tx=0x7f318c02a410 comp rx=0 tx=0).stop 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 -- 172.21.15.29:0/3461453176 shutdown_connections 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 --2- 172.21.15.29:0/3461453176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f319c109e10 0x7f319c110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 --2- 172.21.15.29:0/3461453176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c1094c0 0x7f319c1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 --2- 172.21.15.29:0/3461453176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 0x7f319c103f90 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 -- 172.21.15.29:0/3461453176 >> 172.21.15.29:0/3461453176 conn(0x7f319c0ff610 msgr2=0x7f319c101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 -- 172.21.15.29:0/3461453176 shutdown_connections 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.770+0000 7f31a18a1640 1 -- 172.21.15.29:0/3461453176 wait complete. 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.771+0000 7f31a18a1640 1 Processor -- start 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.771+0000 7f31a18a1640 1 -- start start 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f31a18a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 0x7f319c19c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f31a18a1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f319c1094c0 0x7f319c19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f31a18a1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c109e10 0x7f319c1a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f31a18a1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f319c113980 con 0x7f319c103b80 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f31a18a1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f319c113800 con 0x7f319c1094c0 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f31a18a1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f319c113b00 con 0x7f319c109e10 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 0x7f319c19c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 0x7f319c19c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38000/0 (socket says 172.21.15.29:38000) 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 -- 172.21.15.29:0/54253603 learned_addr learned my addr 172.21.15.29:0/54253603 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319a7fc640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f319c1094c0 0x7f319c19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319b7fe640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c109e10 0x7f319c1a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 -- 172.21.15.29:0/54253603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c109e10 msgr2=0x7f319c1a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c109e10 0x7f319c1a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 -- 172.21.15.29:0/54253603 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f319c1094c0 msgr2=0x7f319c19cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f319c1094c0 0x7f319c19cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 -- 172.21.15.29:0/54253603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319c1a2560 con 0x7f319c103b80 2024-09-21T02:51:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.772+0000 7f319affd640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 0x7f319c19c5f0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f318c009230 tx=0x7f318c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.773+0000 7f31a089f640 1 -- 172.21.15.29:0/54253603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c03f030 con 0x7f319c103b80 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.773+0000 7f31a089f640 1 -- 172.21.15.29:0/54253603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f318c003a80 con 0x7f319c103b80 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.773+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319c1a27f0 con 0x7f319c103b80 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.773+0000 7f31a089f640 1 -- 172.21.15.29:0/54253603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c035070 con 0x7f319c103b80 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.773+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f319c1a8fb0 con 0x7f319c103b80 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.774+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f319c104e00 con 0x7f319c103b80 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.776+0000 7f31a089f640 1 -- 172.21.15.29:0/54253603 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f318c042030 con 0x7f319c103b80 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.776+0000 7f31a089f640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31840b6510 0x7f31840b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.777+0000 7f319a7fc640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31840b6510 0x7f31840b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.777+0000 7f31a089f640 1 -- 172.21.15.29:0/54253603 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f318c108150 con 0x7f319c103b80 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.777+0000 7f319a7fc640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31840b6510 0x7f31840b8a20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f319c105c10 tx=0x7f319000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:51:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:51:59.780+0000 7f31a089f640 1 -- 172.21.15.29:0/54253603 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f318c0d2a40 con 0x7f319c103b80 2024-09-21T02:51:59.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:51:59 smithi190 ceph-mon[33871]: pgmap v775: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 58 KiB/s wr, 36 op/s 2024-09-21T02:52:00.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.032+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f319c1a9180 con 0x7f319c103b80 2024-09-21T02:52:00.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.033+0000 7f31a089f640 1 -- 172.21.15.29:0/54253603 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11971 (secure 0 0 0) 0x7f318c0d7980 con 0x7f319c103b80 2024-09-21T02:52:00.035 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:52:00.035 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-09-21T02:42:31:935404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:42:31.308247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:52:00.035 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-09-21T02:52:00.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31840b6510 msgr2=0x7f31840b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:00.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31840b6510 0x7f31840b8a20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f319c105c10 tx=0x7f319000b390 comp rx=0 tx=0).stop 2024-09-21T02:52:00.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 msgr2=0x7f319c19c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 0x7f319c19c5f0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f318c009230 tx=0x7f318c004270 comp rx=0 tx=0).stop 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 shutdown_connections 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31840b6510 0x7f31840b8a20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c109e10 0x7f319c1a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f319c1094c0 0x7f319c19cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 --2- 172.21.15.29:0/54253603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c103b80 0x7f319c19c5f0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 >> 172.21.15.29:0/54253603 conn(0x7f319c0ff610 msgr2=0x7f319c108bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 shutdown_connections 2024-09-21T02:52:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:00.037+0000 7f31a18a1640 1 -- 172.21.15.29:0/54253603 wait complete. 2024-09-21T02:52:00.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.12 MB/sec execute 577 sec latency 15898.722 ms 2024-09-21T02:52:00.758 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:00 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/54253603' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:52:00.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:00 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/54253603' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:52:00.785 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 294917d3 --release --await' 2024-09-21T02:52:00.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 294917d3 --release --await 2024-09-21T02:52:00.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:00 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/54253603' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:52:01.051 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:52:01.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257605 2.12 MB/sec execute 578 sec latency 16898.811 ms 2024-09-21T02:52:01.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.624+0000 7f4037c14640 1 -- 172.21.15.29:0/3573851341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 msgr2=0x7f4030102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.624+0000 7f4037c14640 1 --2- 172.21.15.29:0/3573851341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 0x7f4030102e70 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f402c009900 tx=0x7f402c02a410 comp rx=0 tx=0).stop 2024-09-21T02:52:01.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.624+0000 7f4037c14640 1 -- 172.21.15.29:0/3573851341 shutdown_connections 2024-09-21T02:52:01.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.624+0000 7f4037c14640 1 --2- 172.21.15.29:0/3573851341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4030108cf0 0x7f403010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.624+0000 7f4037c14640 1 --2- 172.21.15.29:0/3573851341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f40301083a0 0x7f40301087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.624+0000 7f4037c14640 1 --2- 172.21.15.29:0/3573851341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 0x7f4030102e70 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.624+0000 7f4037c14640 1 -- 172.21.15.29:0/3573851341 >> 172.21.15.29:0/3573851341 conn(0x7f4030079730 msgr2=0x7f4030079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.625+0000 7f4037c14640 1 -- 172.21.15.29:0/3573851341 shutdown_connections 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.625+0000 7f4037c14640 1 -- 172.21.15.29:0/3573851341 wait complete. 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.625+0000 7f4037c14640 1 Processor -- start 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.625+0000 7f4037c14640 1 -- start start 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f4037c14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 0x7f403019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f4037c14640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40301083a0 0x7f403019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f4037c14640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4030108cf0 0x7f403019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f4037c14640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4030113930 con 0x7f4030102a60 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f4037c14640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f40301137b0 con 0x7f4030108cf0 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f4037c14640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4030113ab0 con 0x7f40301083a0 2024-09-21T02:52:01.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40359b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 0x7f403019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40359b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 0x7f403019a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38018/0 (socket says 172.21.15.29:38018) 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40359b0640 1 -- 172.21.15.29:0/3611633752 learned_addr learned my addr 172.21.15.29:0/3611633752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40361b1640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4030108cf0 0x7f403019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40351af640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40301083a0 0x7f403019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40359b0640 1 -- 172.21.15.29:0/3611633752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40301083a0 msgr2=0x7f403019a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40359b0640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40301083a0 0x7f403019a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40359b0640 1 -- 172.21.15.29:0/3611633752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4030108cf0 msgr2=0x7f403019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40359b0640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4030108cf0 0x7f403019fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40359b0640 1 -- 172.21.15.29:0/3611633752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40301a0290 con 0x7f4030102a60 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.626+0000 7f40361b1640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4030108cf0 0x7f403019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.627+0000 7f40359b0640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 0x7f403019a320 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f402c009230 tx=0x7f402c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.627+0000 7f4026ffd640 1 -- 172.21.15.29:0/3611633752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c03f030 con 0x7f4030102a60 2024-09-21T02:52:01.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.627+0000 7f4026ffd640 1 -- 172.21.15.29:0/3611633752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f402c003a80 con 0x7f4030102a60 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.627+0000 7f4037c14640 1 -- 172.21.15.29:0/3611633752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40301a0520 con 0x7f4030102a60 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.627+0000 7f4026ffd640 1 -- 172.21.15.29:0/3611633752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c035070 con 0x7f4030102a60 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.627+0000 7f4037c14640 1 -- 172.21.15.29:0/3611633752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40301a6ce0 con 0x7f4030102a60 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.629+0000 7f403e188740 1 Processor -- start 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.629+0000 7f4026ffd640 1 -- 172.21.15.29:0/3611633752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f402c042030 con 0x7f4030102a60 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f403e188740 1 -- start start 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f403e188740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63b7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f403e188740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f403e188740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f403e188740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b3f62a2dc0 con 0x55b3f6011520 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f403e188740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b3f632f1f0 con 0x55b3f5ff0840 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f403e188740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b3f631e620 con 0x55b3f5fe8660 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40361b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63b7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40361b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63b7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38028/0 (socket says 172.21.15.29:38028) 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40361b1640 1 -- 172.21.15.29:0/2152078901 learned_addr learned my addr 172.21.15.29:0/2152078901 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40351af640 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40359b0640 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40361b1640 1 -- 172.21.15.29:0/2152078901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 msgr2=0x55b3f63b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40361b1640 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63b7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40361b1640 1 -- 172.21.15.29:0/2152078901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 msgr2=0x55b3f63b7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40361b1640 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63b7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.630+0000 7f40361b1640 1 -- 172.21.15.29:0/2152078901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3f62e6fd0 con 0x55b3f6011520 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.631+0000 7f40361b1640 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63b7510 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f4028009f90 tx=0x7f402800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f7027d4dae4a6d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.631+0000 7f4026ffd640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f401c0b6510 0x7f401c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.631+0000 7f40359b0640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f401c0b6510 0x7f401c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.631+0000 7f401bfff640 1 -- 172.21.15.29:0/2152078901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028003b70 con 0x55b3f6011520 2024-09-21T02:52:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.631+0000 7f401bfff640 1 -- 172.21.15.29:0/2152078901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4028003d10 con 0x55b3f6011520 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.631+0000 7f401bfff640 1 -- 172.21.15.29:0/2152078901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028011860 con 0x55b3f6011520 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.631+0000 7f4026ffd640 1 -- 172.21.15.29:0/3611633752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f402c107e50 con 0x7f4030102a60 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 -- 172.21.15.29:0/2152078901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 msgr2=0x55b3f63b7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63b7510 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f4028009f90 tx=0x7f402800f7f0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f40359b0640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f401c0b6510 0x7f401c0b8a20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55b3f63b6190 tx=0x7f402c02a410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 -- 172.21.15.29:0/2152078901 shutdown_connections 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63b7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 --2- 172.21.15.29:0/2152078901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63b7510 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 -- 172.21.15.29:0/2152078901 >> 172.21.15.29:0/2152078901 conn(0x55b3f6005910 msgr2=0x55b3f63b5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 -- 172.21.15.29:0/2152078901 shutdown_connections 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 -- 172.21.15.29:0/2152078901 wait complete. 2024-09-21T02:52:01.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.632+0000 7f403e188740 1 -- start start 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f403e188740 1 Processor -- start 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f403e188740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5fe8660 0x55b3f63c2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f403e188740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f403e188740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f6011520 0x55b3f63c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f403e188740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b3f62b9300 con 0x55b3f5fe8660 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f403e188740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b3f6322430 con 0x55b3f5ff0840 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f403e188740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b3f617fbb0 con 0x55b3f6011520 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f403e188740 10 client.0 _ll_register_callbacks cb 0x55b3f62d3120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f40361b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5fe8660 0x55b3f63c2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f40361b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5fe8660 0x55b3f63c2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38044/0 (socket says 172.21.15.29:38044) 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f40361b1640 1 -- 172.21.15.29:0/2292609466 learned_addr learned my addr 172.21.15.29:0/2292609466 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f40351af640 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.633+0000 7f40361b1640 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f6011520 0x55b3f63c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40361b1640 1 -- 172.21.15.29:0/2292609466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f6011520 msgr2=0x55b3f63c35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40361b1640 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f6011520 0x55b3f63c35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40361b1640 1 -- 172.21.15.29:0/2292609466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 msgr2=0x55b3f63c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40361b1640 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40361b1640 1 -- 172.21.15.29:0/2292609466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3f62a8780 con 0x55b3f5fe8660 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40361b1640 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5fe8660 0x55b3f63c2b20 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f4028006010 tx=0x7f4028009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2f4be73985811c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40197fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40197fa640 1 -- 172.21.15.29:0/2292609466 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028003ec0 con 0x55b3f5fe8660 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40197fa640 1 -- 172.21.15.29:0/2292609466 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4028004060 con 0x55b3f5fe8660 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f403e188740 1 -- 172.21.15.29:0/2292609466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4028003d10 con 0x55b3f5fe8660 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40197fa640 1 -- 172.21.15.29:0/2292609466 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4028004260 con 0x55b3f5fe8660 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f403e188740 10 client.17268 fetch_fsmap learned FSMap version 31 2024-09-21T02:52:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f403e188740 1 -- 172.21.15.29:0/2292609466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b3f607cb00 con 0x55b3f5fe8660 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.634+0000 7f40197fa640 1 -- 172.21.15.29:0/2292609466 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4028003d10 con 0x55b3f5fe8660 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.635+0000 7f40197fa640 1 -- 172.21.15.29:0/2292609466 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f4028005040 con 0x55b3f5fe8660 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.635+0000 7f403e188740 10 client.17268 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.635+0000 7f403e188740 10 client.17268 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.635+0000 7f403e188740 20 client.17268 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.635+0000 7f403e188740 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3f5fcfea0 0x55b3f63d4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.635+0000 7f403e188740 4 client.17268 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.635+0000 7f403e188740 1 -- 172.21.15.29:0/2292609466 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3f6331140 con 0x55b3f5fcfea0 2024-09-21T02:52:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.635+0000 7f40351af640 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3f5fcfea0 0x55b3f63d4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.636+0000 7f40197fa640 10 client.17268 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.636+0000 7f40351af640 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3f5fcfea0 0x55b3f63d4cf0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79f942f8cf548c56 server_cookie=e8a51d123d9e5df9 in_seq=0 out_seq=0 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.636+0000 7f40197fa640 1 -- 172.21.15.29:0/2292609466 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b3f6331140 con 0x55b3f5fcfea0 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.636+0000 7f40197fa640 10 client.17268 handle_command_reply: tid=0 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.636+0000 7f403e188740 1 client.17268 shutdown 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 -- 172.21.15.29:0/2292609466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5fe8660 msgr2=0x55b3f63c2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5fe8660 0x55b3f63c2b20 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f4028006010 tx=0x7f4028009b20 comp rx=0 tx=0).stop 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 -- 172.21.15.29:0/2292609466 shutdown_connections 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f6011520 0x55b3f63c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3f5fcfea0 0x55b3f63d4cf0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 --2- 172.21.15.29:0/2292609466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5fe8660 0x55b3f63c2b20 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 -- 172.21.15.29:0/2292609466 >> 172.21.15.29:0/2292609466 conn(0x55b3f6005910 msgr2=0x55b3f63ba6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:01.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f40197fa640 0 client.17268 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:01.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 -- 172.21.15.29:0/2292609466 shutdown_connections 2024-09-21T02:52:01.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 1 -- 172.21.15.29:0/2292609466 wait complete. 2024-09-21T02:52:01.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.637+0000 7f403e188740 20 client.17268 trim_cache size 0 max 16384 2024-09-21T02:52:01.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f403e188740 1 Processor -- start 2024-09-21T02:52:01.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f403e188740 1 -- start start 2024-09-21T02:52:01.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f403e188740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f403e188740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63bfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f403e188740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63c0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f403e188740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f40301b0900 con 0x55b3f6011520 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f403e188740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f40301b2440 con 0x55b3f5ff0840 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f403e188740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40301b0fd0 con 0x55b3f5fe8660 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f40361b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63c0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f40361b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63c0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38048/0 (socket says 172.21.15.29:38048) 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f40361b1640 1 -- 172.21.15.29:0/1435077117 learned_addr learned my addr 172.21.15.29:0/1435077117 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.655+0000 7f40361b1640 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f40351af640 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63bfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f40361b1640 1 -- 172.21.15.29:0/1435077117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 msgr2=0x55b3f63c48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f40361b1640 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63c48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f40361b1640 1 -- 172.21.15.29:0/1435077117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 msgr2=0x55b3f63bfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:01.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f40361b1640 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63bfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f40361b1640 1 -- 172.21.15.29:0/1435077117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3f60f3960 con 0x55b3f6011520 2024-09-21T02:52:01.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f40361b1640 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63c0250 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4028005d70 tx=0x7f402801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5522886bc7cc1125 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:01.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f401b7fe640 1 -- 172.21.15.29:0/1435077117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028005880 con 0x55b3f6011520 2024-09-21T02:52:01.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f401b7fe640 1 -- 172.21.15.29:0/1435077117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4028020030 con 0x55b3f6011520 2024-09-21T02:52:01.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.656+0000 7f401b7fe640 1 -- 172.21.15.29:0/1435077117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028004fb0 con 0x55b3f6011520 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 -- 172.21.15.29:0/1435077117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 msgr2=0x55b3f63c0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63c0250 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4028005d70 tx=0x7f402801e040 comp rx=0 tx=0).stop 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 -- 172.21.15.29:0/1435077117 shutdown_connections 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f6011520 0x55b3f63c0250 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f5ff0840 0x55b3f63bfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 --2- 172.21.15.29:0/1435077117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63c48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 -- 172.21.15.29:0/1435077117 >> 172.21.15.29:0/1435077117 conn(0x55b3f6005910 msgr2=0x55b3f63be150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 -- 172.21.15.29:0/1435077117 shutdown_connections 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.657+0000 7f403e188740 1 -- 172.21.15.29:0/1435077117 wait complete. 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 1 -- start start 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 1 Processor -- start 2024-09-21T02:52:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5ff0840 0x55b3f63c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f6011520 0x55b3f6442960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f40301b1d70 con 0x55b3f5ff0840 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f40301b16a0 con 0x55b3f6011520 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40301ae8e0 con 0x55b3f5fe8660 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f403e188740 10 client.0 _ll_register_callbacks cb 0x7f4028015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f40351af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5ff0840 0x55b3f63c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f40351af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5ff0840 0x55b3f63c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38054/0 (socket says 172.21.15.29:38054) 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f40351af640 1 -- 172.21.15.29:0/2263116747 learned_addr learned my addr 172.21.15.29:0/2263116747 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.658+0000 7f40361b1640 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f6011520 0x55b3f6442960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40361b1640 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40351af640 1 -- 172.21.15.29:0/2263116747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 msgr2=0x55b3f63c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40351af640 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40351af640 1 -- 172.21.15.29:0/2263116747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f6011520 msgr2=0x55b3f6442960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40351af640 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f6011520 0x55b3f6442960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40351af640 1 -- 172.21.15.29:0/2263116747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3f6442f70 con 0x55b3f5ff0840 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40351af640 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5ff0840 0x55b3f63c7b40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4020004700 tx=0x7f40200039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141e6991e0d0907e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40197fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40197fa640 1 -- 172.21.15.29:0/2263116747 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020028060 con 0x55b3f5ff0840 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40197fa640 1 -- 172.21.15.29:0/2263116747 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4020011650 con 0x55b3f5ff0840 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f403e188740 1 -- 172.21.15.29:0/2263116747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4028020030 con 0x55b3f5ff0840 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f40197fa640 1 -- 172.21.15.29:0/2263116747 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f402002f050 con 0x55b3f5ff0840 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f403e188740 10 client.17274 fetch_fsmap learned FSMap version 31 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.659+0000 7f403e188740 1 -- 172.21.15.29:0/2263116747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b3f64442c0 con 0x55b3f5ff0840 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f40197fa640 1 -- 172.21.15.29:0/2263116747 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4028020030 con 0x55b3f5ff0840 2024-09-21T02:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f40197fa640 1 -- 172.21.15.29:0/2263116747 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f402001d720 con 0x55b3f5ff0840 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f403e188740 10 client.17274 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f403e188740 10 client.17274 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f403e188740 20 client.17274 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f403e188740 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3f5fa5250 0x55b3f6446a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f403e188740 4 client.17274 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "294917d3", "release": true, "await": true}] 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f403e188740 1 -- 172.21.15.29:0/2263116747 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "294917d3", "release": true, "await": true}) -- 0x55b3f632cbf0 con 0x55b3f5fa5250 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.660+0000 7f40361b1640 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3f5fa5250 0x55b3f6446a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.661+0000 7f40197fa640 10 client.17274 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.661+0000 7f40361b1640 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3f5fa5250 0x55b3f6446a60 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8605ee31c6d6786 server_cookie=6f179ef3332c0570 in_seq=0 out_seq=0 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f40197fa640 1 -- 172.21.15.29:0/2263116747 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b3f632cbf0 con 0x55b3f5fa5250 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f40197fa640 10 client.17274 handle_command_reply: tid=0 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 client.17274 shutdown 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 -- 172.21.15.29:0/2263116747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5ff0840 msgr2=0x55b3f63c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5ff0840 0x55b3f63c7b40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4020004700 tx=0x7f40200039c0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 -- 172.21.15.29:0/2263116747 shutdown_connections 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3f6011520 0x55b3f6442960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3f5fa5250 0x55b3f6446a60 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f5ff0840 0x55b3f63c7b40 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 --2- 172.21.15.29:0/2263116747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f5fe8660 0x55b3f63c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f40197fa640 0 client.17274 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 -- 172.21.15.29:0/2263116747 >> 172.21.15.29:0/2263116747 conn(0x55b3f6005910 msgr2=0x55b3f63ba610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 -- 172.21.15.29:0/2263116747 shutdown_connections 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 1 -- 172.21.15.29:0/2263116747 wait complete. 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.672+0000 7f403e188740 20 client.17274 trim_cache size 0 max 16384 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 52, 2024-09-21T02:52:01.675 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "294917d3": { 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 52, 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 137.8, 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:52:01.676 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:52:01.677 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:52:01.677 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:52:01.677 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:52:01.677 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:52:01.677 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:52:01.677 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 -- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f401c0b6510 msgr2=0x7f401c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f401c0b6510 0x7f401c0b8a20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55b3f63b6190 tx=0x7f402c02a410 comp rx=0 tx=0).stop 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 -- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 msgr2=0x7f403019a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 0x7f403019a320 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f402c009230 tx=0x7f402c004270 comp rx=0 tx=0).stop 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 -- 172.21.15.29:0/3611633752 shutdown_connections 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f401c0b6510 0x7f401c0b8a20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4030108cf0 0x7f403019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40301083a0 0x7f403019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 --2- 172.21.15.29:0/3611633752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030102a60 0x7f403019a320 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 -- 172.21.15.29:0/3611633752 >> 172.21.15.29:0/3611633752 conn(0x7f4030079730 msgr2=0x7f4030101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 -- 172.21.15.29:0/3611633752 shutdown_connections 2024-09-21T02:52:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:52:01.673+0000 7f4037c14640 1 -- 172.21.15.29:0/3611633752 wait complete. 2024-09-21T02:52:01.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:01 smithi066 ceph-mon[33436]: pgmap v776: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 KiB/s wr, 20 op/s 2024-09-21T02:52:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:01 smithi190 ceph-mon[33871]: pgmap v776: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 KiB/s wr, 20 op/s 2024-09-21T02:52:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:01 smithi029 ceph-mon[31006]: pgmap v776: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 KiB/s wr, 20 op/s 2024-09-21T02:52:02.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 257851 2.12 MB/sec execute 579 sec latency 17471.549 ms 2024-09-21T02:52:02.437 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '294917d3', total seconds elapsed: 21.3 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 -- 172.21.15.29:0/1258305146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c1025a0 msgr2=0x7fcb5c102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 --2- 172.21.15.29:0/1258305146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c1025a0 0x7fcb5c102a10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009900 tx=0x7fcb4c02a410 comp rx=0 tx=0).stop 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 -- 172.21.15.29:0/1258305146 shutdown_connections 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 --2- 172.21.15.29:0/1258305146 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb5c103fc0 0x7fcb5c10a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 --2- 172.21.15.29:0/1258305146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb5c103620 0x7fcb5c103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 --2- 172.21.15.29:0/1258305146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c1025a0 0x7fcb5c102a10 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 -- 172.21.15.29:0/1258305146 >> 172.21.15.29:0/1258305146 conn(0x7fcb5c0fdee0 msgr2=0x7fcb5c100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 -- 172.21.15.29:0/1258305146 shutdown_connections 2024-09-21T02:52:02.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.525+0000 7fcb61182640 1 -- 172.21.15.29:0/1258305146 wait complete. 2024-09-21T02:52:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.526+0000 7fcb61182640 1 Processor -- start 2024-09-21T02:52:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.526+0000 7fcb61182640 1 -- start start 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.526+0000 7fcb61182640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb5c1025a0 0x7fcb5c199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb61182640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c103620 0x7fcb5c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb61182640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb5c103fc0 0x7fcb5c19b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb61182640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c10e080 con 0x7fcb5c103620 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb61182640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c10df00 con 0x7fcb5c103fc0 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb61182640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c10e200 con 0x7fcb5c1025a0 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c103620 0x7fcb5c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb5c1025a0 0x7fcb5c199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c103620 0x7fcb5c19a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38074/0 (socket says 172.21.15.29:38074) 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb5c1025a0 0x7fcb5c199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40024/0 (socket says 172.21.15.29:40024) 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 -- 172.21.15.29:0/3617487357 learned_addr learned my addr 172.21.15.29:0/3617487357 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5b59e640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb5c103fc0 0x7fcb5c19b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 -- 172.21.15.29:0/3617487357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb5c1025a0 msgr2=0x7fcb5c199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb5c1025a0 0x7fcb5c199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 -- 172.21.15.29:0/3617487357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb5c103fc0 msgr2=0x7fcb5c19b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb5c103fc0 0x7fcb5c19b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 -- 172.21.15.29:0/3617487357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5c19b890 con 0x7fcb5c103620 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb5c1025a0 0x7fcb5c199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c103620 0x7fcb5c19a550 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000e9f0 tx=0x7fcb5000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.527+0000 7fcb43fff640 1 -- 172.21.15.29:0/3617487357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5000cd80 con 0x7fcb5c103620 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.528+0000 7fcb61182640 1 -- 172.21.15.29:0/3617487357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5c19bb80 con 0x7fcb5c103620 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.528+0000 7fcb43fff640 1 -- 172.21.15.29:0/3617487357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb500108c0 con 0x7fcb5c103620 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.528+0000 7fcb43fff640 1 -- 172.21.15.29:0/3617487357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb500187c0 con 0x7fcb5c103620 2024-09-21T02:52:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.528+0000 7fcb61182640 1 -- 172.21.15.29:0/3617487357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5c1a2300 con 0x7fcb5c103620 2024-09-21T02:52:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.530+0000 7fcb43fff640 1 -- 172.21.15.29:0/3617487357 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fcb50018980 con 0x7fcb5c103620 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.530+0000 7fcb681cd740 1 Processor -- start 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb681cd740 1 -- start start 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a868d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a87420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb681cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562ec79f46f0 con 0x562ec76b12f0 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb681cd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562ec79e3b20 con 0x562ec76da1b0 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb681cd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562ec79e7930 con 0x562ec76b94d0 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a868d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a868d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38080/0 (socket says 172.21.15.29:38080) 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/3396404311 learned_addr learned my addr 172.21.15.29:0/3396404311 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a87420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5b59e640 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb43fff640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb440b6510 0x7fcb440b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/3396404311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 msgr2=0x562ec7a87420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a87420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/3396404311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 msgr2=0x562ec7a86260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/3396404311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ec79a9530 con 0x562ec76b12f0 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.531+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a87420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.532+0000 7fcb5b59e640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb440b6510 0x7fcb440b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.532+0000 7fcb43fff640 1 -- 172.21.15.29:0/3617487357 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fcb500e7b20 con 0x7fcb5c103620 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.532+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a868d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c02a930 tx=0x7fcb4c031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e926f91256bbf67e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.532+0000 7fcb417fa640 1 -- 172.21.15.29:0/3396404311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c035030 con 0x562ec76b12f0 2024-09-21T02:52:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.532+0000 7fcb417fa640 1 -- 172.21.15.29:0/3396404311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb4c040070 con 0x562ec76b12f0 2024-09-21T02:52:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.532+0000 7fcb417fa640 1 -- 172.21.15.29:0/3396404311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c003cd0 con 0x562ec76b12f0 2024-09-21T02:52:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.532+0000 7fcb5b59e640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb440b6510 0x7fcb440b8a20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcb54005fd0 tx=0x7fcb54004a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 -- 172.21.15.29:0/3396404311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 msgr2=0x562ec7a868d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a868d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c02a930 tx=0x7fcb4c031a30 comp rx=0 tx=0).stop 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 -- 172.21.15.29:0/3396404311 shutdown_connections 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a87420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a868d0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 --2- 172.21.15.29:0/3396404311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 -- 172.21.15.29:0/3396404311 >> 172.21.15.29:0/3396404311 conn(0x562ec76ce300 msgr2=0x562ec7a7d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 -- 172.21.15.29:0/3396404311 shutdown_connections 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.533+0000 7fcb681cd740 1 -- 172.21.15.29:0/3396404311 wait complete. 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 1 -- start start 2024-09-21T02:52:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 1 Processor -- start 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b94d0 0x562ec7a90000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 0x562ec7a90b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562ec791bab0 con 0x562ec76b12f0 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562ec7989730 con 0x562ec76b94d0 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562ec7a87ee0 con 0x562ec76da1b0 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb681cd740 10 client.0 _ll_register_callbacks cb 0x562ec79dd6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38082/0 (socket says 172.21.15.29:38082) 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/2971419452 learned_addr learned my addr 172.21.15.29:0/2971419452 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb5b59e640 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 0x562ec7a90b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.534+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b94d0 0x562ec7a90000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/2971419452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 msgr2=0x562ec7a90b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 0x562ec7a90b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/2971419452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b94d0 msgr2=0x562ec7a90000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b94d0 0x562ec7a90000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/2971419452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ec7745790 con 0x562ec76b12f0 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8f940 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c005760 tx=0x7fcb4c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22aa028504acfbf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2971419452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c035070 con 0x562ec76b12f0 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2971419452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fcb4c03c9d0 con 0x562ec76b12f0 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb681cd740 1 -- 172.21.15.29:0/2971419452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb4c040070 con 0x562ec76b12f0 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2971419452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcb4c03cbd0 con 0x562ec76b12f0 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb681cd740 10 client.17286 fetch_fsmap learned FSMap version 31 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb681cd740 1 -- 172.21.15.29:0/2971419452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562ec77be620 con 0x562ec76b12f0 2024-09-21T02:52:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.535+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2971419452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb4c040070 con 0x562ec76b12f0 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2971419452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fcb4c044410 con 0x562ec76b12f0 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb681cd740 10 client.17286 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb681cd740 10 client.17286 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb681cd740 20 client.17286 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ec7698d80 0x562ec7a9cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb681cd740 4 client.17286 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb681cd740 1 -- 172.21.15.29:0/2971419452 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ec79f6640 con 0x562ec7698d80 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ec7698d80 0x562ec7a9cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ec7698d80 0x562ec7a9cc70 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48b884a0f1360abb server_cookie=2acd69ebfc95116c in_seq=0 out_seq=0 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.536+0000 7fcb3eff5640 10 client.17286 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.537+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2971419452 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562ec79f6640 con 0x562ec7698d80 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.537+0000 7fcb3eff5640 10 client.17286 handle_command_reply: tid=0 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 client.17286 shutdown 2024-09-21T02:52:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 -- 172.21.15.29:0/2971419452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 msgr2=0x562ec7a8f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8f940 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c005760 tx=0x7fcb4c003820 comp rx=0 tx=0).stop 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 -- 172.21.15.29:0/2971419452 shutdown_connections 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 0x562ec7a90b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b94d0 0x562ec7a90000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ec7698d80 0x562ec7a9cc70 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2971419452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8f940 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb3eff5640 0 client.17286 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 -- 172.21.15.29:0/2971419452 >> 172.21.15.29:0/2971419452 conn(0x562ec76ce300 msgr2=0x562ec7a7bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 -- 172.21.15.29:0/2971419452 shutdown_connections 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 1 -- 172.21.15.29:0/2971419452 wait complete. 2024-09-21T02:52:02.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.538+0000 7fcb681cd740 20 client.17286 trim_cache size 0 max 16384 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.555+0000 7fcb681cd740 1 Processor -- start 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.555+0000 7fcb681cd740 1 -- start start 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.555+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.555+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a97a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.555+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb681cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c1aa390 con 0x562ec76b12f0 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb681cd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c1abed0 con 0x562ec76da1b0 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb681cd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c1aaa60 con 0x562ec76b94d0 2024-09-21T02:52:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38090/0 (socket says 172.21.15.29:38090) 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/2348038278 learned_addr learned my addr 172.21.15.29:0/2348038278 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a97a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5b59e640 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/2348038278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 msgr2=0x562ec7a97a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a97a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/2348038278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 msgr2=0x562ec7a86200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 -- 172.21.15.29:0/2348038278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ec7a90ee0 con 0x562ec76b12f0 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5b59e640 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8eb40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c044830 tx=0x7fcb4c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bce9ae182fbf0ec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb40ff9640 1 -- 172.21.15.29:0/2348038278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c035040 con 0x562ec76b12f0 2024-09-21T02:52:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb40ff9640 1 -- 172.21.15.29:0/2348038278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb4c040070 con 0x562ec76b12f0 2024-09-21T02:52:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.556+0000 7fcb40ff9640 1 -- 172.21.15.29:0/2348038278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c04d670 con 0x562ec76b12f0 2024-09-21T02:52:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 -- 172.21.15.29:0/2348038278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 msgr2=0x562ec7a8eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8eb40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c044830 tx=0x7fcb4c005fd0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 -- 172.21.15.29:0/2348038278 shutdown_connections 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76da1b0 0x562ec7a86200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76b94d0 0x562ec7a97a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2348038278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b12f0 0x562ec7a8eb40 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 -- 172.21.15.29:0/2348038278 >> 172.21.15.29:0/2348038278 conn(0x562ec76ce300 msgr2=0x562ec7a7c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 -- 172.21.15.29:0/2348038278 shutdown_connections 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.557+0000 7fcb681cd740 1 -- 172.21.15.29:0/2348038278 wait complete. 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 1 -- start start 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 1 Processor -- start 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b12f0 0x562ec7a976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b94d0 0x562ec7a963b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 0x562ec7a968f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c1ab800 con 0x562ec76b94d0 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c1ab130 con 0x562ec76b12f0 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c1a8370 con 0x562ec76da1b0 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb681cd740 10 client.0 _ll_register_callbacks cb 0x7fcb54007540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b94d0 0x562ec7a963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b94d0 0x562ec7a963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38094/0 (socket says 172.21.15.29:38094) 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5a59c640 1 -- 172.21.15.29:0/2235163409 learned_addr learned my addr 172.21.15.29:0/2235163409 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5b59e640 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 0x562ec7a968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b12f0 0x562ec7a976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5a59c640 1 -- 172.21.15.29:0/2235163409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 msgr2=0x562ec7a968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 0x562ec7a968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5a59c640 1 -- 172.21.15.29:0/2235163409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b12f0 msgr2=0x562ec7a976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b12f0 0x562ec7a976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.558+0000 7fcb5a59c640 1 -- 172.21.15.29:0/2235163409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ec7a97080 con 0x562ec76b94d0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb5a59c640 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b94d0 0x562ec7a963b0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fcb500aaba0 tx=0x7fcb500e7d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313501f55953e73d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2235163409 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb500a90a0 con 0x562ec76b94d0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2235163409 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fcb500a9ba0 con 0x562ec76b94d0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 1 -- 172.21.15.29:0/2235163409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb4c040070 con 0x562ec76b94d0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2235163409 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcb500ad440 con 0x562ec76b94d0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 10 client.17292 fetch_fsmap learned FSMap version 31 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 1 -- 172.21.15.29:0/2235163409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562ec7b07690 con 0x562ec76b94d0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2235163409 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb4c040070 con 0x562ec76b94d0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2235163409 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fcb500a9280 con 0x562ec76b94d0 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 10 client.17292 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:52:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 10 client.17292 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 20 client.17292 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ec766e1c0 0x562ec7b0a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 4 client.17292 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:52:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.559+0000 7fcb681cd740 1 -- 172.21.15.29:0/2235163409 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562ec79f20f0 con 0x562ec766e1c0 2024-09-21T02:52:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.560+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ec766e1c0 0x562ec7b0a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.560+0000 7fcb3eff5640 10 client.17292 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.560+0000 7fcb5ad9d640 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ec766e1c0 0x562ec7b0a080 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15c2fd4e11fc7a85 server_cookie=6432f59c8a6bc3c3 in_seq=0 out_seq=0 2024-09-21T02:52:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb3eff5640 1 -- 172.21.15.29:0/2235163409 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562ec79f20f0 con 0x562ec766e1c0 2024-09-21T02:52:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb3eff5640 10 client.17292 handle_command_reply: tid=0 2024-09-21T02:52:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 client.17292 shutdown 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 -- 172.21.15.29:0/2235163409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b94d0 msgr2=0x562ec7a963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b94d0 0x562ec7a963b0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fcb500aaba0 tx=0x7fcb500e7d20 comp rx=0 tx=0).stop 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 -- 172.21.15.29:0/2235163409 shutdown_connections 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562ec76da1b0 0x562ec7a968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562ec766e1c0 0x562ec7b0a080 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562ec76b94d0 0x562ec7a963b0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 --2- 172.21.15.29:0/2235163409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562ec76b12f0 0x562ec7a976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb3eff5640 0 client.17292 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 -- 172.21.15.29:0/2235163409 >> 172.21.15.29:0/2235163409 conn(0x562ec76ce300 msgr2=0x562ec7a86200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 -- 172.21.15.29:0/2235163409 shutdown_connections 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.596+0000 7fcb681cd740 1 -- 172.21.15.29:0/2235163409 wait complete. 2024-09-21T02:52:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb681cd740 20 client.17292 trim_cache size 0 max 16384 2024-09-21T02:52:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:52:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:52:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:52:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 -- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb440b6510 msgr2=0x7fcb440b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb440b6510 0x7fcb440b8a20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcb54005fd0 tx=0x7fcb54004a90 comp rx=0 tx=0).stop 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 -- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c103620 msgr2=0x7fcb5c19a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c103620 0x7fcb5c19a550 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000e9f0 tx=0x7fcb5000eeb0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 -- 172.21.15.29:0/3617487357 shutdown_connections 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb440b6510 0x7fcb440b8a20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb5c103fc0 0x7fcb5c19b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb5c103620 0x7fcb5c19a550 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 --2- 172.21.15.29:0/3617487357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb5c1025a0 0x7fcb5c199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 -- 172.21.15.29:0/3617487357 >> 172.21.15.29:0/3617487357 conn(0x7fcb5c0fdee0 msgr2=0x7fcb5c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:02.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 -- 172.21.15.29:0/3617487357 shutdown_connections 2024-09-21T02:52:02.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:02.597+0000 7fcb61182640 1 -- 172.21.15.29:0/3617487357 wait complete. 2024-09-21T02:52:03.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 258709 2.13 MB/sec execute 580 sec latency 15.624 ms 2024-09-21T02:52:03.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:03 smithi066 ceph-mon[33436]: pgmap v777: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2024-09-21T02:52:03.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:03 smithi029 ceph-mon[31006]: pgmap v777: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2024-09-21T02:52:03.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:03 smithi190 ceph-mon[33871]: pgmap v777: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2024-09-21T02:52:04.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 259200 2.13 MB/sec execute 581 sec latency 16.199 ms 2024-09-21T02:52:05.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 259436 2.13 MB/sec execute 582 sec latency 19.868 ms 2024-09-21T02:52:05.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:05 smithi066 ceph-mon[33436]: pgmap v778: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 99 KiB/s wr, 30 op/s 2024-09-21T02:52:05.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:05 smithi190 ceph-mon[33871]: pgmap v778: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 99 KiB/s wr, 30 op/s 2024-09-21T02:52:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:05 smithi029 ceph-mon[31006]: pgmap v778: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 99 KiB/s wr, 30 op/s 2024-09-21T02:52:06.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 259969 2.12 MB/sec execute 583 sec latency 17.780 ms 2024-09-21T02:52:07.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 260616 2.13 MB/sec execute 584 sec latency 14.593 ms 2024-09-21T02:52:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:07 smithi066 ceph-mon[33436]: pgmap v779: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 457 KiB/s wr, 42 op/s 2024-09-21T02:52:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:07 smithi190 ceph-mon[33871]: pgmap v779: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 457 KiB/s wr, 42 op/s 2024-09-21T02:52:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:07 smithi029 ceph-mon[31006]: pgmap v779: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 457 KiB/s wr, 42 op/s 2024-09-21T02:52:08.105 INFO:tasks.workunit.client.0.smithi029.stdout: 1 261218 2.13 MB/sec execute 585 sec latency 110.702 ms 2024-09-21T02:52:09.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 261961 2.13 MB/sec execute 586 sec latency 18.592 ms 2024-09-21T02:52:09.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:09 smithi066 ceph-mon[33436]: pgmap v780: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 585 KiB/s wr, 55 op/s 2024-09-21T02:52:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:09 smithi190 ceph-mon[33871]: pgmap v780: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 585 KiB/s wr, 55 op/s 2024-09-21T02:52:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:09 smithi029 ceph-mon[31006]: pgmap v780: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 585 KiB/s wr, 55 op/s 2024-09-21T02:52:10.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 262594 2.14 MB/sec execute 587 sec latency 15.674 ms 2024-09-21T02:52:11.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 262908 2.14 MB/sec execute 588 sec latency 14.500 ms 2024-09-21T02:52:11.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:11 smithi066 ceph-mon[33436]: pgmap v781: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 780 KiB/s wr, 66 op/s 2024-09-21T02:52:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:11 smithi190 ceph-mon[33871]: pgmap v781: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 780 KiB/s wr, 66 op/s 2024-09-21T02:52:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:11 smithi029 ceph-mon[31006]: pgmap v781: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 780 KiB/s wr, 66 op/s 2024-09-21T02:52:12.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 263337 2.13 MB/sec execute 589 sec latency 17.175 ms 2024-09-21T02:52:13.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 263985 2.14 MB/sec execute 590 sec latency 19.290 ms 2024-09-21T02:52:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:13 smithi066 ceph-mon[33436]: pgmap v782: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-09-21T02:52:13.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:13 smithi190 ceph-mon[33871]: pgmap v782: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-09-21T02:52:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:13 smithi029 ceph-mon[31006]: pgmap v782: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-09-21T02:52:14.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 264605 2.14 MB/sec execute 591 sec latency 20.007 ms 2024-09-21T02:52:15.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 265269 2.14 MB/sec execute 592 sec latency 15.957 ms 2024-09-21T02:52:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 -- 172.21.15.29:0/224976556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 msgr2=0x7f43c01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 --2- 172.21.15.29:0/224976556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 0x7f43c01021c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009900 tx=0x7f43b002a410 comp rx=0 tx=0).stop 2024-09-21T02:52:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 -- 172.21.15.29:0/224976556 shutdown_connections 2024-09-21T02:52:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 --2- 172.21.15.29:0/224976556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f43c0103770 0x7f43c010a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 --2- 172.21.15.29:0/224976556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43c0102dd0 0x7f43c01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 --2- 172.21.15.29:0/224976556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 0x7f43c01021c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 -- 172.21.15.29:0/224976556 >> 172.21.15.29:0/224976556 conn(0x7f43c00fd690 msgr2=0x7f43c00ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 -- 172.21.15.29:0/224976556 shutdown_connections 2024-09-21T02:52:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.700+0000 7f43c629b640 1 -- 172.21.15.29:0/224976556 wait complete. 2024-09-21T02:52:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.701+0000 7f43c629b640 1 Processor -- start 2024-09-21T02:52:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.701+0000 7f43c629b640 1 -- start start 2024-09-21T02:52:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43c629b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 0x7f43c019c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43c629b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43c0102dd0 0x7f43c019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43c629b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f43c0103770 0x7f43c019d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43c629b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f43c010cfc0 con 0x7f43c0101d50 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43c629b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f43c010ce40 con 0x7f43c0103770 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43c629b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43c010d140 con 0x7f43c0102dd0 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 0x7f43c019c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 0x7f43c019c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34368/0 (socket says 172.21.15.29:34368) 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 -- 172.21.15.29:0/3230318840 learned_addr learned my addr 172.21.15.29:0/3230318840 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43c0102dd0 0x7f43c019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43c4838640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f43c0103770 0x7f43c019d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 -- 172.21.15.29:0/3230318840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43c0102dd0 msgr2=0x7f43c019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43c0102dd0 0x7f43c019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 -- 172.21.15.29:0/3230318840 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f43c0103770 msgr2=0x7f43c019d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f43c0103770 0x7f43c019d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 -- 172.21.15.29:0/3230318840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c019db20 con 0x7f43c0101d50 2024-09-21T02:52:15.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bffff640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 0x7f43c019c0e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009230 tx=0x7f43b0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:15.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bd7fa640 1 -- 172.21.15.29:0/3230318840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b003f030 con 0x7f43c0101d50 2024-09-21T02:52:15.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bd7fa640 1 -- 172.21.15.29:0/3230318840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43b0004440 con 0x7f43c0101d50 2024-09-21T02:52:15.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43bd7fa640 1 -- 172.21.15.29:0/3230318840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0035070 con 0x7f43c0101d50 2024-09-21T02:52:15.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.702+0000 7f43c629b640 1 -- 172.21.15.29:0/3230318840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c019ddb0 con 0x7f43c0101d50 2024-09-21T02:52:15.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.703+0000 7f43c629b640 1 -- 172.21.15.29:0/3230318840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c019e0c0 con 0x7f43c0101d50 2024-09-21T02:52:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.705+0000 7f43cd213740 1 Processor -- start 2024-09-21T02:52:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.705+0000 7f43bd7fa640 1 -- 172.21.15.29:0/3230318840 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f43b0042030 con 0x7f43c0101d50 2024-09-21T02:52:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.705+0000 7f43cd213740 1 -- start start 2024-09-21T02:52:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43cd213740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275221b0 0x5557278c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43cd213740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557274f92f0 0x5557278ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43cd213740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43cd213740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55572783c6f0 con 0x5557275221b0 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43cd213740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55572782bb20 con 0x5557274f92f0 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43cd213740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55572782f930 con 0x5557275014d0 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43c4838640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275221b0 0x5557278c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43c4838640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275221b0 0x5557278c9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34382/0 (socket says 172.21.15.29:34382) 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43c4838640 1 -- 172.21.15.29:0/3524063958 learned_addr learned my addr 172.21.15.29:0/3524063958 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43bffff640 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557274f92f0 0x5557278ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43c4838640 1 -- 172.21.15.29:0/3524063958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 msgr2=0x5557278cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43c4838640 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43c4838640 1 -- 172.21.15.29:0/3524063958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557274f92f0 msgr2=0x5557278ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43c4838640 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557274f92f0 0x5557278ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43c4838640 1 -- 172.21.15.29:0/3524063958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557277f1530 con 0x5557275221b0 2024-09-21T02:52:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.706+0000 7f43bd7fa640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43a80b6510 0x7f43a80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.707+0000 7f43c4838640 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275221b0 0x5557278c9e50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f43b800b570 tx=0x7f43b800bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d8ee8486c47291 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.707+0000 7f43bffff640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43a80b6510 0x7f43a80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.707+0000 7f43bd7fa640 1 -- 172.21.15.29:0/3230318840 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f43b0107b20 con 0x7f43c0101d50 2024-09-21T02:52:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.707+0000 7f43a67fc640 1 -- 172.21.15.29:0/3524063958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b80076f0 con 0x5557275221b0 2024-09-21T02:52:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.707+0000 7f43a67fc640 1 -- 172.21.15.29:0/3524063958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43b8007d20 con 0x5557275221b0 2024-09-21T02:52:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.707+0000 7f43a67fc640 1 -- 172.21.15.29:0/3524063958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8018da0 con 0x5557275221b0 2024-09-21T02:52:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 -- 172.21.15.29:0/3524063958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275221b0 msgr2=0x5557278c9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43bffff640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43a80b6510 0x7f43a80b8a20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f43b003f000 tx=0x7f43b010b600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275221b0 0x5557278c9e50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f43b800b570 tx=0x7f43b800bb30 comp rx=0 tx=0).stop 2024-09-21T02:52:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 -- 172.21.15.29:0/3524063958 shutdown_connections 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557274f92f0 0x5557278ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 --2- 172.21.15.29:0/3524063958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275221b0 0x5557278c9e50 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 -- 172.21.15.29:0/3524063958 >> 172.21.15.29:0/3524063958 conn(0x555727516300 msgr2=0x5557278c58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 -- 172.21.15.29:0/3524063958 shutdown_connections 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 -- 172.21.15.29:0/3524063958 wait complete. 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.708+0000 7f43cd213740 1 -- start start 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43cd213740 1 Processor -- start 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43cd213740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278d35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43cd213740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278d3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43cd213740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43cd213740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555727763ab0 con 0x5557274f92f0 2024-09-21T02:52:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43cd213740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5557277d1730 con 0x5557275221b0 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43cd213740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5557278cbb30 con 0x5557275014d0 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43cd213740 10 client.0 _ll_register_callbacks cb 0x5557278256b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43c4838640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278d35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43c4838640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278d35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34394/0 (socket says 172.21.15.29:34394) 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43bf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278d3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43c4838640 1 -- 172.21.15.29:0/185573103 learned_addr learned my addr 172.21.15.29:0/185573103 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.709+0000 7f43c4838640 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43bf7fe640 1 -- 172.21.15.29:0/185573103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 msgr2=0x5557278d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43bf7fe640 1 -- 172.21.15.29:0/185573103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 msgr2=0x5557278d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278d35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43bf7fe640 1 -- 172.21.15.29:0/185573103 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55572758d790 con 0x5557275014d0 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278d3c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43b40082a0 tx=0x7f43b400a950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82acdd7a3b4fc4d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f439f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f439f7fe640 1 -- 172.21.15.29:0/185573103 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b401c070 con 0x5557275014d0 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43cd213740 1 -- 172.21.15.29:0/185573103 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43b8007d20 con 0x5557275014d0 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f439f7fe640 1 -- 172.21.15.29:0/185573103 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f43b4005220 con 0x5557275014d0 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f439f7fe640 1 -- 172.21.15.29:0/185573103 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f43b4005480 con 0x5557275014d0 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43cd213740 10 client.26833 fetch_fsmap learned FSMap version 31 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.710+0000 7f43cd213740 1 -- 172.21.15.29:0/185573103 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555727606620 con 0x5557275014d0 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f439f7fe640 1 -- 172.21.15.29:0/185573103 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43b8007d20 con 0x5557275014d0 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f439f7fe640 1 -- 172.21.15.29:0/185573103 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f43b401f020 con 0x5557275014d0 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f43cd213740 10 client.26833 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f43cd213740 10 client.26833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f43cd213740 20 client.26833 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f43cd213740 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5557274e0d80 0x5557278e0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f43cd213740 4 client.26833 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:52:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f43cd213740 1 -- 172.21.15.29:0/185573103 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55572783e640 con 0x5557274e0d80 2024-09-21T02:52:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.711+0000 7f43c4838640 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5557274e0d80 0x5557278e0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.712+0000 7f439f7fe640 10 client.26833 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.712+0000 7f43c4838640 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5557274e0d80 0x5557278e0db0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc4142be70082892 server_cookie=ec980821301f015c in_seq=0 out_seq=0 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f439f7fe640 1 -- 172.21.15.29:0/185573103 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55572783e640 con 0x5557274e0d80 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f439f7fe640 10 client.26833 handle_command_reply: tid=0 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 client.26833 shutdown 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 -- 172.21.15.29:0/185573103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 msgr2=0x5557278d3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278d3c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43b40082a0 tx=0x7f43b400a950 comp rx=0 tx=0).stop 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 -- 172.21.15.29:0/185573103 shutdown_connections 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275014d0 0x5557278d3c90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5557274e0d80 0x5557278e0db0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 --2- 172.21.15.29:0/185573103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278d35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f439f7fe640 0 client.26833 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 -- 172.21.15.29:0/185573103 >> 172.21.15.29:0/185573103 conn(0x555727516300 msgr2=0x5557278caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 -- 172.21.15.29:0/185573103 shutdown_connections 2024-09-21T02:52:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 1 -- 172.21.15.29:0/185573103 wait complete. 2024-09-21T02:52:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.713+0000 7f43cd213740 20 client.26833 trim_cache size 0 max 16384 2024-09-21T02:52:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.729+0000 7f43cd213740 1 Processor -- start 2024-09-21T02:52:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.729+0000 7f43cd213740 1 -- start start 2024-09-21T02:52:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43cd213740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557274f92f0 0x5557278d3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43cd213740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275014d0 0x5557278cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43cd213740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278cc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43cd213740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f43c01ae820 con 0x5557275014d0 2024-09-21T02:52:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43cd213740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f43c01b0360 con 0x5557275221b0 2024-09-21T02:52:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43cd213740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43c01aeef0 con 0x5557274f92f0 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275014d0 0x5557278cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275014d0 0x5557278cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34410/0 (socket says 172.21.15.29:34410) 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 -- 172.21.15.29:0/3280156215 learned_addr learned my addr 172.21.15.29:0/3280156215 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43c4838640 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557274f92f0 0x5557278d3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43c4838640 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 -- 172.21.15.29:0/3280156215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557274f92f0 msgr2=0x5557278d3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557274f92f0 0x5557278d3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 -- 172.21.15.29:0/3280156215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 msgr2=0x5557278cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 -- 172.21.15.29:0/3280156215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557278e1110 con 0x5557275014d0 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43c4838640 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275014d0 0x5557278cbf50 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f43b4004990 tx=0x7f43b4005f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de46a446e67f4603 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43a5ffb640 1 -- 172.21.15.29:0/3280156215 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b402f070 con 0x5557275014d0 2024-09-21T02:52:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.730+0000 7f43a5ffb640 1 -- 172.21.15.29:0/3280156215 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43b400dc60 con 0x5557275014d0 2024-09-21T02:52:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.731+0000 7f43a5ffb640 1 -- 172.21.15.29:0/3280156215 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b401c070 con 0x5557275014d0 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.731+0000 7f43cd213740 1 -- 172.21.15.29:0/3280156215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275014d0 msgr2=0x5557278cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.731+0000 7f43cd213740 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275014d0 0x5557278cbf50 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f43b4004990 tx=0x7f43b4005f50 comp rx=0 tx=0).stop 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.731+0000 7f43cd213740 1 -- 172.21.15.29:0/3280156215 shutdown_connections 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.731+0000 7f43cd213740 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275221b0 0x5557278cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.731+0000 7f43cd213740 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557275014d0 0x5557278cbf50 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.731+0000 7f43cd213740 1 --2- 172.21.15.29:0/3280156215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557274f92f0 0x5557278d3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.731+0000 7f43cd213740 1 -- 172.21.15.29:0/3280156215 >> 172.21.15.29:0/3280156215 conn(0x555727516300 msgr2=0x5557278d35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 -- 172.21.15.29:0/3280156215 shutdown_connections 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 -- 172.21.15.29:0/3280156215 wait complete. 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 -- start start 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 Processor -- start 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278daf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275014d0 0x5557278d9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275221b0 0x5557278da150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f43c01afc90 con 0x5557274f92f0 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f43c01af5c0 con 0x5557275014d0 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43c01ac800 con 0x5557275221b0 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.732+0000 7f43cd213740 10 client.0 _ll_register_callbacks cb 0x7f43b40040f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278daf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278daf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34412/0 (socket says 172.21.15.29:34412) 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 -- 172.21.15.29:0/3045775453 learned_addr learned my addr 172.21.15.29:0/3045775453 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43bf7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275014d0 0x5557278d9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275221b0 0x5557278da150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 -- 172.21.15.29:0/3045775453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275221b0 msgr2=0x5557278da150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275221b0 0x5557278da150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 -- 172.21.15.29:0/3045775453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275014d0 msgr2=0x5557278d9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275014d0 0x5557278d9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 -- 172.21.15.29:0/3045775453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557278da690 con 0x5557274f92f0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275014d0 0x5557278d9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43c4838640 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278daf50 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f43b8000f80 tx=0x7f43b800bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b0a66248364404 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f439f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f439f7fe640 1 -- 172.21.15.29:0/3045775453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8009260 con 0x5557274f92f0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f439f7fe640 1 -- 172.21.15.29:0/3045775453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f43b8018950 con 0x5557274f92f0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.733+0000 7f43cd213740 1 -- 172.21.15.29:0/3045775453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43b400dc60 con 0x5557274f92f0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f439f7fe640 1 -- 172.21.15.29:0/3045775453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f43b8018b50 con 0x5557274f92f0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43cd213740 10 client.17310 fetch_fsmap learned FSMap version 31 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43cd213740 1 -- 172.21.15.29:0/3045775453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5557278d9500 con 0x5557274f92f0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f439f7fe640 1 -- 172.21.15.29:0/3045775453 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43b400dc60 con 0x5557274f92f0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f439f7fe640 1 -- 172.21.15.29:0/3045775453 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f43b8023040 con 0x5557274f92f0 2024-09-21T02:52:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43cd213740 10 client.17310 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T02:52:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43cd213740 10 client.17310 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43cd213740 20 client.17310 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43cd213740 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5557274b61c0 0x55572794dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43cd213740 4 client.17310 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:52:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43cd213740 1 -- 172.21.15.29:0/3045775453 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55572783a0f0 con 0x5557274b61c0 2024-09-21T02:52:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.734+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5557274b61c0 0x55572794dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.735+0000 7f439f7fe640 10 client.17310 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.735+0000 7f43bf7fe640 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5557274b61c0 0x55572794dc00 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cbee1877c95e6ce server_cookie=28a3cd98918f2491 in_seq=0 out_seq=0 2024-09-21T02:52:15.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:15 smithi066 ceph-mon[33436]: pgmap v783: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-09-21T02:52:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.783+0000 7f439f7fe640 1 -- 172.21.15.29:0/3045775453 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55572783a0f0 con 0x5557274b61c0 2024-09-21T02:52:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.783+0000 7f439f7fe640 10 client.17310 handle_command_reply: tid=0 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.783+0000 7f43cd213740 1 client.17310 shutdown 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 -- 172.21.15.29:0/3045775453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 msgr2=0x5557278daf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278daf50 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f43b8000f80 tx=0x7f43b800bb30 comp rx=0 tx=0).stop 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 -- 172.21.15.29:0/3045775453 shutdown_connections 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557275221b0 0x5557278da150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5557275014d0 0x5557278d9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5557274b61c0 0x55572794dc00 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 --2- 172.21.15.29:0/3045775453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557274f92f0 0x5557278daf50 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f439f7fe640 0 client.17310 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 -- 172.21.15.29:0/3045775453 >> 172.21.15.29:0/3045775453 conn(0x555727516300 msgr2=0x5557278c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 -- 172.21.15.29:0/3045775453 shutdown_connections 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 1 -- 172.21.15.29:0/3045775453 wait complete. 2024-09-21T02:52:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.784+0000 7f43cd213740 20 client.17310 trim_cache size 0 max 16384 2024-09-21T02:52:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:52:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:52:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:52:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 -- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43a80b6510 msgr2=0x7f43a80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43a80b6510 0x7f43a80b8a20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f43b003f000 tx=0x7f43b010b600 comp rx=0 tx=0).stop 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 -- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 msgr2=0x7f43c019c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 0x7f43c019c0e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009230 tx=0x7f43b0004040 comp rx=0 tx=0).stop 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 -- 172.21.15.29:0/3230318840 shutdown_connections 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f43a80b6510 0x7f43a80b8a20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f43c0103770 0x7f43c019d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43c0102dd0 0x7f43c019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 --2- 172.21.15.29:0/3230318840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0101d50 0x7f43c019c0e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 -- 172.21.15.29:0/3230318840 >> 172.21.15.29:0/3230318840 conn(0x7f43c00fd690 msgr2=0x7f43c01926a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 -- 172.21.15.29:0/3230318840 shutdown_connections 2024-09-21T02:52:15.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:15.785+0000 7f43c629b640 1 -- 172.21.15.29:0/3230318840 wait complete. 2024-09-21T02:52:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:15 smithi190 ceph-mon[33871]: pgmap v783: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-09-21T02:52:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:15 smithi029 ceph-mon[31006]: pgmap v783: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-09-21T02:52:16.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 265848 2.14 MB/sec execute 593 sec latency 46.333 ms 2024-09-21T02:52:17.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 266370 2.15 MB/sec execute 594 sec latency 15.246 ms 2024-09-21T02:52:17.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:17 smithi066 ceph-mon[33436]: pgmap v784: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.6 MiB/s wr, 119 op/s 2024-09-21T02:52:17.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:17 smithi190 ceph-mon[33871]: pgmap v784: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.6 MiB/s wr, 119 op/s 2024-09-21T02:52:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:17 smithi029 ceph-mon[31006]: pgmap v784: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.6 MiB/s wr, 119 op/s 2024-09-21T02:52:18.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 266681 2.14 MB/sec execute 595 sec latency 17.246 ms 2024-09-21T02:52:19.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 267311 2.14 MB/sec execute 596 sec latency 18.115 ms 2024-09-21T02:52:19.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:19 smithi066 ceph-mon[33436]: pgmap v785: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-09-21T02:52:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:19 smithi190 ceph-mon[33871]: pgmap v785: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-09-21T02:52:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:19 smithi029 ceph-mon[31006]: pgmap v785: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-09-21T02:52:20.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 267952 2.15 MB/sec execute 597 sec latency 14.251 ms 2024-09-21T02:52:21.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 268566 2.15 MB/sec execute 598 sec latency 17.122 ms 2024-09-21T02:52:21.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:21 smithi066 ceph-mon[33436]: pgmap v786: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T02:52:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:21 smithi190 ceph-mon[33871]: pgmap v786: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T02:52:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:21 smithi029 ceph-mon[31006]: pgmap v786: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T02:52:22.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 269374 2.15 MB/sec execute 599 sec latency 17.061 ms 2024-09-21T02:52:23.106 INFO:tasks.workunit.client.0.smithi029.stdout: 1 cleanup 600 sec 2024-09-21T02:52:23.602 INFO:tasks.workunit.client.0.smithi029.stdout: 0 cleanup 601 sec 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Operation Count AvgLat MaxLat 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: ---------------------------------------- 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: NTCreateX 41360 6.961 27564.949 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Close 30401 0.054 3.080 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Rename 1757 7.238 157.154 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Unlink 8354 6.092 23499.391 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Qpathinfo 37561 2.023 117.465 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Qfileinfo 6557 0.078 1.001 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Qfsinfo 6895 0.331 3.462 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Sfileinfo 3366 0.666 2.293 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: Find 14528 10.815 146.450 2024-09-21T02:52:23.603 INFO:tasks.workunit.client.0.smithi029.stdout: WriteX 20487 1.096 18811.983 2024-09-21T02:52:23.604 INFO:tasks.workunit.client.0.smithi029.stdout: ReadX 65147 0.004 0.240 2024-09-21T02:52:23.604 INFO:tasks.workunit.client.0.smithi029.stdout: LockX 136 0.674 1.430 2024-09-21T02:52:23.604 INFO:tasks.workunit.client.0.smithi029.stdout: UnlockX 136 0.562 1.097 2024-09-21T02:52:23.604 INFO:tasks.workunit.client.0.smithi029.stdout: Flush 2897 3.519 208.741 2024-09-21T02:52:23.604 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-09-21T02:52:23.604 INFO:tasks.workunit.client.0.smithi029.stdout:Throughput 2.15387 MB/sec 1 clients 1 procs max_latency=27565.237 ms 2024-09-21T02:52:23.604 INFO:tasks.workunit.client.0.smithi029.stderr:+ dbench 10 2024-09-21T02:52:23.604 INFO:tasks.workunit.client.0.smithi029.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-21T02:52:23.605 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-09-21T02:52:23.605 INFO:tasks.workunit.client.0.smithi029.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-21T02:52:23.605 INFO:tasks.workunit.client.0.smithi029.stdout:8 of 10 processes prepared for launch 0 sec 2024-09-21T02:52:23.704 INFO:tasks.workunit.client.0.smithi029.stdout:10 of 10 processes prepared for launch 0 sec 2024-09-21T02:52:23.704 INFO:tasks.workunit.client.0.smithi029.stdout:releasing clients 2024-09-21T02:52:23.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:23 smithi066 ceph-mon[33436]: pgmap v787: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-09-21T02:52:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:23 smithi190 ceph-mon[33871]: pgmap v787: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-09-21T02:52:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:23 smithi029 ceph-mon[31006]: pgmap v787: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-09-21T02:52:24.704 INFO:tasks.workunit.client.0.smithi029.stdout: 10 261 98.72 MB/sec warmup 1 sec latency 36.735 ms 2024-09-21T02:52:25.704 INFO:tasks.workunit.client.0.smithi029.stdout: 10 477 83.53 MB/sec warmup 2 sec latency 83.459 ms 2024-09-21T02:52:25.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:25 smithi066 ceph-mon[33436]: pgmap v788: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-09-21T02:52:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:25 smithi190 ceph-mon[33871]: pgmap v788: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-09-21T02:52:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:25 smithi029 ceph-mon[31006]: pgmap v788: 129 pgs: 129 active+clean; 49 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-09-21T02:52:26.704 INFO:tasks.workunit.client.0.smithi029.stdout: 10 687 70.65 MB/sec warmup 3 sec latency 120.341 ms 2024-09-21T02:52:27.704 INFO:tasks.workunit.client.0.smithi029.stdout: 10 855 53.92 MB/sec warmup 4 sec latency 88.545 ms 2024-09-21T02:52:27.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:27 smithi066 ceph-mon[33436]: pgmap v789: 129 pgs: 129 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 5.2 MiB/s wr, 138 op/s 2024-09-21T02:52:27.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:27 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:52:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:27 smithi190 ceph-mon[33871]: pgmap v789: 129 pgs: 129 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 5.2 MiB/s wr, 138 op/s 2024-09-21T02:52:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:27 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:52:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:27 smithi029 ceph-mon[31006]: pgmap v789: 129 pgs: 129 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 5.2 MiB/s wr, 138 op/s 2024-09-21T02:52:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:27 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:52:28.704 INFO:tasks.workunit.client.0.smithi029.stdout: 10 997 44.11 MB/sec warmup 5 sec latency 122.377 ms 2024-09-21T02:52:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.920+0000 7fdf7255e640 1 -- 172.21.15.29:0/1797262121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf6c103630 msgr2=0x7fdf6c103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.920+0000 7fdf7255e640 1 --2- 172.21.15.29:0/1797262121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf6c103630 0x7fdf6c103a40 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009900 tx=0x7fdf6002a410 comp rx=0 tx=0).stop 2024-09-21T02:52:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.920+0000 7fdf7255e640 1 -- 172.21.15.29:0/1797262121 shutdown_connections 2024-09-21T02:52:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.920+0000 7fdf7255e640 1 --2- 172.21.15.29:0/1797262121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf6c103fd0 0x7fdf6c10a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.920+0000 7fdf7255e640 1 --2- 172.21.15.29:0/1797262121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf6c103630 0x7fdf6c103a40 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.920+0000 7fdf7255e640 1 --2- 172.21.15.29:0/1797262121 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf6c1025b0 0x7fdf6c102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.920+0000 7fdf7255e640 1 -- 172.21.15.29:0/1797262121 >> 172.21.15.29:0/1797262121 conn(0x7fdf6c0fdf30 msgr2=0x7fdf6c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.921+0000 7fdf7255e640 1 -- 172.21.15.29:0/1797262121 shutdown_connections 2024-09-21T02:52:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.921+0000 7fdf7255e640 1 -- 172.21.15.29:0/1797262121 wait complete. 2024-09-21T02:52:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.921+0000 7fdf7255e640 1 Processor -- start 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.921+0000 7fdf7255e640 1 -- start start 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf7255e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf6c1025b0 0x7fdf6c198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf7255e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf6c103630 0x7fdf6c199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf7255e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf6c103fd0 0x7fdf6c199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf7255e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c10e0a0 con 0x7fdf6c103fd0 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf7255e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c10df20 con 0x7fdf6c1025b0 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf7255e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c10e220 con 0x7fdf6c103630 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf6c1025b0 0x7fdf6c198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf6c103630 0x7fdf6c199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf6c1025b0 0x7fdf6c198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:41250/0 (socket says 172.21.15.29:41250) 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6bfff640 1 -- 172.21.15.29:0/920538810 learned_addr learned my addr 172.21.15.29:0/920538810 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf70afb640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf6c103fd0 0x7fdf6c199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6bfff640 1 -- 172.21.15.29:0/920538810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf6c103630 msgr2=0x7fdf6c199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf6c103630 0x7fdf6c199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6bfff640 1 -- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf6c103fd0 msgr2=0x7fdf6c199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf6c103fd0 0x7fdf6c199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.922+0000 7fdf6bfff640 1 -- 172.21.15.29:0/920538810 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c19a7f0 con 0x7fdf6c1025b0 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.923+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf6c1025b0 0x7fdf6c198db0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c00ea60 tx=0x7fdf5c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.923+0000 7fdf697fa640 1 -- 172.21.15.29:0/920538810 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c00cde0 con 0x7fdf6c1025b0 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.923+0000 7fdf7255e640 1 -- 172.21.15.29:0/920538810 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6c19aae0 con 0x7fdf6c1025b0 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.923+0000 7fdf697fa640 1 -- 172.21.15.29:0/920538810 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf5c0035f0 con 0x7fdf6c1025b0 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.923+0000 7fdf697fa640 1 -- 172.21.15.29:0/920538810 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c010690 con 0x7fdf6c1025b0 2024-09-21T02:52:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.923+0000 7fdf7255e640 1 -- 172.21.15.29:0/920538810 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6c06b780 con 0x7fdf6c1025b0 2024-09-21T02:52:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf794cd740 1 Processor -- start 2024-09-21T02:52:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf794cd740 1 -- start start 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf697fa640 1 -- 172.21.15.29:0/920538810 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fdf5c010830 con 0x7fdf6c1025b0 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d075260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d0758d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca84d0 0x563e1d076420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf794cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e1cfe36f0 con 0x563e1ccc91b0 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf794cd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x563e1cfd2b20 con 0x563e1cca02f0 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf794cd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563e1cfd6930 con 0x563e1cca84d0 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d0758d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf6b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca84d0 0x563e1d076420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d0758d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:41260/0 (socket says 172.21.15.29:41260) 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf6b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca84d0 0x563e1d076420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56362/0 (socket says 172.21.15.29:56362) 2024-09-21T02:52:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf6bfff640 1 -- 172.21.15.29:0/1724782621 learned_addr learned my addr 172.21.15.29:0/1724782621 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.926+0000 7fdf70afb640 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf6bfff640 1 -- 172.21.15.29:0/1724782621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca84d0 msgr2=0x563e1d076420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca84d0 0x563e1d076420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf6bfff640 1 -- 172.21.15.29:0/1724782621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 msgr2=0x563e1d075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf6bfff640 1 -- 172.21.15.29:0/1724782621 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e1cf98530 con 0x563e1cca02f0 2024-09-21T02:52:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca84d0 0x563e1d076420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:52:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d0758d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0eda80 tx=0x7fdf5c0edf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a807f13f22c1b2c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf697fa640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf540b6510 0x7fdf540b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf527fc640 1 -- 172.21.15.29:0/1724782621 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0f6070 con 0x563e1cca02f0 2024-09-21T02:52:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf70afb640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf540b6510 0x7fdf540b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf527fc640 1 -- 172.21.15.29:0/1724782621 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf5c0ad210 con 0x563e1cca02f0 2024-09-21T02:52:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf527fc640 1 -- 172.21.15.29:0/1724782621 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0ace70 con 0x563e1cca02f0 2024-09-21T02:52:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.927+0000 7fdf697fa640 1 -- 172.21.15.29:0/920538810 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdf5c0e78f0 con 0x7fdf6c1025b0 2024-09-21T02:52:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf70afb640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf540b6510 0x7fdf540b8a20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdf64005fd0 tx=0x7fdf640096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf794cd740 1 -- 172.21.15.29:0/1724782621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 msgr2=0x563e1d0758d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf794cd740 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d0758d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0eda80 tx=0x7fdf5c0edf40 comp rx=0 tx=0).stop 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf794cd740 1 -- 172.21.15.29:0/1724782621 shutdown_connections 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf794cd740 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca84d0 0x563e1d076420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf794cd740 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d0758d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf794cd740 1 --2- 172.21.15.29:0/1724782621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf794cd740 1 -- 172.21.15.29:0/1724782621 >> 172.21.15.29:0/1724782621 conn(0x563e1ccbd300 msgr2=0x563e1d06c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.928+0000 7fdf794cd740 1 -- 172.21.15.29:0/1724782621 shutdown_connections 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.929+0000 7fdf794cd740 1 -- 172.21.15.29:0/1724782621 wait complete. 2024-09-21T02:52:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.929+0000 7fdf794cd740 1 -- start start 2024-09-21T02:52:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.929+0000 7fdf794cd740 1 Processor -- start 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca02f0 0x563e1d07ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca84d0 0x563e1d07f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1ccc91b0 0x563e1d07fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca02f0 0x563e1d07ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf794cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e1cf0aab0 con 0x563e1cca02f0 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca02f0 0x563e1d07ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47720/0 (socket says 172.21.15.29:47720) 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6bfff640 1 -- 172.21.15.29:0/3175321670 learned_addr learned my addr 172.21.15.29:0/3175321670 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x563e1cf78730 con 0x563e1cca84d0 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563e1d076f40 con 0x563e1ccc91b0 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca84d0 0x563e1d07f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf794cd740 10 client.0 _ll_register_callbacks cb 0x563e1cfcc6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf70afb640 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1ccc91b0 0x563e1d07fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/3175321670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1ccc91b0 msgr2=0x563e1d07fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1ccc91b0 0x563e1d07fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/3175321670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca02f0 msgr2=0x563e1d07ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca02f0 0x563e1d07ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/3175321670 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e1cd34790 con 0x563e1cca84d0 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.930+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca02f0 0x563e1d07ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca84d0 0x563e1d07f0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdf60005760 tx=0x7fdf600318b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b34a8b5af8de9e67 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3175321670 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60035040 con 0x563e1cca84d0 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3175321670 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdf60004240 con 0x563e1cca84d0 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf5c0ad210 con 0x563e1cca84d0 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3175321670 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdf6004e050 con 0x563e1cca84d0 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf794cd740 10 client.26702 fetch_fsmap learned FSMap version 32 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563e1cdad620 con 0x563e1cca84d0 2024-09-21T02:52:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.931+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3175321670 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf5c0ad210 con 0x563e1cca84d0 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3175321670 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fdf6003c440 con 0x563e1cca84d0 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf794cd740 10 client.26702 fetch_fsmap finished waiting for FSMap version 32 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf794cd740 10 client.26702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf794cd740 20 client.26702 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563e1cc87d80 0x563e1d08c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf794cd740 4 client.26702 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e1cfe5640 con 0x563e1cc87d80 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563e1cc87d80 0x563e1d08c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf4b7fe640 10 client.26702 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.932+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563e1cc87d80 0x563e1d08c1e0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a0446af5569b093 server_cookie=a86c145c7dabf230 in_seq=0 out_seq=0 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.933+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3175321670 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563e1cfe5640 con 0x563e1cc87d80 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.933+0000 7fdf4b7fe640 10 client.26702 handle_command_reply: tid=0 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.933+0000 7fdf794cd740 1 client.26702 shutdown 2024-09-21T02:52:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca84d0 msgr2=0x563e1d07f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca84d0 0x563e1d07f0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdf60005760 tx=0x7fdf600318b0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 shutdown_connections 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1ccc91b0 0x563e1d07fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca84d0 0x563e1d07f0c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563e1cc87d80 0x563e1d08c1e0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3175321670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca02f0 0x563e1d07ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf4b7fe640 0 client.26702 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 >> 172.21.15.29:0/3175321670 conn(0x563e1ccbd300 msgr2=0x563e1d06aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 shutdown_connections 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 1 -- 172.21.15.29:0/3175321670 wait complete. 2024-09-21T02:52:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.934+0000 7fdf794cd740 20 client.26702 trim_cache size 0 max 16384 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.956+0000 7fdf794cd740 1 Processor -- start 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.956+0000 7fdf794cd740 1 -- start start 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca02f0 0x563e1d075870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca84d0 0x563e1d07ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1ccc91b0 0x563e1d07ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf794cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c1aa360 con 0x563e1cca84d0 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf794cd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c1abf60 con 0x563e1ccc91b0 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf794cd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c1aaa30 con 0x563e1cca02f0 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca84d0 0x563e1d07ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca84d0 0x563e1d07ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47736/0 (socket says 172.21.15.29:47736) 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/2623285350 learned_addr learned my addr 172.21.15.29:0/2623285350 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca02f0 0x563e1d075870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf70afb640 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1ccc91b0 0x563e1d07ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/2623285350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca02f0 msgr2=0x563e1d075870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca02f0 0x563e1d075870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/2623285350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1ccc91b0 msgr2=0x563e1d07ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1ccc91b0 0x563e1d07ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.957+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/2623285350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e1d07cbc0 con 0x563e1cca84d0 2024-09-21T02:52:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.958+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca84d0 0x563e1d07ea00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009850 tx=0x7fdf60031420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928e863e4be72b71 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.958+0000 7fdf51ffb640 1 -- 172.21.15.29:0/2623285350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60035040 con 0x563e1cca84d0 2024-09-21T02:52:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.958+0000 7fdf51ffb640 1 -- 172.21.15.29:0/2623285350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf600044c0 con 0x563e1cca84d0 2024-09-21T02:52:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.958+0000 7fdf51ffb640 1 -- 172.21.15.29:0/2623285350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf600039f0 con 0x563e1cca84d0 2024-09-21T02:52:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 -- 172.21.15.29:0/2623285350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca84d0 msgr2=0x563e1d07ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca84d0 0x563e1d07ea00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009850 tx=0x7fdf60031420 comp rx=0 tx=0).stop 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 -- 172.21.15.29:0/2623285350 shutdown_connections 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1ccc91b0 0x563e1d07ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1cca84d0 0x563e1d07ea00 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 --2- 172.21.15.29:0/2623285350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cca02f0 0x563e1d075870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 -- 172.21.15.29:0/2623285350 >> 172.21.15.29:0/2623285350 conn(0x563e1ccbd300 msgr2=0x563e1d06c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 -- 172.21.15.29:0/2623285350 shutdown_connections 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.959+0000 7fdf794cd740 1 -- 172.21.15.29:0/2623285350 wait complete. 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.960+0000 7fdf794cd740 1 -- start start 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.960+0000 7fdf794cd740 1 Processor -- start 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.960+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d086470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.960+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d085130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.960+0000 7fdf794cd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cc5d1c0 0x563e1d085670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf794cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c1ab7d0 con 0x563e1ccc91b0 2024-09-21T02:52:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf794cd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c1ab100 con 0x563e1cca02f0 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf794cd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c1a8340 con 0x563e1cc5d1c0 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d085130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d085130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47748/0 (socket says 172.21.15.29:47748) 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/3832403541 learned_addr learned my addr 172.21.15.29:0/3832403541 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d086470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf794cd740 10 client.0 _ll_register_callbacks cb 0x7fdf60009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf70afb640 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cc5d1c0 0x563e1d085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/3832403541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cc5d1c0 msgr2=0x563e1d085670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cc5d1c0 0x563e1d085670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/3832403541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 msgr2=0x563e1d086470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d086470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.961+0000 7fdf6b7fe640 1 -- 172.21.15.29:0/3832403541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e1d085be0 con 0x563e1ccc91b0 2024-09-21T02:52:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf6b7fe640 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d085130 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fdf60005760 tx=0x7fdf6003ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b501e19e49165c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3832403541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60035070 con 0x563e1ccc91b0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf794cd740 1 -- 172.21.15.29:0/3832403541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563e1cfe10f0 con 0x563e1ccc91b0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3832403541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdf60004030 con 0x563e1ccc91b0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3832403541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdf60064050 con 0x563e1ccc91b0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf794cd740 10 client.17328 fetch_fsmap learned FSMap version 32 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf794cd740 1 -- 172.21.15.29:0/3832403541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563e1d084a10 con 0x563e1ccc91b0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.962+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3832403541 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x563e1cfe10f0 con 0x563e1ccc91b0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3832403541 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fdf6005b430 con 0x563e1ccc91b0 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf794cd740 10 client.17328 fetch_fsmap finished waiting for FSMap version 32 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf794cd740 10 client.17328 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf794cd740 20 client.17328 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563e1d0fbaa0 0x563e1d0fdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf794cd740 4 client.17328 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:52:28.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf794cd740 1 -- 172.21.15.29:0/3832403541 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e1cfde520 con 0x563e1d0fbaa0 2024-09-21T02:52:28.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563e1d0fbaa0 0x563e1d0fdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:28.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.963+0000 7fdf4b7fe640 10 client.17328 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:28.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:28.964+0000 7fdf6bfff640 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563e1d0fbaa0 0x563e1d0fdec0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ecb78e105d7cbe server_cookie=e18c59028421bbb7 in_seq=0 out_seq=0 2024-09-21T02:52:29.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.281+0000 7fdf4b7fe640 1 -- 172.21.15.29:0/3832403541 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e1cfde520 con 0x563e1d0fbaa0 2024-09-21T02:52:29.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.281+0000 7fdf4b7fe640 10 client.17328 handle_command_reply: tid=0 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.281+0000 7fdf794cd740 1 client.17328 shutdown 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.281+0000 7fdf794cd740 1 -- 172.21.15.29:0/3832403541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 msgr2=0x563e1d085130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.281+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d085130 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fdf60005760 tx=0x7fdf6003ca80 comp rx=0 tx=0).stop 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.281+0000 7fdf794cd740 1 -- 172.21.15.29:0/3832403541 shutdown_connections 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.281+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e1cc5d1c0 0x563e1d085670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.281+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563e1d0fbaa0 0x563e1d0fdec0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e1ccc91b0 0x563e1d085130 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf4b7fe640 0 client.17328 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf794cd740 1 --2- 172.21.15.29:0/3832403541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563e1cca02f0 0x563e1d086470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf794cd740 1 -- 172.21.15.29:0/3832403541 >> 172.21.15.29:0/3832403541 conn(0x563e1ccbd300 msgr2=0x563e1d075760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf794cd740 1 -- 172.21.15.29:0/3832403541 shutdown_connections 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf794cd740 1 -- 172.21.15.29:0/3832403541 wait complete. 2024-09-21T02:52:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf794cd740 20 client.17328 trim_cache size 0 max 16384 2024-09-21T02:52:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:52:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:52:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:52:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf7255e640 1 -- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf540b6510 msgr2=0x7fdf540b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf7255e640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf540b6510 0x7fdf540b8a20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdf64005fd0 tx=0x7fdf640096a0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf7255e640 1 -- 172.21.15.29:0/920538810 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf6c1025b0 msgr2=0x7fdf6c198db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.282+0000 7fdf7255e640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf6c1025b0 0x7fdf6c198db0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c00ea60 tx=0x7fdf5c00ef20 comp rx=0 tx=0).stop 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.283+0000 7fdf7255e640 1 -- 172.21.15.29:0/920538810 shutdown_connections 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.283+0000 7fdf7255e640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdf540b6510 0x7fdf540b8a20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.283+0000 7fdf7255e640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf6c103fd0 0x7fdf6c199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.283+0000 7fdf7255e640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf6c103630 0x7fdf6c199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.283+0000 7fdf7255e640 1 --2- 172.21.15.29:0/920538810 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf6c1025b0 0x7fdf6c198db0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.283+0000 7fdf7255e640 1 -- 172.21.15.29:0/920538810 >> 172.21.15.29:0/920538810 conn(0x7fdf6c0fdf30 msgr2=0x7fdf6c106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.283+0000 7fdf7255e640 1 -- 172.21.15.29:0/920538810 shutdown_connections 2024-09-21T02:52:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:29.283+0000 7fdf7255e640 1 -- 172.21.15.29:0/920538810 wait complete. 2024-09-21T02:52:29.704 INFO:tasks.workunit.client.0.smithi029.stdout: 10 1120 38.73 MB/sec warmup 6 sec latency 284.651 ms 2024-09-21T02:52:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:29 smithi066 ceph-mon[33436]: pgmap v790: 129 pgs: 129 active+clean; 184 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 13 MiB/s wr, 185 op/s 2024-09-21T02:52:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:29 smithi190 ceph-mon[33871]: pgmap v790: 129 pgs: 129 active+clean; 184 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 13 MiB/s wr, 185 op/s 2024-09-21T02:52:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:29 smithi029 ceph-mon[31006]: pgmap v790: 129 pgs: 129 active+clean; 184 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 13 MiB/s wr, 185 op/s 2024-09-21T02:52:30.704 INFO:tasks.workunit.client.0.smithi029.stdout: 10 1263 33.79 MB/sec warmup 7 sec latency 142.764 ms 2024-09-21T02:52:30.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:30 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:52:30.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:30 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:52:30.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:30 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:52:31.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 1420 30.81 MB/sec warmup 8 sec latency 147.502 ms 2024-09-21T02:52:31.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:31 smithi066 ceph-mon[33436]: pgmap v791: 129 pgs: 129 active+clean; 226 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 18 MiB/s wr, 225 op/s 2024-09-21T02:52:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:31 smithi190 ceph-mon[33871]: pgmap v791: 129 pgs: 129 active+clean; 226 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 18 MiB/s wr, 225 op/s 2024-09-21T02:52:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:31 smithi029 ceph-mon[31006]: pgmap v791: 129 pgs: 129 active+clean; 226 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 18 MiB/s wr, 225 op/s 2024-09-21T02:52:32.704 INFO:tasks.workunit.client.0.smithi029.stdout: 10 1558 29.04 MB/sec warmup 9 sec latency 311.527 ms 2024-09-21T02:52:33.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 1683 26.34 MB/sec warmup 10 sec latency 210.027 ms 2024-09-21T02:52:33.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:33 smithi066 ceph-mon[33436]: pgmap v792: 129 pgs: 129 active+clean; 226 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 19 MiB/s wr, 297 op/s 2024-09-21T02:52:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:33 smithi190 ceph-mon[33871]: pgmap v792: 129 pgs: 129 active+clean; 226 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 19 MiB/s wr, 297 op/s 2024-09-21T02:52:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:33 smithi029 ceph-mon[31006]: pgmap v792: 129 pgs: 129 active+clean; 226 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 19 MiB/s wr, 297 op/s 2024-09-21T02:52:34.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 1827 24.13 MB/sec warmup 11 sec latency 87.270 ms 2024-09-21T02:52:35.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2012 22.70 MB/sec warmup 12 sec latency 80.706 ms 2024-09-21T02:52:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:35 smithi066 ceph-mon[33436]: pgmap v793: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 19 MiB/s wr, 395 op/s 2024-09-21T02:52:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:35 smithi190 ceph-mon[33871]: pgmap v793: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 19 MiB/s wr, 395 op/s 2024-09-21T02:52:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:35 smithi029 ceph-mon[31006]: pgmap v793: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 19 MiB/s wr, 395 op/s 2024-09-21T02:52:36.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2185 21.28 MB/sec warmup 13 sec latency 107.628 ms 2024-09-21T02:52:37.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2326 20.17 MB/sec warmup 14 sec latency 110.980 ms 2024-09-21T02:52:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:37 smithi066 ceph-mon[33436]: pgmap v794: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 19 MiB/s wr, 392 op/s 2024-09-21T02:52:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:37 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:52:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:37 smithi190 ceph-mon[33871]: pgmap v794: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 19 MiB/s wr, 392 op/s 2024-09-21T02:52:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:37 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:52:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:37 smithi029 ceph-mon[31006]: pgmap v794: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 19 MiB/s wr, 392 op/s 2024-09-21T02:52:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:37 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:52:38.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2516 19.61 MB/sec warmup 15 sec latency 88.350 ms 2024-09-21T02:52:38.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:52:38.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:52:38.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:52:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:52:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:52:38.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:52:38.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:52:38.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:52:38.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:52:39.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2626 18.68 MB/sec warmup 16 sec latency 100.072 ms 2024-09-21T02:52:40.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:39 smithi190 ceph-mon[33871]: pgmap v795: 129 pgs: 129 active+clean; 233 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 16 MiB/s wr, 404 op/s 2024-09-21T02:52:40.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:39 smithi029 ceph-mon[31006]: pgmap v795: 129 pgs: 129 active+clean; 233 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 16 MiB/s wr, 404 op/s 2024-09-21T02:52:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:39 smithi066 ceph-mon[33436]: pgmap v795: 129 pgs: 129 active+clean; 233 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 16 MiB/s wr, 404 op/s 2024-09-21T02:52:40.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2764 18.11 MB/sec warmup 17 sec latency 86.381 ms 2024-09-21T02:52:41.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2939 17.99 MB/sec warmup 18 sec latency 86.754 ms 2024-09-21T02:52:42.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:41 smithi190 ceph-mon[33871]: pgmap v796: 129 pgs: 129 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 7.4 MiB/s wr, 373 op/s 2024-09-21T02:52:42.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:41 smithi029 ceph-mon[31006]: pgmap v796: 129 pgs: 129 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 7.4 MiB/s wr, 373 op/s 2024-09-21T02:52:42.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:41 smithi066 ceph-mon[33436]: pgmap v796: 129 pgs: 129 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 7.4 MiB/s wr, 373 op/s 2024-09-21T02:52:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.425+0000 7fdd89871640 1 -- 172.21.15.29:0/759800916 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd8410de30 msgr2=0x7fdd84110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.425+0000 7fdd89871640 1 --2- 172.21.15.29:0/759800916 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd8410de30 0x7fdd84110250 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009900 tx=0x7fdd7802a420 comp rx=0 tx=0).stop 2024-09-21T02:52:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.425+0000 7fdd89871640 1 -- 172.21.15.29:0/759800916 shutdown_connections 2024-09-21T02:52:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.425+0000 7fdd89871640 1 --2- 172.21.15.29:0/759800916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd84110940 0x7fdd84112e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.425+0000 7fdd89871640 1 --2- 172.21.15.29:0/759800916 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd8410de30 0x7fdd84110250 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.425+0000 7fdd89871640 1 --2- 172.21.15.29:0/759800916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd8410b4d0 0x7fdd8410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.426+0000 7fdd89871640 1 -- 172.21.15.29:0/759800916 >> 172.21.15.29:0/759800916 conn(0x7fdd84070640 msgr2=0x7fdd84072af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.426+0000 7fdd89871640 1 -- 172.21.15.29:0/759800916 shutdown_connections 2024-09-21T02:52:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.426+0000 7fdd89871640 1 -- 172.21.15.29:0/759800916 wait complete. 2024-09-21T02:52:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.427+0000 7fdd89871640 1 Processor -- start 2024-09-21T02:52:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.427+0000 7fdd89871640 1 -- start start 2024-09-21T02:52:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.427+0000 7fdd89871640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd8410b4d0 0x7fdd841a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.427+0000 7fdd89871640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd8410de30 0x7fdd841a9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.427+0000 7fdd89871640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd84110940 0x7fdd841a9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.427+0000 7fdd89871640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84115690 con 0x7fdd84110940 2024-09-21T02:52:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.427+0000 7fdd89871640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84115510 con 0x7fdd8410b4d0 2024-09-21T02:52:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.427+0000 7fdd89871640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84115810 con 0x7fdd8410de30 2024-09-21T02:52:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.428+0000 7fdd83fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd8410de30 0x7fdd841a9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.428+0000 7fdd83fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd8410de30 0x7fdd841a9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56946/0 (socket says 172.21.15.29:56946) 2024-09-21T02:52:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.428+0000 7fdd83fff640 1 -- 172.21.15.29:0/3493102986 learned_addr learned my addr 172.21.15.29:0/3493102986 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.429+0000 7fdd83fff640 1 -- 172.21.15.29:0/3493102986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd8410b4d0 msgr2=0x7fdd841a8b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T02:52:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.429+0000 7fdd83fff640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd8410b4d0 0x7fdd841a8b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.429+0000 7fdd83fff640 1 -- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd84110940 msgr2=0x7fdd841a9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.429+0000 7fdd83fff640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd84110940 0x7fdd841a9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.429+0000 7fdd83fff640 1 -- 172.21.15.29:0/3493102986 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd841aa560 con 0x7fdd8410de30 2024-09-21T02:52:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.430+0000 7fdd83fff640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd8410de30 0x7fdd841a9190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdd78005760 tx=0x7fdd78003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.430+0000 7fdd81ffb640 1 -- 172.21.15.29:0/3493102986 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7803f070 con 0x7fdd8410de30 2024-09-21T02:52:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.430+0000 7fdd89871640 1 -- 172.21.15.29:0/3493102986 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd841aa7f0 con 0x7fdd8410de30 2024-09-21T02:52:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.430+0000 7fdd81ffb640 1 -- 172.21.15.29:0/3493102986 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd78031ba0 con 0x7fdd8410de30 2024-09-21T02:52:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.430+0000 7fdd81ffb640 1 -- 172.21.15.29:0/3493102986 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78035030 con 0x7fdd8410de30 2024-09-21T02:52:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.431+0000 7fdd89871640 1 -- 172.21.15.29:0/3493102986 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd841b0f40 con 0x7fdd8410de30 2024-09-21T02:52:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.434+0000 7fdd91a47740 1 Processor -- start 2024-09-21T02:52:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd91a47740 1 -- start start 2024-09-21T02:52:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7bd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7bd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 0x55657e7be200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd91a47740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55657e72f6f0 con 0x55657e3ec2f0 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd91a47740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55657e71eb20 con 0x55657e4151b0 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd91a47740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55657e722930 con 0x55657e3f44d0 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd89070640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7bd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd8886f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7bd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd8886f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7bd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52344/0 (socket says 172.21.15.29:52344) 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd8886f640 1 -- 172.21.15.29:0/3698722381 learned_addr learned my addr 172.21.15.29:0/3698722381 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.435+0000 7fdd81ffb640 1 -- 172.21.15.29:0/3493102986 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fdd78042070 con 0x7fdd8410de30 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd89070640 1 -- 172.21.15.29:0/3698722381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 msgr2=0x55657e7be200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd89070640 1 --2- 172.21.15.29:0/3698722381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 0x55657e7be200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd89070640 1 -- 172.21.15.29:0/3698722381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 msgr2=0x55657e7bd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd89070640 1 --2- 172.21.15.29:0/3698722381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7bd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd89070640 1 -- 172.21.15.29:0/3698722381 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55657e6e4530 con 0x55657e4151b0 2024-09-21T02:52:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd89070640 1 --2- 172.21.15.29:0/3698722381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7bd040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd740096a0 tx=0x7fdd74009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60e0ef931b9f251c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd8886f640 1 --2- 172.21.15.29:0/3698722381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7bd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd6affd640 1 -- 172.21.15.29:0/3698722381 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74015020 con 0x55657e4151b0 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd6affd640 1 -- 172.21.15.29:0/3698722381 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd7400bd50 con 0x55657e4151b0 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.436+0000 7fdd6affd640 1 -- 172.21.15.29:0/3698722381 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74011e60 con 0x55657e4151b0 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.437+0000 7fdd81ffb640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd6c0b6590 0x7fdd6c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.437+0000 7fdd81ffb640 1 -- 172.21.15.29:0/3493102986 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdd78108010 con 0x7fdd8410de30 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.437+0000 7fdd8886f640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd6c0b6590 0x7fdd6c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 -- 172.21.15.29:0/3698722381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 msgr2=0x55657e7bd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 --2- 172.21.15.29:0/3698722381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7bd040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd740096a0 tx=0x7fdd74009b60 comp rx=0 tx=0).stop 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 -- 172.21.15.29:0/3698722381 shutdown_connections 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 --2- 172.21.15.29:0/3698722381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 0x55657e7be200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 --2- 172.21.15.29:0/3698722381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7bd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 --2- 172.21.15.29:0/3698722381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7bd040 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 -- 172.21.15.29:0/3698722381 >> 172.21.15.29:0/3698722381 conn(0x55657e409300 msgr2=0x55657e7b8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd8886f640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd6c0b6590 0x7fdd6c0b8aa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0099a0 tx=0x7fdd7c00f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 -- 172.21.15.29:0/3698722381 shutdown_connections 2024-09-21T02:52:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 -- 172.21.15.29:0/3698722381 wait complete. 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 -- start start 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.438+0000 7fdd91a47740 1 Processor -- start 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e3ec2f0 0x55657e7c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e4151b0 0x55657e7c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd91a47740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55657e656ab0 con 0x55657e3f44d0 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd91a47740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55657e6c4730 con 0x55657e3ec2f0 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd91a47740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55657e7bed20 con 0x55657e4151b0 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd91a47740 10 client.0 _ll_register_callbacks cb 0x55657e7186b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd89070640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e4151b0 0x55657e7c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52356/0 (socket says 172.21.15.29:52356) 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd83fff640 1 -- 172.21.15.29:0/201491748 learned_addr learned my addr 172.21.15.29:0/201491748 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.439+0000 7fdd8886f640 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e3ec2f0 0x55657e7c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd89070640 1 -- 172.21.15.29:0/201491748 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e3ec2f0 msgr2=0x55657e7c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd89070640 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e3ec2f0 0x55657e7c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd89070640 1 -- 172.21.15.29:0/201491748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 msgr2=0x55657e7c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd89070640 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd89070640 1 -- 172.21.15.29:0/201491748 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55657e480790 con 0x55657e4151b0 2024-09-21T02:52:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd83fff640 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:52:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd89070640 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e4151b0 0x55657e7c7ac0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009670 tx=0x7fdd74011e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=877e0564a732532 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd63fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd63fff640 1 -- 172.21.15.29:0/201491748 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74015030 con 0x55657e4151b0 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd63fff640 1 -- 172.21.15.29:0/201491748 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdd7400b6c0 con 0x55657e4151b0 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd91a47740 1 -- 172.21.15.29:0/201491748 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd7400bd50 con 0x55657e4151b0 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd63fff640 1 -- 172.21.15.29:0/201491748 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdd7402d050 con 0x55657e4151b0 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.440+0000 7fdd91a47740 10 client.26863 fetch_fsmap learned FSMap version 33 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd91a47740 1 -- 172.21.15.29:0/201491748 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55657e4f9620 con 0x55657e4151b0 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd63fff640 1 -- 172.21.15.29:0/201491748 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd7400bd50 con 0x55657e4151b0 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd63fff640 1 -- 172.21.15.29:0/201491748 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7fdd74024ba0 con 0x55657e4151b0 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd91a47740 10 client.26863 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd91a47740 10 client.26863 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd91a47740 20 client.26863 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd91a47740 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55657e3d3d80 0x55657e7d3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd91a47740 4 client.26863 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:52:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd91a47740 1 -- 172.21.15.29:0/201491748 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55657e731640 con 0x55657e3d3d80 2024-09-21T02:52:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.441+0000 7fdd8886f640 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55657e3d3d80 0x55657e7d3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.442+0000 7fdd63fff640 10 client.26863 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.442+0000 7fdd8886f640 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55657e3d3d80 0x55657e7d3e10 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2623c41fd3b37a06 server_cookie=caedc1c352e42c02 in_seq=0 out_seq=0 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd63fff640 1 -- 172.21.15.29:0/201491748 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55657e731640 con 0x55657e3d3d80 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd63fff640 10 client.26863 handle_command_reply: tid=0 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 client.26863 shutdown 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 -- 172.21.15.29:0/201491748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e4151b0 msgr2=0x55657e7c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e4151b0 0x55657e7c7ac0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009670 tx=0x7fdd74011e10 comp rx=0 tx=0).stop 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 -- 172.21.15.29:0/201491748 shutdown_connections 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e4151b0 0x55657e7c7ac0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55657e3d3d80 0x55657e7d3e10 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 --2- 172.21.15.29:0/201491748 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e3ec2f0 0x55657e7c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 -- 172.21.15.29:0/201491748 >> 172.21.15.29:0/201491748 conn(0x55657e409300 msgr2=0x55657e7b7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:42.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd63fff640 0 client.26863 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:42.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 -- 172.21.15.29:0/201491748 shutdown_connections 2024-09-21T02:52:42.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 1 -- 172.21.15.29:0/201491748 wait complete. 2024-09-21T02:52:42.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.444+0000 7fdd91a47740 20 client.26863 trim_cache size 0 max 16384 2024-09-21T02:52:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.466+0000 7fdd91a47740 1 Processor -- start 2024-09-21T02:52:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd91a47740 1 -- start start 2024-09-21T02:52:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3ec2f0 0x55657e7c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd91a47740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841bb1f0 con 0x55657e3f44d0 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd91a47740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841bcdf0 con 0x55657e4151b0 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd91a47740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841bb8c0 con 0x55657e3ec2f0 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd89070640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd8886f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3ec2f0 0x55657e7c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52366/0 (socket says 172.21.15.29:52366) 2024-09-21T02:52:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd89070640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7c5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:35524/0 (socket says 172.21.15.29:35524) 2024-09-21T02:52:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd83fff640 1 -- 172.21.15.29:0/1264236371 learned_addr learned my addr 172.21.15.29:0/1264236371 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd89070640 1 -- 172.21.15.29:0/1264236371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3ec2f0 msgr2=0x55657e7c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd89070640 1 --2- 172.21.15.29:0/1264236371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3ec2f0 0x55657e7c4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd89070640 1 -- 172.21.15.29:0/1264236371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 msgr2=0x55657e7c52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd89070640 1 --2- 172.21.15.29:0/1264236371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.467+0000 7fdd89070640 1 -- 172.21.15.29:0/1264236371 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55657e7c72c0 con 0x55657e4151b0 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.468+0000 7fdd89070640 1 --2- 172.21.15.29:0/1264236371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7c5ab0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd740096a0 tx=0x7fdd74006e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec9299b7670d1a58 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.468+0000 7fdd6a7fc640 1 -- 172.21.15.29:0/1264236371 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74036070 con 0x55657e4151b0 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.468+0000 7fdd6a7fc640 1 -- 172.21.15.29:0/1264236371 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd7401a7b0 con 0x55657e4151b0 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.468+0000 7fdd6a7fc640 1 -- 172.21.15.29:0/1264236371 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74015040 con 0x55657e4151b0 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.469+0000 7fdd91a47740 1 -- 172.21.15.29:0/1264236371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 msgr2=0x55657e7c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.469+0000 7fdd91a47740 1 --2- 172.21.15.29:0/1264236371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7c5ab0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd740096a0 tx=0x7fdd74006e40 comp rx=0 tx=0).stop 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.469+0000 7fdd91a47740 1 -- 172.21.15.29:0/1264236371 shutdown_connections 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.469+0000 7fdd91a47740 1 --2- 172.21.15.29:0/1264236371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7c5ab0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.469+0000 7fdd91a47740 1 --2- 172.21.15.29:0/1264236371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3f44d0 0x55657e7c52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.469+0000 7fdd91a47740 1 --2- 172.21.15.29:0/1264236371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3ec2f0 0x55657e7c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.469+0000 7fdd91a47740 1 -- 172.21.15.29:0/1264236371 >> 172.21.15.29:0/1264236371 conn(0x55657e409300 msgr2=0x55657e7b8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.470+0000 7fdd91a47740 1 -- 172.21.15.29:0/1264236371 shutdown_connections 2024-09-21T02:52:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.470+0000 7fdd91a47740 1 -- 172.21.15.29:0/1264236371 wait complete. 2024-09-21T02:52:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.470+0000 7fdd91a47740 1 -- start start 2024-09-21T02:52:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.470+0000 7fdd91a47740 1 Processor -- start 2024-09-21T02:52:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.471+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.471+0000 7fdd91a47740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 0x55657e7cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.471+0000 7fdd8886f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.471+0000 7fdd8886f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52374/0 (socket says 172.21.15.29:52374) 2024-09-21T02:52:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.471+0000 7fdd8886f640 1 -- 172.21.15.29:0/275900151 learned_addr learned my addr 172.21.15.29:0/275900151 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd91a47740 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841bc660 con 0x55657e3ec2f0 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd83fff640 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 0x55657e7cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd89070640 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841bbf90 con 0x55657e4151b0 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841b91d0 con 0x55657e3f44d0 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd91a47740 10 client.0 _ll_register_callbacks cb 0x7fdd74007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd8886f640 1 -- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 msgr2=0x55657e7cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd8886f640 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 0x55657e7cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd8886f640 1 -- 172.21.15.29:0/275900151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 msgr2=0x55657e7cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd8886f640 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd8886f640 1 -- 172.21.15.29:0/275900151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55657e7ccbb0 con 0x55657e3ec2f0 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.472+0000 7fdd83fff640 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 0x55657e7cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.473+0000 7fdd8886f640 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7cb220 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c00dfd0 tx=0x7fdd7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2049bf6714176c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.473+0000 7fdd63fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:52:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.473+0000 7fdd63fff640 1 -- 172.21.15.29:0/275900151 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c028070 con 0x55657e3ec2f0 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.473+0000 7fdd63fff640 1 -- 172.21.15.29:0/275900151 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdd7c026940 con 0x55657e3ec2f0 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.473+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd7401a7b0 con 0x55657e3ec2f0 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd63fff640 1 -- 172.21.15.29:0/275900151 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdd7c037050 con 0x55657e3ec2f0 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd91a47740 10 client.17343 fetch_fsmap learned FSMap version 33 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55657e7cdfa0 con 0x55657e3ec2f0 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd63fff640 1 -- 172.21.15.29:0/275900151 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd7401a7b0 con 0x55657e3ec2f0 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd63fff640 1 -- 172.21.15.29:0/275900151 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7fdd7c017070 con 0x55657e3ec2f0 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd91a47740 10 client.17343 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd91a47740 10 client.17343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd91a47740 20 client.17343 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd91a47740 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55657e3a91c0 0x55657e8407a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd91a47740 4 client.17343 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.474+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55657e72d0f0 con 0x55657e3a91c0 2024-09-21T02:52:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.475+0000 7fdd83fff640 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55657e3a91c0 0x55657e8407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.475+0000 7fdd63fff640 10 client.17343 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.475+0000 7fdd83fff640 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55657e3a91c0 0x55657e8407a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b547aa3412d2962b server_cookie=a5a780701927277c in_seq=0 out_seq=0 2024-09-21T02:52:42.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.621+0000 7fdd63fff640 1 -- 172.21.15.29:0/275900151 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55657e72d0f0 con 0x55657e3a91c0 2024-09-21T02:52:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.621+0000 7fdd63fff640 10 client.17343 handle_command_reply: tid=0 2024-09-21T02:52:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.621+0000 7fdd91a47740 1 client.17343 shutdown 2024-09-21T02:52:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 msgr2=0x55657e7cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7cb220 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c00dfd0 tx=0x7fdd7c004a40 comp rx=0 tx=0).stop 2024-09-21T02:52:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 shutdown_connections 2024-09-21T02:52:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55657e4151b0 0x55657e7cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55657e3f44d0 0x55657e7cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55657e3a91c0 0x55657e8407a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd63fff640 0 client.17343 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 --2- 172.21.15.29:0/275900151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55657e3ec2f0 0x55657e7cb220 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 >> 172.21.15.29:0/275900151 conn(0x55657e409300 msgr2=0x55657e7c4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 shutdown_connections 2024-09-21T02:52:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 1 -- 172.21.15.29:0/275900151 wait complete. 2024-09-21T02:52:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.622+0000 7fdd91a47740 20 client.17343 trim_cache size 0 max 16384 2024-09-21T02:52:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:52:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:52:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:52:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 -- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd6c0b6590 msgr2=0x7fdd6c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd6c0b6590 0x7fdd6c0b8aa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0099a0 tx=0x7fdd7c00f040 comp rx=0 tx=0).stop 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 -- 172.21.15.29:0/3493102986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd8410de30 msgr2=0x7fdd841a9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd8410de30 0x7fdd841a9190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdd78005760 tx=0x7fdd78003c10 comp rx=0 tx=0).stop 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 -- 172.21.15.29:0/3493102986 shutdown_connections 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdd6c0b6590 0x7fdd6c0b8aa0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd84110940 0x7fdd841a9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd8410de30 0x7fdd841a9190 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 --2- 172.21.15.29:0/3493102986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd8410b4d0 0x7fdd841a8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 -- 172.21.15.29:0/3493102986 >> 172.21.15.29:0/3493102986 conn(0x7fdd84070640 msgr2=0x7fdd8410c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 -- 172.21.15.29:0/3493102986 shutdown_connections 2024-09-21T02:52:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:42.623+0000 7fdd89871640 1 -- 172.21.15.29:0/3493102986 wait complete. 2024-09-21T02:52:42.778 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3139 18.06 MB/sec warmup 19 sec latency 234.351 ms 2024-09-21T02:52:43.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3339 17.69 MB/sec warmup 20 sec latency 227.249 ms 2024-09-21T02:52:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:43 smithi190 ceph-mon[33871]: pgmap v797: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 3.6 MiB/s wr, 334 op/s 2024-09-21T02:52:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:43 smithi029 ceph-mon[31006]: pgmap v797: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 3.6 MiB/s wr, 334 op/s 2024-09-21T02:52:44.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:43 smithi066 ceph-mon[33436]: pgmap v797: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 3.6 MiB/s wr, 334 op/s 2024-09-21T02:52:44.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3550 17.52 MB/sec warmup 21 sec latency 110.750 ms 2024-09-21T02:52:45.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3636 16.78 MB/sec warmup 22 sec latency 96.128 ms 2024-09-21T02:52:46.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:45 smithi190 ceph-mon[33871]: pgmap v798: 129 pgs: 129 active+clean; 240 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 3.6 MiB/s wr, 313 op/s 2024-09-21T02:52:46.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:45 smithi029 ceph-mon[31006]: pgmap v798: 129 pgs: 129 active+clean; 240 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 3.6 MiB/s wr, 313 op/s 2024-09-21T02:52:46.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:45 smithi066 ceph-mon[33436]: pgmap v798: 129 pgs: 129 active+clean; 240 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 3.6 MiB/s wr, 313 op/s 2024-09-21T02:52:46.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3709 16.13 MB/sec warmup 23 sec latency 103.847 ms 2024-09-21T02:52:47.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3813 15.81 MB/sec warmup 24 sec latency 93.389 ms 2024-09-21T02:52:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:47 smithi190 ceph-mon[33871]: pgmap v799: 129 pgs: 129 active+clean; 246 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 3.8 MiB/s wr, 239 op/s 2024-09-21T02:52:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:47 smithi029 ceph-mon[31006]: pgmap v799: 129 pgs: 129 active+clean; 246 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 3.8 MiB/s wr, 239 op/s 2024-09-21T02:52:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:47 smithi066 ceph-mon[33436]: pgmap v799: 129 pgs: 129 active+clean; 246 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 3.8 MiB/s wr, 239 op/s 2024-09-21T02:52:48.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4016 16.11 MB/sec warmup 25 sec latency 172.634 ms 2024-09-21T02:52:49.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4208 15.89 MB/sec warmup 26 sec latency 86.613 ms 2024-09-21T02:52:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:49 smithi190 ceph-mon[33871]: pgmap v800: 129 pgs: 129 active+clean; 247 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.1 MiB/s wr, 275 op/s 2024-09-21T02:52:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:49 smithi029 ceph-mon[31006]: pgmap v800: 129 pgs: 129 active+clean; 247 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.1 MiB/s wr, 275 op/s 2024-09-21T02:52:50.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:49 smithi066 ceph-mon[33436]: pgmap v800: 129 pgs: 129 active+clean; 247 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.1 MiB/s wr, 275 op/s 2024-09-21T02:52:50.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4323 15.49 MB/sec warmup 27 sec latency 89.625 ms 2024-09-21T02:52:51.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4450 15.17 MB/sec warmup 28 sec latency 116.689 ms 2024-09-21T02:52:52.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:51 smithi190 ceph-mon[33871]: pgmap v801: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-09-21T02:52:52.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:51 smithi029 ceph-mon[31006]: pgmap v801: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-09-21T02:52:52.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:51 smithi066 ceph-mon[33436]: pgmap v801: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-09-21T02:52:52.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4552 14.74 MB/sec warmup 29 sec latency 90.433 ms 2024-09-21T02:52:53.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4671 14.36 MB/sec warmup 30 sec latency 92.443 ms 2024-09-21T02:52:54.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:53 smithi190 ceph-mon[33871]: pgmap v802: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.6 MiB/s wr, 257 op/s 2024-09-21T02:52:54.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:53 smithi029 ceph-mon[31006]: pgmap v802: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.6 MiB/s wr, 257 op/s 2024-09-21T02:52:54.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:53 smithi066 ceph-mon[33436]: pgmap v802: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.6 MiB/s wr, 257 op/s 2024-09-21T02:52:54.705 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4827 14.02 MB/sec warmup 31 sec latency 108.312 ms 2024-09-21T02:52:55.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4957 13.70 MB/sec warmup 32 sec latency 129.291 ms 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 -- 172.21.15.29:0/4048891734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7948103650 msgr2=0x7f7948103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 --2- 172.21.15.29:0/4048891734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7948103650 0x7f7948103a60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7938009900 tx=0x7f793802a410 comp rx=0 tx=0).stop 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 -- 172.21.15.29:0/4048891734 shutdown_connections 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 --2- 172.21.15.29:0/4048891734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7948103ff0 0x7f794810a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 --2- 172.21.15.29:0/4048891734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7948103650 0x7f7948103a60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 --2- 172.21.15.29:0/4048891734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79481025d0 0x7f7948102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 -- 172.21.15.29:0/4048891734 >> 172.21.15.29:0/4048891734 conn(0x7f79480fdf30 msgr2=0x7f79481003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 -- 172.21.15.29:0/4048891734 shutdown_connections 2024-09-21T02:52:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.779+0000 7f7950557640 1 -- 172.21.15.29:0/4048891734 wait complete. 2024-09-21T02:52:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.780+0000 7f7950557640 1 Processor -- start 2024-09-21T02:52:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.780+0000 7f7950557640 1 -- start start 2024-09-21T02:52:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.780+0000 7f7950557640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79481025d0 0x7f79481a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.780+0000 7f7950557640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7948103650 0x7f79481a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f7950557640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7948103ff0 0x7f79481a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f7950557640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f794810e100 con 0x7f7948103650 2024-09-21T02:52:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f7950557640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f794810df80 con 0x7f7948103ff0 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f7950557640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f794810e280 con 0x7f79481025d0 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7948103650 0x7f79481a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7948103650 0x7f79481a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51080/0 (socket says 172.21.15.29:51080) 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 -- 172.21.15.29:0/2388413315 learned_addr learned my addr 172.21.15.29:0/2388413315 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794e2f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79481025d0 0x7f79481a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794eaf4640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7948103ff0 0x7f79481a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 -- 172.21.15.29:0/2388413315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79481025d0 msgr2=0x7f79481a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79481025d0 0x7f79481a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 -- 172.21.15.29:0/2388413315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7948103ff0 msgr2=0x7f79481a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7948103ff0 0x7f79481a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 -- 172.21.15.29:0/2388413315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79481a1d20 con 0x7f7948103650 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f794daf2640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7948103650 0x7f79481a0d10 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f7938009850 tx=0x7f7938003e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f793f7fe640 1 -- 172.21.15.29:0/2388413315 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793803f070 con 0x7f7948103650 2024-09-21T02:52:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.781+0000 7f7950557640 1 -- 172.21.15.29:0/2388413315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79481a1fb0 con 0x7f7948103650 2024-09-21T02:52:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.782+0000 7f7950557640 1 -- 172.21.15.29:0/2388413315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79481a86c0 con 0x7f7948103650 2024-09-21T02:52:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.782+0000 7f793f7fe640 1 -- 172.21.15.29:0/2388413315 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f793803bd00 con 0x7f7948103650 2024-09-21T02:52:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.782+0000 7f793f7fe640 1 -- 172.21.15.29:0/2388413315 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938035030 con 0x7f7948103650 2024-09-21T02:52:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.785+0000 7f79574c4740 1 Processor -- start 2024-09-21T02:52:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.785+0000 7f79574c4740 1 -- start start 2024-09-21T02:52:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.785+0000 7f793f7fe640 1 -- 172.21.15.29:0/2388413315 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f7938042070 con 0x7f7948103650 2024-09-21T02:52:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f79574c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468fd1b0 0x559746ca4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f79574c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746ca54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f79574c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468dc4d0 0x559746ca6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f79574c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559746c176f0 con 0x5597468fd1b0 2024-09-21T02:52:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f79574c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559746c06b20 con 0x5597468d42f0 2024-09-21T02:52:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f79574c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559746c0a930 con 0x5597468dc4d0 2024-09-21T02:52:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794eaf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468fd1b0 0x559746ca4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794e2f3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746ca54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794eaf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468fd1b0 0x559746ca4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51096/0 (socket says 172.21.15.29:51096) 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794eaf4640 1 -- 172.21.15.29:0/3835946400 learned_addr learned my addr 172.21.15.29:0/3835946400 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794daf2640 1 --2- 172.21.15.29:0/3835946400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468dc4d0 0x559746ca6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794eaf4640 1 -- 172.21.15.29:0/3835946400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468dc4d0 msgr2=0x559746ca6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794eaf4640 1 --2- 172.21.15.29:0/3835946400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468dc4d0 0x559746ca6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794eaf4640 1 -- 172.21.15.29:0/3835946400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 msgr2=0x559746ca54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794eaf4640 1 --2- 172.21.15.29:0/3835946400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746ca54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.786+0000 7f794eaf4640 1 -- 172.21.15.29:0/3835946400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559746bcc530 con 0x5597468fd1b0 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.787+0000 7f794eaf4640 1 --2- 172.21.15.29:0/3835946400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468fd1b0 0x559746ca4e50 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f7940009fd0 tx=0x7f79400118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7c5e8c8c56ffff server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.787+0000 7f793cbfe640 1 -- 172.21.15.29:0/3835946400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940004d20 con 0x5597468fd1b0 2024-09-21T02:52:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.787+0000 7f793cbfe640 1 -- 172.21.15.29:0/3835946400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7940004050 con 0x5597468fd1b0 2024-09-21T02:52:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.787+0000 7f793cbfe640 1 -- 172.21.15.29:0/3835946400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940005260 con 0x5597468fd1b0 2024-09-21T02:52:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.787+0000 7f793f7fe640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79340b65d0 0x7f79340b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.787+0000 7f794e2f3640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79340b65d0 0x7f79340b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.787+0000 7f793f7fe640 1 -- 172.21.15.29:0/2388413315 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7938107870 con 0x7f7948103650 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 -- 172.21.15.29:0/3835946400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468fd1b0 msgr2=0x559746ca4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 --2- 172.21.15.29:0/3835946400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468fd1b0 0x559746ca4e50 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f7940009fd0 tx=0x7f79400118d0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f794e2f3640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79340b65d0 0x7f79340b8ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7944009da0 tx=0x7f7944005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 -- 172.21.15.29:0/3835946400 shutdown_connections 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 --2- 172.21.15.29:0/3835946400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468dc4d0 0x559746ca6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 --2- 172.21.15.29:0/3835946400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746ca54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 --2- 172.21.15.29:0/3835946400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468fd1b0 0x559746ca4e50 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 -- 172.21.15.29:0/3835946400 >> 172.21.15.29:0/3835946400 conn(0x5597468f1300 msgr2=0x559746ca08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 -- 172.21.15.29:0/3835946400 shutdown_connections 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.788+0000 7f79574c4740 1 -- 172.21.15.29:0/3835946400 wait complete. 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.789+0000 7f79574c4740 1 -- start start 2024-09-21T02:52:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.789+0000 7f79574c4740 1 Processor -- start 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f79574c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746cae660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f79574c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746caed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f79574c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468fd1b0 0x559746caf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f79574c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559746b3eab0 con 0x5597468dc4d0 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f794e2f3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746cae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f794e2f3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746cae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:39364/0 (socket says 172.21.15.29:39364) 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f794e2f3640 1 -- 172.21.15.29:0/3017628245 learned_addr learned my addr 172.21.15.29:0/3017628245 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f794daf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746caed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559746bac730 con 0x5597468d42f0 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559746ca6b30 con 0x5597468fd1b0 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f794eaf4640 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468fd1b0 0x559746caf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.790+0000 7f79574c4740 10 client.0 _ll_register_callbacks cb 0x559746c006b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f794e2f3640 1 -- 172.21.15.29:0/3017628245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468fd1b0 msgr2=0x559746caf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f794e2f3640 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468fd1b0 0x559746caf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f794e2f3640 1 -- 172.21.15.29:0/3017628245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 msgr2=0x559746caed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f794e2f3640 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746caed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f794e2f3640 1 -- 172.21.15.29:0/3017628245 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559746968790 con 0x5597468d42f0 2024-09-21T02:52:55.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7940004050 con 0x5597468d42f0 2024-09-21T02:52:55.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f794e2f3640 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746cae660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f794401e9f0 tx=0x7f7944004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94af56c0b434c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:55.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f7931ffb640 10 client.26735 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:52:55.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.791+0000 7f7931ffb640 1 -- 172.21.15.29:0/3017628245 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794402e060 con 0x5597468d42f0 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.792+0000 7f7931ffb640 1 -- 172.21.15.29:0/3017628245 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7944018030 con 0x5597468d42f0 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.792+0000 7f7931ffb640 1 -- 172.21.15.29:0/3017628245 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7944029420 con 0x5597468d42f0 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.792+0000 7f79574c4740 10 client.26735 fetch_fsmap learned FSMap version 33 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.792+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({fsmap=33}) -- 0x5597469e1620 con 0x5597468d42f0 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.793+0000 7f7931ffb640 1 -- 172.21.15.29:0/3017628245 <== mon.2 v2:172.21.15.66:3300/0 4 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f79440295c0 con 0x5597468d42f0 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.793+0000 7f79574c4740 10 client.26735 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.793+0000 7f79574c4740 10 client.26735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.793+0000 7f79574c4740 20 client.26735 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.793+0000 7f79574c4740 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5597468bbd80 0x559746cbbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.793+0000 7f79574c4740 4 client.26735 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.793+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559746c19640 con 0x5597468bbd80 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.793+0000 7f794daf2640 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5597468bbd80 0x559746cbbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.794+0000 7f7931ffb640 10 client.26735 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.794+0000 7f794daf2640 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5597468bbd80 0x559746cbbcc0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7cb50492adea20f server_cookie=856127d34578d059 in_seq=0 out_seq=0 2024-09-21T02:52:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f7931ffb640 1 -- 172.21.15.29:0/3017628245 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559746c19640 con 0x5597468bbd80 2024-09-21T02:52:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f7931ffb640 10 client.26735 handle_command_reply: tid=0 2024-09-21T02:52:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f79574c4740 1 client.26735 shutdown 2024-09-21T02:52:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 msgr2=0x559746cae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f79574c4740 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746cae660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f794401e9f0 tx=0x7f7944004730 comp rx=0 tx=0).stop 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 shutdown_connections 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f79574c4740 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468fd1b0 0x559746caf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f79574c4740 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5597468bbd80 0x559746cbbcc0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f7931ffb640 0 client.26735 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f79574c4740 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746caed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.798+0000 7f79574c4740 1 --2- 172.21.15.29:0/3017628245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468d42f0 0x559746cae660 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.799+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 >> 172.21.15.29:0/3017628245 conn(0x5597468f1300 msgr2=0x559746c9eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.799+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 shutdown_connections 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.799+0000 7f79574c4740 1 -- 172.21.15.29:0/3017628245 wait complete. 2024-09-21T02:52:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.799+0000 7f79574c4740 20 client.26735 trim_cache size 0 max 16384 2024-09-21T02:52:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.823+0000 7f79574c4740 1 Processor -- start 2024-09-21T02:52:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.823+0000 7f79574c4740 1 -- start start 2024-09-21T02:52:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.823+0000 7f79574c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cad760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f79574c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746c9eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f79574c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746c9f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f79574c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f792c00fc00 con 0x5597468dc4d0 2024-09-21T02:52:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f79574c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f792c011800 con 0x5597468fd1b0 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f79574c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f792c0102d0 con 0x5597468d42f0 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794daf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746c9eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794daf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746c9eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51116/0 (socket says 172.21.15.29:51116) 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794daf2640 1 -- 172.21.15.29:0/2230849844 learned_addr learned my addr 172.21.15.29:0/2230849844 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794eaf4640 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746c9f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794e2f3640 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794daf2640 1 -- 172.21.15.29:0/2230849844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 msgr2=0x559746cad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794daf2640 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794daf2640 1 -- 172.21.15.29:0/2230849844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 msgr2=0x559746c9f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794daf2640 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746c9f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794daf2640 1 -- 172.21.15.29:0/2230849844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559746ca63f0 con 0x5597468dc4d0 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.824+0000 7f794e2f3640 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:52:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.825+0000 7f794daf2640 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746c9eea0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f79380cd430 tx=0x7f793810d550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3555c2594d3eb461 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.825+0000 7f7933fff640 1 -- 172.21.15.29:0/2230849844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938049040 con 0x5597468dc4d0 2024-09-21T02:52:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.825+0000 7f7933fff640 1 -- 172.21.15.29:0/2230849844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f793800b0b0 con 0x5597468dc4d0 2024-09-21T02:52:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.825+0000 7f7933fff640 1 -- 172.21.15.29:0/2230849844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793810ae60 con 0x5597468dc4d0 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 -- 172.21.15.29:0/2230849844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 msgr2=0x559746c9eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746c9eea0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f79380cd430 tx=0x7f793810d550 comp rx=0 tx=0).stop 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 -- 172.21.15.29:0/2230849844 shutdown_connections 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746c9f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746c9eea0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 --2- 172.21.15.29:0/2230849844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 -- 172.21.15.29:0/2230849844 >> 172.21.15.29:0/2230849844 conn(0x5597468f1300 msgr2=0x559746cad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 -- 172.21.15.29:0/2230849844 shutdown_connections 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.826+0000 7f79574c4740 1 -- 172.21.15.29:0/2230849844 wait complete. 2024-09-21T02:52:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 1 -- start start 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 1 Processor -- start 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cb60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746cb4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746cb52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f792c011070 con 0x5597468dc4d0 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f792c0109a0 con 0x5597468fd1b0 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f792c00dbe0 con 0x5597468d42f0 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f79574c4740 10 client.0 _ll_register_callbacks cb 0x7f7944009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f794e2f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cb60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f794e2f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cb60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:59772/0 (socket says 172.21.15.29:59772) 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f794e2f3640 1 -- 172.21.15.29:0/866969636 learned_addr learned my addr 172.21.15.29:0/866969636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f794daf2640 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746cb4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.827+0000 7f794eaf4640 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746cb52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f794daf2640 1 -- 172.21.15.29:0/866969636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 msgr2=0x559746cb60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f794daf2640 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cb60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f794daf2640 1 -- 172.21.15.29:0/866969636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 msgr2=0x559746cb52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f794daf2640 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746cb52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f794daf2640 1 -- 172.21.15.29:0/866969636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559746cb5820 con 0x5597468dc4d0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f794eaf4640 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746cb52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f794daf2640 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746cb4da0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f79380ca9a0 tx=0x7f79380cd260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e390ba745742fd17 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f7931ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f7931ffb640 1 -- 172.21.15.29:0/866969636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793810ae60 con 0x5597468dc4d0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f7931ffb640 1 -- 172.21.15.29:0/866969636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f79380d6c10 con 0x5597468dc4d0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.828+0000 7f79574c4740 1 -- 172.21.15.29:0/866969636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f793800b0b0 con 0x5597468dc4d0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f7931ffb640 1 -- 172.21.15.29:0/866969636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7938049040 con 0x5597468dc4d0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f79574c4740 10 client.17361 fetch_fsmap learned FSMap version 33 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f79574c4740 1 -- 172.21.15.29:0/866969636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559746cb4670 con 0x5597468dc4d0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f7931ffb640 1 -- 172.21.15.29:0/866969636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f793800b0b0 con 0x5597468dc4d0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f7931ffb640 1 -- 172.21.15.29:0/866969636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f793804c060 con 0x5597468dc4d0 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f79574c4740 10 client.17361 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:52:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f79574c4740 10 client.17361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:52:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f79574c4740 20 client.17361 populate_metadata read hostname 'smithi029' 2024-09-21T02:52:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f79574c4740 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5597468911c0 0x559746d28df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:52:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f79574c4740 4 client.17361 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:52:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f79574c4740 1 -- 172.21.15.29:0/866969636 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559746c150f0 con 0x5597468911c0 2024-09-21T02:52:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.829+0000 7f794e2f3640 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5597468911c0 0x559746d28df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:52:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.830+0000 7f7931ffb640 10 client.17361 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.830+0000 7f794e2f3640 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5597468911c0 0x559746d28df0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84bd12af1f394749 server_cookie=a53ca5b2cc54f3c7 in_seq=0 out_seq=0 2024-09-21T02:52:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.998+0000 7f7931ffb640 1 -- 172.21.15.29:0/866969636 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559746c150f0 con 0x5597468911c0 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.998+0000 7f7931ffb640 10 client.17361 handle_command_reply: tid=0 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 client.17361 shutdown 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 -- 172.21.15.29:0/866969636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 msgr2=0x559746cb4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746cb4da0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f79380ca9a0 tx=0x7f79380cd260 comp rx=0 tx=0).stop 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 -- 172.21.15.29:0/866969636 shutdown_connections 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5597468fd1b0 0x559746cb52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5597468911c0 0x559746d28df0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f7931ffb640 0 client.17361 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597468dc4d0 0x559746cb4da0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 --2- 172.21.15.29:0/866969636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597468d42f0 0x559746cb60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 -- 172.21.15.29:0/866969636 >> 172.21.15.29:0/866969636 conn(0x5597468f1300 msgr2=0x559746cada10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 -- 172.21.15.29:0/866969636 shutdown_connections 2024-09-21T02:52:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 1 -- 172.21.15.29:0/866969636 wait complete. 2024-09-21T02:52:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:55.999+0000 7f79574c4740 20 client.17361 trim_cache size 0 max 16384 2024-09-21T02:52:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:52:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:52:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:52:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:52:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.000+0000 7f7950557640 1 -- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79340b65d0 msgr2=0x7f79340b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.000+0000 7f7950557640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79340b65d0 0x7f79340b8ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7944009da0 tx=0x7f7944005f90 comp rx=0 tx=0).stop 2024-09-21T02:52:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.000+0000 7f7950557640 1 -- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7948103650 msgr2=0x7f79481a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:52:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.000+0000 7f7950557640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7948103650 0x7f79481a0d10 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f7938009850 tx=0x7f7938003e50 comp rx=0 tx=0).stop 2024-09-21T02:52:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.001+0000 7f7950557640 1 -- 172.21.15.29:0/2388413315 shutdown_connections 2024-09-21T02:52:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.001+0000 7f7950557640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79340b65d0 0x7f79340b8ae0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.001+0000 7f7950557640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7948103ff0 0x7f79481a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.001+0000 7f7950557640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7948103650 0x7f79481a0d10 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.001+0000 7f7950557640 1 --2- 172.21.15.29:0/2388413315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79481025d0 0x7f79481a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:52:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.001+0000 7f7950557640 1 -- 172.21.15.29:0/2388413315 >> 172.21.15.29:0/2388413315 conn(0x7f79480fdf30 msgr2=0x7f794806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:52:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.001+0000 7f7950557640 1 -- 172.21.15.29:0/2388413315 shutdown_connections 2024-09-21T02:52:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:52:56.001+0000 7f7950557640 1 -- 172.21.15.29:0/2388413315 wait complete. 2024-09-21T02:52:56.062 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:55 smithi066 ceph-mon[33436]: pgmap v803: 129 pgs: 129 active+clean; 254 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 4.6 MiB/s wr, 267 op/s 2024-09-21T02:52:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:55 smithi190 ceph-mon[33871]: pgmap v803: 129 pgs: 129 active+clean; 254 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 4.6 MiB/s wr, 267 op/s 2024-09-21T02:52:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:55 smithi029 ceph-mon[31006]: pgmap v803: 129 pgs: 129 active+clean; 254 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 4.6 MiB/s wr, 267 op/s 2024-09-21T02:52:56.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5067 13.38 MB/sec warmup 33 sec latency 187.124 ms 2024-09-21T02:52:57.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5193 13.30 MB/sec warmup 34 sec latency 320.960 ms 2024-09-21T02:52:58.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:57 smithi190 ceph-mon[33871]: pgmap v804: 129 pgs: 129 active+clean; 259 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 4.1 MiB/s wr, 246 op/s 2024-09-21T02:52:58.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:57 smithi029 ceph-mon[31006]: pgmap v804: 129 pgs: 129 active+clean; 259 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 4.1 MiB/s wr, 246 op/s 2024-09-21T02:52:58.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:57 smithi066 ceph-mon[33436]: pgmap v804: 129 pgs: 129 active+clean; 259 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 4.1 MiB/s wr, 246 op/s 2024-09-21T02:52:58.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5323 13.10 MB/sec warmup 35 sec latency 492.841 ms 2024-09-21T02:52:59.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5465 12.81 MB/sec warmup 36 sec latency 86.147 ms 2024-09-21T02:53:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:52:59 smithi190 ceph-mon[33871]: pgmap v805: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-09-21T02:53:00.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:52:59 smithi029 ceph-mon[31006]: pgmap v805: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-09-21T02:53:00.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:52:59 smithi066 ceph-mon[33436]: pgmap v805: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-09-21T02:53:00.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5650 13.03 MB/sec warmup 37 sec latency 97.882 ms 2024-09-21T02:53:01.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5795 12.82 MB/sec warmup 38 sec latency 126.534 ms 2024-09-21T02:53:02.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:01 smithi190 ceph-mon[33871]: pgmap v806: 129 pgs: 129 active+clean; 241 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.6 MiB/s wr, 265 op/s 2024-09-21T02:53:02.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:01 smithi029 ceph-mon[31006]: pgmap v806: 129 pgs: 129 active+clean; 241 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.6 MiB/s wr, 265 op/s 2024-09-21T02:53:02.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:01 smithi066 ceph-mon[33436]: pgmap v806: 129 pgs: 129 active+clean; 241 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.6 MiB/s wr, 265 op/s 2024-09-21T02:53:02.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5947 12.55 MB/sec warmup 39 sec latency 85.940 ms 2024-09-21T02:53:03.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6104 12.38 MB/sec warmup 40 sec latency 146.211 ms 2024-09-21T02:53:04.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:03 smithi190 ceph-mon[33871]: pgmap v807: 129 pgs: 129 active+clean; 237 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 3.7 MiB/s wr, 276 op/s 2024-09-21T02:53:04.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:03 smithi029 ceph-mon[31006]: pgmap v807: 129 pgs: 129 active+clean; 237 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 3.7 MiB/s wr, 276 op/s 2024-09-21T02:53:04.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:03 smithi066 ceph-mon[33436]: pgmap v807: 129 pgs: 129 active+clean; 237 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 3.7 MiB/s wr, 276 op/s 2024-09-21T02:53:04.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6254 12.18 MB/sec warmup 41 sec latency 246.771 ms 2024-09-21T02:53:05.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6384 11.92 MB/sec warmup 42 sec latency 87.814 ms 2024-09-21T02:53:06.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:05 smithi190 ceph-mon[33871]: pgmap v808: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 3.4 MiB/s wr, 314 op/s 2024-09-21T02:53:06.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:05 smithi029 ceph-mon[31006]: pgmap v808: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 3.4 MiB/s wr, 314 op/s 2024-09-21T02:53:06.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:05 smithi066 ceph-mon[33436]: pgmap v808: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 3.4 MiB/s wr, 314 op/s 2024-09-21T02:53:06.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6567 11.92 MB/sec warmup 43 sec latency 126.600 ms 2024-09-21T02:53:07.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6703 11.75 MB/sec warmup 44 sec latency 152.849 ms 2024-09-21T02:53:08.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:07 smithi190 ceph-mon[33871]: pgmap v809: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 3.1 MiB/s wr, 298 op/s 2024-09-21T02:53:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:07 smithi029 ceph-mon[31006]: pgmap v809: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 3.1 MiB/s wr, 298 op/s 2024-09-21T02:53:08.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:07 smithi066 ceph-mon[33436]: pgmap v809: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 3.1 MiB/s wr, 298 op/s 2024-09-21T02:53:08.552 INFO:tasks.quiescer.fs.[cephfs]:Total duration (37.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:53:08.552 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.85 2024-09-21T02:53:08.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:53:08.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6832 11.65 MB/sec warmup 45 sec latency 227.810 ms 2024-09-21T02:53:08.876 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:53:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 -- 172.21.15.29:0/936400951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31281025b0 msgr2=0x7f3128102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 --2- 172.21.15.29:0/936400951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31281025b0 0x7f3128102a20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3118009860 tx=0x7f311802a340 comp rx=0 tx=0).stop 2024-09-21T02:53:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 -- 172.21.15.29:0/936400951 shutdown_connections 2024-09-21T02:53:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 --2- 172.21.15.29:0/936400951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3128103fd0 0x7f312810a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 --2- 172.21.15.29:0/936400951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3128103630 0x7f3128103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 --2- 172.21.15.29:0/936400951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31281025b0 0x7f3128102a20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 -- 172.21.15.29:0/936400951 >> 172.21.15.29:0/936400951 conn(0x7f31280fdf30 msgr2=0x7f31281003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 -- 172.21.15.29:0/936400951 shutdown_connections 2024-09-21T02:53:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.150+0000 7f312d67e640 1 -- 172.21.15.29:0/936400951 wait complete. 2024-09-21T02:53:09.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.151+0000 7f312d67e640 1 Processor -- start 2024-09-21T02:53:09.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.151+0000 7f312d67e640 1 -- start start 2024-09-21T02:53:09.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.151+0000 7f312d67e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31281025b0 0x7f312810cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.151+0000 7f312d67e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3128103630 0x7f312810b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.151+0000 7f312d67e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3128103fd0 0x7f312810bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f312d67e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f312810f1f0 con 0x7f31281025b0 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f312d67e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f312810f070 con 0x7f3128103fd0 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f312d67e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f312810f370 con 0x7f3128103630 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f3126ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31281025b0 0x7f312810cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f31277fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3128103fd0 0x7f312810bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f311ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3128103630 0x7f312810b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f3126ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31281025b0 0x7f312810cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59646/0 (socket says 172.21.15.29:59646) 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f3126ffd640 1 -- 172.21.15.29:0/1240804493 learned_addr learned my addr 172.21.15.29:0/1240804493 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f31277fe640 1 -- 172.21.15.29:0/1240804493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3128103630 msgr2=0x7f312810b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f31277fe640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3128103630 0x7f312810b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f31277fe640 1 -- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31281025b0 msgr2=0x7f312810cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f31277fe640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31281025b0 0x7f312810cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f31277fe640 1 -- 172.21.15.29:0/1240804493 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312810c250 con 0x7f3128103fd0 2024-09-21T02:53:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.152+0000 7f31277fe640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3128103fd0 0x7f312810bb20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f312000ef00 tx=0x7f312000c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.153+0000 7f3124ff9640 1 -- 172.21.15.29:0/1240804493 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120004b20 con 0x7f3128103fd0 2024-09-21T02:53:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.153+0000 7f312d67e640 1 -- 172.21.15.29:0/1240804493 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31281a3ef0 con 0x7f3128103fd0 2024-09-21T02:53:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.153+0000 7f3124ff9640 1 -- 172.21.15.29:0/1240804493 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3120003e00 con 0x7f3128103fd0 2024-09-21T02:53:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.153+0000 7f312d67e640 1 -- 172.21.15.29:0/1240804493 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31281a4300 con 0x7f3128103fd0 2024-09-21T02:53:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.153+0000 7f3124ff9640 1 -- 172.21.15.29:0/1240804493 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120004820 con 0x7f3128103fd0 2024-09-21T02:53:09.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.156+0000 7f3124ff9640 1 -- 172.21.15.29:0/1240804493 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f312001d020 con 0x7f3128103fd0 2024-09-21T02:53:09.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.156+0000 7f31345e4740 1 Processor -- start 2024-09-21T02:53:09.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.156+0000 7f31345e4740 1 -- start start 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f31345e4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 0x56055e19fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f31345e4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f31345e4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f31345e4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56055e1126f0 con 0x56055ddcf2f0 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f31345e4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56055e101b20 con 0x56055ddd74d0 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f31345e4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56055e105930 con 0x56055ddf81b0 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f3126ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:34080/0 (socket says 172.21.15.29:34080) 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 -- 172.21.15.29:0/3849747621 learned_addr learned my addr 172.21.15.29:0/3849747621 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 --2- 172.21.15.29:0/3849747621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 0x56055e19fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 -- 172.21.15.29:0/3849747621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 msgr2=0x56055e19fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 --2- 172.21.15.29:0/3849747621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 0x56055e19fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 -- 172.21.15.29:0/3849747621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 msgr2=0x56055e1a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 --2- 172.21.15.29:0/3849747621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.157+0000 7f311ffff640 1 -- 172.21.15.29:0/3849747621 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055e0c7530 con 0x56055ddd74d0 2024-09-21T02:53:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.158+0000 7f3124ff9640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31140b6590 0x7f31140b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.158+0000 7f311ffff640 1 --2- 172.21.15.29:0/3849747621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a1010 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f31100093e0 tx=0x7f3110009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=560fd137f8a38110 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.158+0000 7f3126ffd640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31140b6590 0x7f31140b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.158+0000 7f311d7fa640 1 -- 172.21.15.29:0/3849747621 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110004050 con 0x56055ddd74d0 2024-09-21T02:53:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.158+0000 7f311d7fa640 1 -- 172.21.15.29:0/3849747621 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f311000f7c0 con 0x56055ddd74d0 2024-09-21T02:53:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.158+0000 7f311d7fa640 1 -- 172.21.15.29:0/3849747621 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110003850 con 0x56055ddd74d0 2024-09-21T02:53:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.158+0000 7f3124ff9640 1 -- 172.21.15.29:0/1240804493 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f31200e7450 con 0x7f3128103fd0 2024-09-21T02:53:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 -- 172.21.15.29:0/3849747621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 msgr2=0x56055e1a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 --2- 172.21.15.29:0/3849747621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a1010 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f31100093e0 tx=0x7f3110009370 comp rx=0 tx=0).stop 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f3126ffd640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31140b6590 0x7f31140b8aa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3118005e00 tx=0x7f31180058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 -- 172.21.15.29:0/3849747621 shutdown_connections 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 --2- 172.21.15.29:0/3849747621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a1010 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 --2- 172.21.15.29:0/3849747621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 --2- 172.21.15.29:0/3849747621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 0x56055e19fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 -- 172.21.15.29:0/3849747621 >> 172.21.15.29:0/3849747621 conn(0x56055ddec300 msgr2=0x56055e19b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 -- 172.21.15.29:0/3849747621 shutdown_connections 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.159+0000 7f31345e4740 1 -- 172.21.15.29:0/3849747621 wait complete. 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.160+0000 7f31345e4740 1 -- start start 2024-09-21T02:53:09.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.161+0000 7f31345e4740 1 Processor -- start 2024-09-21T02:53:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.161+0000 7f31345e4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.161+0000 7f31345e4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.161+0000 7f3126ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.161+0000 7f3126ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59664/0 (socket says 172.21.15.29:59664) 2024-09-21T02:53:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.161+0000 7f3126ffd640 1 -- 172.21.15.29:0/432047579 learned_addr learned my addr 172.21.15.29:0/432047579 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.162+0000 7f31345e4740 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 0x56055e1aa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.162+0000 7f311ffff640 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.162+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56055e039ab0 con 0x56055ddcf2f0 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.162+0000 7f31277fe640 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 0x56055e1aa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.162+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56055e0a7730 con 0x56055ddd74d0 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.162+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56055e1a1b30 con 0x56055ddf81b0 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.163+0000 7f31345e4740 10 client.0 _ll_register_callbacks cb 0x56055e0fb6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.163+0000 7f311ffff640 1 -- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 msgr2=0x56055e1aa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.163+0000 7f311ffff640 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 0x56055e1aa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.163+0000 7f311ffff640 1 -- 172.21.15.29:0/432047579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 msgr2=0x56055e1a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.163+0000 7f311ffff640 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.163+0000 7f311ffff640 1 -- 172.21.15.29:0/432047579 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055de63790 con 0x56055ddd74d0 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.163+0000 7f311ffff640 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a9d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f31100093e0 tx=0x7f311000fdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86baeae7983eb329 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.163+0000 7f310e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.164+0000 7f310e7fc640 1 -- 172.21.15.29:0/432047579 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31100035f0 con 0x56055ddd74d0 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.164+0000 7f310e7fc640 1 -- 172.21.15.29:0/432047579 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3110003790 con 0x56055ddd74d0 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.164+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f311000f7c0 con 0x56055ddd74d0 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.164+0000 7f310e7fc640 1 -- 172.21.15.29:0/432047579 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3110030050 con 0x56055ddd74d0 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.164+0000 7f31345e4740 10 client.26753 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.164+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56055dedc620 con 0x56055ddd74d0 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.164+0000 7f310e7fc640 1 -- 172.21.15.29:0/432047579 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f311000f7c0 con 0x56055ddd74d0 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.165+0000 7f310e7fc640 1 -- 172.21.15.29:0/432047579 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f311001f8d0 con 0x56055ddd74d0 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.165+0000 7f31345e4740 10 client.26753 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.165+0000 7f31345e4740 10 client.26753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.165+0000 7f31345e4740 20 client.26753 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.165+0000 7f31345e4740 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56055ddb6d80 0x56055e1b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.165+0000 7f31345e4740 4 client.26753 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:53:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.165+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56055e114640 con 0x56055ddb6d80 2024-09-21T02:53:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.165+0000 7f3126ffd640 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56055ddb6d80 0x56055e1b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.166+0000 7f310e7fc640 10 client.26753 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.166+0000 7f3126ffd640 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56055ddb6d80 0x56055e1b6b90 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc4d795893806d12 server_cookie=720f2dcf3cd629b9 in_seq=0 out_seq=0 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f310e7fc640 1 -- 172.21.15.29:0/432047579 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56055e114640 con 0x56055ddb6d80 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f310e7fc640 10 client.26753 handle_command_reply: tid=0 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 client.26753 shutdown 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 msgr2=0x56055e1a9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a9d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f31100093e0 tx=0x7f311000fdc0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 shutdown_connections 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddf81b0 0x56055e1aa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddd74d0 0x56055e1a9d80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56055ddb6d80 0x56055e1b6b90 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f310e7fc640 0 client.26753 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 --2- 172.21.15.29:0/432047579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddcf2f0 0x56055e1a96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 >> 172.21.15.29:0/432047579 conn(0x56055ddec300 msgr2=0x56055e199ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 shutdown_connections 2024-09-21T02:53:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 1 -- 172.21.15.29:0/432047579 wait complete. 2024-09-21T02:53:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.167+0000 7f31345e4740 20 client.26753 trim_cache size 0 max 16384 2024-09-21T02:53:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.195+0000 7f31345e4740 1 Processor -- start 2024-09-21T02:53:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.195+0000 7f31345e4740 1 -- start start 2024-09-21T02:53:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f31345e4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddb6d80 0x56055e1a8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f31345e4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddcf2f0 0x56055e1b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f31345e4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddd74d0 0x56055e1a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f31345e4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f31281ae720 con 0x56055ddb6d80 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f311ffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddcf2f0 0x56055e1b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f311ffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddcf2f0 0x56055e1b17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:34100/0 (socket says 172.21.15.29:34100) 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f311ffff640 1 -- 172.21.15.29:0/3353679480 learned_addr learned my addr 172.21.15.29:0/3353679480 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f31277fe640 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddd74d0 0x56055e1a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f31345e4740 1 -- 172.21.15.29:0/3353679480 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f31281b0320 con 0x56055ddcf2f0 2024-09-21T02:53:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.196+0000 7f31345e4740 1 -- 172.21.15.29:0/3353679480 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f31281aedf0 con 0x56055ddd74d0 2024-09-21T02:53:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.197+0000 7f3126ffd640 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddb6d80 0x56055e1a8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.197+0000 7f31277fe640 1 -- 172.21.15.29:0/3353679480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddcf2f0 msgr2=0x56055e1b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.197+0000 7f31277fe640 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddcf2f0 0x56055e1b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.197+0000 7f31277fe640 1 -- 172.21.15.29:0/3353679480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddb6d80 msgr2=0x56055e1a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.197+0000 7f31277fe640 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddb6d80 0x56055e1a8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.197+0000 7f31277fe640 1 -- 172.21.15.29:0/3353679480 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055e1a4ef0 con 0x56055ddd74d0 2024-09-21T02:53:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.197+0000 7f3126ffd640 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddb6d80 0x56055e1a8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:53:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.197+0000 7f31277fe640 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddd74d0 0x56055e1a96c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f31200e4bd0 tx=0x7f31200a9230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56fb733150a300a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.198+0000 7f311cff9640 1 -- 172.21.15.29:0/3353679480 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312001a040 con 0x56055ddd74d0 2024-09-21T02:53:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.198+0000 7f311cff9640 1 -- 172.21.15.29:0/3353679480 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31200ad960 con 0x56055ddd74d0 2024-09-21T02:53:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.198+0000 7f311cff9640 1 -- 172.21.15.29:0/3353679480 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31200ac4c0 con 0x56055ddd74d0 2024-09-21T02:53:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.199+0000 7f31345e4740 1 -- 172.21.15.29:0/3353679480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddd74d0 msgr2=0x56055e1a96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.199+0000 7f31345e4740 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddd74d0 0x56055e1a96c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f31200e4bd0 tx=0x7f31200a9230 comp rx=0 tx=0).stop 2024-09-21T02:53:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.199+0000 7f31345e4740 1 -- 172.21.15.29:0/3353679480 shutdown_connections 2024-09-21T02:53:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.199+0000 7f31345e4740 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddd74d0 0x56055e1a96c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.199+0000 7f31345e4740 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddcf2f0 0x56055e1b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.199+0000 7f31345e4740 1 --2- 172.21.15.29:0/3353679480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddb6d80 0x56055e1a8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.199+0000 7f31345e4740 1 -- 172.21.15.29:0/3353679480 >> 172.21.15.29:0/3353679480 conn(0x56055ddec300 msgr2=0x56055e1a7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.199+0000 7f31345e4740 1 -- 172.21.15.29:0/3353679480 shutdown_connections 2024-09-21T02:53:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.200+0000 7f31345e4740 1 -- 172.21.15.29:0/3353679480 wait complete. 2024-09-21T02:53:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.200+0000 7f31345e4740 1 -- start start 2024-09-21T02:53:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.200+0000 7f31345e4740 1 Processor -- start 2024-09-21T02:53:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f31345e4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddb6d80 0x56055e1ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f31345e4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddcf2f0 0x56055e1aec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f31345e4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddd74d0 0x56055e1af800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f31345e4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f31281afb90 con 0x56055ddd74d0 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f31345e4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f31281af4c0 con 0x56055ddb6d80 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f31345e4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f31281ac700 con 0x56055ddcf2f0 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f31345e4740 10 client.0 _ll_register_callbacks cb 0x7f3110005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddb6d80 0x56055e1ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f311ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddcf2f0 0x56055e1aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddb6d80 0x56055e1ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:34104/0 (socket says 172.21.15.29:34104) 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 -- 172.21.15.29:0/4005144209 learned_addr learned my addr 172.21.15.29:0/4005144209 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f31277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddd74d0 0x56055e1af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 -- 172.21.15.29:0/4005144209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddcf2f0 msgr2=0x56055e1aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddcf2f0 0x56055e1aec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 -- 172.21.15.29:0/4005144209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddd74d0 msgr2=0x56055e1af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddd74d0 0x56055e1af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 -- 172.21.15.29:0/4005144209 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055e1aff60 con 0x56055ddb6d80 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f3126ffd640 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddb6d80 0x56055e1ae5d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3118005cd0 tx=0x7f311802a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2bfa4783c91f8e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.201+0000 7f310e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.202+0000 7f310e7fc640 1 -- 172.21.15.29:0/4005144209 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311805f060 con 0x56055ddb6d80 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.202+0000 7f310e7fc640 1 -- 172.21.15.29:0/4005144209 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f31180041c0 con 0x56055ddb6d80 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.202+0000 7f31345e4740 1 -- 172.21.15.29:0/4005144209 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31200ad960 con 0x56055ddb6d80 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.202+0000 7f310e7fc640 1 -- 172.21.15.29:0/4005144209 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3118066050 con 0x56055ddb6d80 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.202+0000 7f31345e4740 10 client.26759 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.202+0000 7f31345e4740 1 -- 172.21.15.29:0/4005144209 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56055e1b12f0 con 0x56055ddb6d80 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.202+0000 7f310e7fc640 1 -- 172.21.15.29:0/4005144209 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31200ad960 con 0x56055ddb6d80 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.202+0000 7f310e7fc640 1 -- 172.21.15.29:0/4005144209 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f311803d040 con 0x56055ddb6d80 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f31345e4740 10 client.26759 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f31345e4740 10 client.26759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:53:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f31345e4740 20 client.26759 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f31345e4740 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56055ddf81b0 0x56055e2235d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f31345e4740 4 client.26759 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:53:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f31345e4740 1 -- 172.21.15.29:0/4005144209 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56055e1100f0 con 0x56055ddf81b0 2024-09-21T02:53:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f311ffff640 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56055ddf81b0 0x56055e2235d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f310e7fc640 10 client.26759 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.203+0000 7f311ffff640 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56055ddf81b0 0x56055e2235d0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e810c05ca74ab06a server_cookie=78f3fd15d10b2eac in_seq=0 out_seq=0 2024-09-21T02:53:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f310e7fc640 1 -- 172.21.15.29:0/4005144209 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56055e1100f0 con 0x56055ddf81b0 2024-09-21T02:53:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f310e7fc640 10 client.26759 handle_command_reply: tid=0 2024-09-21T02:53:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 client.26759 shutdown 2024-09-21T02:53:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 -- 172.21.15.29:0/4005144209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddb6d80 msgr2=0x56055e1ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddb6d80 0x56055e1ae5d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3118005cd0 tx=0x7f311802a340 comp rx=0 tx=0).stop 2024-09-21T02:53:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 -- 172.21.15.29:0/4005144209 shutdown_connections 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56055ddf81b0 0x56055e2235d0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56055ddd74d0 0x56055e1af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56055ddcf2f0 0x56055e1aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 --2- 172.21.15.29:0/4005144209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56055ddb6d80 0x56055e1ae5d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f31345e4740 1 -- 172.21.15.29:0/4005144209 >> 172.21.15.29:0/4005144209 conn(0x56055ddec300 msgr2=0x56055e1a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.349+0000 7f310e7fc640 0 client.26759 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.350+0000 7f31345e4740 1 -- 172.21.15.29:0/4005144209 shutdown_connections 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.350+0000 7f31345e4740 1 -- 172.21.15.29:0/4005144209 wait complete. 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.350+0000 7f31345e4740 20 client.26759 trim_cache size 0 max 16384 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:53:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:53:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:53:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.352+0000 7f312d67e640 1 -- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31140b6590 msgr2=0x7f31140b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.352+0000 7f312d67e640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31140b6590 0x7f31140b8aa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3118005e00 tx=0x7f31180058b0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.352+0000 7f312d67e640 1 -- 172.21.15.29:0/1240804493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3128103fd0 msgr2=0x7f312810bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.352+0000 7f312d67e640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3128103fd0 0x7f312810bb20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f312000ef00 tx=0x7f312000c740 comp rx=0 tx=0).stop 2024-09-21T02:53:09.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.354+0000 7f312d67e640 1 -- 172.21.15.29:0/1240804493 shutdown_connections 2024-09-21T02:53:09.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.354+0000 7f312d67e640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f31140b6590 0x7f31140b8aa0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.354+0000 7f312d67e640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3128103fd0 0x7f312810bb20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.354+0000 7f312d67e640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3128103630 0x7f312810b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.354+0000 7f312d67e640 1 --2- 172.21.15.29:0/1240804493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31281025b0 0x7f312810cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:09.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.354+0000 7f312d67e640 1 -- 172.21.15.29:0/1240804493 >> 172.21.15.29:0/1240804493 conn(0x7f31280fdf30 msgr2=0x7f3128106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:09.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.354+0000 7f312d67e640 1 -- 172.21.15.29:0/1240804493 shutdown_connections 2024-09-21T02:53:09.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:09.354+0000 7f312d67e640 1 -- 172.21.15.29:0/1240804493 wait complete. 2024-09-21T02:53:09.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6918 11.44 MB/sec warmup 46 sec latency 236.475 ms 2024-09-21T02:53:10.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:09 smithi029 ceph-mon[31006]: pgmap v810: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-09-21T02:53:10.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:09 smithi190 ceph-mon[33871]: pgmap v810: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-09-21T02:53:10.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:09 smithi066 ceph-mon[33436]: pgmap v810: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-09-21T02:53:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 -- 172.21.15.29:0/183700812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc22810de60 msgr2=0x7fc228110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 --2- 172.21.15.29:0/183700812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc22810de60 0x7fc228110330 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc220009ca0 tx=0x7fc22002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 -- 172.21.15.29:0/183700812 shutdown_connections 2024-09-21T02:53:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 --2- 172.21.15.29:0/183700812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc22810de60 0x7fc228110330 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 --2- 172.21.15.29:0/183700812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc228105380 0x7fc22810d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 --2- 172.21.15.29:0/183700812 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc228104a30 0x7fc228104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 -- 172.21.15.29:0/183700812 >> 172.21.15.29:0/183700812 conn(0x7fc228079790 msgr2=0x7fc228079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 -- 172.21.15.29:0/183700812 shutdown_connections 2024-09-21T02:53:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.356+0000 7fc22d399640 1 -- 172.21.15.29:0/183700812 wait complete. 2024-09-21T02:53:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.357+0000 7fc22d399640 1 Processor -- start 2024-09-21T02:53:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.357+0000 7fc22d399640 1 -- start start 2024-09-21T02:53:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.357+0000 7fc22d399640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc228104a30 0x7fc228102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.357+0000 7fc22d399640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc228105380 0x7fc228102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.357+0000 7fc22d399640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc22810de60 0x7fc2281031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.357+0000 7fc22d399640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc2281139a0 con 0x7fc228104a30 2024-09-21T02:53:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.357+0000 7fc22d399640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc228113820 con 0x7fc228105380 2024-09-21T02:53:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.357+0000 7fc22d399640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc228113b20 con 0x7fc22810de60 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc2267fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc228105380 0x7fc228102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc226ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc228104a30 0x7fc228102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc226ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc228104a30 0x7fc228102770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59700/0 (socket says 172.21.15.29:59700) 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc226ffd640 1 -- 172.21.15.29:0/941244933 learned_addr learned my addr 172.21.15.29:0/941244933 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc2277fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc22810de60 0x7fc2281031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc2267fc640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc228105380 0x7fc228102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:34120/0 (socket says 172.21.15.29:34120) 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc226ffd640 1 -- 172.21.15.29:0/941244933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc22810de60 msgr2=0x7fc2281031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc226ffd640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc22810de60 0x7fc2281031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc226ffd640 1 -- 172.21.15.29:0/941244933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc228105380 msgr2=0x7fc228102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc226ffd640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc228105380 0x7fc228102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.358+0000 7fc226ffd640 1 -- 172.21.15.29:0/941244933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2281a85f0 con 0x7fc228104a30 2024-09-21T02:53:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.359+0000 7fc226ffd640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc228104a30 0x7fc228102770 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc21800e9f0 tx=0x7fc21800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.359+0000 7fc20ffff640 1 -- 172.21.15.29:0/941244933 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21800cd80 con 0x7fc228104a30 2024-09-21T02:53:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.359+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2281a88e0 con 0x7fc228104a30 2024-09-21T02:53:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.359+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2281a8e70 con 0x7fc228104a30 2024-09-21T02:53:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.359+0000 7fc20ffff640 1 -- 172.21.15.29:0/941244933 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2180108c0 con 0x7fc228104a30 2024-09-21T02:53:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.359+0000 7fc20ffff640 1 -- 172.21.15.29:0/941244933 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218018780 con 0x7fc228104a30 2024-09-21T02:53:10.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.360+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc228104e40 con 0x7fc228104a30 2024-09-21T02:53:10.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.363+0000 7fc20ffff640 1 -- 172.21.15.29:0/941244933 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc218010a70 con 0x7fc228104a30 2024-09-21T02:53:10.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.364+0000 7fc20ffff640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2100b6590 0x7fc2100b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:10.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.365+0000 7fc2267fc640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2100b6590 0x7fc2100b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:10.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.365+0000 7fc20ffff640 1 -- 172.21.15.29:0/941244933 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc2180e7d90 con 0x7fc228104a30 2024-09-21T02:53:10.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.365+0000 7fc2267fc640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2100b6590 0x7fc2100b8aa0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc22810fcd0 tx=0x7fc21c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:10.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.369+0000 7fc20ffff640 1 -- 172.21.15.29:0/941244933 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc2180b2680 con 0x7fc228104a30 2024-09-21T02:53:10.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7102 11.63 MB/sec warmup 47 sec latency 260.328 ms 2024-09-21T02:53:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.761+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc228103a70 con 0x7fc228104a30 2024-09-21T02:53:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.762+0000 7fc20ffff640 1 -- 172.21.15.29:0/941244933 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11979 (secure 0 0 0) 0x7fc2180b75c0 con 0x7fc228104a30 2024-09-21T02:53:10.764 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:53:10.764 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"btime":"2024-09-21T02:52:29:480744+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:52:28.665505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:53:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2024-09-21T02:53:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2100b6590 msgr2=0x7fc2100b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2100b6590 0x7fc2100b8aa0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc22810fcd0 tx=0x7fc21c005f90 comp rx=0 tx=0).stop 2024-09-21T02:53:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc228104a30 msgr2=0x7fc228102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc228104a30 0x7fc228102770 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc21800e9f0 tx=0x7fc21800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 shutdown_connections 2024-09-21T02:53:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc2100b6590 0x7fc2100b8aa0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc22810de60 0x7fc2281031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc228105380 0x7fc228102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 --2- 172.21.15.29:0/941244933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc228104a30 0x7fc228102770 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 >> 172.21.15.29:0/941244933 conn(0x7fc228079790 msgr2=0x7fc228100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 shutdown_connections 2024-09-21T02:53:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:10.766+0000 7fc22d399640 1 -- 172.21.15.29:0/941244933 wait complete. 2024-09-21T02:53:11.079 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:10 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/941244933' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:11.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:10 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/941244933' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:11.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:10 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/941244933' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:11.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7292 11.69 MB/sec warmup 48 sec latency 96.954 ms 2024-09-21T02:53:11.865 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 138.85 --await-for 45.0' 2024-09-21T02:53:11.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 138.85 --await-for 45.0 2024-09-21T02:53:12.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:11 smithi190 ceph-mon[33871]: pgmap v811: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-09-21T02:53:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:11 smithi029 ceph-mon[31006]: pgmap v811: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-09-21T02:53:12.249 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:53:12.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:11 smithi066 ceph-mon[33436]: pgmap v811: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-09-21T02:53:12.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7540 11.83 MB/sec warmup 49 sec latency 112.723 ms 2024-09-21T02:53:12.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.841+0000 7f1cff56d640 1 -- 172.21.15.29:0/3334828448 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf8103bb0 msgr2=0x7f1cf8103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.841+0000 7f1cff56d640 1 --2- 172.21.15.29:0/3334828448 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf8103bb0 0x7f1cf8103fc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009900 tx=0x7f1cf402a410 comp rx=0 tx=0).stop 2024-09-21T02:53:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.841+0000 7f1cff56d640 1 -- 172.21.15.29:0/3334828448 shutdown_connections 2024-09-21T02:53:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.841+0000 7f1cff56d640 1 --2- 172.21.15.29:0/3334828448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf8109e40 0x7f1cf8110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.841+0000 7f1cff56d640 1 --2- 172.21.15.29:0/3334828448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf81094f0 0x7f1cf8109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.841+0000 7f1cff56d640 1 --2- 172.21.15.29:0/3334828448 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf8103bb0 0x7f1cf8103fc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.841+0000 7f1cff56d640 1 -- 172.21.15.29:0/3334828448 >> 172.21.15.29:0/3334828448 conn(0x7f1cf80ff600 msgr2=0x7f1cf8101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.842+0000 7f1cff56d640 1 -- 172.21.15.29:0/3334828448 shutdown_connections 2024-09-21T02:53:12.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.842+0000 7f1cff56d640 1 -- 172.21.15.29:0/3334828448 wait complete. 2024-09-21T02:53:12.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cff56d640 1 Processor -- start 2024-09-21T02:53:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cff56d640 1 -- start start 2024-09-21T02:53:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cff56d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf8103bb0 0x7f1cf8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cff56d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf81094f0 0x7f1cf8075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cff56d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf8109e40 0x7f1cf8076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cfd309640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf8103bb0 0x7f1cf8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cfd309640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf8103bb0 0x7f1cf8075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48084/0 (socket says 172.21.15.29:48084) 2024-09-21T02:53:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cfdb0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf8109e40 0x7f1cf8076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf81094f0 0x7f1cf8075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf81094f0 0x7f1cf8075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:34138/0 (socket says 172.21.15.29:34138) 2024-09-21T02:53:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cfdb0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf8109e40 0x7f1cf8076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59718/0 (socket says 172.21.15.29:59718) 2024-09-21T02:53:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cfcb08640 1 -- 172.21.15.29:0/821564280 learned_addr learned my addr 172.21.15.29:0/821564280 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf8113a00 con 0x7f1cf8109e40 2024-09-21T02:53:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf8113880 con 0x7f1cf81094f0 2024-09-21T02:53:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.843+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf8113b80 con 0x7f1cf8103bb0 2024-09-21T02:53:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.844+0000 7f1cfdb0a640 1 -- 172.21.15.29:0/821564280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf8103bb0 msgr2=0x7f1cf8075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.844+0000 7f1cfdb0a640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf8103bb0 0x7f1cf8075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.844+0000 7f1cfdb0a640 1 -- 172.21.15.29:0/821564280 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf81094f0 msgr2=0x7f1cf8075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.844+0000 7f1cfdb0a640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf81094f0 0x7f1cf8075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.844+0000 7f1cfdb0a640 1 -- 172.21.15.29:0/821564280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf81a84a0 con 0x7f1cf8109e40 2024-09-21T02:53:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.844+0000 7f1cfdb0a640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf8109e40 0x7f1cf8076190 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000b540 tx=0x7f1cf000ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.844+0000 7f1cee7fc640 1 -- 172.21.15.29:0/821564280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0004b20 con 0x7f1cf8109e40 2024-09-21T02:53:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.844+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf8199f50 con 0x7f1cf8109e40 2024-09-21T02:53:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.845+0000 7f1cee7fc640 1 -- 172.21.15.29:0/821564280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cf0004cc0 con 0x7f1cf8109e40 2024-09-21T02:53:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.845+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf819a460 con 0x7f1cf8109e40 2024-09-21T02:53:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.845+0000 7f1cee7fc640 1 -- 172.21.15.29:0/821564280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf001d950 con 0x7f1cf8109e40 2024-09-21T02:53:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.848+0000 7f1d05ae1740 1 Processor -- start 2024-09-21T02:53:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1cee7fc640 1 -- 172.21.15.29:0/821564280 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f1cf0024020 con 0x7f1cf8109e40 2024-09-21T02:53:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1d05ae1740 1 -- start start 2024-09-21T02:53:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab84e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7b5990 0x55ee2ab853b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab858f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ee2aafbbb0 con 0x55ee2a7bdb70 2024-09-21T02:53:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55ee2aaeafe0 con 0x55ee2a7b5990 2024-09-21T02:53:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ee2aaeedf0 con 0x55ee2a7de850 2024-09-21T02:53:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab84e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab84e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48100/0 (socket says 172.21.15.29:48100) 2024-09-21T02:53:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1cfcb08640 1 -- 172.21.15.29:0/422349948 learned_addr learned my addr 172.21.15.29:0/422349948 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab858f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.849+0000 7f1cfd309640 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7b5990 0x55ee2ab853b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cee7fc640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1ce40b6510 0x7f1ce40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cfd309640 1 -- 172.21.15.29:0/422349948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 msgr2=0x55ee2ab84e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cfd309640 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab84e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cfd309640 1 -- 172.21.15.29:0/422349948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7bdb70 msgr2=0x55ee2ab858f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cfd309640 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab858f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cfd309640 1 -- 172.21.15.29:0/422349948 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee2a858ba0 con 0x55ee2a7b5990 2024-09-21T02:53:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cee7fc640 1 -- 172.21.15.29:0/821564280 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1cf00ef7a0 con 0x7f1cf8109e40 2024-09-21T02:53:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cfdb0a640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1ce40b6510 0x7f1ce40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.850+0000 7f1cfd309640 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7b5990 0x55ee2ab853b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009850 tx=0x7f1cf400b050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b55137f0f924f75 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.851+0000 7f1cfdb0a640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1ce40b6510 0x7f1ce40b8a20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1cf00b5250 tx=0x7f1cf0005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.851+0000 7f1ce37fe640 1 -- 172.21.15.29:0/422349948 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4035070 con 0x55ee2a7b5990 2024-09-21T02:53:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.851+0000 7f1ce37fe640 1 -- 172.21.15.29:0/422349948 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cf4040030 con 0x55ee2a7b5990 2024-09-21T02:53:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1ce37fe640 1 -- 172.21.15.29:0/422349948 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf403c6a0 con 0x55ee2a7b5990 2024-09-21T02:53:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 -- 172.21.15.29:0/422349948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7b5990 msgr2=0x55ee2ab853b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7b5990 0x55ee2ab853b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009850 tx=0x7f1cf400b050 comp rx=0 tx=0).stop 2024-09-21T02:53:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 -- 172.21.15.29:0/422349948 shutdown_connections 2024-09-21T02:53:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab858f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7b5990 0x55ee2ab853b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/422349948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab84e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 -- 172.21.15.29:0/422349948 >> 172.21.15.29:0/422349948 conn(0x55ee2a7d2c40 msgr2=0x55ee2ab82d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:12.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 -- 172.21.15.29:0/422349948 shutdown_connections 2024-09-21T02:53:12.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.852+0000 7f1d05ae1740 1 -- 172.21.15.29:0/422349948 wait complete. 2024-09-21T02:53:12.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.853+0000 7f1d05ae1740 1 -- start start 2024-09-21T02:53:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.853+0000 7f1d05ae1740 1 Processor -- start 2024-09-21T02:53:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab90970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab90eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 0x55ee2ab9c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ee2a94c7b0 con 0x55ee2a7de850 2024-09-21T02:53:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55ee2aa85e90 con 0x55ee2a7bdb70 2024-09-21T02:53:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ee2ab93690 con 0x55ee2a7b5990 2024-09-21T02:53:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab90970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab90970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48106/0 (socket says 172.21.15.29:48106) 2024-09-21T02:53:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1cfcb08640 1 -- 172.21.15.29:0/2204144762 learned_addr learned my addr 172.21.15.29:0/2204144762 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1d05ae1740 10 client.0 _ll_register_callbacks cb 0x55ee2aae8710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.854+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 0x55ee2ab9c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.855+0000 7f1cfd309640 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab90eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.855+0000 7f1cfcb08640 1 -- 172.21.15.29:0/2204144762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 msgr2=0x55ee2ab90eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.855+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab90eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.855+0000 7f1cfcb08640 1 -- 172.21.15.29:0/2204144762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 msgr2=0x55ee2ab9c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.855+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 0x55ee2ab9c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.855+0000 7f1cfcb08640 1 -- 172.21.15.29:0/2204144762 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee2ab9cf10 con 0x55ee2a7b5990 2024-09-21T02:53:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.856+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab90970 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8011d80 tx=0x7f1ce8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1f82ba770171f41 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.856+0000 7f1ce0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:53:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.856+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/2204144762 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce800fe60 con 0x55ee2a7b5990 2024-09-21T02:53:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.856+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/2204144762 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1ce8011050 con 0x55ee2a7b5990 2024-09-21T02:53:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.856+0000 7f1d05ae1740 1 -- 172.21.15.29:0/2204144762 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cf4040030 con 0x55ee2a7b5990 2024-09-21T02:53:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.857+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/2204144762 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1ce8011250 con 0x55ee2a7b5990 2024-09-21T02:53:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.857+0000 7f1d05ae1740 10 client.26920 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.857+0000 7f1d05ae1740 1 -- 172.21.15.29:0/2204144762 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ee2ab9f7f0 con 0x55ee2a7b5990 2024-09-21T02:53:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.857+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/2204144762 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cf4040030 con 0x55ee2a7b5990 2024-09-21T02:53:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.858+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/2204144762 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f1ce8004b40 con 0x55ee2a7b5990 2024-09-21T02:53:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.858+0000 7f1d05ae1740 10 client.26920 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.858+0000 7f1d05ae1740 10 client.26920 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:53:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.858+0000 7f1d05ae1740 20 client.26920 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.858+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ee2a79d1d0 0x55ee2aba2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.858+0000 7f1d05ae1740 4 client.26920 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:53:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.858+0000 7f1d05ae1740 1 -- 172.21.15.29:0/2204144762 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee2aafdb00 con 0x55ee2a79d1d0 2024-09-21T02:53:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.859+0000 7f1cfd309640 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ee2a79d1d0 0x55ee2aba2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.859+0000 7f1cfd309640 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ee2a79d1d0 0x55ee2aba2dc0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b8f2a4d1a09fbf4 server_cookie=6cc60dbd4a496773 in_seq=0 out_seq=0 2024-09-21T02:53:12.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.859+0000 7f1ce0ff9640 10 client.26920 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:12.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/2204144762 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ee2aafdb00 con 0x55ee2a79d1d0 2024-09-21T02:53:12.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1ce0ff9640 10 client.26920 handle_command_reply: tid=0 2024-09-21T02:53:12.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1d05ae1740 1 client.26920 shutdown 2024-09-21T02:53:12.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1d05ae1740 1 -- 172.21.15.29:0/2204144762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 msgr2=0x55ee2ab90970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab90970 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8011d80 tx=0x7f1ce8004730 comp rx=0 tx=0).stop 2024-09-21T02:53:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1d05ae1740 1 -- 172.21.15.29:0/2204144762 shutdown_connections 2024-09-21T02:53:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ee2a79d1d0 0x55ee2aba2dc0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 0x55ee2ab9c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab90eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.860+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/2204144762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab90970 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.861+0000 7f1d05ae1740 1 -- 172.21.15.29:0/2204144762 >> 172.21.15.29:0/2204144762 conn(0x55ee2a7d2c40 msgr2=0x55ee2ab88050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.861+0000 7f1ce0ff9640 0 client.26920 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.861+0000 7f1d05ae1740 1 -- 172.21.15.29:0/2204144762 shutdown_connections 2024-09-21T02:53:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.861+0000 7f1d05ae1740 1 -- 172.21.15.29:0/2204144762 wait complete. 2024-09-21T02:53:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.861+0000 7f1d05ae1740 20 client.26920 trim_cache size 0 max 16384 2024-09-21T02:53:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.885+0000 7f1d05ae1740 1 Processor -- start 2024-09-21T02:53:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.885+0000 7f1d05ae1740 1 -- start start 2024-09-21T02:53:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.885+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7b5990 0x55ee2ab9c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab923a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab8d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf81ae830 con 0x55ee2a7b5990 2024-09-21T02:53:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf81b0430 con 0x55ee2a7bdb70 2024-09-21T02:53:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1d05ae1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf81aef00 con 0x55ee2a7de850 2024-09-21T02:53:12.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7b5990 0x55ee2ab9c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfd309640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab923a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7b5990 0x55ee2ab9c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59762/0 (socket says 172.21.15.29:59762) 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfcb08640 1 -- 172.21.15.29:0/1120713525 learned_addr learned my addr 172.21.15.29:0/1120713525 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab8d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfd309640 1 -- 172.21.15.29:0/1120713525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 msgr2=0x55ee2ab8d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfd309640 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab8d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfd309640 1 -- 172.21.15.29:0/1120713525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7b5990 msgr2=0x55ee2ab9c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfd309640 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7b5990 0x55ee2ab9c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfd309640 1 -- 172.21.15.29:0/1120713525 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee2ab87160 con 0x55ee2a7bdb70 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab8d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.886+0000 7f1cfd309640 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab923a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4005760 tx=0x7f1cf4047040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c44939ce35c27960 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.887+0000 7f1ce2ffd640 1 -- 172.21.15.29:0/1120713525 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4035040 con 0x55ee2a7bdb70 2024-09-21T02:53:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.887+0000 7f1ce2ffd640 1 -- 172.21.15.29:0/1120713525 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cf4040070 con 0x55ee2a7bdb70 2024-09-21T02:53:12.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.887+0000 7f1ce2ffd640 1 -- 172.21.15.29:0/1120713525 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4052880 con 0x55ee2a7bdb70 2024-09-21T02:53:12.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 -- 172.21.15.29:0/1120713525 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 msgr2=0x55ee2ab923a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:12.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab923a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4005760 tx=0x7f1cf4047040 comp rx=0 tx=0).stop 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 -- 172.21.15.29:0/1120713525 shutdown_connections 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7de850 0x55ee2ab8d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab923a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/1120713525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7b5990 0x55ee2ab9c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 -- 172.21.15.29:0/1120713525 >> 172.21.15.29:0/1120713525 conn(0x55ee2a7d2c40 msgr2=0x55ee2ab91c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 -- 172.21.15.29:0/1120713525 shutdown_connections 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.888+0000 7f1d05ae1740 1 -- 172.21.15.29:0/1120713525 wait complete. 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.889+0000 7f1d05ae1740 1 -- start start 2024-09-21T02:53:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.889+0000 7f1d05ae1740 1 Processor -- start 2024-09-21T02:53:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab953b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab95950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1d05ae1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 0x55ee2ac06870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1cfcb08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab953b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48134/0 (socket says 172.21.15.29:48134) 2024-09-21T02:53:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1cfd309640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab95950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1cfcb08640 1 -- 172.21.15.29:0/693824560 learned_addr learned my addr 172.21.15.29:0/693824560 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf81afca0 con 0x55ee2a7de850 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf81af5d0 con 0x55ee2a7bdb70 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf81ac810 con 0x55ee2a7b5990 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1d05ae1740 10 client.0 _ll_register_callbacks cb 0x7f1ce8009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.890+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 0x55ee2ac06870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1cfcb08640 1 -- 172.21.15.29:0/693824560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 msgr2=0x55ee2ab95950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab95950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1cfcb08640 1 -- 172.21.15.29:0/693824560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 msgr2=0x55ee2ac06870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 0x55ee2ac06870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1cfcb08640 1 -- 172.21.15.29:0/693824560 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee2ac06ee0 con 0x55ee2a7b5990 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1cfd309640 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab95950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1cfcb08640 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab953b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8011d80 tx=0x7f1ce8009d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fff25e86827240b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1ce0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/693824560 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8046040 con 0x55ee2a7b5990 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/693824560 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1ce8032dc0 con 0x55ee2a7b5990 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.891+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cf4040070 con 0x55ee2a7b5990 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.892+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/693824560 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1ce804f050 con 0x55ee2a7b5990 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.892+0000 7f1d05ae1740 10 client.26926 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.892+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ee2ac0e290 con 0x55ee2a7b5990 2024-09-21T02:53:12.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.892+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/693824560 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cf4040070 con 0x55ee2a7b5990 2024-09-21T02:53:12.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.893+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/693824560 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f1ce8004b10 con 0x55ee2a7b5990 2024-09-21T02:53:12.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.893+0000 7f1d05ae1740 10 client.26926 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:12.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.893+0000 7f1d05ae1740 10 client.26926 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:53:12.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.893+0000 7f1d05ae1740 20 client.26926 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:12.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.893+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ee2a772580 0x55ee2ac10a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:12.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.893+0000 7f1d05ae1740 4 client.26926 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.85, "await_for": 45.0}] 2024-09-21T02:53:12.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.893+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.85, "await_for": 45.0}) -- 0x55ee2aaf95b0 con 0x55ee2a772580 2024-09-21T02:53:12.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.893+0000 7f1cfd309640 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ee2a772580 0x55ee2ac10a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:12.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.894+0000 7f1ce0ff9640 10 client.26926 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:12.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:12.894+0000 7f1cfd309640 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ee2a772580 0x55ee2ac10a90 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5f626bf6f0079d server_cookie=3b3371ac700ae163 in_seq=0 out_seq=0 2024-09-21T02:53:13.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.519+0000 7f1ce0ff9640 1 -- 172.21.15.29:0/693824560 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ee2aaf95b0 con 0x55ee2a772580 2024-09-21T02:53:13.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.519+0000 7f1ce0ff9640 10 client.26926 handle_command_reply: tid=0 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.519+0000 7f1d05ae1740 1 client.26926 shutdown 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.519+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 msgr2=0x55ee2ab953b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.519+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab953b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8011d80 tx=0x7f1ce8009d90 comp rx=0 tx=0).stop 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.519+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 shutdown_connections 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.519+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ee2a772580 0x55ee2ac10a90 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ee2a7de850 0x55ee2ac06870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ee2a7bdb70 0x55ee2ab95950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1ce0ff9640 0 client.26926 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1d05ae1740 1 --2- 172.21.15.29:0/693824560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ee2a7b5990 0x55ee2ab953b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 >> 172.21.15.29:0/693824560 conn(0x55ee2a7d2c40 msgr2=0x55ee2ab85e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 shutdown_connections 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1d05ae1740 1 -- 172.21.15.29:0/693824560 wait complete. 2024-09-21T02:53:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1d05ae1740 20 client.26926 trim_cache size 0 max 16384 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 33, 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 54, 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "7a7ee7ad": { 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 54, 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:53:13.523 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 138.8, 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:13.524 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:53:13.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1ce40b6510 msgr2=0x7f1ce40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:13.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1cff56d640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1ce40b6510 0x7f1ce40b8a20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1cf00b5250 tx=0x7f1cf0005f70 comp rx=0 tx=0).stop 2024-09-21T02:53:13.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf8109e40 msgr2=0x7f1cf8076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:13.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.520+0000 7f1cff56d640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf8109e40 0x7f1cf8076190 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000b540 tx=0x7f1cf000ba00 comp rx=0 tx=0).stop 2024-09-21T02:53:13.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.521+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 shutdown_connections 2024-09-21T02:53:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.521+0000 7f1cff56d640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1ce40b6510 0x7f1ce40b8a20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.521+0000 7f1cff56d640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf8109e40 0x7f1cf8076190 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.521+0000 7f1cff56d640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1cf81094f0 0x7f1cf8075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.521+0000 7f1cff56d640 1 --2- 172.21.15.29:0/821564280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf8103bb0 0x7f1cf8075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.521+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 >> 172.21.15.29:0/821564280 conn(0x7f1cf80ff600 msgr2=0x7f1cf8108a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.521+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 shutdown_connections 2024-09-21T02:53:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:13.521+0000 7f1cff56d640 1 -- 172.21.15.29:0/821564280 wait complete. 2024-09-21T02:53:13.707 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7582 11.62 MB/sec warmup 50 sec latency 767.424 ms 2024-09-21T02:53:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:13 smithi190 ceph-mon[33871]: pgmap v812: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-09-21T02:53:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:13 smithi029 ceph-mon[31006]: pgmap v812: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-09-21T02:53:14.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:13 smithi066 ceph-mon[33436]: pgmap v812: 129 pgs: 129 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-09-21T02:53:14.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 11.39 MB/sec warmup 51 sec latency 1767.520 ms 2024-09-21T02:53:14.783 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7a7ee7ad', quiesce took 6.2 seconds. Will release after: 12.650000000000002 2024-09-21T02:53:15.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 11.18 MB/sec warmup 52 sec latency 2767.647 ms 2024-09-21T02:53:16.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:15 smithi190 ceph-mon[33871]: pgmap v813: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 3.5 MiB/s wr, 274 op/s 2024-09-21T02:53:16.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:15 smithi029 ceph-mon[31006]: pgmap v813: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 3.5 MiB/s wr, 274 op/s 2024-09-21T02:53:16.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:15 smithi066 ceph-mon[33436]: pgmap v813: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 3.5 MiB/s wr, 274 op/s 2024-09-21T02:53:16.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 10.96 MB/sec warmup 53 sec latency 3767.740 ms 2024-09-21T02:53:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:16 smithi190 ceph-mon[33871]: pgmap v814: 129 pgs: 129 active+clean; 253 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 3.4 MiB/s wr, 229 op/s 2024-09-21T02:53:17.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:16 smithi029 ceph-mon[31006]: pgmap v814: 129 pgs: 129 active+clean; 253 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 3.4 MiB/s wr, 229 op/s 2024-09-21T02:53:17.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:16 smithi066 ceph-mon[33436]: pgmap v814: 129 pgs: 129 active+clean; 253 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 3.4 MiB/s wr, 229 op/s 2024-09-21T02:53:17.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 10.76 MB/sec warmup 54 sec latency 4767.855 ms 2024-09-21T02:53:18.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 10.57 MB/sec warmup 55 sec latency 5768.000 ms 2024-09-21T02:53:19.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:19 smithi190 ceph-mon[33871]: pgmap v815: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 3.3 MiB/s wr, 242 op/s 2024-09-21T02:53:19.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:19 smithi029 ceph-mon[31006]: pgmap v815: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 3.3 MiB/s wr, 242 op/s 2024-09-21T02:53:19.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 10.38 MB/sec warmup 56 sec latency 6768.147 ms 2024-09-21T02:53:19.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:19 smithi066 ceph-mon[33436]: pgmap v815: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 3.3 MiB/s wr, 242 op/s 2024-09-21T02:53:20.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 10.19 MB/sec warmup 57 sec latency 7768.252 ms 2024-09-21T02:53:21.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:21 smithi190 ceph-mon[33871]: pgmap v816: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-09-21T02:53:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:21 smithi029 ceph-mon[31006]: pgmap v816: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-09-21T02:53:21.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 10.02 MB/sec warmup 58 sec latency 8768.344 ms 2024-09-21T02:53:21.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:21 smithi066 ceph-mon[33436]: pgmap v816: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-09-21T02:53:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 -- 172.21.15.29:0/3994081560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 msgr2=0x7fc7fc103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 --2- 172.21.15.29:0/3994081560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 0x7fc7fc103a30 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009900 tx=0x7fc7e402a410 comp rx=0 tx=0).stop 2024-09-21T02:53:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 -- 172.21.15.29:0/3994081560 shutdown_connections 2024-09-21T02:53:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 --2- 172.21.15.29:0/3994081560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7fc103fc0 0x7fc7fc10a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 --2- 172.21.15.29:0/3994081560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 0x7fc7fc103a30 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 --2- 172.21.15.29:0/3994081560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7fc1025a0 0x7fc7fc102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 -- 172.21.15.29:0/3994081560 >> 172.21.15.29:0/3994081560 conn(0x7fc7fc0fdee0 msgr2=0x7fc7fc100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 -- 172.21.15.29:0/3994081560 shutdown_connections 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.463+0000 7fc8028be640 1 -- 172.21.15.29:0/3994081560 wait complete. 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.464+0000 7fc8028be640 1 Processor -- start 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.464+0000 7fc8028be640 1 -- start start 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.464+0000 7fc8028be640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7fc1025a0 0x7fc7fc199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.464+0000 7fc8028be640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 0x7fc7fc19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.464+0000 7fc8028be640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7fc103fc0 0x7fc7fc19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.464+0000 7fc8028be640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc10f220 con 0x7fc7fc103620 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.464+0000 7fc8028be640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc10f0a0 con 0x7fc7fc1025a0 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.464+0000 7fc8028be640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc10f3a0 con 0x7fc7fc103fc0 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 0x7fc7fc19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 0x7fc7fc19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42352/0 (socket says 172.21.15.29:42352) 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 -- 172.21.15.29:0/1514797155 learned_addr learned my addr 172.21.15.29:0/1514797155 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7fc1025a0 0x7fc7fc199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc800e5b640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7fc103fc0 0x7fc7fc19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 -- 172.21.15.29:0/1514797155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7fc103fc0 msgr2=0x7fc7fc19b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7fc103fc0 0x7fc7fc19b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 -- 172.21.15.29:0/1514797155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7fc1025a0 msgr2=0x7fc7fc199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7fc1025a0 0x7fc7fc199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 -- 172.21.15.29:0/1514797155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7fc19b8c0 con 0x7fc7fc103620 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc800e5b640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7fc103fc0 0x7fc7fc19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7fc1025a0 0x7fc7fc199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 0x7fc7fc19a4f0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4005760 tx=0x7fc7e4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f9ffb640 1 -- 172.21.15.29:0/1514797155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e403f070 con 0x7fc7fc103620 2024-09-21T02:53:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f9ffb640 1 -- 172.21.15.29:0/1514797155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7e4031da0 con 0x7fc7fc103620 2024-09-21T02:53:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc8028be640 1 -- 172.21.15.29:0/1514797155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7fc19bb50 con 0x7fc7fc103620 2024-09-21T02:53:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc7f9ffb640 1 -- 172.21.15.29:0/1514797155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4035030 con 0x7fc7fc103620 2024-09-21T02:53:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.465+0000 7fc8028be640 1 -- 172.21.15.29:0/1514797155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7fc1a22a0 con 0x7fc7fc103620 2024-09-21T02:53:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc809823740 1 Processor -- start 2024-09-21T02:53:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc809823740 1 -- start start 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc7f9ffb640 1 -- 172.21.15.29:0/1514797155 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc7e4042070 con 0x7fc7fc103620 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc809823740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe66e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc809823740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa962f0 0x55811fe674c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc809823740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 0x55811fe68010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc809823740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55811fdd96f0 con 0x55811fabf1b0 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc809823740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55811fdc8b20 con 0x55811fa962f0 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc809823740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55811fdcc930 con 0x55811fa9e4d0 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc800e5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe66e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc800e5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe66e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42354/0 (socket says 172.21.15.29:42354) 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc800e5b640 1 -- 172.21.15.29:0/2436901898 learned_addr learned my addr 172.21.15.29:0/2436901898 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa962f0 0x55811fe674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.468+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 0x55811fe68010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc800e5b640 1 -- 172.21.15.29:0/2436901898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 msgr2=0x55811fe68010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc800e5b640 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 0x55811fe68010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc800e5b640 1 -- 172.21.15.29:0/2436901898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa962f0 msgr2=0x55811fe674c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc800e5b640 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa962f0 0x55811fe674c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc800e5b640 1 -- 172.21.15.29:0/2436901898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55811fd8e530 con 0x55811fabf1b0 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc800e5b640 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe66e50 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc7f40057b0 tx=0x7fc7f4009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e160bc65a9ca88 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc7f9ffb640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc7e80b6590 0x7fc7e80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc7f27fc640 1 -- 172.21.15.29:0/2436901898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4015070 con 0x55811fabf1b0 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc7f27fc640 1 -- 172.21.15.29:0/2436901898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7f4005aa0 con 0x55811fabf1b0 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc7f27fc640 1 -- 172.21.15.29:0/2436901898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4010750 con 0x55811fabf1b0 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc7e80b6590 0x7fc7e80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.469+0000 7fc7f9ffb640 1 -- 172.21.15.29:0/1514797155 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc7e4108030 con 0x7fc7fc103620 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 -- 172.21.15.29:0/2436901898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 msgr2=0x55811fe66e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc7e80b6590 0x7fc7e80b8aa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec005fd0 tx=0x7fc7ec0094a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe66e50 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc7f40057b0 tx=0x7fc7f4009770 comp rx=0 tx=0).stop 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 -- 172.21.15.29:0/2436901898 shutdown_connections 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 0x55811fe68010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa962f0 0x55811fe674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 --2- 172.21.15.29:0/2436901898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe66e50 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 -- 172.21.15.29:0/2436901898 >> 172.21.15.29:0/2436901898 conn(0x55811fab3300 msgr2=0x55811fe628e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 -- 172.21.15.29:0/2436901898 shutdown_connections 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 -- 172.21.15.29:0/2436901898 wait complete. 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.470+0000 7fc809823740 1 -- start start 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc809823740 1 Processor -- start 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc809823740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe70610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc809823740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa9e4d0 0x55811fe70cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc809823740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe71840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc809823740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55811fd00ab0 con 0x55811fabf1b0 2024-09-21T02:53:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc809823740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55811fd6e730 con 0x55811fa9e4d0 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc809823740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55811fe68b30 con 0x55811fa962f0 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc809823740 10 client.0 _ll_register_callbacks cb 0x55811fdc26b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc800e5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe71840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc800e5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe71840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42370/0 (socket says 172.21.15.29:42370) 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc800e5b640 1 -- 172.21.15.29:0/1088217463 learned_addr learned my addr 172.21.15.29:0/1088217463 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe70610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.471+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa9e4d0 0x55811fe70cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc800e5b640 1 -- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 msgr2=0x55811fe70610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc800e5b640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe70610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc800e5b640 1 -- 172.21.15.29:0/1088217463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa9e4d0 msgr2=0x55811fe70cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc800e5b640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa9e4d0 0x55811fe70cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc800e5b640 1 -- 172.21.15.29:0/1088217463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55811fb2a790 con 0x55811fabf1b0 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa9e4d0 0x55811fe70cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe70610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc800e5b640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe71840 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4005780 tx=0x7fc7f40054f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b860faa0ec5e99e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc7db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc7db7fe640 1 -- 172.21.15.29:0/1088217463 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4029060 con 0x55811fabf1b0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc7db7fe640 1 -- 172.21.15.29:0/1088217463 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc7f4011680 con 0x55811fabf1b0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc809823740 1 -- 172.21.15.29:0/1088217463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55811fddb640 con 0x55811fabf1b0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc7db7fe640 1 -- 172.21.15.29:0/1088217463 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc7f4030050 con 0x55811fabf1b0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc809823740 10 client.17415 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc809823740 1 -- 172.21.15.29:0/1088217463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55811fba3620 con 0x55811fabf1b0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.472+0000 7fc7db7fe640 1 -- 172.21.15.29:0/1088217463 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55811fddb640 con 0x55811fabf1b0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc7db7fe640 1 -- 172.21.15.29:0/1088217463 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7fc7f400f040 con 0x55811fabf1b0 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc809823740 10 client.17415 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc809823740 10 client.17415 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc809823740 20 client.17415 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc809823740 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55811fa7dd80 0x55811fe7dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc809823740 4 client.17415 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc809823740 1 -- 172.21.15.29:0/1088217463 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55811fdd70f0 con 0x55811fa7dd80 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55811fa7dd80 0x55811fe7dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc7db7fe640 10 client.17415 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.473+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55811fa7dd80 0x55811fe7dab0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c14df2c243c4c8d9 server_cookie=3ebdf6d05836cd6d in_seq=0 out_seq=0 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.474+0000 7fc7db7fe640 1 -- 172.21.15.29:0/1088217463 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55811fdd70f0 con 0x55811fa7dd80 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.474+0000 7fc7db7fe640 10 client.17415 handle_command_reply: tid=0 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.474+0000 7fc809823740 1 client.17415 shutdown 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 -- 172.21.15.29:0/1088217463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 msgr2=0x55811fe71840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe71840 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4005780 tx=0x7fc7f40054f0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 -- 172.21.15.29:0/1088217463 shutdown_connections 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55811fa7dd80 0x55811fe7dab0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fabf1b0 0x55811fe71840 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa9e4d0 0x55811fe70cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc7db7fe640 0 client.17415 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 --2- 172.21.15.29:0/1088217463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe70610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 -- 172.21.15.29:0/1088217463 >> 172.21.15.29:0/1088217463 conn(0x55811fab3300 msgr2=0x55811fe60ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 -- 172.21.15.29:0/1088217463 shutdown_connections 2024-09-21T02:53:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 1 -- 172.21.15.29:0/1088217463 wait complete. 2024-09-21T02:53:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.475+0000 7fc809823740 20 client.17415 trim_cache size 0 max 16384 2024-09-21T02:53:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.489+0000 7fc809823740 1 Processor -- start 2024-09-21T02:53:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc809823740 1 -- start start 2024-09-21T02:53:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc809823740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 0x55811fe71e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc809823740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe78720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc809823740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa9e4d0 0x55811fe60ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc809823740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1ac590 con 0x55811fa9e4d0 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc809823740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1ae190 con 0x55811fa7dd80 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc809823740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1acc60 con 0x55811fa962f0 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa9e4d0 0x55811fe60ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa9e4d0 0x55811fe60ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42372/0 (socket says 172.21.15.29:42372) 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 -- 172.21.15.29:0/578931636 learned_addr learned my addr 172.21.15.29:0/578931636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe78720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 0x55811fe71e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 -- 172.21.15.29:0/578931636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 msgr2=0x55811fe78720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe78720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 -- 172.21.15.29:0/578931636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 msgr2=0x55811fe71e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 0x55811fe71e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 -- 172.21.15.29:0/578931636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55811fe71040 con 0x55811fa9e4d0 2024-09-21T02:53:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe78720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.490+0000 7fc800e5b640 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa9e4d0 0x55811fe60ea0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4010dd0 tx=0x7fc7f4005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec26f07eb056046f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc7f1ffb640 1 -- 172.21.15.29:0/578931636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f401d020 con 0x55811fa9e4d0 2024-09-21T02:53:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc7f1ffb640 1 -- 172.21.15.29:0/578931636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7f4011b10 con 0x55811fa9e4d0 2024-09-21T02:53:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc7f1ffb640 1 -- 172.21.15.29:0/578931636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4029070 con 0x55811fa9e4d0 2024-09-21T02:53:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc809823740 1 -- 172.21.15.29:0/578931636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa9e4d0 msgr2=0x55811fe60ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc809823740 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa9e4d0 0x55811fe60ea0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4010dd0 tx=0x7fc7f4005a30 comp rx=0 tx=0).stop 2024-09-21T02:53:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc809823740 1 -- 172.21.15.29:0/578931636 shutdown_connections 2024-09-21T02:53:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc809823740 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa9e4d0 0x55811fe60ea0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc809823740 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa962f0 0x55811fe78720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.491+0000 7fc809823740 1 --2- 172.21.15.29:0/578931636 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 0x55811fe71e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 -- 172.21.15.29:0/578931636 >> 172.21.15.29:0/578931636 conn(0x55811fab3300 msgr2=0x55811fe71780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 -- 172.21.15.29:0/578931636 shutdown_connections 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 -- 172.21.15.29:0/578931636 wait complete. 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 -- start start 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 Processor -- start 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 0x55811fe75410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa962f0 0x55811fe75ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 0x55811fe76640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1ada00 con 0x55811fa962f0 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1ad330 con 0x55811fa7dd80 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1aa570 con 0x55811fa9e4d0 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.492+0000 7fc809823740 10 client.0 _ll_register_callbacks cb 0x7fc7ec009340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa962f0 0x55811fe75ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa962f0 0x55811fe75ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42388/0 (socket says 172.21.15.29:42388) 2024-09-21T02:53:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 -- 172.21.15.29:0/3945930455 learned_addr learned my addr 172.21.15.29:0/3945930455 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 0x55811fe75410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc800e5b640 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 0x55811fe76640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 -- 172.21.15.29:0/3945930455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 msgr2=0x55811fe76640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 0x55811fe76640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 -- 172.21.15.29:0/3945930455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 msgr2=0x55811fe75410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 0x55811fe75410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 -- 172.21.15.29:0/3945930455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55811fe76da0 con 0x55811fa962f0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7f3fff640 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa962f0 0x55811fe75ad0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40cb0b0 tx=0x7fc7e40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367c0b8639c1e391 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7db7fe640 1 -- 172.21.15.29:0/3945930455 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e40d0060 con 0x55811fa962f0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7db7fe640 1 -- 172.21.15.29:0/3945930455 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc7e40d25c0 con 0x55811fa962f0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc809823740 1 -- 172.21.15.29:0/3945930455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f4011b10 con 0x55811fa962f0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc7db7fe640 1 -- 172.21.15.29:0/3945930455 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc7e4049040 con 0x55811fa962f0 2024-09-21T02:53:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc809823740 10 client.17421 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.493+0000 7fc809823740 1 -- 172.21.15.29:0/3945930455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55811fe78160 con 0x55811fa962f0 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc7db7fe640 1 -- 172.21.15.29:0/3945930455 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7f4011b10 con 0x55811fa962f0 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc7db7fe640 1 -- 172.21.15.29:0/3945930455 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7fc7e40b50d0 con 0x55811fa962f0 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc809823740 10 client.17421 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc809823740 10 client.17421 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc809823740 20 client.17421 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc809823740 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55811fabf1b0 0x55811feea480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc809823740 4 client.17421 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc809823740 1 -- 172.21.15.29:0/3945930455 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55811fdd4520 con 0x55811fabf1b0 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.494+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55811fabf1b0 0x55811feea480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.495+0000 7fc7db7fe640 10 client.17421 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.495+0000 7fc7fbfff640 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55811fabf1b0 0x55811feea480 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84a0889d7c52699d server_cookie=4005fb01112d598f in_seq=0 out_seq=0 2024-09-21T02:53:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.578+0000 7fc7db7fe640 1 -- 172.21.15.29:0/3945930455 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55811fdd4520 con 0x55811fabf1b0 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.578+0000 7fc7db7fe640 10 client.17421 handle_command_reply: tid=0 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.578+0000 7fc809823740 1 client.17421 shutdown 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 -- 172.21.15.29:0/3945930455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa962f0 msgr2=0x55811fe75ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa962f0 0x55811fe75ad0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40cb0b0 tx=0x7fc7e40045e0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 -- 172.21.15.29:0/3945930455 shutdown_connections 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55811fa9e4d0 0x55811fe76640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55811fabf1b0 0x55811feea480 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55811fa962f0 0x55811fe75ad0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 --2- 172.21.15.29:0/3945930455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55811fa7dd80 0x55811fe75410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc7db7fe640 0 client.17421 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 -- 172.21.15.29:0/3945930455 >> 172.21.15.29:0/3945930455 conn(0x55811fab3300 msgr2=0x55811fe60ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 -- 172.21.15.29:0/3945930455 shutdown_connections 2024-09-21T02:53:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 1 -- 172.21.15.29:0/3945930455 wait complete. 2024-09-21T02:53:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.579+0000 7fc809823740 20 client.17421 trim_cache size 0 max 16384 2024-09-21T02:53:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:53:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:53:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:53:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 -- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc7e80b6590 msgr2=0x7fc7e80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc7e80b6590 0x7fc7e80b8aa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec005fd0 tx=0x7fc7ec0094a0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 -- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 msgr2=0x7fc7fc19a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 0x7fc7fc19a4f0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4005760 tx=0x7fc7e4003d50 comp rx=0 tx=0).stop 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 -- 172.21.15.29:0/1514797155 shutdown_connections 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc7e80b6590 0x7fc7e80b8aa0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7fc103fc0 0x7fc7fc19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7fc103620 0x7fc7fc19a4f0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 --2- 172.21.15.29:0/1514797155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7fc1025a0 0x7fc7fc199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 -- 172.21.15.29:0/1514797155 >> 172.21.15.29:0/1514797155 conn(0x7fc7fc0fdee0 msgr2=0x7fc7fc106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 -- 172.21.15.29:0/1514797155 shutdown_connections 2024-09-21T02:53:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:22.580+0000 7fc8028be640 1 -- 172.21.15.29:0/1514797155 wait complete. 2024-09-21T02:53:22.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 9.85 MB/sec warmup 59 sec latency 9768.424 ms 2024-09-21T02:53:23.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:23 smithi190 ceph-mon[33871]: pgmap v817: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-09-21T02:53:23.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:23 smithi029 ceph-mon[31006]: pgmap v817: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-09-21T02:53:23.708 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 9.69 MB/sec warmup 60 sec latency 10768.498 ms 2024-09-21T02:53:23.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:23 smithi066 ceph-mon[33436]: pgmap v817: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-09-21T02:53:24.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 9.53 MB/sec warmup 61 sec latency 11768.614 ms 2024-09-21T02:53:25.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:25 smithi190 ceph-mon[33871]: pgmap v818: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-09-21T02:53:25.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:25 smithi029 ceph-mon[31006]: pgmap v818: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-09-21T02:53:25.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 9.37 MB/sec warmup 62 sec latency 12768.744 ms 2024-09-21T02:53:25.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:25 smithi066 ceph-mon[33436]: pgmap v818: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-09-21T02:53:26.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 9.22 MB/sec warmup 63 sec latency 13768.852 ms 2024-09-21T02:53:27.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:53:27.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:27 smithi190 ceph-mon[33871]: pgmap v819: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.1 MiB/s wr, 84 op/s 2024-09-21T02:53:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:27 smithi029 ceph-mon[31006]: pgmap v819: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.1 MiB/s wr, 84 op/s 2024-09-21T02:53:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:53:27.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 9.08 MB/sec warmup 64 sec latency 14768.910 ms 2024-09-21T02:53:27.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:27 smithi066 ceph-mon[33436]: pgmap v819: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.1 MiB/s wr, 84 op/s 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.223+0000 7f17098d1640 1 -- 172.21.15.29:0/1380028054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 msgr2=0x7f1704103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.223+0000 7f17098d1640 1 --2- 172.21.15.29:0/1380028054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 0x7f1704103f90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009900 tx=0x7f16f402a410 comp rx=0 tx=0).stop 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.224+0000 7f17098d1640 1 -- 172.21.15.29:0/1380028054 shutdown_connections 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.224+0000 7f17098d1640 1 --2- 172.21.15.29:0/1380028054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1704109e10 0x7f1704110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.224+0000 7f17098d1640 1 --2- 172.21.15.29:0/1380028054 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17041094c0 0x7f17041098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.224+0000 7f17098d1640 1 --2- 172.21.15.29:0/1380028054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 0x7f1704103f90 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.224+0000 7f17098d1640 1 -- 172.21.15.29:0/1380028054 >> 172.21.15.29:0/1380028054 conn(0x7f17040ff590 msgr2=0x7f17041019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.224+0000 7f17098d1640 1 -- 172.21.15.29:0/1380028054 shutdown_connections 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.224+0000 7f17098d1640 1 -- 172.21.15.29:0/1380028054 wait complete. 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.224+0000 7f17098d1640 1 Processor -- start 2024-09-21T02:53:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17098d1640 1 -- start start 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17098d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 0x7f170419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17098d1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17041094c0 0x7f170419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17098d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1704109e10 0x7f170419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17098d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1704113980 con 0x7f1704103b80 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17098d1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1704113800 con 0x7f17041094c0 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17098d1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1704113b00 con 0x7f1704109e10 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 0x7f170419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 0x7f170419a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53440/0 (socket says 172.21.15.29:53440) 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 -- 172.21.15.29:0/2392380005 learned_addr learned my addr 172.21.15.29:0/2392380005 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17027fc640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17041094c0 0x7f170419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f17037fe640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1704109e10 0x7f170419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 -- 172.21.15.29:0/2392380005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1704109e10 msgr2=0x7f170419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1704109e10 0x7f170419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 -- 172.21.15.29:0/2392380005 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17041094c0 msgr2=0x7f170419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17041094c0 0x7f170419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 -- 172.21.15.29:0/2392380005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17041a03a0 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.225+0000 7f1702ffd640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 0x7f170419a430 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009230 tx=0x7f16f4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.226+0000 7f17088cf640 1 -- 172.21.15.29:0/2392380005 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f403f030 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.226+0000 7f17088cf640 1 -- 172.21.15.29:0/2392380005 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16f4003a80 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.226+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17041a0630 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.226+0000 7f17088cf640 1 -- 172.21.15.29:0/2392380005 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4035070 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.226+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17041a6df0 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.227+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1704103f90 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.228+0000 7f17088cf640 1 -- 172.21.15.29:0/2392380005 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f16f4042030 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.229+0000 7f17088cf640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ec0b6510 0x7f16ec0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.229+0000 7f17027fc640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ec0b6510 0x7f16ec0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.229+0000 7f17088cf640 1 -- 172.21.15.29:0/2392380005 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f16f4108260 con 0x7f1704103b80 2024-09-21T02:53:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.230+0000 7f17027fc640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ec0b6510 0x7f16ec0b8a20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1704102db0 tx=0x7f16f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.232+0000 7f17088cf640 1 -- 172.21.15.29:0/2392380005 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f16f40ce770 con 0x7f1704103b80 2024-09-21T02:53:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.460+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f170419b560 con 0x7f1704103b80 2024-09-21T02:53:28.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.461+0000 7f17088cf640 1 -- 172.21.15.29:0/2392380005 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11979 (secure 0 0 0) 0x7f16f40d2b50 con 0x7f1704103b80 2024-09-21T02:53:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:53:28.463 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"btime":"2024-09-21T02:52:29:480744+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:52:28.665505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:53:28.463 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ec0b6510 msgr2=0x7f16ec0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ec0b6510 0x7f16ec0b8a20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1704102db0 tx=0x7f16f8005f90 comp rx=0 tx=0).stop 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 msgr2=0x7f170419a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 0x7f170419a430 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009230 tx=0x7f16f4004270 comp rx=0 tx=0).stop 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 shutdown_connections 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f16ec0b6510 0x7f16ec0b8a20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1704109e10 0x7f170419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17041094c0 0x7f170419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 --2- 172.21.15.29:0/2392380005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704103b80 0x7f170419a430 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 >> 172.21.15.29:0/2392380005 conn(0x7f17040ff590 msgr2=0x7f17041019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 shutdown_connections 2024-09-21T02:53:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:28.463+0000 7f17098d1640 1 -- 172.21.15.29:0/2392380005 wait complete. 2024-09-21T02:53:28.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 8.94 MB/sec warmup 65 sec latency 15768.966 ms 2024-09-21T02:53:29.215 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 7a7ee7ad --release --await' 2024-09-21T02:53:29.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 7a7ee7ad --release --await 2024-09-21T02:53:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:53:29.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:29 smithi029 ceph-mon[31006]: pgmap v820: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 584 KiB/s wr, 75 op/s 2024-09-21T02:53:29.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:29 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2392380005' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:29 smithi190 ceph-mon[33871]: pgmap v820: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 584 KiB/s wr, 75 op/s 2024-09-21T02:53:29.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:29 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2392380005' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:29.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7585 8.80 MB/sec warmup 66 sec latency 16768.996 ms 2024-09-21T02:53:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:29 smithi066 ceph-mon[33436]: pgmap v820: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 584 KiB/s wr, 75 op/s 2024-09-21T02:53:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:29 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2392380005' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 -- 172.21.15.29:0/1494681662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781025f0 msgr2=0x7ff578102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 --2- 172.21.15.29:0/1494681662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781025f0 0x7ff578102a00 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff574009900 tx=0x7ff57402a480 comp rx=0 tx=0).stop 2024-09-21T02:53:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 -- 172.21.15.29:0/1494681662 shutdown_connections 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 --2- 172.21.15.29:0/1494681662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5781101a0 0x7ff5781125c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 --2- 172.21.15.29:0/1494681662 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff578102f40 0x7ff57810fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 --2- 172.21.15.29:0/1494681662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781025f0 0x7ff578102a00 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 -- 172.21.15.29:0/1494681662 >> 172.21.15.29:0/1494681662 conn(0x7ff578079740 msgr2=0x7ff578079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 -- 172.21.15.29:0/1494681662 shutdown_connections 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.021+0000 7ff57f330640 1 -- 172.21.15.29:0/1494681662 wait complete. 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57f330640 1 Processor -- start 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57f330640 1 -- start start 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57f330640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5781025f0 0x7ff57819c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57f330640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff578102f40 0x7ff57819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57f330640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781101a0 0x7ff5781a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57f330640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff578115b40 con 0x7ff5781101a0 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57f330640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff5781159c0 con 0x7ff5781025f0 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57f330640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff578115cc0 con 0x7ff578102f40 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d8cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781101a0 0x7ff5781a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d8cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781101a0 0x7ff5781a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53448/0 (socket says 172.21.15.29:53448) 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d8cd640 1 -- 172.21.15.29:0/512098681 learned_addr learned my addr 172.21.15.29:0/512098681 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5781025f0 0x7ff57819c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff578102f40 0x7ff57819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d8cd640 1 -- 172.21.15.29:0/512098681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff578102f40 msgr2=0x7ff57819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d8cd640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff578102f40 0x7ff57819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d8cd640 1 -- 172.21.15.29:0/512098681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5781025f0 msgr2=0x7ff57819c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d8cd640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5781025f0 0x7ff57819c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.022+0000 7ff57d8cd640 1 -- 172.21.15.29:0/512098681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5781a24c0 con 0x7ff5781101a0 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.023+0000 7ff57d8cd640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781101a0 0x7ff5781a1e50 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff57000eb10 tx=0x7ff57000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.023+0000 7ff56e7fc640 1 -- 172.21.15.29:0/512098681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57000ce20 con 0x7ff5781101a0 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.023+0000 7ff56e7fc640 1 -- 172.21.15.29:0/512098681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff570005020 con 0x7ff5781101a0 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.023+0000 7ff56e7fc640 1 -- 172.21.15.29:0/512098681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff570010690 con 0x7ff5781101a0 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.023+0000 7ff57f330640 1 -- 172.21.15.29:0/512098681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5781a27b0 con 0x7ff5781101a0 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.023+0000 7ff57f330640 1 -- 172.21.15.29:0/512098681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5781a8fd0 con 0x7ff5781101a0 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff5858a4740 1 Processor -- start 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff56e7fc640 1 -- 172.21.15.29:0/512098681 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff570004a40 con 0x7ff5781101a0 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff5858a4740 1 -- start start 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47ba510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47baa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47baf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff5858a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dac46a5dc0 con 0x55dac43f3840 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff5858a4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55dac47321f0 con 0x55dac43eb660 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.026+0000 7ff5858a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55dac4721620 con 0x55dac4414520 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47baf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47baf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53462/0 (socket says 172.21.15.29:53462) 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 -- 172.21.15.29:0/662184545 learned_addr learned my addr 172.21.15.29:0/662184545 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47ba510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47baa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff56e7fc640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5640b65c0 0x7ff5640b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 -- 172.21.15.29:0/662184545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 msgr2=0x55dac47ba510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47ba510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 -- 172.21.15.29:0/662184545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 msgr2=0x55dac47baa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47baa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 -- 172.21.15.29:0/662184545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dac46e9fd0 con 0x55dac43f3840 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57d8cd640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5640b65c0 0x7ff5640b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff56e7fc640 1 -- 172.21.15.29:0/512098681 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff5700e7ad0 con 0x7ff5781101a0 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47baf90 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff568011770 tx=0x7ff56800a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb1ce602803239c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.027+0000 7ff5637fe640 1 -- 172.21.15.29:0/662184545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568005020 con 0x55dac43f3840 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5637fe640 1 -- 172.21.15.29:0/662184545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff56800f840 con 0x55dac43f3840 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5637fe640 1 -- 172.21.15.29:0/662184545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568005650 con 0x55dac43f3840 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5858a4740 1 -- 172.21.15.29:0/662184545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 msgr2=0x55dac47baf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff57d8cd640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5640b65c0 0x7ff5640b8ad0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff5700aab50 tx=0x7ff570071e10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5858a4740 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47baf90 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff568011770 tx=0x7ff56800a760 comp rx=0 tx=0).stop 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5858a4740 1 -- 172.21.15.29:0/662184545 shutdown_connections 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5858a4740 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47baf90 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5858a4740 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47baa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5858a4740 1 --2- 172.21.15.29:0/662184545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47ba510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.028+0000 7ff5858a4740 1 -- 172.21.15.29:0/662184545 >> 172.21.15.29:0/662184545 conn(0x55dac4408910 msgr2=0x55dac47b8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.029+0000 7ff5858a4740 1 -- 172.21.15.29:0/662184545 shutdown_connections 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.029+0000 7ff5858a4740 1 -- 172.21.15.29:0/662184545 wait complete. 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.029+0000 7ff5858a4740 1 -- start start 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.029+0000 7ff5858a4740 1 Processor -- start 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47d1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff5858a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dac46bc300 con 0x55dac43f3840 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff5858a4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55dac4725430 con 0x55dac43eb660 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff5858a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55dac4582bb0 con 0x55dac4414520 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff5858a4740 10 client.0 _ll_register_callbacks cb 0x55dac46d6120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53464/0 (socket says 172.21.15.29:53464) 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 -- 172.21.15.29:0/1405371909 learned_addr learned my addr 172.21.15.29:0/1405371909 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47d1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 -- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 msgr2=0x55dac47d1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47d1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 -- 172.21.15.29:0/1405371909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 msgr2=0x55dac47c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 -- 172.21.15.29:0/1405371909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dac46ab780 con 0x55dac43f3840 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47d1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47c60a0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff57402a9a0 tx=0x7ff574031f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5b7d808dae19b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.030+0000 7ff560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff560ff9640 1 -- 172.21.15.29:0/1405371909 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574035040 con 0x55dac43f3840 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff560ff9640 1 -- 172.21.15.29:0/1405371909 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff574004240 con 0x55dac43f3840 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 1 -- 172.21.15.29:0/1405371909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff56800f840 con 0x55dac43f3840 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff560ff9640 1 -- 172.21.15.29:0/1405371909 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff57404e050 con 0x55dac43f3840 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 10 client.17439 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 1 -- 172.21.15.29:0/1405371909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dac447fb00 con 0x55dac43f3840 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff560ff9640 1 -- 172.21.15.29:0/1405371909 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff56800f840 con 0x55dac43f3840 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff560ff9640 1 -- 172.21.15.29:0/1405371909 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7ff57403c440 con 0x55dac43f3840 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 10 client.17439 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 10 client.17439 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 20 client.17439 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dac43d2ea0 0x55dac47d8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 4 client.17439 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.031+0000 7ff5858a4740 1 -- 172.21.15.29:0/1405371909 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dac4734140 con 0x55dac43d2ea0 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.032+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dac43d2ea0 0x55dac47d8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.032+0000 7ff560ff9640 10 client.17439 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.032+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dac43d2ea0 0x55dac47d8050 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd28b51d22903fc server_cookie=c93d24c875829a5e in_seq=0 out_seq=0 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff560ff9640 1 -- 172.21.15.29:0/1405371909 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dac4734140 con 0x55dac43d2ea0 2024-09-21T02:53:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff560ff9640 10 client.17439 handle_command_reply: tid=0 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 client.17439 shutdown 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 -- 172.21.15.29:0/1405371909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 msgr2=0x55dac47c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47c60a0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff57402a9a0 tx=0x7ff574031f20 comp rx=0 tx=0).stop 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 -- 172.21.15.29:0/1405371909 shutdown_connections 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac47d1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dac43d2ea0 0x55dac47d8050 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47c60a0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1405371909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 -- 172.21.15.29:0/1405371909 >> 172.21.15.29:0/1405371909 conn(0x55dac4408910 msgr2=0x55dac47bd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff560ff9640 0 client.17439 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 -- 172.21.15.29:0/1405371909 shutdown_connections 2024-09-21T02:53:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 1 -- 172.21.15.29:0/1405371909 wait complete. 2024-09-21T02:53:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.033+0000 7ff5858a4740 20 client.17439 trim_cache size 0 max 16384 2024-09-21T02:53:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.050+0000 7ff5858a4740 1 Processor -- start 2024-09-21T02:53:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.050+0000 7ff5858a4740 1 -- start start 2024-09-21T02:53:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac43eb660 0x55dac47bbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43f3840 0x55dac47d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac4414520 0x55dac47c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff5858a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff5781b2bb0 con 0x55dac4414520 2024-09-21T02:53:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff5858a4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff5781b47b0 con 0x55dac43f3840 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff5858a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff5781b3280 con 0x55dac43eb660 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac4414520 0x55dac47c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac4414520 0x55dac47c7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53470/0 (socket says 172.21.15.29:53470) 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 -- 172.21.15.29:0/2847063625 learned_addr learned my addr 172.21.15.29:0/2847063625 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac43eb660 0x55dac47bbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43f3840 0x55dac47d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 -- 172.21.15.29:0/2847063625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac43eb660 msgr2=0x55dac47bbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac43eb660 0x55dac47bbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 -- 172.21.15.29:0/2847063625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43f3840 msgr2=0x55dac47d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43f3840 0x55dac47d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.051+0000 7ff57c8cb640 1 -- 172.21.15.29:0/2847063625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dac44f6960 con 0x55dac4414520 2024-09-21T02:53:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.052+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac4414520 0x55dac47c7a40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x55dac47c8860 tx=0x7ff568013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8baf160178e17bc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.052+0000 7ff562ffd640 1 -- 172.21.15.29:0/2847063625 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56800a6f0 con 0x55dac4414520 2024-09-21T02:53:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.052+0000 7ff562ffd640 1 -- 172.21.15.29:0/2847063625 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff56802e070 con 0x55dac4414520 2024-09-21T02:53:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.052+0000 7ff562ffd640 1 -- 172.21.15.29:0/2847063625 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568004660 con 0x55dac4414520 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.052+0000 7ff5858a4740 1 -- 172.21.15.29:0/2847063625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac4414520 msgr2=0x55dac47c7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.052+0000 7ff5858a4740 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac4414520 0x55dac47c7a40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x55dac47c8860 tx=0x7ff568013040 comp rx=0 tx=0).stop 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 -- 172.21.15.29:0/2847063625 shutdown_connections 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac4414520 0x55dac47c7a40 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43f3840 0x55dac47d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 --2- 172.21.15.29:0/2847063625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac43eb660 0x55dac47bbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 -- 172.21.15.29:0/2847063625 >> 172.21.15.29:0/2847063625 conn(0x55dac4408910 msgr2=0x55dac47c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 -- 172.21.15.29:0/2847063625 shutdown_connections 2024-09-21T02:53:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 -- 172.21.15.29:0/2847063625 wait complete. 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 -- start start 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.053+0000 7ff5858a4740 1 Processor -- start 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47ca690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff5858a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac483ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff5858a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff5781b4020 con 0x55dac43f3840 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff5858a4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff5781b3950 con 0x55dac43eb660 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff5858a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff5781b0b90 con 0x55dac4414520 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff5858a4740 10 client.0 _ll_register_callbacks cb 0x7ff574009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53480/0 (socket says 172.21.15.29:53480) 2024-09-21T02:53:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 -- 172.21.15.29:0/1263147966 learned_addr learned my addr 172.21.15.29:0/1263147966 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac483ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 -- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 msgr2=0x55dac483ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac483ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 -- 172.21.15.29:0/1263147966 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 msgr2=0x55dac47ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 -- 172.21.15.29:0/1263147966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dac483c090 con 0x55dac43f3840 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac483ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff57d0cc640 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47cac30 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff574000c00 tx=0x7ff57403c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef1794b42e07cc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff560ff9640 1 -- 172.21.15.29:0/1263147966 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574035050 con 0x55dac43f3840 2024-09-21T02:53:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff5858a4740 1 -- 172.21.15.29:0/1263147966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff56802e070 con 0x55dac43f3840 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.054+0000 7ff560ff9640 1 -- 172.21.15.29:0/1263147966 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff574031a50 con 0x55dac43f3840 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff560ff9640 1 -- 172.21.15.29:0/1263147966 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff574004030 con 0x55dac43f3840 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff5858a4740 10 client.17445 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff5858a4740 1 -- 172.21.15.29:0/1263147966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dac4843450 con 0x55dac43f3840 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff560ff9640 1 -- 172.21.15.29:0/1263147966 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff56802e070 con 0x55dac43f3840 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff560ff9640 1 -- 172.21.15.29:0/1263147966 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7ff574003990 con 0x55dac43f3840 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff5858a4740 10 client.17445 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff5858a4740 10 client.17445 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff5858a4740 20 client.17445 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dac43a8250 0x55dac4845cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff5858a4740 4 client.17445 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7a7ee7ad", "release": true, "await": true}] 2024-09-21T02:53:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff5858a4740 1 -- 172.21.15.29:0/1263147966 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7a7ee7ad", "release": true, "await": true}) -- 0x55dac472fbf0 con 0x55dac43a8250 2024-09-21T02:53:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.055+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dac43a8250 0x55dac4845cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.056+0000 7ff57c8cb640 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dac43a8250 0x55dac4845cf0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eca9ae553d4fa8c server_cookie=208364f5a2a5741d in_seq=0 out_seq=0 2024-09-21T02:53:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.056+0000 7ff560ff9640 10 client.17445 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:30.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.120+0000 7ff560ff9640 1 -- 172.21.15.29:0/1263147966 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55dac472fbf0 con 0x55dac43a8250 2024-09-21T02:53:30.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.120+0000 7ff560ff9640 10 client.17445 handle_command_reply: tid=0 2024-09-21T02:53:30.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.120+0000 7ff5858a4740 1 client.17445 shutdown 2024-09-21T02:53:30.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 -- 172.21.15.29:0/1263147966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 msgr2=0x55dac47cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47cac30 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff574000c00 tx=0x7ff57403c670 comp rx=0 tx=0).stop 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 -- 172.21.15.29:0/1263147966 shutdown_connections 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dac4414520 0x55dac483ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55dac43a8250 0x55dac4845cf0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dac43f3840 0x55dac47cac30 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 --2- 172.21.15.29:0/1263147966 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55dac43eb660 0x55dac47ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 -- 172.21.15.29:0/1263147966 >> 172.21.15.29:0/1263147966 conn(0x55dac4408910 msgr2=0x55dac47c4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff560ff9640 0 client.17445 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 -- 172.21.15.29:0/1263147966 shutdown_connections 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 1 -- 172.21.15.29:0/1263147966 wait complete. 2024-09-21T02:53:30.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.121+0000 7ff5858a4740 20 client.17445 trim_cache size 0 max 16384 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 33, 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 56, 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "7a7ee7ad": { 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 56, 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:53:30.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 138.8, 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:30.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:30.127 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:53:30.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 -- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5640b65c0 msgr2=0x7ff5640b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5640b65c0 0x7ff5640b8ad0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff5700aab50 tx=0x7ff570071e10 comp rx=0 tx=0).stop 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 -- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781101a0 msgr2=0x7ff5781a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781101a0 0x7ff5781a1e50 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff57000eb10 tx=0x7ff57000ef10 comp rx=0 tx=0).stop 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 -- 172.21.15.29:0/512098681 shutdown_connections 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff5640b65c0 0x7ff5640b8ad0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5781101a0 0x7ff5781a1e50 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff578102f40 0x7ff57819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 --2- 172.21.15.29:0/512098681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5781025f0 0x7ff57819c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 -- 172.21.15.29:0/512098681 >> 172.21.15.29:0/512098681 conn(0x7ff578079740 msgr2=0x7ff578100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.122+0000 7ff57f330640 1 -- 172.21.15.29:0/512098681 shutdown_connections 2024-09-21T02:53:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:30.123+0000 7ff57f330640 1 -- 172.21.15.29:0/512098681 wait complete. 2024-09-21T02:53:30.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7693 8.80 MB/sec warmup 67 sec latency 17121.832 ms 2024-09-21T02:53:31.229 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7a7ee7ad', total seconds elapsed: 22.7 2024-09-21T02:53:31.229 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.85 2024-09-21T02:53:31.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:53:31.566 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:53:31.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:31 smithi190 ceph-mon[33871]: pgmap v821: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 402 KiB/s wr, 48 op/s 2024-09-21T02:53:31.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:31 smithi029 ceph-mon[31006]: pgmap v821: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 402 KiB/s wr, 48 op/s 2024-09-21T02:53:31.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7781 8.72 MB/sec warmup 68 sec latency 162.539 ms 2024-09-21T02:53:31.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:31 smithi066 ceph-mon[33436]: pgmap v821: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 402 KiB/s wr, 48 op/s 2024-09-21T02:53:32.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 -- 172.21.15.29:0/2387247140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc102a60 msgr2=0x7f18fc102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:32.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 --2- 172.21.15.29:0/2387247140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc102a60 0x7f18fc102e70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009900 tx=0x7f18f002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 -- 172.21.15.29:0/2387247140 shutdown_connections 2024-09-21T02:53:32.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 --2- 172.21.15.29:0/2387247140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc108cf0 0x7f18fc10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 --2- 172.21.15.29:0/2387247140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18fc1083a0 0x7f18fc1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 --2- 172.21.15.29:0/2387247140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc102a60 0x7f18fc102e70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 -- 172.21.15.29:0/2387247140 >> 172.21.15.29:0/2387247140 conn(0x7f18fc079730 msgr2=0x7f18fc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:32.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 -- 172.21.15.29:0/2387247140 shutdown_connections 2024-09-21T02:53:32.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.349+0000 7f1902a82640 1 -- 172.21.15.29:0/2387247140 wait complete. 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f1902a82640 1 Processor -- start 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f1902a82640 1 -- start start 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f1902a82640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc102a60 0x7f18fc19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f1902a82640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc1083a0 0x7f18fc19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f1902a82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18fc108cf0 0x7f18fc19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f1902a82640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc113920 con 0x7f18fc108cf0 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f1902a82640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc1137a0 con 0x7f18fc102a60 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f1902a82640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc113aa0 con 0x7f18fc1083a0 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f18fbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc1083a0 0x7f18fc19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f18fbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc1083a0 0x7f18fc19a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51388/0 (socket says 172.21.15.29:51388) 2024-09-21T02:53:32.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f18fbfff640 1 -- 172.21.15.29:0/144338931 learned_addr learned my addr 172.21.15.29:0/144338931 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f190081e640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc102a60 0x7f18fc19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.350+0000 7f190101f640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18fc108cf0 0x7f18fc19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18fbfff640 1 -- 172.21.15.29:0/144338931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc102a60 msgr2=0x7f18fc19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18fbfff640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc102a60 0x7f18fc19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18fbfff640 1 -- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18fc108cf0 msgr2=0x7f18fc19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18fbfff640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18fc108cf0 0x7f18fc19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18fbfff640 1 -- 172.21.15.29:0/144338931 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18fc1a02c0 con 0x7f18fc1083a0 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18fbfff640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc1083a0 0x7f18fc19a8f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009db0 tx=0x7f18e800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18f9ffb640 1 -- 172.21.15.29:0/144338931 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e8005020 con 0x7f18fc1083a0 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18f9ffb640 1 -- 172.21.15.29:0/144338931 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f18e8005650 con 0x7f18fc1083a0 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18fc1a05b0 con 0x7f18fc1083a0 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.351+0000 7f18f9ffb640 1 -- 172.21.15.29:0/144338931 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e801a670 con 0x7f18fc1083a0 2024-09-21T02:53:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.352+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18fc10fc90 con 0x7f18fc1083a0 2024-09-21T02:53:32.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.353+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18fc103ce0 con 0x7f18fc1083a0 2024-09-21T02:53:32.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.354+0000 7f18f9ffb640 1 -- 172.21.15.29:0/144338931 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f18e80051c0 con 0x7f18fc1083a0 2024-09-21T02:53:32.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.355+0000 7f18f9ffb640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f18ec0b6590 0x7f18ec0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:32.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.355+0000 7f190081e640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f18ec0b6590 0x7f18ec0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:32.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.355+0000 7f18f9ffb640 1 -- 172.21.15.29:0/144338931 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f18e80eb240 con 0x7f18fc1083a0 2024-09-21T02:53:32.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.356+0000 7f190081e640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f18ec0b6590 0x7f18ec0b8aa0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f18f00057c0 tx=0x7f18f00058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:32.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.363+0000 7f18f9ffb640 1 -- 172.21.15.29:0/144338931 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f18e80b5b30 con 0x7f18fc1083a0 2024-09-21T02:53:32.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7857 8.61 MB/sec warmup 69 sec latency 104.300 ms 2024-09-21T02:53:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.715+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18fc10aac0 con 0x7f18fc1083a0 2024-09-21T02:53:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.716+0000 7f18f9ffb640 1 -- 172.21.15.29:0/144338931 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11979 (secure 0 0 0) 0x7f18e80baa70 con 0x7f18fc1083a0 2024-09-21T02:53:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:53:32.718 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"btime":"2024-09-21T02:52:29:480744+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:52:28.665505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:53:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2024-09-21T02:53:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.719+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f18ec0b6590 msgr2=0x7f18ec0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.719+0000 7f1902a82640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f18ec0b6590 0x7f18ec0b8aa0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f18f00057c0 tx=0x7f18f00058d0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.719+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc1083a0 msgr2=0x7f18fc19a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.719+0000 7f1902a82640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc1083a0 0x7f18fc19a8f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009db0 tx=0x7f18e800c6a0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.720+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 shutdown_connections 2024-09-21T02:53:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.720+0000 7f1902a82640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f18ec0b6590 0x7f18ec0b8aa0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.720+0000 7f1902a82640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18fc108cf0 0x7f18fc19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.720+0000 7f1902a82640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18fc1083a0 0x7f18fc19a8f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.720+0000 7f1902a82640 1 --2- 172.21.15.29:0/144338931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc102a60 0x7f18fc19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.720+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 >> 172.21.15.29:0/144338931 conn(0x7f18fc079730 msgr2=0x7f18fc1014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.720+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 shutdown_connections 2024-09-21T02:53:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:32.720+0000 7f1902a82640 1 -- 172.21.15.29:0/144338931 wait complete. 2024-09-21T02:53:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:33 smithi029 ceph-mon[31006]: pgmap v822: 129 pgs: 129 active+clean; 252 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 356 KiB/s wr, 52 op/s 2024-09-21T02:53:33.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/144338931' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:33.533 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 138.85 --await-for 45.0' 2024-09-21T02:53:33.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 138.85 --await-for 45.0 2024-09-21T02:53:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:33 smithi190 ceph-mon[33871]: pgmap v822: 129 pgs: 129 active+clean; 252 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 356 KiB/s wr, 52 op/s 2024-09-21T02:53:33.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:33 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/144338931' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:33.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8052 8.79 MB/sec warmup 70 sec latency 80.332 ms 2024-09-21T02:53:33.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:33 smithi066 ceph-mon[33436]: pgmap v822: 129 pgs: 129 active+clean; 252 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 356 KiB/s wr, 52 op/s 2024-09-21T02:53:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:33 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/144338931' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 -- 172.21.15.29:0/428640687 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f34102a80 msgr2=0x7f0f34102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/428640687 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f34102a80 0x7f0f34102e90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009900 tx=0x7f0f2402a3f0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 -- 172.21.15.29:0/428640687 shutdown_connections 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/428640687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f34108d10 0x7f0f3410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/428640687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f341083c0 0x7f0f341087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/428640687 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f34102a80 0x7f0f34102e90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 -- 172.21.15.29:0/428640687 >> 172.21.15.29:0/428640687 conn(0x7f0f34079760 msgr2=0x7f0f34079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 -- 172.21.15.29:0/428640687 shutdown_connections 2024-09-21T02:53:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.502+0000 7f0f39dd6640 1 -- 172.21.15.29:0/428640687 wait complete. 2024-09-21T02:53:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.503+0000 7f0f39dd6640 1 Processor -- start 2024-09-21T02:53:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.503+0000 7f0f39dd6640 1 -- start start 2024-09-21T02:53:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.503+0000 7f0f39dd6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f34102a80 0x7f0f3410d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.503+0000 7f0f39dd6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f341083c0 0x7f0f341094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.503+0000 7f0f39dd6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f34108d10 0x7f0f34109c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.503+0000 7f0f39dd6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f34112880 con 0x7f0f34102a80 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f39dd6640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0f34112700 con 0x7f0f341083c0 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f39dd6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f34112a00 con 0x7f0f34108d10 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f33fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f34108d10 0x7f0f34109c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f33fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f34108d10 0x7f0f34109c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39654/0 (socket says 172.21.15.29:39654) 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f33fff640 1 -- 172.21.15.29:0/3545345326 learned_addr learned my addr 172.21.15.29:0/3545345326 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f341083c0 0x7f0f341094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f337fe640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f34102a80 0x7f0f3410d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f33fff640 1 -- 172.21.15.29:0/3545345326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f341083c0 msgr2=0x7f0f341094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f33fff640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f341083c0 0x7f0f341094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f33fff640 1 -- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f34102a80 msgr2=0x7f0f3410d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f33fff640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f34102a80 0x7f0f3410d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f33fff640 1 -- 172.21.15.29:0/3545345326 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f341aca80 con 0x7f0f34108d10 2024-09-21T02:53:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.504+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f341083c0 0x7f0f341094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:53:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.505+0000 7f0f33fff640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f34108d10 0x7f0f34109c10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00eb10 tx=0x7f0f2c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.505+0000 7f0f30ff9640 1 -- 172.21.15.29:0/3545345326 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c004b20 con 0x7f0f34108d10 2024-09-21T02:53:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.505+0000 7f0f39dd6640 1 -- 172.21.15.29:0/3545345326 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f341accb0 con 0x7f0f34108d10 2024-09-21T02:53:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.505+0000 7f0f30ff9640 1 -- 172.21.15.29:0/3545345326 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f2c004cc0 con 0x7f0f34108d10 2024-09-21T02:53:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.505+0000 7f0f39dd6640 1 -- 172.21.15.29:0/3545345326 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f341ad190 con 0x7f0f34108d10 2024-09-21T02:53:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.505+0000 7f0f30ff9640 1 -- 172.21.15.29:0/3545345326 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c010690 con 0x7f0f34108d10 2024-09-21T02:53:34.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.508+0000 7f0f30ff9640 1 -- 172.21.15.29:0/3545345326 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f0f2c005000 con 0x7f0f34108d10 2024-09-21T02:53:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f30ff9640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f1c0b6590 0x7f0f1c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f4034a740 1 Processor -- start 2024-09-21T02:53:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f4034a740 1 -- start start 2024-09-21T02:53:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f337fe640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f1c0b6590 0x7f0f1c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f30ff9640 1 -- 172.21.15.29:0/3545345326 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0f2c0e7aa0 con 0x7f0f34108d10 2024-09-21T02:53:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8405e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a84068f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f4034a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e3a837cbb0 con 0x55e3a805f850 2024-09-21T02:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f4034a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e3a836bfe0 con 0x55e3a8036990 2024-09-21T02:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f4034a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e3a836fdf0 con 0x55e3a803eb70 2024-09-21T02:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.509+0000 7f0f33fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a84068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f33fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a84068f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39660/0 (socket says 172.21.15.29:39660) 2024-09-21T02:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f33fff640 1 -- 172.21.15.29:0/3922742608 learned_addr learned my addr 172.21.15.29:0/3922742608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f337fe640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f1c0b6590 0x7f0f1c0b8aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0f3410cf60 tx=0x7f0f24005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f33fff640 1 -- 172.21.15.29:0/3922742608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 msgr2=0x55e3a84063b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f33fff640 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84063b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f33fff640 1 -- 172.21.15.29:0/3922742608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 msgr2=0x55e3a8405e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f337fe640 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8405e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f33fff640 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8405e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f33fff640 1 -- 172.21.15.29:0/3922742608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3a80d9ba0 con 0x55e3a803eb70 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f337fe640 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8405e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.510+0000 7f0f33fff640 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a84068f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0e7dc0 tx=0x7f0f2c0ad680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=138a10974c66a7fd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.511+0000 7f0f19ffb640 1 -- 172.21.15.29:0/3922742608 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0b2020 con 0x55e3a803eb70 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.511+0000 7f0f19ffb640 1 -- 172.21.15.29:0/3922742608 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f2c0ad3c0 con 0x55e3a803eb70 2024-09-21T02:53:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.511+0000 7f0f19ffb640 1 -- 172.21.15.29:0/3922742608 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0b72d0 con 0x55e3a803eb70 2024-09-21T02:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 -- 172.21.15.29:0/3922742608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 msgr2=0x55e3a84068f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a84068f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0e7dc0 tx=0x7f0f2c0ad680 comp rx=0 tx=0).stop 2024-09-21T02:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 -- 172.21.15.29:0/3922742608 shutdown_connections 2024-09-21T02:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a84068f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3922742608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8405e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 -- 172.21.15.29:0/3922742608 >> 172.21.15.29:0/3922742608 conn(0x55e3a8053c40 msgr2=0x55e3a8403d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 -- 172.21.15.29:0/3922742608 shutdown_connections 2024-09-21T02:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.512+0000 7f0f4034a740 1 -- 172.21.15.29:0/3922742608 wait complete. 2024-09-21T02:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.513+0000 7f0f4034a740 1 -- start start 2024-09-21T02:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.513+0000 7f0f4034a740 1 Processor -- start 2024-09-21T02:53:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.513+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a8036990 0x55e3a8411a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.513+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a803eb70 0x55e3a8411f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.513+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a841d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.513+0000 7f0f4034a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e3a81cd7b0 con 0x55e3a805f850 2024-09-21T02:53:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.513+0000 7f0f4034a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8306e90 con 0x55e3a803eb70 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.513+0000 7f0f4034a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e3a84145f0 con 0x55e3a8036990 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a8036990 0x55e3a8411a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a8036990 0x55e3a8411a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39674/0 (socket says 172.21.15.29:39674) 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 -- 172.21.15.29:0/249837762 learned_addr learned my addr 172.21.15.29:0/249837762 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f4034a740 10 client.0 _ll_register_callbacks cb 0x55e3a8369710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a803eb70 0x55e3a8411f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f33fff640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a841d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 -- 172.21.15.29:0/249837762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a803eb70 msgr2=0x55e3a8411f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a803eb70 0x55e3a8411f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 -- 172.21.15.29:0/249837762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 msgr2=0x55e3a841d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a841d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 -- 172.21.15.29:0/249837762 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3a841de80 con 0x55e3a8036990 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a803eb70 0x55e3a8411f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f33fff640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a841d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f337fe640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a8036990 0x55e3a8411a50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0f24033000 tx=0x7f0f2402a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f049f9f1d04cde91 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.514+0000 7f0f12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:53:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.515+0000 7f0f12ffd640 1 -- 172.21.15.29:0/249837762 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2404d070 con 0x55e3a8036990 2024-09-21T02:53:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.515+0000 7f0f12ffd640 1 -- 172.21.15.29:0/249837762 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0f24003a00 con 0x55e3a8036990 2024-09-21T02:53:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.515+0000 7f0f4034a740 1 -- 172.21.15.29:0/249837762 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f2c0ad3c0 con 0x55e3a8036990 2024-09-21T02:53:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.515+0000 7f0f12ffd640 1 -- 172.21.15.29:0/249837762 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0f2405b050 con 0x55e3a8036990 2024-09-21T02:53:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.515+0000 7f0f4034a740 10 client.26971 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.515+0000 7f0f4034a740 1 -- 172.21.15.29:0/249837762 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e3a8420750 con 0x55e3a8036990 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.515+0000 7f0f12ffd640 1 -- 172.21.15.29:0/249837762 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f2c0ad3c0 con 0x55e3a8036990 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.516+0000 7f0f12ffd640 1 -- 172.21.15.29:0/249837762 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f0f24052980 con 0x55e3a8036990 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.516+0000 7f0f4034a740 10 client.26971 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.516+0000 7f0f4034a740 10 client.26971 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.516+0000 7f0f4034a740 20 client.26971 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.516+0000 7f0f4034a740 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e3a801e1d0 0x55e3a8423d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.516+0000 7f0f4034a740 4 client.26971 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.516+0000 7f0f4034a740 1 -- 172.21.15.29:0/249837762 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3a837eb00 con 0x55e3a801e1d0 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.516+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e3a801e1d0 0x55e3a8423d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.517+0000 7f0f12ffd640 10 client.26971 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.517+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e3a801e1d0 0x55e3a8423d20 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e98867584e6b827 server_cookie=cbfce666e104b081 in_seq=0 out_seq=0 2024-09-21T02:53:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f12ffd640 1 -- 172.21.15.29:0/249837762 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e3a837eb00 con 0x55e3a801e1d0 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f12ffd640 10 client.26971 handle_command_reply: tid=0 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 client.26971 shutdown 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 -- 172.21.15.29:0/249837762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a8036990 msgr2=0x55e3a8411a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a8036990 0x55e3a8411a50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0f24033000 tx=0x7f0f2402a3f0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 -- 172.21.15.29:0/249837762 shutdown_connections 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e3a801e1d0 0x55e3a8423d20 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a841d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a803eb70 0x55e3a8411f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f12ffd640 0 client.26971 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 --2- 172.21.15.29:0/249837762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a8036990 0x55e3a8411a50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.518+0000 7f0f4034a740 1 -- 172.21.15.29:0/249837762 >> 172.21.15.29:0/249837762 conn(0x55e3a8053c40 msgr2=0x55e3a8409000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.519+0000 7f0f4034a740 1 -- 172.21.15.29:0/249837762 shutdown_connections 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.519+0000 7f0f4034a740 1 -- 172.21.15.29:0/249837762 wait complete. 2024-09-21T02:53:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.519+0000 7f0f4034a740 20 client.26971 trim_cache size 0 max 16384 2024-09-21T02:53:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.542+0000 7f0f4034a740 1 Processor -- start 2024-09-21T02:53:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.542+0000 7f0f4034a740 1 -- start start 2024-09-21T02:53:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a840af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a840b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a840ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f4034a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2000fc00 con 0x55e3a805f850 2024-09-21T02:53:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f4034a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0f20011800 con 0x55e3a8036990 2024-09-21T02:53:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f4034a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f200102d0 con 0x55e3a803eb70 2024-09-21T02:53:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f32ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a840b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a840ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f32ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a840b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:39678/0 (socket says 172.21.15.29:39678) 2024-09-21T02:53:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f32ffd640 1 -- 172.21.15.29:0/2402742351 learned_addr learned my addr 172.21.15.29:0/2402742351 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.543+0000 7f0f337fe640 1 --2- 172.21.15.29:0/2402742351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a840af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f33fff640 1 -- 172.21.15.29:0/2402742351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 msgr2=0x55e3a840b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f33fff640 1 --2- 172.21.15.29:0/2402742351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a840b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f33fff640 1 -- 172.21.15.29:0/2402742351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 msgr2=0x55e3a840af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f33fff640 1 --2- 172.21.15.29:0/2402742351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a840af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f33fff640 1 -- 172.21.15.29:0/2402742351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3a840d0e0 con 0x55e3a805f850 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f33fff640 1 --2- 172.21.15.29:0/2402742351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a840ea40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0e7dc0 tx=0x7f0f2c00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646cc2afaf11c47c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f197fa640 1 -- 172.21.15.29:0/2402742351 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0b2050 con 0x55e3a805f850 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f197fa640 1 -- 172.21.15.29:0/2402742351 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f2c0adfb0 con 0x55e3a805f850 2024-09-21T02:53:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.544+0000 7f0f197fa640 1 -- 172.21.15.29:0/2402742351 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0bcb10 con 0x55e3a805f850 2024-09-21T02:53:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 -- 172.21.15.29:0/2402742351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 msgr2=0x55e3a840ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 --2- 172.21.15.29:0/2402742351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a840ea40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0e7dc0 tx=0x7f0f2c00cee0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 -- 172.21.15.29:0/2402742351 shutdown_connections 2024-09-21T02:53:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 --2- 172.21.15.29:0/2402742351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a840ea40 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 --2- 172.21.15.29:0/2402742351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a840b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 --2- 172.21.15.29:0/2402742351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a840af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 -- 172.21.15.29:0/2402742351 >> 172.21.15.29:0/2402742351 conn(0x55e3a8053c40 msgr2=0x55e3a84100d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 -- 172.21.15.29:0/2402742351 shutdown_connections 2024-09-21T02:53:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 -- 172.21.15.29:0/2402742351 wait complete. 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.546+0000 7f0f4034a740 1 -- start start 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f4034a740 1 Processor -- start 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84879b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a8487f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f4034a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8415f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f4034a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f20011070 con 0x55e3a805f850 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f4034a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0f200109a0 con 0x55e3a8036990 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f4034a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2000dbe0 con 0x55e3a803eb70 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f4034a740 10 client.0 _ll_register_callbacks cb 0x7f0f2c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84879b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84879b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:37880/0 (socket says 172.21.15.29:37880) 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.547+0000 7f0f337fe640 1 -- 172.21.15.29:0/3199656479 learned_addr learned my addr 172.21.15.29:0/3199656479 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f33fff640 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8415f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a8487f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f32ffd640 1 -- 172.21.15.29:0/3199656479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 msgr2=0x55e3a84879b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84879b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f32ffd640 1 -- 172.21.15.29:0/3199656479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 msgr2=0x55e3a8415f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8415f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f32ffd640 1 -- 172.21.15.29:0/3199656479 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3a84166e0 con 0x55e3a803eb70 2024-09-21T02:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f337fe640 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84879b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f32ffd640 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a8487f70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0f28000c00 tx=0x7f0f28013a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a5cb3d38ed19254 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.548+0000 7f0f12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.549+0000 7f0f12ffd640 1 -- 172.21.15.29:0/3199656479 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28029020 con 0x55e3a803eb70 2024-09-21T02:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.549+0000 7f0f4034a740 1 -- 172.21.15.29:0/3199656479 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f2c0adfb0 con 0x55e3a803eb70 2024-09-21T02:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.549+0000 7f0f12ffd640 1 -- 172.21.15.29:0/3199656479 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0f28005220 con 0x55e3a803eb70 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.549+0000 7f0f12ffd640 1 -- 172.21.15.29:0/3199656479 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0f28031050 con 0x55e3a803eb70 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.549+0000 7f0f4034a740 10 client.26977 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.549+0000 7f0f4034a740 1 -- 172.21.15.29:0/3199656479 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e3a848f110 con 0x55e3a803eb70 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.550+0000 7f0f12ffd640 1 -- 172.21.15.29:0/3199656479 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f2c0adfb0 con 0x55e3a803eb70 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.550+0000 7f0f12ffd640 1 -- 172.21.15.29:0/3199656479 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f0f28010660 con 0x55e3a803eb70 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.550+0000 7f0f4034a740 10 client.26977 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.550+0000 7f0f4034a740 10 client.26977 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.550+0000 7f0f4034a740 20 client.26977 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.550+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e3a7ff3580 0x55e3a8491a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.550+0000 7f0f4034a740 4 client.26977 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.85, "await_for": 45.0}] 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.550+0000 7f0f4034a740 1 -- 172.21.15.29:0/3199656479 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.85, "await_for": 45.0}) -- 0x55e3a837a5b0 con 0x55e3a7ff3580 2024-09-21T02:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.551+0000 7f0f337fe640 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e3a7ff3580 0x55e3a8491a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:34.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.552+0000 7f0f337fe640 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e3a7ff3580 0x55e3a8491a30 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6fe15236a46930 server_cookie=658645620e52e4b6 in_seq=0 out_seq=0 2024-09-21T02:53:34.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:34.552+0000 7f0f12ffd640 10 client.26977 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:34.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8178 8.83 MB/sec warmup 71 sec latency 165.231 ms 2024-09-21T02:53:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.047+0000 7f0f12ffd640 1 -- 172.21.15.29:0/3199656479 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e3a837a5b0 con 0x55e3a7ff3580 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.047+0000 7f0f12ffd640 10 client.26977 handle_command_reply: tid=0 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.047+0000 7f0f4034a740 1 client.26977 shutdown 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 -- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 msgr2=0x55e3a8487f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a8487f70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0f28000c00 tx=0x7f0f28013a50 comp rx=0 tx=0).stop 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 -- 172.21.15.29:0/3199656479 shutdown_connections 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e3a7ff3580 0x55e3a8491a30 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a805f850 0x55e3a8415f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3a803eb70 0x55e3a8487f70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f12ffd640 0 client.26977 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 --2- 172.21.15.29:0/3199656479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e3a8036990 0x55e3a84879b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 -- 172.21.15.29:0/3199656479 >> 172.21.15.29:0/3199656479 conn(0x55e3a8053c40 msgr2=0x55e3a840af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 -- 172.21.15.29:0/3199656479 shutdown_connections 2024-09-21T02:53:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 1 -- 172.21.15.29:0/3199656479 wait complete. 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.048+0000 7f0f4034a740 20 client.26977 trim_cache size 0 max 16384 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 33, 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 58, 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "2b162bc5": { 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 58, 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:53:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 138.8, 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:35.053 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:53:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 -- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f1c0b6590 msgr2=0x7f0f1c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f1c0b6590 0x7f0f1c0b8aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0f3410cf60 tx=0x7f0f24005fd0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 -- 172.21.15.29:0/3545345326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f34108d10 msgr2=0x7f0f34109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f34108d10 0x7f0f34109c10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00eb10 tx=0x7f0f2c00c2d0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 -- 172.21.15.29:0/3545345326 shutdown_connections 2024-09-21T02:53:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f1c0b6590 0x7f0f1c0b8aa0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f34108d10 0x7f0f34109c10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f341083c0 0x7f0f341094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 --2- 172.21.15.29:0/3545345326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f34102a80 0x7f0f3410d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 -- 172.21.15.29:0/3545345326 >> 172.21.15.29:0/3545345326 conn(0x7f0f34079760 msgr2=0x7f0f3410d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 -- 172.21.15.29:0/3545345326 shutdown_connections 2024-09-21T02:53:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:35.049+0000 7f0f39dd6640 1 -- 172.21.15.29:0/3545345326 wait complete. 2024-09-21T02:53:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:35 smithi190 ceph-mon[33871]: pgmap v823: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 447 KiB/s wr, 74 op/s 2024-09-21T02:53:35.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:35 smithi029 ceph-mon[31006]: pgmap v823: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 447 KiB/s wr, 74 op/s 2024-09-21T02:53:35.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 -- 172.21.15.29:0/16938671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e90102ae0 msgr2=0x7f6e90102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 --2- 172.21.15.29:0/16938671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e90102ae0 0x7f6e90102ef0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009900 tx=0x7f6e8002a410 comp rx=0 tx=0).stop 2024-09-21T02:53:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 -- 172.21.15.29:0/16938671 shutdown_connections 2024-09-21T02:53:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 --2- 172.21.15.29:0/16938671 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e901040d0 0x7f6e9010ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 --2- 172.21.15.29:0/16938671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e901034c0 0x7f6e90103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 --2- 172.21.15.29:0/16938671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e90102ae0 0x7f6e90102ef0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 -- 172.21.15.29:0/16938671 >> 172.21.15.29:0/16938671 conn(0x7f6e900fd690 msgr2=0x7f6e900ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 -- 172.21.15.29:0/16938671 shutdown_connections 2024-09-21T02:53:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.682+0000 7f6e95620640 1 -- 172.21.15.29:0/16938671 wait complete. 2024-09-21T02:53:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e95620640 1 Processor -- start 2024-09-21T02:53:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e95620640 1 -- start start 2024-09-21T02:53:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e95620640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e90102ae0 0x7f6e90101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e95620640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e901034c0 0x7f6e901a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e95620640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e901040d0 0x7f6e901a3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e95620640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9010e0b0 con 0x7f6e901040d0 2024-09-21T02:53:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e95620640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9010df30 con 0x7f6e901034c0 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e95620640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9010e230 con 0x7f6e90102ae0 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e901040d0 0x7f6e901a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e901040d0 0x7f6e901a3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53174/0 (socket says 172.21.15.29:53174) 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/3575127419 learned_addr learned my addr 172.21.15.29:0/3575127419 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e901034c0 0x7f6e901a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8effd640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e90102ae0 0x7f6e90101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/3575127419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e90102ae0 msgr2=0x7f6e90101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e90102ae0 0x7f6e90101fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/3575127419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e901034c0 msgr2=0x7f6e901a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.683+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e901034c0 0x7f6e901a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.684+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/3575127419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e901a3b70 con 0x7f6e901040d0 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.684+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e901034c0 0x7f6e901a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.684+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e901040d0 0x7f6e901a3560 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800eb10 tx=0x7f6e8800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.684+0000 7f6e77fff640 1 -- 172.21.15.29:0/3575127419 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88004b20 con 0x7f6e901040d0 2024-09-21T02:53:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.684+0000 7f6e77fff640 1 -- 172.21.15.29:0/3575127419 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e88004cc0 con 0x7f6e901040d0 2024-09-21T02:53:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.684+0000 7f6e95620640 1 -- 172.21.15.29:0/3575127419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e901a3e60 con 0x7f6e901040d0 2024-09-21T02:53:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.684+0000 7f6e77fff640 1 -- 172.21.15.29:0/3575127419 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88010690 con 0x7f6e901040d0 2024-09-21T02:53:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.684+0000 7f6e95620640 1 -- 172.21.15.29:0/3575127419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e901a42d0 con 0x7f6e901040d0 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.686+0000 7f6e9c667740 1 Processor -- start 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e77fff640 1 -- 172.21.15.29:0/3575127419 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f6e88005020 con 0x7f6e901040d0 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e9c667740 1 -- start start 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 0x55de796f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 0x55de796f8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e9c667740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55de796696f0 con 0x55de7934f1b0 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e9c667740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55de79658b20 con 0x55de793262f0 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e9c667740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55de7965c930 con 0x55de7932e4d0 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e8e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e8e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796f6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53182/0 (socket says 172.21.15.29:53182) 2024-09-21T02:53:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e8e7fc640 1 -- 172.21.15.29:0/3977133937 learned_addr learned my addr 172.21.15.29:0/3977133937 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 0x55de796f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.687+0000 7f6e8effd640 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 0x55de796f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e8e7fc640 1 -- 172.21.15.29:0/3977133937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 msgr2=0x55de796f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 0x55de796f8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e8e7fc640 1 -- 172.21.15.29:0/3977133937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 msgr2=0x55de796f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 0x55de796f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e8e7fc640 1 -- 172.21.15.29:0/3977133937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de7961e530 con 0x55de7934f1b0 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e77fff640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e780b6510 0x7f6e780b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e8effd640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e780b6510 0x7f6e780b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796f6e50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6e84013c90 tx=0x7f6e84009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8db8509a48b0ba4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e77fff640 1 -- 172.21.15.29:0/3575127419 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6e880e7cc0 con 0x7f6e901040d0 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e757fa640 1 -- 172.21.15.29:0/3977133937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84013120 con 0x55de7934f1b0 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e757fa640 1 -- 172.21.15.29:0/3977133937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e8400fc50 con 0x55de7934f1b0 2024-09-21T02:53:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.688+0000 7f6e757fa640 1 -- 172.21.15.29:0/3977133937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84010d50 con 0x55de7934f1b0 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 -- 172.21.15.29:0/3977133937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 msgr2=0x55de796f6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e8effd640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e780b6510 0x7f6e780b8a20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6e80000c00 tx=0x7f6e80005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796f6e50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6e84013c90 tx=0x7f6e84009c60 comp rx=0 tx=0).stop 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 -- 172.21.15.29:0/3977133937 shutdown_connections 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 0x55de796f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 0x55de796f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3977133937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796f6e50 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 -- 172.21.15.29:0/3977133937 >> 172.21.15.29:0/3977133937 conn(0x55de79343300 msgr2=0x55de796f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 -- 172.21.15.29:0/3977133937 shutdown_connections 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 -- 172.21.15.29:0/3977133937 wait complete. 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 -- start start 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.689+0000 7f6e9c667740 1 Processor -- start 2024-09-21T02:53:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 0x55de797005b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 0x55de79700c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de797017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e9c667740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55de79590ab0 con 0x55de7934f1b0 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e9c667740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55de795fe730 con 0x55de793262f0 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e9c667740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55de796f8ad0 con 0x55de7932e4d0 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e9c667740 10 client.0 _ll_register_callbacks cb 0x55de796526b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de797017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de797017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53186/0 (socket says 172.21.15.29:53186) 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/3347152618 learned_addr learned my addr 172.21.15.29:0/3347152618 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 0x55de79700c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8effd640 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 0x55de797005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/3347152618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 msgr2=0x55de79700c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 0x55de79700c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/3347152618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 msgr2=0x55de797005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 0x55de797005b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/3347152618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de793ba790 con 0x55de7934f1b0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de797017e0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6e880049d0 tx=0x7f6e880ad960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363d3002fd4224cd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.690+0000 7f6e72ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e72ff5640 1 -- 172.21.15.29:0/3347152618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e880b2070 con 0x55de7934f1b0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e72ff5640 1 -- 172.21.15.29:0/3347152618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6e880a9050 con 0x55de7934f1b0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 1 -- 172.21.15.29:0/3347152618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e8400fc50 con 0x55de7934f1b0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e72ff5640 1 -- 172.21.15.29:0/3347152618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6e880ae1d0 con 0x55de7934f1b0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 10 client.17478 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 1 -- 172.21.15.29:0/3347152618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55de79433620 con 0x55de7934f1b0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e72ff5640 1 -- 172.21.15.29:0/3347152618 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e8400fc50 con 0x55de7934f1b0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e72ff5640 1 -- 172.21.15.29:0/3347152618 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f6e880b2210 con 0x55de7934f1b0 2024-09-21T02:53:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 10 client.17478 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 10 client.17478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 20 client.17478 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55de7930dd80 0x55de7970de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 4 client.17478 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.691+0000 7f6e9c667740 1 -- 172.21.15.29:0/3347152618 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de7966b640 con 0x55de7930dd80 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.692+0000 7f6e8effd640 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55de7930dd80 0x55de7970de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.692+0000 7f6e8effd640 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55de7930dd80 0x55de7970de40 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53ec8c444e28f5fc server_cookie=5cfe3b4979d525bb in_seq=0 out_seq=0 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.692+0000 7f6e72ff5640 10 client.17478 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e72ff5640 1 -- 172.21.15.29:0/3347152618 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55de7966b640 con 0x55de7930dd80 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e72ff5640 10 client.17478 handle_command_reply: tid=0 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 client.17478 shutdown 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 -- 172.21.15.29:0/3347152618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 msgr2=0x55de797017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de797017e0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6e880049d0 tx=0x7f6e880ad960 comp rx=0 tx=0).stop 2024-09-21T02:53:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 -- 172.21.15.29:0/3347152618 shutdown_connections 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55de7930dd80 0x55de7970de40 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de797017e0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de7932e4d0 0x55de79700c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e72ff5640 0 client.17478 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 --2- 172.21.15.29:0/3347152618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de793262f0 0x55de797005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 -- 172.21.15.29:0/3347152618 >> 172.21.15.29:0/3347152618 conn(0x55de79343300 msgr2=0x55de796f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 -- 172.21.15.29:0/3347152618 shutdown_connections 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 1 -- 172.21.15.29:0/3347152618 wait complete. 2024-09-21T02:53:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.693+0000 7f6e9c667740 20 client.17478 trim_cache size 0 max 16384 2024-09-21T02:53:35.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e9c667740 1 Processor -- start 2024-09-21T02:53:35.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e9c667740 1 -- start start 2024-09-21T02:53:35.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 8.71 MB/sec warmup 72 sec latency 1165.258 ms 2024-09-21T02:53:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de796f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7932e4d0 0x55de796f7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796feb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e9c667740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901ae720 con 0x55de7934f1b0 2024-09-21T02:53:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e9c667740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901b0320 con 0x55de7932e4d0 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e9c667740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901aedf0 con 0x55de793262f0 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796feb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796feb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53202/0 (socket says 172.21.15.29:53202) 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/676449609 learned_addr learned my addr 172.21.15.29:0/676449609 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8effd640 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de796f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7932e4d0 0x55de796f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/676449609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 msgr2=0x55de796f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de796f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/676449609 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7932e4d0 msgr2=0x55de796f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7932e4d0 0x55de796f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 -- 172.21.15.29:0/676449609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de796f2e10 con 0x55de7934f1b0 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8effd640 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de796f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.708+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796feb40 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6e880aa0a0 tx=0x7f6e880ad9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a9352fa3231450 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e74ff9640 1 -- 172.21.15.29:0/676449609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e880b2040 con 0x55de7934f1b0 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e74ff9640 1 -- 172.21.15.29:0/676449609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e880b25b0 con 0x55de7934f1b0 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e74ff9640 1 -- 172.21.15.29:0/676449609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e880b22d0 con 0x55de7934f1b0 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 -- 172.21.15.29:0/676449609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 msgr2=0x55de796feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796feb40 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6e880aa0a0 tx=0x7f6e880ad9d0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 -- 172.21.15.29:0/676449609 shutdown_connections 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7934f1b0 0x55de796feb40 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7932e4d0 0x55de796f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 --2- 172.21.15.29:0/676449609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de796f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 -- 172.21.15.29:0/676449609 >> 172.21.15.29:0/676449609 conn(0x55de79343300 msgr2=0x55de796f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 -- 172.21.15.29:0/676449609 shutdown_connections 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.709+0000 7f6e9c667740 1 -- 172.21.15.29:0/676449609 wait complete. 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 1 -- start start 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 1 Processor -- start 2024-09-21T02:53:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de79708470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7932e4d0 0x55de79707130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7934f1b0 0x55de79707730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901afb90 con 0x55de7932e4d0 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901af4c0 con 0x55de7934f1b0 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901ac700 con 0x55de793262f0 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e9c667740 10 client.0 _ll_register_callbacks cb 0x7f6e880a9970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7932e4d0 0x55de79707130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7932e4d0 0x55de79707130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53218/0 (socket says 172.21.15.29:53218) 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8e7fc640 1 -- 172.21.15.29:0/2150799206 learned_addr learned my addr 172.21.15.29:0/2150799206 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8f7fe640 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7934f1b0 0x55de79707730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8effd640 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de79708470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8e7fc640 1 -- 172.21.15.29:0/2150799206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 msgr2=0x55de79708470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de79708470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8e7fc640 1 -- 172.21.15.29:0/2150799206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7934f1b0 msgr2=0x55de79707730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7934f1b0 0x55de79707730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.710+0000 7f6e8e7fc640 1 -- 172.21.15.29:0/2150799206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de79707f10 con 0x55de7932e4d0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e8e7fc640 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7932e4d0 0x55de79707130 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6e84013c90 tx=0x7f6e8400fe50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66619859892f94e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e72ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e72ff5640 1 -- 172.21.15.29:0/2150799206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84005450 con 0x55de7932e4d0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e72ff5640 1 -- 172.21.15.29:0/2150799206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f6e840055f0 con 0x55de7932e4d0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 1 -- 172.21.15.29:0/2150799206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e880b25b0 con 0x55de7932e4d0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e72ff5640 1 -- 172.21.15.29:0/2150799206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6e84031050 con 0x55de7932e4d0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 10 client.17484 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 1 -- 172.21.15.29:0/2150799206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55de79778800 con 0x55de7932e4d0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e72ff5640 1 -- 172.21.15.29:0/2150799206 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e880b25b0 con 0x55de7932e4d0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e72ff5640 1 -- 172.21.15.29:0/2150799206 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f6e840204c0 con 0x55de7932e4d0 2024-09-21T02:53:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 10 client.17484 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 10 client.17484 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:53:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 20 client.17484 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55de792e31c0 0x55de7977b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 4 client.17484 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:53:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.711+0000 7f6e9c667740 1 -- 172.21.15.29:0/2150799206 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de796670f0 con 0x55de792e31c0 2024-09-21T02:53:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.712+0000 7f6e8effd640 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55de792e31c0 0x55de7977b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.712+0000 7f6e8effd640 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55de792e31c0 0x55de7977b010 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47919c7a332af946 server_cookie=8e7bc2f7848d5fbd in_seq=0 out_seq=0 2024-09-21T02:53:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.712+0000 7f6e72ff5640 10 client.17484 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:35 smithi066 ceph-mon[33436]: pgmap v823: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 447 KiB/s wr, 74 op/s 2024-09-21T02:53:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.792+0000 7f6e72ff5640 1 -- 172.21.15.29:0/2150799206 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de796670f0 con 0x55de792e31c0 2024-09-21T02:53:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.792+0000 7f6e72ff5640 10 client.17484 handle_command_reply: tid=0 2024-09-21T02:53:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.792+0000 7f6e9c667740 1 client.17484 shutdown 2024-09-21T02:53:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 -- 172.21.15.29:0/2150799206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7932e4d0 msgr2=0x55de79707130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7932e4d0 0x55de79707130 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6e84013c90 tx=0x7f6e8400fe50 comp rx=0 tx=0).stop 2024-09-21T02:53:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 -- 172.21.15.29:0/2150799206 shutdown_connections 2024-09-21T02:53:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55de7934f1b0 0x55de79707730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55de792e31c0 0x55de7977b010 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de7932e4d0 0x55de79707130 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 --2- 172.21.15.29:0/2150799206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55de793262f0 0x55de79708470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e72ff5640 0 client.17484 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 -- 172.21.15.29:0/2150799206 >> 172.21.15.29:0/2150799206 conn(0x55de79343300 msgr2=0x55de796f15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 -- 172.21.15.29:0/2150799206 shutdown_connections 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 1 -- 172.21.15.29:0/2150799206 wait complete. 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e9c667740 20 client.17484 trim_cache size 0 max 16384 2024-09-21T02:53:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:53:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:53:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:53:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e95620640 1 -- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e780b6510 msgr2=0x7f6e780b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e95620640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e780b6510 0x7f6e780b8a20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6e80000c00 tx=0x7f6e80005d50 comp rx=0 tx=0).stop 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e95620640 1 -- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e901040d0 msgr2=0x7f6e901a3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e95620640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e901040d0 0x7f6e901a3560 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800eb10 tx=0x7f6e8800c2d0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.793+0000 7f6e95620640 1 -- 172.21.15.29:0/3575127419 shutdown_connections 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.794+0000 7f6e95620640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f6e780b6510 0x7f6e780b8a20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.794+0000 7f6e95620640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e901040d0 0x7f6e901a3560 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.794+0000 7f6e95620640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e901034c0 0x7f6e901a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.794+0000 7f6e95620640 1 --2- 172.21.15.29:0/3575127419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e90102ae0 0x7f6e90101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.794+0000 7f6e95620640 1 -- 172.21.15.29:0/3575127419 >> 172.21.15.29:0/3575127419 conn(0x7f6e900fd690 msgr2=0x7f6e90106b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.794+0000 7f6e95620640 1 -- 172.21.15.29:0/3575127419 shutdown_connections 2024-09-21T02:53:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:35.794+0000 7f6e95620640 1 -- 172.21.15.29:0/3575127419 wait complete. 2024-09-21T02:53:36.156 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2b162bc5', quiesce took 4.9 seconds. Will release after: 13.950000000000001 2024-09-21T02:53:36.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 8.59 MB/sec warmup 73 sec latency 2165.342 ms 2024-09-21T02:53:37.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:37 smithi190 ceph-mon[33871]: pgmap v824: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 846 KiB/s wr, 77 op/s 2024-09-21T02:53:37.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:37 smithi029 ceph-mon[31006]: pgmap v824: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 846 KiB/s wr, 77 op/s 2024-09-21T02:53:37.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 8.48 MB/sec warmup 74 sec latency 3165.469 ms 2024-09-21T02:53:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:37 smithi066 ceph-mon[33436]: pgmap v824: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 846 KiB/s wr, 77 op/s 2024-09-21T02:53:38.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:38 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:53:38.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:38 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:53:38.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 8.36 MB/sec warmup 75 sec latency 4165.558 ms 2024-09-21T02:53:38.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:38 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:53:39.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:39 smithi190 ceph-mon[33871]: pgmap v825: 129 pgs: 129 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-09-21T02:53:39.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:53:39.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:53:39.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:53:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:39 smithi029 ceph-mon[31006]: pgmap v825: 129 pgs: 129 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-09-21T02:53:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:53:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:53:39.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:53:39.709 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 8.25 MB/sec warmup 76 sec latency 5165.662 ms 2024-09-21T02:53:39.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:39 smithi066 ceph-mon[33436]: pgmap v825: 129 pgs: 129 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-09-21T02:53:39.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:53:39.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:53:39.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:53:40.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 8.15 MB/sec warmup 77 sec latency 6165.761 ms 2024-09-21T02:53:41.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:41 smithi190 ceph-mon[33871]: pgmap v826: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:53:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:41 smithi029 ceph-mon[31006]: pgmap v826: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:53:41.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 8.04 MB/sec warmup 78 sec latency 7165.876 ms 2024-09-21T02:53:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:41 smithi066 ceph-mon[33436]: pgmap v826: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-09-21T02:53:42.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.94 MB/sec warmup 79 sec latency 8165.985 ms 2024-09-21T02:53:43.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:43 smithi190 ceph-mon[33871]: pgmap v827: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:53:43.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:43 smithi029 ceph-mon[31006]: pgmap v827: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:53:43.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.84 MB/sec warmup 80 sec latency 9166.085 ms 2024-09-21T02:53:43.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:43 smithi066 ceph-mon[33436]: pgmap v827: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-09-21T02:53:44.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.74 MB/sec warmup 81 sec latency 10166.205 ms 2024-09-21T02:53:45.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:45 smithi190 ceph-mon[33871]: pgmap v828: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-09-21T02:53:45.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:45 smithi029 ceph-mon[31006]: pgmap v828: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-09-21T02:53:45.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.65 MB/sec warmup 82 sec latency 11166.268 ms 2024-09-21T02:53:45.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:45 smithi066 ceph-mon[33436]: pgmap v828: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-09-21T02:53:46.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.56 MB/sec warmup 83 sec latency 12166.357 ms 2024-09-21T02:53:47.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:47 smithi190 ceph-mon[33871]: pgmap v829: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-09-21T02:53:47.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:47 smithi029 ceph-mon[31006]: pgmap v829: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-09-21T02:53:47.710 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.47 MB/sec warmup 84 sec latency 13166.458 ms 2024-09-21T02:53:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:47 smithi066 ceph-mon[33436]: pgmap v829: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-09-21T02:53:48.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.38 MB/sec warmup 85 sec latency 14166.560 ms 2024-09-21T02:53:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 -- 172.21.15.29:0/2599730159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 msgr2=0x7f72c4102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 --2- 172.21.15.29:0/2599730159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 0x7f72c4102a10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009900 tx=0x7f72b402a410 comp rx=0 tx=0).stop 2024-09-21T02:53:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 -- 172.21.15.29:0/2599730159 shutdown_connections 2024-09-21T02:53:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 --2- 172.21.15.29:0/2599730159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72c4103fc0 0x7f72c410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 --2- 172.21.15.29:0/2599730159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c4103620 0x7f72c4103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 --2- 172.21.15.29:0/2599730159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 0x7f72c4102a10 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 -- 172.21.15.29:0/2599730159 >> 172.21.15.29:0/2599730159 conn(0x7f72c40fdee0 msgr2=0x7f72c4100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 -- 172.21.15.29:0/2599730159 shutdown_connections 2024-09-21T02:53:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.905+0000 7f72c9713640 1 -- 172.21.15.29:0/2599730159 wait complete. 2024-09-21T02:53:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.906+0000 7f72c9713640 1 Processor -- start 2024-09-21T02:53:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.906+0000 7f72c9713640 1 -- start start 2024-09-21T02:53:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c9713640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 0x7f72c4198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c9713640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c4103620 0x7f72c41993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c9713640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72c4103fc0 0x7f72c4199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c9713640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f72c410e080 con 0x7f72c41025a0 2024-09-21T02:53:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c9713640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f72c410df00 con 0x7f72c4103fc0 2024-09-21T02:53:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c9713640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f72c410e200 con 0x7f72c4103620 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 0x7f72c4198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 0x7f72c4198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55800/0 (socket says 172.21.15.29:55800) 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 -- 172.21.15.29:0/262213573 learned_addr learned my addr 172.21.15.29:0/262213573 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c27fc640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c4103620 0x7f72c41993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c37fe640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72c4103fc0 0x7f72c4199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 -- 172.21.15.29:0/262213573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c4103620 msgr2=0x7f72c41993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c4103620 0x7f72c41993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 -- 172.21.15.29:0/262213573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72c4103fc0 msgr2=0x7f72c4199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72c4103fc0 0x7f72c4199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 -- 172.21.15.29:0/262213573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c419a7a0 con 0x7f72c41025a0 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c37fe640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72c4103fc0 0x7f72c4199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72c2ffd640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 0x7f72c4198d60 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009230 tx=0x7f72b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.907+0000 7f72abfff640 1 -- 172.21.15.29:0/262213573 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b403f030 con 0x7f72c41025a0 2024-09-21T02:53:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.908+0000 7f72c9713640 1 -- 172.21.15.29:0/262213573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c419aa30 con 0x7f72c41025a0 2024-09-21T02:53:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.908+0000 7f72abfff640 1 -- 172.21.15.29:0/262213573 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72b4032e20 con 0x7f72c41025a0 2024-09-21T02:53:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.908+0000 7f72abfff640 1 -- 172.21.15.29:0/262213573 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4035070 con 0x7f72c41025a0 2024-09-21T02:53:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.908+0000 7f72c9713640 1 -- 172.21.15.29:0/262213573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c419ad40 con 0x7f72c41025a0 2024-09-21T02:53:48.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72d067d740 1 Processor -- start 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72d067d740 1 -- start start 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72abfff640 1 -- 172.21.15.29:0/262213573 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f72b4042030 con 0x7f72c41025a0 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72d067d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72d067d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bcb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72d067d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 0x564e12bcc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72d067d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e12b3d6f0 con 0x564e128231b0 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72d067d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564e12b2cb20 con 0x564e127fa2f0 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.910+0000 7f72d067d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564e12b30930 con 0x564e128024d0 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55806/0 (socket says 172.21.15.29:55806) 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 -- 172.21.15.29:0/3063975565 learned_addr learned my addr 172.21.15.29:0/3063975565 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c2ffd640 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bcb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c27fc640 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 0x564e12bcc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 -- 172.21.15.29:0/3063975565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 msgr2=0x564e12bcc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 0x564e12bcc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 -- 172.21.15.29:0/3063975565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 msgr2=0x564e12bcb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bcb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 -- 172.21.15.29:0/3063975565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e12af2530 con 0x564e128231b0 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c37fe640 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcae50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009f70 tx=0x7f72bc00f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d7056d1ff21a81 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72abfff640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72ac0b6510 0x7f72ac0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72a97fa640 1 -- 172.21.15.29:0/3063975565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc0039c0 con 0x564e128231b0 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72c2ffd640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72ac0b6510 0x7f72ac0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72a97fa640 1 -- 172.21.15.29:0/3063975565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72bc004030 con 0x564e128231b0 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.911+0000 7f72a97fa640 1 -- 172.21.15.29:0/3063975565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc018650 con 0x564e128231b0 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72abfff640 1 -- 172.21.15.29:0/262213573 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f72b4107d40 con 0x7f72c41025a0 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 -- 172.21.15.29:0/3063975565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 msgr2=0x564e12bcae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcae50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009f70 tx=0x7f72bc00f980 comp rx=0 tx=0).stop 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72c2ffd640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72ac0b6510 0x7f72ac0b8a20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f72b4004650 tx=0x7f72b4049de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 -- 172.21.15.29:0/3063975565 shutdown_connections 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 0x564e12bcc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bcb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 --2- 172.21.15.29:0/3063975565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcae50 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 -- 172.21.15.29:0/3063975565 >> 172.21.15.29:0/3063975565 conn(0x564e12817300 msgr2=0x564e12bc68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 -- 172.21.15.29:0/3063975565 shutdown_connections 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.912+0000 7f72d067d740 1 -- 172.21.15.29:0/3063975565 wait complete. 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 1 -- start start 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 1 Processor -- start 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e127fa2f0 0x564e12bd45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e128024d0 0x564e12bd4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 0x564e12bd5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e12a64ab0 con 0x564e127fa2f0 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564e12ad2730 con 0x564e128024d0 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564e12bccb30 con 0x564e128231b0 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72d067d740 10 client.0 _ll_register_callbacks cb 0x564e12b266b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e127fa2f0 0x564e12bd45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e127fa2f0 0x564e12bd45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55816/0 (socket says 172.21.15.29:55816) 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72c37fe640 1 -- 172.21.15.29:0/1552025114 learned_addr learned my addr 172.21.15.29:0/1552025114 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.913+0000 7f72c27fc640 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e128024d0 0x564e12bd4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72c37fe640 1 -- 172.21.15.29:0/1552025114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 msgr2=0x564e12bd5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72c37fe640 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 0x564e12bd5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72c37fe640 1 -- 172.21.15.29:0/1552025114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e128024d0 msgr2=0x564e12bd4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72c37fe640 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e128024d0 0x564e12bd4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72c37fe640 1 -- 172.21.15.29:0/1552025114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e1288e790 con 0x564e127fa2f0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72c37fe640 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e127fa2f0 0x564e12bd45d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0097d0 tx=0x7f72bc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48251643d5468d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1552025114 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc003dc0 con 0x564e127fa2f0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1552025114 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f72bc004a90 con 0x564e127fa2f0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72d067d740 1 -- 172.21.15.29:0/1552025114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72bc004030 con 0x564e127fa2f0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1552025114 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f72bc030050 con 0x564e127fa2f0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72d067d740 10 client.17496 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.914+0000 7f72d067d740 1 -- 172.21.15.29:0/1552025114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564e12907620 con 0x564e127fa2f0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1552025114 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72bc004030 con 0x564e127fa2f0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1552025114 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f72bc021050 con 0x564e127fa2f0 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72d067d740 10 client.17496 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72d067d740 10 client.17496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72d067d740 20 client.17496 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72d067d740 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e127e1d80 0x564e12be19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72d067d740 4 client.17496 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:53:48.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72d067d740 1 -- 172.21.15.29:0/1552025114 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e12b3f640 con 0x564e127e1d80 2024-09-21T02:53:48.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.915+0000 7f72c27fc640 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e127e1d80 0x564e12be19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.916+0000 7f72c27fc640 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e127e1d80 0x564e12be19d0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=535b749be51ce532 server_cookie=daff3251e487f2d4 in_seq=0 out_seq=0 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.916+0000 7f72a6ff5640 10 client.17496 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1552025114 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564e12b3f640 con 0x564e127e1d80 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72a6ff5640 10 client.17496 handle_command_reply: tid=0 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 client.17496 shutdown 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 -- 172.21.15.29:0/1552025114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e127fa2f0 msgr2=0x564e12bd45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e127fa2f0 0x564e12bd45d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0097d0 tx=0x7f72bc0039c0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 -- 172.21.15.29:0/1552025114 shutdown_connections 2024-09-21T02:53:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 0x564e12bd5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e128024d0 0x564e12bd4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e127e1d80 0x564e12be19d0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 --2- 172.21.15.29:0/1552025114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e127fa2f0 0x564e12bd45d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72a6ff5640 0 client.17496 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 -- 172.21.15.29:0/1552025114 >> 172.21.15.29:0/1552025114 conn(0x564e12817300 msgr2=0x564e12bcbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 -- 172.21.15.29:0/1552025114 shutdown_connections 2024-09-21T02:53:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 1 -- 172.21.15.29:0/1552025114 wait complete. 2024-09-21T02:53:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.917+0000 7f72d067d740 20 client.17496 trim_cache size 0 max 16384 2024-09-21T02:53:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72d067d740 1 Processor -- start 2024-09-21T02:53:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72d067d740 1 -- start start 2024-09-21T02:53:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72d067d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72d067d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 0x564e12bd37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72d067d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72d067d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41aa360 con 0x564e128231b0 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72d067d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41abf60 con 0x564e127fa2f0 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72d067d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41aaa30 con 0x564e128024d0 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55818/0 (socket says 172.21.15.29:55818) 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.932+0000 7f72c37fe640 1 -- 172.21.15.29:0/2114569905 learned_addr learned my addr 172.21.15.29:0/2114569905 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72c27fc640 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 0x564e12bd37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72c37fe640 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72c37fe640 1 -- 172.21.15.29:0/2114569905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 msgr2=0x564e12bd37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72c37fe640 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 0x564e12bd37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72c37fe640 1 -- 172.21.15.29:0/2114569905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 msgr2=0x564e12bd31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72c37fe640 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72c37fe640 1 -- 172.21.15.29:0/2114569905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e12bcfd90 con 0x564e128231b0 2024-09-21T02:53:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72c37fe640 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcadf0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f72bc004570 tx=0x7f72bc00f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdf59da418d9aaa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72a8ff9640 1 -- 172.21.15.29:0/2114569905 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc0225d0 con 0x564e128231b0 2024-09-21T02:53:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72a8ff9640 1 -- 172.21.15.29:0/2114569905 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72bc027070 con 0x564e128231b0 2024-09-21T02:53:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.933+0000 7f72a8ff9640 1 -- 172.21.15.29:0/2114569905 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc004020 con 0x564e128231b0 2024-09-21T02:53:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 -- 172.21.15.29:0/2114569905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 msgr2=0x564e12bcadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcadf0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f72bc004570 tx=0x7f72bc00f980 comp rx=0 tx=0).stop 2024-09-21T02:53:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 -- 172.21.15.29:0/2114569905 shutdown_connections 2024-09-21T02:53:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128231b0 0x564e12bcadf0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128024d0 0x564e12bd37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 --2- 172.21.15.29:0/2114569905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 -- 172.21.15.29:0/2114569905 >> 172.21.15.29:0/2114569905 conn(0x564e12817300 msgr2=0x564e12bd2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 -- 172.21.15.29:0/2114569905 shutdown_connections 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 -- 172.21.15.29:0/2114569905 wait complete. 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 -- start start 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 Processor -- start 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128024d0 0x564e12bd9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.934+0000 7f72d067d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 0x564e12bdab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72d067d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41ab7d0 con 0x564e128024d0 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72d067d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41ab100 con 0x564e127fa2f0 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72d067d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41a8340 con 0x564e128231b0 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72d067d740 10 client.0 _ll_register_callbacks cb 0x7f72bc009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128024d0 0x564e12bd9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128024d0 0x564e12bd9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55830/0 (socket says 172.21.15.29:55830) 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 -- 172.21.15.29:0/1799191595 learned_addr learned my addr 172.21.15.29:0/1799191595 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c37fe640 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 0x564e12bdab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c37fe640 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 -- 172.21.15.29:0/1799191595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 msgr2=0x564e12bdab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 0x564e12bdab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 -- 172.21.15.29:0/1799191595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 msgr2=0x564e12bd98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 -- 172.21.15.29:0/1799191595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e12bdb260 con 0x564e128024d0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c37fe640 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72c27fc640 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128024d0 0x564e12bd9f90 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x564e12be2140 tx=0x7f72b8012900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b61faf42df63f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1799191595 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b80035f0 con 0x564e128024d0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72d067d740 1 -- 172.21.15.29:0/1799191595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72bc027070 con 0x564e128024d0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.935+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1799191595 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f72b8003790 con 0x564e128024d0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1799191595 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f72b802e050 con 0x564e128024d0 2024-09-21T02:53:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72d067d740 10 client.17502 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72d067d740 1 -- 172.21.15.29:0/1799191595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564e12bdc650 con 0x564e128024d0 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1799191595 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72bc027070 con 0x564e128024d0 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1799191595 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f72b8011a30 con 0x564e128024d0 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72d067d740 10 client.17502 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72d067d740 10 client.17502 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72d067d740 20 client.17502 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72d067d740 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e127b71c0 0x564e12c4e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72d067d740 4 client.17502 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72d067d740 1 -- 172.21.15.29:0/1799191595 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e12b3b0f0 con 0x564e127b71c0 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.936+0000 7f72c37fe640 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e127b71c0 0x564e12c4e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.937+0000 7f72c37fe640 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e127b71c0 0x564e12c4e8f0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3bd03692d647ce8 server_cookie=99cec610c5225d3a in_seq=0 out_seq=0 2024-09-21T02:53:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.937+0000 7f72a6ff5640 10 client.17502 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72a6ff5640 1 -- 172.21.15.29:0/1799191595 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564e12b3b0f0 con 0x564e127b71c0 2024-09-21T02:53:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72a6ff5640 10 client.17502 handle_command_reply: tid=0 2024-09-21T02:53:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 client.17502 shutdown 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 -- 172.21.15.29:0/1799191595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128024d0 msgr2=0x564e12bd9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128024d0 0x564e12bd9f90 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x564e12be2140 tx=0x7f72b8012900 comp rx=0 tx=0).stop 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 -- 172.21.15.29:0/1799191595 shutdown_connections 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564e128231b0 0x564e12bdab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564e127b71c0 0x564e12c4e8f0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e128024d0 0x564e12bd9f90 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 --2- 172.21.15.29:0/1799191595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564e127fa2f0 0x564e12bd98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72a6ff5640 0 client.17502 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 -- 172.21.15.29:0/1799191595 >> 172.21.15.29:0/1799191595 conn(0x564e12817300 msgr2=0x564e12bd2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 -- 172.21.15.29:0/1799191595 shutdown_connections 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 1 -- 172.21.15.29:0/1799191595 wait complete. 2024-09-21T02:53:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72d067d740 20 client.17502 trim_cache size 0 max 16384 2024-09-21T02:53:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:53:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:53:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:53:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 -- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72ac0b6510 msgr2=0x7f72ac0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72ac0b6510 0x7f72ac0b8a20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f72b4004650 tx=0x7f72b4049de0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 -- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 msgr2=0x7f72c4198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 0x7f72c4198d60 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009230 tx=0x7f72b40042f0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 -- 172.21.15.29:0/262213573 shutdown_connections 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f72ac0b6510 0x7f72ac0b8a20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72c4103fc0 0x7f72c4199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c4103620 0x7f72c41993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 --2- 172.21.15.29:0/262213573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c41025a0 0x7f72c4198d60 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.968+0000 7f72c9713640 1 -- 172.21.15.29:0/262213573 >> 172.21.15.29:0/262213573 conn(0x7f72c40fdee0 msgr2=0x7f72c4106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.969+0000 7f72c9713640 1 -- 172.21.15.29:0/262213573 shutdown_connections 2024-09-21T02:53:48.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:53:48.969+0000 7f72c9713640 1 -- 172.21.15.29:0/262213573 wait complete. 2024-09-21T02:53:49.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:49 smithi190 ceph-mon[33871]: pgmap v830: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 634 KiB/s wr, 90 op/s 2024-09-21T02:53:49.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:49 smithi029 ceph-mon[31006]: pgmap v830: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 634 KiB/s wr, 90 op/s 2024-09-21T02:53:49.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.29 MB/sec warmup 86 sec latency 15166.656 ms 2024-09-21T02:53:49.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:49 smithi066 ceph-mon[33436]: pgmap v830: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 634 KiB/s wr, 90 op/s 2024-09-21T02:53:50.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:53:50.380 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:53:50.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.21 MB/sec warmup 87 sec latency 16166.712 ms 2024-09-21T02:53:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 -- 172.21.15.29:0/3343154932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 msgr2=0x7f9f8810d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 --2- 172.21.15.29:0/3343154932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 0x7f9f8810d8d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009900 tx=0x7f9f7802a410 comp rx=0 tx=0).stop 2024-09-21T02:53:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 -- 172.21.15.29:0/3343154932 shutdown_connections 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 --2- 172.21.15.29:0/3343154932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f8810de10 0x7f9f881102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 --2- 172.21.15.29:0/3343154932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 0x7f9f8810d8d0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 --2- 172.21.15.29:0/3343154932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f88104860 0x7f9f88104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 -- 172.21.15.29:0/3343154932 >> 172.21.15.29:0/3343154932 conn(0x7f9f88079730 msgr2=0x7f9f88079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 -- 172.21.15.29:0/3343154932 shutdown_connections 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.895+0000 7f9f90173640 1 -- 172.21.15.29:0/3343154932 wait complete. 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f90173640 1 Processor -- start 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f90173640 1 -- start start 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f90173640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f88104860 0x7f9f8819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f90173640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 0x7f9f8819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f90173640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f8810de10 0x7f9f8819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f90173640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881138d0 con 0x7f9f88105290 2024-09-21T02:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f90173640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88113750 con 0x7f9f88104860 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f90173640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88113a50 con 0x7f9f8810de10 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f8d70e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 0x7f9f8819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f8d70e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 0x7f9f8819a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55838/0 (socket says 172.21.15.29:55838) 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f8d70e640 1 -- 172.21.15.29:0/542319394 learned_addr learned my addr 172.21.15.29:0/542319394 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f8e710640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f8810de10 0x7f9f8819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.896+0000 7f9f8df0f640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f88104860 0x7f9f8819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f8d70e640 1 -- 172.21.15.29:0/542319394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f8810de10 msgr2=0x7f9f8819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f8d70e640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f8810de10 0x7f9f8819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f8d70e640 1 -- 172.21.15.29:0/542319394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f88104860 msgr2=0x7f9f8819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f8d70e640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f88104860 0x7f9f8819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f8d70e640 1 -- 172.21.15.29:0/542319394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f881a0350 con 0x7f9f88105290 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f8d70e640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 0x7f9f8819a980 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f9f78005940 tx=0x7f9f780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f7effd640 1 -- 172.21.15.29:0/542319394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7803f030 con 0x7f9f88105290 2024-09-21T02:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f881a05e0 con 0x7f9f88105290 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f7effd640 1 -- 172.21.15.29:0/542319394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f78003c90 con 0x7f9f88105290 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f7effd640 1 -- 172.21.15.29:0/542319394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f78035070 con 0x7f9f88105290 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.897+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f881a6da0 con 0x7f9f88105290 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.899+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f88104c70 con 0x7f9f88105290 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.900+0000 7f9f7effd640 1 -- 172.21.15.29:0/542319394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9f78042030 con 0x7f9f88105290 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.900+0000 7f9f7effd640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f740b6510 0x7f9f740b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.901+0000 7f9f8df0f640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f740b6510 0x7f9f740b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.901+0000 7f9f7effd640 1 -- 172.21.15.29:0/542319394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9f78107f20 con 0x7f9f88105290 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.901+0000 7f9f8df0f640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f740b6510 0x7f9f740b8a20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9f840058c0 tx=0x7f9f8400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:50.905+0000 7f9f7effd640 1 -- 172.21.15.29:0/542319394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9f780d2810 con 0x7f9f88105290 2024-09-21T02:53:51.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.158+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f88108f60 con 0x7f9f88105290 2024-09-21T02:53:51.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.159+0000 7f9f7effd640 1 -- 172.21.15.29:0/542319394 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11979 (secure 0 0 0) 0x7f9f780d7750 con 0x7f9f88105290 2024-09-21T02:53:51.162 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:53:51.162 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"btime":"2024-09-21T02:52:29:480744+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:52:28.665505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:53:51.162 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2024-09-21T02:53:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f740b6510 msgr2=0x7f9f740b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f740b6510 0x7f9f740b8a20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9f840058c0 tx=0x7f9f8400a2d0 comp rx=0 tx=0).stop 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 msgr2=0x7f9f8819a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 0x7f9f8819a980 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f9f78005940 tx=0x7f9f780042f0 comp rx=0 tx=0).stop 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 shutdown_connections 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f740b6510 0x7f9f740b8a20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f8810de10 0x7f9f8819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f88105290 0x7f9f8819a980 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 --2- 172.21.15.29:0/542319394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f88104860 0x7f9f8819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 >> 172.21.15.29:0/542319394 conn(0x7f9f88079730 msgr2=0x7f9f881009c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 shutdown_connections 2024-09-21T02:53:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:51.162+0000 7f9f90173640 1 -- 172.21.15.29:0/542319394 wait complete. 2024-09-21T02:53:51.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:51 smithi190 ceph-mon[33871]: pgmap v831: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 197 KiB/s wr, 35 op/s 2024-09-21T02:53:51.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:51 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/542319394' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:51.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:51 smithi029 ceph-mon[31006]: pgmap v831: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 197 KiB/s wr, 35 op/s 2024-09-21T02:53:51.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:51 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/542319394' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:51.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.13 MB/sec warmup 88 sec latency 17166.758 ms 2024-09-21T02:53:51.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:51 smithi066 ceph-mon[33436]: pgmap v831: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 197 KiB/s wr, 35 op/s 2024-09-21T02:53:51.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:51 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/542319394' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:53:52.061 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 2b162bc5 --release --await' 2024-09-21T02:53:52.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 2b162bc5 --release --await 2024-09-21T02:53:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:53:52.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8184 7.05 MB/sec warmup 89 sec latency 18166.850 ms 2024-09-21T02:53:52.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 -- 172.21.15.29:0/1248735782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 msgr2=0x7fd3dc1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 --2- 172.21.15.29:0/1248735782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 0x7fd3dc1087e0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009900 tx=0x7fd3d002a410 comp rx=0 tx=0).stop 2024-09-21T02:53:52.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 -- 172.21.15.29:0/1248735782 shutdown_connections 2024-09-21T02:53:52.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 --2- 172.21.15.29:0/1248735782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc108d20 0x7fd3dc10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 --2- 172.21.15.29:0/1248735782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 0x7fd3dc1087e0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 --2- 172.21.15.29:0/1248735782 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3dc102ae0 0x7fd3dc102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 -- 172.21.15.29:0/1248735782 >> 172.21.15.29:0/1248735782 conn(0x7fd3dc079760 msgr2=0x7fd3dc079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 -- 172.21.15.29:0/1248735782 shutdown_connections 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.847+0000 7fd3e2373640 1 -- 172.21.15.29:0/1248735782 wait complete. 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.848+0000 7fd3e2373640 1 Processor -- start 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.848+0000 7fd3e2373640 1 -- start start 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.848+0000 7fd3e2373640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3dc102ae0 0x7fd3dc19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.848+0000 7fd3e2373640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 0x7fd3dc19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.848+0000 7fd3e2373640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc108d20 0x7fd3dc19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.848+0000 7fd3e2373640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc112890 con 0x7fd3dc1083d0 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.848+0000 7fd3e2373640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc112710 con 0x7fd3dc102ae0 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.848+0000 7fd3e2373640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc112a10 con 0x7fd3dc108d20 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 0x7fd3dc19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 0x7fd3dc19a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55852/0 (socket says 172.21.15.29:55852) 2024-09-21T02:53:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 -- 172.21.15.29:0/775123155 learned_addr learned my addr 172.21.15.29:0/775123155 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3dc102ae0 0x7fd3dc19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3e0910640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc108d20 0x7fd3dc19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 -- 172.21.15.29:0/775123155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc108d20 msgr2=0x7fd3dc19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc108d20 0x7fd3dc19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 -- 172.21.15.29:0/775123155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3dc102ae0 msgr2=0x7fd3dc19a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3dc102ae0 0x7fd3dc19a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 -- 172.21.15.29:0/775123155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc1a02c0 con 0x7fd3dc1083d0 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3dc102ae0 0x7fd3dc19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3e0910640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc108d20 0x7fd3dc19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 0x7fd3dc19a9b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd3d002a930 tx=0x7fd3d0003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.849+0000 7fd3d97fa640 1 -- 172.21.15.29:0/775123155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d003f070 con 0x7fd3dc1083d0 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.850+0000 7fd3e2373640 1 -- 172.21.15.29:0/775123155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3dc1a0550 con 0x7fd3dc1083d0 2024-09-21T02:53:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.850+0000 7fd3d97fa640 1 -- 172.21.15.29:0/775123155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3d0003e10 con 0x7fd3dc1083d0 2024-09-21T02:53:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.850+0000 7fd3d97fa640 1 -- 172.21.15.29:0/775123155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0035030 con 0x7fd3dc1083d0 2024-09-21T02:53:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.850+0000 7fd3e2373640 1 -- 172.21.15.29:0/775123155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3dc1a0a80 con 0x7fd3dc1083d0 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.852+0000 7fd3e88e7740 1 Processor -- start 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.852+0000 7fd3e88e7740 1 -- start start 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.852+0000 7fd3d97fa640 1 -- 172.21.15.29:0/775123155 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd3d0042070 con 0x7fd3dc1083d0 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011e3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011e3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5638010cedc0 con 0x563800e14660 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56380115b1f0 con 0x563800e1c840 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56380114a620 con 0x563800e3d520 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011e3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011e3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55856/0 (socket says 172.21.15.29:55856) 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3dbfff640 1 -- 172.21.15.29:0/3101532115 learned_addr learned my addr 172.21.15.29:0/3101532115 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3e0910640 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011e3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3dbfff640 1 -- 172.21.15.29:0/3101532115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 msgr2=0x5638011e3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011e3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3dbfff640 1 -- 172.21.15.29:0/3101532115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 msgr2=0x5638011e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3dbfff640 1 -- 172.21.15.29:0/3101532115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563801112fd0 con 0x563800e14660 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.853+0000 7fd3d97fa640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd3c40b6540 0x7fd3c40b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.854+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011e3a50 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc008c60 tx=0x7fd3cc00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79f00b3d155ce1b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.854+0000 7fd3e0910640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd3c40b6540 0x7fd3c40b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.854+0000 7fd3d97fa640 1 -- 172.21.15.29:0/775123155 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd3d0107f10 con 0x7fd3dc1083d0 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.854+0000 7fd3c27fc640 1 -- 172.21.15.29:0/3101532115 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc014070 con 0x563800e14660 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.854+0000 7fd3c27fc640 1 -- 172.21.15.29:0/3101532115 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3cc0047a0 con 0x563800e14660 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.854+0000 7fd3c27fc640 1 -- 172.21.15.29:0/3101532115 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc010690 con 0x563800e14660 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3101532115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 msgr2=0x5638011e3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e0910640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd3c40b6540 0x7fd3c40b8a50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4005fd0 tx=0x7fd3d4009780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011e3a50 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc008c60 tx=0x7fd3cc00e720 comp rx=0 tx=0).stop 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3101532115 shutdown_connections 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011e3a50 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3101532115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011e3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3101532115 >> 172.21.15.29:0/3101532115 conn(0x563800e31910 msgr2=0x5638011e1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3101532115 shutdown_connections 2024-09-21T02:53:52.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3101532115 wait complete. 2024-09-21T02:53:52.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.855+0000 7fd3e88e7740 1 -- start start 2024-09-21T02:53:52.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e88e7740 1 Processor -- start 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011f3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5638010e5300 con 0x563800e14660 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56380114e430 con 0x563800e1c840 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563800fabbb0 con 0x563800e3d520 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e88e7740 10 client.0 _ll_register_callbacks cb 0x5638010ff120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011f2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55862/0 (socket says 172.21.15.29:55862) 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3dbfff640 1 -- 172.21.15.29:0/3739111145 learned_addr learned my addr 172.21.15.29:0/3739111145 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3e0910640 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3dbfff640 1 -- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 msgr2=0x5638011f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3dbfff640 1 -- 172.21.15.29:0/3739111145 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 msgr2=0x5638011f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.856+0000 7fd3dbfff640 1 -- 172.21.15.29:0/3739111145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638010d4780 con 0x563800e14660 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3e0910640 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011f2f10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc008fa0 tx=0x7fd3cc0050f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de34a08d2965180 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3739111145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc027060 con 0x563800e14660 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3739111145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd3cc005470 con 0x563800e14660 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3739111145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3cc0047a0 con 0x563800e14660 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3739111145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd3cc02e050 con 0x563800e14660 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3e88e7740 10 client.17520 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3739111145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563800ea8b00 con 0x563800e14660 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.857+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3739111145 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3cc0047a0 con 0x563800e14660 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3739111145 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7fd3cc018410 con 0x563800e14660 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3e88e7740 10 client.17520 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3e88e7740 10 client.17520 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3e88e7740 20 client.17520 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563800dfbea0 0x563801205170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3e88e7740 4 client.17520 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3739111145 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56380115d140 con 0x563800dfbea0 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563800dfbea0 0x563801205170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3bb7fe640 10 client.17520 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.858+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563800dfbea0 0x563801205170 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=997fb0e3b67036e7 server_cookie=4a3af74021928b3d in_seq=0 out_seq=0 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.859+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3739111145 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56380115d140 con 0x563800dfbea0 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.859+0000 7fd3bb7fe640 10 client.17520 handle_command_reply: tid=0 2024-09-21T02:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.859+0000 7fd3e88e7740 1 client.17520 shutdown 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.859+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3739111145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 msgr2=0x5638011f2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.859+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011f2f10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc008fa0 tx=0x7fd3cc0050f0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3739111145 shutdown_connections 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e1c840 0x5638011f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563800dfbea0 0x563801205170 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3739111145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638011f2f10 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3bb7fe640 0 client.17520 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3739111145 >> 172.21.15.29:0/3739111145 conn(0x563800e31910 msgr2=0x5638011e4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3739111145 shutdown_connections 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3739111145 wait complete. 2024-09-21T02:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.860+0000 7fd3e88e7740 20 client.17520 trim_cache size 0 max 16384 2024-09-21T02:53:52.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3e88e7740 1 Processor -- start 2024-09-21T02:53:52.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3e88e7740 1 -- start start 2024-09-21T02:53:52.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e14660 0x5638011e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e1c840 0x5638011f0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc1ae760 con 0x563800e1c840 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc1b0360 con 0x563800e14660 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc1aee30 con 0x563800e3d520 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e1c840 0x5638011f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e1c840 0x5638011f0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55876/0 (socket says 172.21.15.29:55876) 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3db7fe640 1 -- 172.21.15.29:0/3999237527 learned_addr learned my addr 172.21.15.29:0/3999237527 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.877+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e14660 0x5638011e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3e0910640 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3db7fe640 1 -- 172.21.15.29:0/3999237527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 msgr2=0x5638011f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3db7fe640 1 -- 172.21.15.29:0/3999237527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e14660 msgr2=0x5638011e4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e14660 0x5638011e4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3db7fe640 1 -- 172.21.15.29:0/3999237527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638011f1e70 con 0x563800e1c840 2024-09-21T02:53:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3e0910640 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:53:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e1c840 0x5638011f0100 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00ca2f0 tx=0x7fd3d0105560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81707ee3fb75aa39 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/3999237527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0049040 con 0x563800e1c840 2024-09-21T02:53:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/3999237527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3d00cd830 con 0x563800e1c840 2024-09-21T02:53:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.878+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/3999237527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00c9f90 con 0x563800e1c840 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3999237527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e1c840 msgr2=0x5638011f0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e1c840 0x5638011f0100 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00ca2f0 tx=0x7fd3d0105560 comp rx=0 tx=0).stop 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3999237527 shutdown_connections 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e3d520 0x5638011f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e1c840 0x5638011f0100 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/3999237527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e14660 0x5638011e4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3999237527 >> 172.21.15.29:0/3999237527 conn(0x563800e31910 msgr2=0x5638011e4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3999237527 shutdown_connections 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 -- 172.21.15.29:0/3999237527 wait complete. 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.879+0000 7fd3e88e7740 1 -- start start 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e88e7740 1 Processor -- start 2024-09-21T02:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638012065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e1c840 0x563801206b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e88e7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e3d520 0x56380126e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc1afbd0 con 0x563800e14660 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc1af500 con 0x563800e3d520 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e88e7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc1ac740 con 0x563800e1c840 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e88e7740 10 client.0 _ll_register_callbacks cb 0x7fd3d4009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638012065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638012065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55886/0 (socket says 172.21.15.29:55886) 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3dbfff640 1 -- 172.21.15.29:0/613095538 learned_addr learned my addr 172.21.15.29:0/613095538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3e0910640 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e3d520 0x56380126e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.880+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e1c840 0x563801206b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3dbfff640 1 -- 172.21.15.29:0/613095538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e1c840 msgr2=0x563801206b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e1c840 0x563801206b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3dbfff640 1 -- 172.21.15.29:0/613095538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e3d520 msgr2=0x56380126e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e3d520 0x56380126e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3dbfff640 1 -- 172.21.15.29:0/613095538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563800f1f960 con 0x563800e14660 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3dbfff640 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638012065c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc008fd0 tx=0x7fd3cc018640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68e0f95d70cb9bf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/613095538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc037060 con 0x563800e14660 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/613095538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd3cc0189d0 con 0x563800e14660 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3e88e7740 1 -- 172.21.15.29:0/613095538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3d00cd830 con 0x563800e14660 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/613095538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd3cc03e050 con 0x563800e14660 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3e88e7740 10 client.17526 fetch_fsmap learned FSMap version 33 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3e88e7740 1 -- 172.21.15.29:0/613095538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56380126fd90 con 0x563800e14660 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.881+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/613095538 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3d00cd830 con 0x563800e14660 2024-09-21T02:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/613095538 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7fd3cc018bd0 con 0x563800e14660 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3e88e7740 10 client.17526 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3e88e7740 10 client.17526 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3e88e7740 20 client.17526 populate_metadata read hostname 'smithi029' 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563800dd1250 0x5638012725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3e88e7740 4 client.17526 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2b162bc5", "release": true, "await": true}] 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3e88e7740 1 -- 172.21.15.29:0/613095538 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2b162bc5", "release": true, "await": true}) -- 0x563801158bf0 con 0x563800dd1250 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563800dd1250 0x5638012725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3bb7fe640 10 client.17526 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.882+0000 7fd3db7fe640 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563800dd1250 0x5638012725b0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=408bfd2a4b07023 server_cookie=4260acec28700ae7 in_seq=0 out_seq=0 2024-09-21T02:53:52.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.980+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/613095538 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563801158bf0 con 0x563800dd1250 2024-09-21T02:53:52.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.980+0000 7fd3bb7fe640 10 client.17526 handle_command_reply: tid=0 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.980+0000 7fd3e88e7740 1 client.17526 shutdown 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.980+0000 7fd3e88e7740 1 -- 172.21.15.29:0/613095538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 msgr2=0x5638012065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.980+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638012065c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc008fd0 tx=0x7fd3cc018640 comp rx=0 tx=0).stop 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.980+0000 7fd3e88e7740 1 -- 172.21.15.29:0/613095538 shutdown_connections 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.980+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x563800e3d520 0x56380126e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.981+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563800e1c840 0x563801206b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.981+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x563800dd1250 0x5638012725b0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.981+0000 7fd3e88e7740 1 --2- 172.21.15.29:0/613095538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563800e14660 0x5638012065c0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.981+0000 7fd3e88e7740 1 -- 172.21.15.29:0/613095538 >> 172.21.15.29:0/613095538 conn(0x563800e31910 msgr2=0x5638011e4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.981+0000 7fd3bb7fe640 0 client.17526 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.981+0000 7fd3e88e7740 1 -- 172.21.15.29:0/613095538 shutdown_connections 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.981+0000 7fd3e88e7740 1 -- 172.21.15.29:0/613095538 wait complete. 2024-09-21T02:53:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.981+0000 7fd3e88e7740 20 client.17526 trim_cache size 0 max 16384 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 33, 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 60, 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "2b162bc5": { 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 60, 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:53:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 138.8, 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:53:52.987 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:53:52.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 -- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd3c40b6540 msgr2=0x7fd3c40b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd3c40b6540 0x7fd3c40b8a50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4005fd0 tx=0x7fd3d4009780 comp rx=0 tx=0).stop 2024-09-21T02:53:52.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 -- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 msgr2=0x7fd3dc19a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 0x7fd3dc19a9b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd3d002a930 tx=0x7fd3d0003c20 comp rx=0 tx=0).stop 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 -- 172.21.15.29:0/775123155 shutdown_connections 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd3c40b6540 0x7fd3c40b8a50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc108d20 0x7fd3dc19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3dc1083d0 0x7fd3dc19a9b0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 --2- 172.21.15.29:0/775123155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3dc102ae0 0x7fd3dc19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 -- 172.21.15.29:0/775123155 >> 172.21.15.29:0/775123155 conn(0x7fd3dc079760 msgr2=0x7fd3dc06bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 -- 172.21.15.29:0/775123155 shutdown_connections 2024-09-21T02:53:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:53:52.982+0000 7fd3e2373640 1 -- 172.21.15.29:0/775123155 wait complete. 2024-09-21T02:53:53.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:53 smithi029 ceph-mon[31006]: pgmap v832: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 22 op/s 2024-09-21T02:53:53.623 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:53 smithi190 ceph-mon[33871]: pgmap v832: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 22 op/s 2024-09-21T02:53:53.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8298 7.06 MB/sec warmup 90 sec latency 18454.936 ms 2024-09-21T02:53:53.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:53 smithi066 ceph-mon[33436]: pgmap v832: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 22 op/s 2024-09-21T02:53:53.859 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2b162bc5', total seconds elapsed: 22.6 2024-09-21T02:53:54.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8444 7.06 MB/sec warmup 91 sec latency 82.574 ms 2024-09-21T02:53:55.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:55 smithi190 ceph-mon[33871]: pgmap v833: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 51 KiB/s wr, 23 op/s 2024-09-21T02:53:55.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:55 smithi029 ceph-mon[31006]: pgmap v833: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 51 KiB/s wr, 23 op/s 2024-09-21T02:53:55.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8542 7.00 MB/sec warmup 92 sec latency 91.649 ms 2024-09-21T02:53:55.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:55 smithi066 ceph-mon[33436]: pgmap v833: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 51 KiB/s wr, 23 op/s 2024-09-21T02:53:56.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8666 6.97 MB/sec warmup 93 sec latency 98.814 ms 2024-09-21T02:53:57.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:57 smithi190 ceph-mon[33871]: pgmap v834: 129 pgs: 129 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 327 KiB/s wr, 32 op/s 2024-09-21T02:53:57.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:57 smithi029 ceph-mon[31006]: pgmap v834: 129 pgs: 129 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 327 KiB/s wr, 32 op/s 2024-09-21T02:53:57.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8787 6.93 MB/sec warmup 94 sec latency 155.621 ms 2024-09-21T02:53:57.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:57 smithi066 ceph-mon[33436]: pgmap v834: 129 pgs: 129 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 327 KiB/s wr, 32 op/s 2024-09-21T02:53:58.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8900 6.88 MB/sec warmup 95 sec latency 232.135 ms 2024-09-21T02:53:59.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:53:59 smithi190 ceph-mon[33871]: pgmap v835: 129 pgs: 129 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 741 KiB/s wr, 55 op/s 2024-09-21T02:53:59.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:53:59 smithi029 ceph-mon[31006]: pgmap v835: 129 pgs: 129 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 741 KiB/s wr, 55 op/s 2024-09-21T02:53:59.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9044 6.87 MB/sec warmup 96 sec latency 127.299 ms 2024-09-21T02:53:59.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:53:59 smithi066 ceph-mon[33436]: pgmap v835: 129 pgs: 129 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 741 KiB/s wr, 55 op/s 2024-09-21T02:54:00.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9189 6.86 MB/sec warmup 97 sec latency 102.921 ms 2024-09-21T02:54:01.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:01 smithi190 ceph-mon[33871]: pgmap v836: 129 pgs: 129 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-09-21T02:54:01.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:01 smithi029 ceph-mon[31006]: pgmap v836: 129 pgs: 129 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-09-21T02:54:01.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9340 6.84 MB/sec warmup 98 sec latency 94.541 ms 2024-09-21T02:54:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:01 smithi066 ceph-mon[33436]: pgmap v836: 129 pgs: 129 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.3 MiB/s wr, 101 op/s 2024-09-21T02:54:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 -- 172.21.15.29:0/4192572718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0608106bb0 msgr2=0x7f0608108fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 --2- 172.21.15.29:0/4192572718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0608106bb0 0x7f0608108fd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009900 tx=0x7f05f402a410 comp rx=0 tx=0).stop 2024-09-21T02:54:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 -- 172.21.15.29:0/4192572718 shutdown_connections 2024-09-21T02:54:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 --2- 172.21.15.29:0/4192572718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0608109510 0x7f060810b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 --2- 172.21.15.29:0/4192572718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0608106bb0 0x7f0608108fd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 --2- 172.21.15.29:0/4192572718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0608101a60 0x7f0608101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 -- 172.21.15.29:0/4192572718 >> 172.21.15.29:0/4192572718 conn(0x7f06080fd680 msgr2=0x7f06080ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 -- 172.21.15.29:0/4192572718 shutdown_connections 2024-09-21T02:54:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.101+0000 7f060eddf640 1 -- 172.21.15.29:0/4192572718 wait complete. 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.102+0000 7f060eddf640 1 Processor -- start 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.102+0000 7f060eddf640 1 -- start start 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060eddf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0608101a60 0x7f060819c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060eddf640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0608106bb0 0x7f060819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060eddf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0608109510 0x7f060819d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060eddf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f060810f1c0 con 0x7f0608101a60 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060eddf640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f060810f040 con 0x7f0608106bb0 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060eddf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f060810f340 con 0x7f0608109510 2024-09-21T02:54:02.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060cb7b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0608101a60 0x7f060819c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f0607fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0608106bb0 0x7f060819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060cb7b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0608101a60 0x7f060819c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48800/0 (socket says 172.21.15.29:48800) 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060cb7b640 1 -- 172.21.15.29:0/2980868143 learned_addr learned my addr 172.21.15.29:0/2980868143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060d37c640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0608109510 0x7f060819d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060cb7b640 1 -- 172.21.15.29:0/2980868143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0608109510 msgr2=0x7f060819d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060cb7b640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0608109510 0x7f060819d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060cb7b640 1 -- 172.21.15.29:0/2980868143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0608106bb0 msgr2=0x7f060819c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060cb7b640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0608106bb0 0x7f060819c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.103+0000 7f060cb7b640 1 -- 172.21.15.29:0/2980868143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060819db20 con 0x7f0608101a60 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.104+0000 7f060cb7b640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0608101a60 0x7f060819c0e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e9f0 tx=0x7f05fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.104+0000 7f0605ffb640 1 -- 172.21.15.29:0/2980868143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00cd80 con 0x7f0608101a60 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.104+0000 7f0605ffb640 1 -- 172.21.15.29:0/2980868143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05fc0054b0 con 0x7f0608101a60 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.104+0000 7f0605ffb640 1 -- 172.21.15.29:0/2980868143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc010430 con 0x7f0608101a60 2024-09-21T02:54:02.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.104+0000 7f060eddf640 1 -- 172.21.15.29:0/2980868143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f060819de10 con 0x7f0608101a60 2024-09-21T02:54:02.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.104+0000 7f060eddf640 1 -- 172.21.15.29:0/2980868143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06081a4500 con 0x7f0608101a60 2024-09-21T02:54:02.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0615e27740 1 Processor -- start 2024-09-21T02:54:02.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0615e27740 1 -- start start 2024-09-21T02:54:02.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0605ffb640 1 -- 172.21.15.29:0/2980868143 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f05fc0046c0 con 0x7f0608101a60 2024-09-21T02:54:02.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0615e27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce47d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0615e27740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0615e27740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce47e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0615e27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ccce3e76f0 con 0x55ccce0cd1b0 2024-09-21T02:54:02.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0615e27740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55ccce3d6b20 con 0x55ccce0a42f0 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.108+0000 7f0615e27740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ccce3da930 con 0x55ccce0ac4d0 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:55842/0 (socket says 172.21.15.29:55842) 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 -- 172.21.15.29:0/110031528 learned_addr learned my addr 172.21.15.29:0/110031528 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f0607fff640 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce47e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060d37c640 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce47d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 -- 172.21.15.29:0/110031528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 msgr2=0x55ccce47e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce47e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 -- 172.21.15.29:0/110031528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 msgr2=0x55ccce47d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce47d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 -- 172.21.15.29:0/110031528 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccce39c530 con 0x55ccce0a42f0 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f060cb7b640 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47dcd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0eba30 tx=0x7f05fc0ebb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf6e3a4c3db4d12a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.109+0000 7f0605ffb640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05f80b6590 0x7f05f80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.110+0000 7f05f2ffd640 1 -- 172.21.15.29:0/110031528 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc0fb070 con 0x55ccce0a42f0 2024-09-21T02:54:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.110+0000 7f05f2ffd640 1 -- 172.21.15.29:0/110031528 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05fc0ad340 con 0x55ccce0a42f0 2024-09-21T02:54:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.110+0000 7f05f2ffd640 1 -- 172.21.15.29:0/110031528 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc0ad180 con 0x55ccce0a42f0 2024-09-21T02:54:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.110+0000 7f060d37c640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05f80b6590 0x7f05f80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.110+0000 7f0605ffb640 1 -- 172.21.15.29:0/2980868143 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f05fc0e7a20 con 0x7f0608101a60 2024-09-21T02:54:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 -- 172.21.15.29:0/110031528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 msgr2=0x55ccce47dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f060d37c640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05f80b6590 0x7f05f80b8aa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0608102860 tx=0x7f0600009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47dcd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0eba30 tx=0x7f05fc0ebb20 comp rx=0 tx=0).stop 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 -- 172.21.15.29:0/110031528 shutdown_connections 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce47e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47dcd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 --2- 172.21.15.29:0/110031528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce47d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 -- 172.21.15.29:0/110031528 >> 172.21.15.29:0/110031528 conn(0x55ccce0c1300 msgr2=0x55ccce4708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 -- 172.21.15.29:0/110031528 shutdown_connections 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 -- 172.21.15.29:0/110031528 wait complete. 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 -- start start 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.111+0000 7f0615e27740 1 Processor -- start 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f0615e27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce486e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f0615e27740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce487500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f0615e27740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce488070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f060cb7b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce486e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f0615e27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ccce30eab0 con 0x55ccce0a42f0 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f060cb7b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce486e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48818/0 (socket says 172.21.15.29:48818) 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f060cb7b640 1 -- 172.21.15.29:0/2980761453 learned_addr learned my addr 172.21.15.29:0/2980761453 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55ccce37c730 con 0x55ccce0cd1b0 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f0607fff640 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce487500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ccce47f340 con 0x55ccce0ac4d0 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f0615e27740 10 client.0 _ll_register_callbacks cb 0x55ccce3d06b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.112+0000 7f060d37c640 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce488070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0607fff640 1 -- 172.21.15.29:0/2980761453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 msgr2=0x55ccce488070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0607fff640 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce488070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0607fff640 1 -- 172.21.15.29:0/2980761453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 msgr2=0x55ccce486e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0607fff640 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce486e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0607fff640 1 -- 172.21.15.29:0/2980761453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccce138790 con 0x55ccce0ac4d0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0607fff640 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce487500 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f05f4005760 tx=0x7f05f4031870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c0d9b17924d81f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f05e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f05e7fff640 1 -- 172.21.15.29:0/2980761453 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4035040 con 0x55ccce0ac4d0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f05e7fff640 1 -- 172.21.15.29:0/2980761453 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f05f4004240 con 0x55ccce0ac4d0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05fc0ad340 con 0x55ccce0ac4d0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f05e7fff640 1 -- 172.21.15.29:0/2980761453 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f05f404e050 con 0x55ccce0ac4d0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0615e27740 10 client.27034 fetch_fsmap learned FSMap version 33 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.113+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ccce1b1620 con 0x55ccce0ac4d0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f05e7fff640 1 -- 172.21.15.29:0/2980761453 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05fc0ad340 con 0x55ccce0ac4d0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f05e7fff640 1 -- 172.21.15.29:0/2980761453 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f05f403c440 con 0x55ccce0ac4d0 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f0615e27740 10 client.27034 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f0615e27740 10 client.27034 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f0615e27740 20 client.27034 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f0615e27740 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ccce08bd80 0x55ccce494720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f0615e27740 4 client.27034 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:54:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccce3e9640 con 0x55ccce08bd80 2024-09-21T02:54:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.114+0000 7f060cb7b640 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ccce08bd80 0x55ccce494720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.115+0000 7f05e7fff640 10 client.27034 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.115+0000 7f060cb7b640 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ccce08bd80 0x55ccce494720 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c19378fd6b8dc72e server_cookie=886c3f4653ad73b6 in_seq=0 out_seq=0 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.116+0000 7f05e7fff640 1 -- 172.21.15.29:0/2980761453 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ccce3e9640 con 0x55ccce08bd80 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.116+0000 7f05e7fff640 10 client.27034 handle_command_reply: tid=0 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.116+0000 7f0615e27740 1 client.27034 shutdown 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.116+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 msgr2=0x55ccce487500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.116+0000 7f0615e27740 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce487500 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f05f4005760 tx=0x7f05f4031870 comp rx=0 tx=0).stop 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 shutdown_connections 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce488070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce487500 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ccce08bd80 0x55ccce494720 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 1 --2- 172.21.15.29:0/2980761453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce486e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f05e7fff640 0 client.27034 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:02.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 >> 172.21.15.29:0/2980761453 conn(0x55ccce0c1300 msgr2=0x55ccce46eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:02.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 shutdown_connections 2024-09-21T02:54:02.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 1 -- 172.21.15.29:0/2980761453 wait complete. 2024-09-21T02:54:02.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.117+0000 7f0615e27740 20 client.27034 trim_cache size 0 max 16384 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.145+0000 7f0615e27740 1 Processor -- start 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.145+0000 7f0615e27740 1 -- start start 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.145+0000 7f0615e27740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.145+0000 7f0615e27740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce470df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.145+0000 7f0615e27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce471330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.145+0000 7f0615e27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f06081aa350 con 0x55ccce0cd1b0 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.145+0000 7f0615e27740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f06081abf50 con 0x55ccce0a42f0 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.145+0000 7f0615e27740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06081aaa20 con 0x55ccce0ac4d0 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f060d37c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce471330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f060cb7b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f060d37c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce471330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48822/0 (socket says 172.21.15.29:48822) 2024-09-21T02:54:02.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f060d37c640 1 -- 172.21.15.29:0/3750624346 learned_addr learned my addr 172.21.15.29:0/3750624346 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:02.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f0607fff640 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce470df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f0607fff640 1 -- 172.21.15.29:0/3750624346 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 msgr2=0x55ccce47e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f0607fff640 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f0607fff640 1 -- 172.21.15.29:0/3750624346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 msgr2=0x55ccce471330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f0607fff640 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce471330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f0607fff640 1 -- 172.21.15.29:0/3750624346 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccce486bd0 con 0x55ccce0ac4d0 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f060d37c640 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce471330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f060cb7b640 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.146+0000 7f0607fff640 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce470df0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009900 tx=0x7f05f4009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f79383780cef811 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.147+0000 7f05f27fc640 1 -- 172.21.15.29:0/3750624346 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f404b070 con 0x55ccce0ac4d0 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.147+0000 7f05f27fc640 1 -- 172.21.15.29:0/3750624346 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05f4005850 con 0x55ccce0ac4d0 2024-09-21T02:54:02.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.147+0000 7f05f27fc640 1 -- 172.21.15.29:0/3750624346 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4035040 con 0x55ccce0ac4d0 2024-09-21T02:54:02.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 -- 172.21.15.29:0/3750624346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 msgr2=0x55ccce470df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce470df0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009900 tx=0x7f05f4009640 comp rx=0 tx=0).stop 2024-09-21T02:54:02.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 -- 172.21.15.29:0/3750624346 shutdown_connections 2024-09-21T02:54:02.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0cd1b0 0x55ccce471330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce470df0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 --2- 172.21.15.29:0/3750624346 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0a42f0 0x55ccce47e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 -- 172.21.15.29:0/3750624346 >> 172.21.15.29:0/3750624346 conn(0x55ccce0c1300 msgr2=0x55ccce470290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:02.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 -- 172.21.15.29:0/3750624346 shutdown_connections 2024-09-21T02:54:02.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.148+0000 7f0615e27740 1 -- 172.21.15.29:0/3750624346 wait complete. 2024-09-21T02:54:02.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.149+0000 7f0615e27740 1 -- start start 2024-09-21T02:54:02.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.149+0000 7f0615e27740 1 Processor -- start 2024-09-21T02:54:02.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.149+0000 7f0615e27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce48e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.149+0000 7f0615e27740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce48eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0615e27740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce48d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0607fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce48eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0607fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce48eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40950/0 (socket says 172.21.15.29:40950) 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0607fff640 1 -- 172.21.15.29:0/220393088 learned_addr learned my addr 172.21.15.29:0/220393088 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f06081ab7c0 con 0x55ccce0a42f0 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f06081ab0f0 con 0x55ccce0cd1b0 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06081a8330 con 0x55ccce0ac4d0 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f060d37c640 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce48d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f060cb7b640 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce48e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0615e27740 10 client.0 _ll_register_callbacks cb 0x7f05f4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f060d37c640 1 -- 172.21.15.29:0/220393088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 msgr2=0x55ccce48eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f060d37c640 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce48eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f060d37c640 1 -- 172.21.15.29:0/220393088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 msgr2=0x55ccce48e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f060d37c640 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce48e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f060d37c640 1 -- 172.21.15.29:0/220393088 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccce48dd30 con 0x55ccce0cd1b0 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.150+0000 7f0607fff640 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce48eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f060d37c640 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce48d5c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0600005fd0 tx=0x7f0600004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b169be95e51a9c88 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f05e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f05e7fff640 1 -- 172.21.15.29:0/220393088 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600004b20 con 0x55ccce0cd1b0 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f05e7fff640 1 -- 172.21.15.29:0/220393088 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0600004cc0 con 0x55ccce0cd1b0 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05f4005850 con 0x55ccce0cd1b0 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f05e7fff640 1 -- 172.21.15.29:0/220393088 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0600034050 con 0x55ccce0cd1b0 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f0615e27740 10 client.26903 fetch_fsmap learned FSMap version 33 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ccce48cc80 con 0x55ccce0cd1b0 2024-09-21T02:54:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.151+0000 7f05e7fff640 1 -- 172.21.15.29:0/220393088 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05f4005850 con 0x55ccce0cd1b0 2024-09-21T02:54:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f05e7fff640 1 -- 172.21.15.29:0/220393088 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f06000055d0 con 0x55ccce0cd1b0 2024-09-21T02:54:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f0615e27740 10 client.26903 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:54:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f0615e27740 10 client.26903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f0615e27740 20 client.26903 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f0615e27740 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ccce0611c0 0x55ccce501840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:02.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f0615e27740 4 client.26903 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:54:02.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ccce3e50f0 con 0x55ccce0611c0 2024-09-21T02:54:02.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f060cb7b640 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ccce0611c0 0x55ccce501840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:02.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.152+0000 7f05e7fff640 10 client.26903 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:02.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.153+0000 7f060cb7b640 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ccce0611c0 0x55ccce501840 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c934e43f3a199616 server_cookie=12e679bfe6ecc68a in_seq=0 out_seq=0 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f05e7fff640 1 -- 172.21.15.29:0/220393088 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ccce3e50f0 con 0x55ccce0611c0 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f05e7fff640 10 client.26903 handle_command_reply: tid=0 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 client.26903 shutdown 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 msgr2=0x55ccce48d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce48d5c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0600005fd0 tx=0x7f0600004a40 comp rx=0 tx=0).stop 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 shutdown_connections 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ccce0cd1b0 0x55ccce48d5c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccce0ac4d0 0x55ccce48eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ccce0611c0 0x55ccce501840 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 --2- 172.21.15.29:0/220393088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ccce0a42f0 0x55ccce48e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 >> 172.21.15.29:0/220393088 conn(0x55ccce0c1300 msgr2=0x55ccce470220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f05e7fff640 0 client.26903 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 shutdown_connections 2024-09-21T02:54:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 1 -- 172.21.15.29:0/220393088 wait complete. 2024-09-21T02:54:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.290+0000 7f0615e27740 20 client.26903 trim_cache size 0 max 16384 2024-09-21T02:54:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:54:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:54:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:54:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:54:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 -- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05f80b6590 msgr2=0x7f05f80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05f80b6590 0x7f05f80b8aa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0608102860 tx=0x7f0600009500 comp rx=0 tx=0).stop 2024-09-21T02:54:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 -- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0608101a60 msgr2=0x7f060819c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0608101a60 0x7f060819c0e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e9f0 tx=0x7f05fc00edf0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 -- 172.21.15.29:0/2980868143 shutdown_connections 2024-09-21T02:54:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f05f80b6590 0x7f05f80b8aa0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0608109510 0x7f060819d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0608106bb0 0x7f060819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 --2- 172.21.15.29:0/2980868143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0608101a60 0x7f060819c0e0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 -- 172.21.15.29:0/2980868143 >> 172.21.15.29:0/2980868143 conn(0x7f06080fd680 msgr2=0x7f0608103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 -- 172.21.15.29:0/2980868143 shutdown_connections 2024-09-21T02:54:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:02.291+0000 7f060eddf640 1 -- 172.21.15.29:0/2980868143 wait complete. 2024-09-21T02:54:02.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9472 6.82 MB/sec warmup 99 sec latency 173.855 ms 2024-09-21T02:54:03.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:03 smithi190 ceph-mon[33871]: pgmap v837: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-09-21T02:54:03.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:03 smithi029 ceph-mon[31006]: pgmap v837: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-09-21T02:54:03.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9642 6.92 MB/sec warmup 100 sec latency 158.290 ms 2024-09-21T02:54:03.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:03 smithi066 ceph-mon[33436]: pgmap v837: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-09-21T02:54:04.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9794 6.94 MB/sec warmup 101 sec latency 97.550 ms 2024-09-21T02:54:05.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:05 smithi029 ceph-mon[31006]: pgmap v838: 129 pgs: 129 active+clean; 242 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 2.3 MiB/s wr, 222 op/s 2024-09-21T02:54:05.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9945 6.91 MB/sec warmup 102 sec latency 112.052 ms 2024-09-21T02:54:05.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:05 smithi066 ceph-mon[33436]: pgmap v838: 129 pgs: 129 active+clean; 242 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 2.3 MiB/s wr, 222 op/s 2024-09-21T02:54:05.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:05 smithi190 ceph-mon[33871]: pgmap v838: 129 pgs: 129 active+clean; 242 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 2.3 MiB/s wr, 222 op/s 2024-09-21T02:54:06.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10138 6.92 MB/sec warmup 103 sec latency 84.703 ms 2024-09-21T02:54:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:07 smithi029 ceph-mon[31006]: pgmap v839: 129 pgs: 129 active+clean; 237 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 2.8 MiB/s wr, 277 op/s 2024-09-21T02:54:07.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10293 6.88 MB/sec warmup 104 sec latency 71.267 ms 2024-09-21T02:54:07.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:07 smithi066 ceph-mon[33436]: pgmap v839: 129 pgs: 129 active+clean; 237 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 2.8 MiB/s wr, 277 op/s 2024-09-21T02:54:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:07 smithi190 ceph-mon[33871]: pgmap v839: 129 pgs: 129 active+clean; 237 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 2.8 MiB/s wr, 277 op/s 2024-09-21T02:54:08.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10452 6.88 MB/sec warmup 105 sec latency 94.224 ms 2024-09-21T02:54:09.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:09 smithi029 ceph-mon[31006]: pgmap v840: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 3.0 MiB/s wr, 322 op/s 2024-09-21T02:54:09.711 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10630 6.91 MB/sec warmup 106 sec latency 96.263 ms 2024-09-21T02:54:09.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:09 smithi066 ceph-mon[33436]: pgmap v840: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 3.0 MiB/s wr, 322 op/s 2024-09-21T02:54:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:09 smithi190 ceph-mon[33871]: pgmap v840: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 3.0 MiB/s wr, 322 op/s 2024-09-21T02:54:10.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10753 6.91 MB/sec warmup 107 sec latency 99.444 ms 2024-09-21T02:54:11.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:11 smithi029 ceph-mon[31006]: pgmap v841: 129 pgs: 129 active+clean; 241 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 2.8 MiB/s wr, 313 op/s 2024-09-21T02:54:11.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10890 6.91 MB/sec warmup 108 sec latency 127.304 ms 2024-09-21T02:54:11.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:11 smithi066 ceph-mon[33436]: pgmap v841: 129 pgs: 129 active+clean; 241 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 2.8 MiB/s wr, 313 op/s 2024-09-21T02:54:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:11 smithi190 ceph-mon[33871]: pgmap v841: 129 pgs: 129 active+clean; 241 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 2.8 MiB/s wr, 313 op/s 2024-09-21T02:54:12.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11024 6.90 MB/sec warmup 109 sec latency 101.521 ms 2024-09-21T02:54:13.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11262 7.06 MB/sec warmup 110 sec latency 111.525 ms 2024-09-21T02:54:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:13 smithi066 ceph-mon[33436]: pgmap v842: 129 pgs: 129 active+clean; 247 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 3.1 MiB/s wr, 317 op/s 2024-09-21T02:54:13.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:13 smithi190 ceph-mon[33871]: pgmap v842: 129 pgs: 129 active+clean; 247 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 3.1 MiB/s wr, 317 op/s 2024-09-21T02:54:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:13 smithi029 ceph-mon[31006]: pgmap v842: 129 pgs: 129 active+clean; 247 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 3.1 MiB/s wr, 317 op/s 2024-09-21T02:54:14.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11367 7.06 MB/sec warmup 111 sec latency 416.677 ms 2024-09-21T02:54:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.440+0000 7f484bb21640 1 -- 172.21.15.29:0/3290319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4844103620 msgr2=0x7f4844103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.440+0000 7f484bb21640 1 --2- 172.21.15.29:0/3290319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4844103620 0x7f4844103a30 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4834009900 tx=0x7f483402a410 comp rx=0 tx=0).stop 2024-09-21T02:54:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.440+0000 7f484bb21640 1 -- 172.21.15.29:0/3290319 shutdown_connections 2024-09-21T02:54:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.440+0000 7f484bb21640 1 --2- 172.21.15.29:0/3290319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4844103fc0 0x7f484410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.440+0000 7f484bb21640 1 --2- 172.21.15.29:0/3290319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4844103620 0x7f4844103a30 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.440+0000 7f484bb21640 1 --2- 172.21.15.29:0/3290319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48441025a0 0x7f4844102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.440+0000 7f484bb21640 1 -- 172.21.15.29:0/3290319 >> 172.21.15.29:0/3290319 conn(0x7f48440fdee0 msgr2=0x7f4844100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.440+0000 7f484bb21640 1 -- 172.21.15.29:0/3290319 shutdown_connections 2024-09-21T02:54:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.441+0000 7f484bb21640 1 -- 172.21.15.29:0/3290319 wait complete. 2024-09-21T02:54:15.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.441+0000 7f484bb21640 1 Processor -- start 2024-09-21T02:54:15.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f484bb21640 1 -- start start 2024-09-21T02:54:15.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f484bb21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48441025a0 0x7f4844199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f484bb21640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4844103620 0x7f484419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f484bb21640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4844103fc0 0x7f484419b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f484bb21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f484410f1e0 con 0x7f48441025a0 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f484bb21640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f484410f060 con 0x7f4844103620 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f484bb21640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f484410f360 con 0x7f4844103fc0 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f48498bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48441025a0 0x7f4844199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f48498bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48441025a0 0x7f4844199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58172/0 (socket says 172.21.15.29:58172) 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f48498bd640 1 -- 172.21.15.29:0/4229780282 learned_addr learned my addr 172.21.15.29:0/4229780282 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f484a0be640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4844103fc0 0x7f484419b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.442+0000 7f48490bc640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4844103620 0x7f484419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f48498bd640 1 -- 172.21.15.29:0/4229780282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4844103fc0 msgr2=0x7f484419b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f48498bd640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4844103fc0 0x7f484419b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f48498bd640 1 -- 172.21.15.29:0/4229780282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4844103620 msgr2=0x7f484419a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f48498bd640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4844103620 0x7f484419a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f48498bd640 1 -- 172.21.15.29:0/4229780282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484419b8c0 con 0x7f48441025a0 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f48498bd640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48441025a0 0x7f4844199e80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f484000e9f0 tx=0x7f484000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f483affd640 1 -- 172.21.15.29:0/4229780282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484000cd80 con 0x7f48441025a0 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f484bb21640 1 -- 172.21.15.29:0/4229780282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f484419bbb0 con 0x7f48441025a0 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f483affd640 1 -- 172.21.15.29:0/4229780282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48400108c0 con 0x7f48441025a0 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f483affd640 1 -- 172.21.15.29:0/4229780282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48400187c0 con 0x7f48441025a0 2024-09-21T02:54:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.443+0000 7f484bb21640 1 -- 172.21.15.29:0/4229780282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f484410b690 con 0x7f48441025a0 2024-09-21T02:54:15.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.447+0000 7f4852a88740 1 Processor -- start 2024-09-21T02:54:15.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.447+0000 7f4852a88740 1 -- start start 2024-09-21T02:54:15.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f4852a88740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7881b0 0x560d7fb38660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f4852a88740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb38cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f4852a88740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb39820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f4852a88740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560d7faa26f0 con 0x560d7f75f2f0 2024-09-21T02:54:15.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f4852a88740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560d7fa91b20 con 0x560d7f7881b0 2024-09-21T02:54:15.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f4852a88740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560d7fa95930 con 0x560d7f7674d0 2024-09-21T02:54:15.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f484a0be640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7881b0 0x560d7fb38660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f484a0be640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7881b0 0x560d7fb38660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:39152/0 (socket says 172.21.15.29:39152) 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f484a0be640 1 -- 172.21.15.29:0/2912753486 learned_addr learned my addr 172.21.15.29:0/2912753486 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f48490bc640 1 --2- 172.21.15.29:0/2912753486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb39820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f484a0be640 1 -- 172.21.15.29:0/2912753486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 msgr2=0x560d7fb39820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f484a0be640 1 --2- 172.21.15.29:0/2912753486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb39820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f484a0be640 1 -- 172.21.15.29:0/2912753486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 msgr2=0x560d7fb38cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f484a0be640 1 --2- 172.21.15.29:0/2912753486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb38cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.448+0000 7f484a0be640 1 -- 172.21.15.29:0/2912753486 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d7fa57530 con 0x560d7f7881b0 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.449+0000 7f483affd640 1 -- 172.21.15.29:0/4229780282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4840010430 con 0x7f48441025a0 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.449+0000 7f484a0be640 1 --2- 172.21.15.29:0/2912753486 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7881b0 0x560d7fb38660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f483c00e380 tx=0x7f483c0094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaaa2fc16ba48eaf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.449+0000 7f482ffff640 1 -- 172.21.15.29:0/2912753486 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c010630 con 0x560d7f7881b0 2024-09-21T02:54:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.449+0000 7f482ffff640 1 -- 172.21.15.29:0/2912753486 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f483c010c60 con 0x560d7f7881b0 2024-09-21T02:54:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.449+0000 7f482ffff640 1 -- 172.21.15.29:0/2912753486 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c019ca0 con 0x560d7f7881b0 2024-09-21T02:54:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.450+0000 7f483affd640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48300b6610 0x7f48300b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.450+0000 7f483affd640 1 -- 172.21.15.29:0/4229780282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f48400e79d0 con 0x7f48441025a0 2024-09-21T02:54:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.450+0000 7f48498bd640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48300b6610 0x7f48300b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 -- 172.21.15.29:0/2912753486 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7881b0 msgr2=0x560d7fb38660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 --2- 172.21.15.29:0/2912753486 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7881b0 0x560d7fb38660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f483c00e380 tx=0x7f483c0094e0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f48498bd640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48300b6610 0x7f48300b8b20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f48400ad530 tx=0x7f48400a9f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 -- 172.21.15.29:0/2912753486 shutdown_connections 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 --2- 172.21.15.29:0/2912753486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb39820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 --2- 172.21.15.29:0/2912753486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb38cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 --2- 172.21.15.29:0/2912753486 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7881b0 0x560d7fb38660 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 -- 172.21.15.29:0/2912753486 >> 172.21.15.29:0/2912753486 conn(0x560d7f77c300 msgr2=0x560d7fb2b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 -- 172.21.15.29:0/2912753486 shutdown_connections 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.451+0000 7f4852a88740 1 -- 172.21.15.29:0/2912753486 wait complete. 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.452+0000 7f4852a88740 1 -- start start 2024-09-21T02:54:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.452+0000 7f4852a88740 1 Processor -- start 2024-09-21T02:54:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.452+0000 7f4852a88740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f75f2f0 0x560d7fb41e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.452+0000 7f4852a88740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb42550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.452+0000 7f4852a88740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7881b0 0x560d7fb430c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.452+0000 7f4852a88740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560d7f9c9ab0 con 0x560d7f7881b0 2024-09-21T02:54:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.452+0000 7f4852a88740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560d7fa37730 con 0x560d7f75f2f0 2024-09-21T02:54:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.452+0000 7f4852a88740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560d7fb3a340 con 0x560d7f7674d0 2024-09-21T02:54:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f4852a88740 10 client.0 _ll_register_callbacks cb 0x560d7fa8b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f484a0be640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f75f2f0 0x560d7fb41e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f48490bc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb42550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f484a0be640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7881b0 0x560d7fb430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f48490bc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb42550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38182/0 (socket says 172.21.15.29:38182) 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f48490bc640 1 -- 172.21.15.29:0/4288353567 learned_addr learned my addr 172.21.15.29:0/4288353567 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f484a0be640 1 -- 172.21.15.29:0/4288353567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 msgr2=0x560d7fb42550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f484a0be640 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb42550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f484a0be640 1 -- 172.21.15.29:0/4288353567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7881b0 msgr2=0x560d7fb430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f484a0be640 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7881b0 0x560d7fb430c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f484a0be640 1 -- 172.21.15.29:0/4288353567 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d7f7f3790 con 0x560d7f75f2f0 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.454+0000 7f484a0be640 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f75f2f0 0x560d7fb41e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f483c0011e0 tx=0x7f483c0094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb6d961d21953ab server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.453+0000 7f48490bc640 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb42550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.454+0000 7f482d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:54:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.454+0000 7f482d7fa640 1 -- 172.21.15.29:0/4288353567 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c004ca0 con 0x560d7f75f2f0 2024-09-21T02:54:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.454+0000 7f482d7fa640 1 -- 172.21.15.29:0/4288353567 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f483c0105e0 con 0x560d7f75f2f0 2024-09-21T02:54:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.454+0000 7f4852a88740 1 -- 172.21.15.29:0/4288353567 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f483c010c60 con 0x560d7f75f2f0 2024-09-21T02:54:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.454+0000 7f482d7fa640 1 -- 172.21.15.29:0/4288353567 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f483c004e40 con 0x560d7f75f2f0 2024-09-21T02:54:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.455+0000 7f4852a88740 10 client.26915 fetch_fsmap learned FSMap version 33 2024-09-21T02:54:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.455+0000 7f4852a88740 1 -- 172.21.15.29:0/4288353567 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560d7f86c620 con 0x560d7f75f2f0 2024-09-21T02:54:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.455+0000 7f482d7fa640 1 -- 172.21.15.29:0/4288353567 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f483c0107e0 con 0x560d7f75f2f0 2024-09-21T02:54:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.455+0000 7f482d7fa640 1 -- 172.21.15.29:0/4288353567 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f483c030030 con 0x560d7f75f2f0 2024-09-21T02:54:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.455+0000 7f4852a88740 10 client.26915 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:54:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.455+0000 7f4852a88740 10 client.26915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.455+0000 7f4852a88740 20 client.26915 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.456+0000 7f4852a88740 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560d7f746d80 0x560d7fb4f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.456+0000 7f4852a88740 4 client.26915 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:54:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.456+0000 7f4852a88740 1 -- 172.21.15.29:0/4288353567 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f483c010c60 con 0x560d7f746d80 2024-09-21T02:54:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.457+0000 7f48490bc640 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560d7f746d80 0x560d7fb4f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.457+0000 7f482d7fa640 10 client.26915 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.457+0000 7f48490bc640 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560d7f746d80 0x560d7fb4f4c0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20927ccc6036a5a server_cookie=660e0481ac5d42dc in_seq=0 out_seq=0 2024-09-21T02:54:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.459+0000 7f482d7fa640 1 -- 172.21.15.29:0/4288353567 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f483c010c60 con 0x560d7f746d80 2024-09-21T02:54:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.459+0000 7f482d7fa640 10 client.26915 handle_command_reply: tid=0 2024-09-21T02:54:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.459+0000 7f4852a88740 1 client.26915 shutdown 2024-09-21T02:54:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f4852a88740 1 -- 172.21.15.29:0/4288353567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f75f2f0 msgr2=0x560d7fb41e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f4852a88740 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f75f2f0 0x560d7fb41e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f483c0011e0 tx=0x7f483c0094e0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f4852a88740 1 -- 172.21.15.29:0/4288353567 shutdown_connections 2024-09-21T02:54:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f4852a88740 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560d7f746d80 0x560d7fb4f4c0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f482d7fa640 0 client.26915 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f4852a88740 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7881b0 0x560d7fb430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f4852a88740 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f7674d0 0x560d7fb42550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f4852a88740 1 --2- 172.21.15.29:0/4288353567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f75f2f0 0x560d7fb41e90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.461+0000 7f4852a88740 1 -- 172.21.15.29:0/4288353567 >> 172.21.15.29:0/4288353567 conn(0x560d7f77c300 msgr2=0x560d7fb29ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.462+0000 7f4852a88740 1 -- 172.21.15.29:0/4288353567 shutdown_connections 2024-09-21T02:54:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.462+0000 7f4852a88740 1 -- 172.21.15.29:0/4288353567 wait complete. 2024-09-21T02:54:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.463+0000 7f4852a88740 20 client.26915 trim_cache size 0 max 16384 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.482+0000 7f4852a88740 1 Processor -- start 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.482+0000 7f4852a88740 1 -- start start 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.482+0000 7f4852a88740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f746d80 0x560d7fb2b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.482+0000 7f4852a88740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f75f2f0 0x560d7fb265c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.482+0000 7f4852a88740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7674d0 0x560d7fb2be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.482+0000 7f4852a88740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f48441aa390 con 0x560d7f7674d0 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.482+0000 7f4852a88740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f48441abf90 con 0x560d7f746d80 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.482+0000 7f4852a88740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48441aaa60 con 0x560d7f75f2f0 2024-09-21T02:54:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7674d0 0x560d7fb2be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7674d0 0x560d7fb2be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58198/0 (socket says 172.21.15.29:58198) 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 -- 172.21.15.29:0/1314240977 learned_addr learned my addr 172.21.15.29:0/1314240977 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f48490bc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f75f2f0 0x560d7fb265c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 --2- 172.21.15.29:0/1314240977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f746d80 0x560d7fb2b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 -- 172.21.15.29:0/1314240977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f75f2f0 msgr2=0x560d7fb265c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 --2- 172.21.15.29:0/1314240977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f75f2f0 0x560d7fb265c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 -- 172.21.15.29:0/1314240977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f746d80 msgr2=0x560d7fb2b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 --2- 172.21.15.29:0/1314240977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f746d80 0x560d7fb2b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 -- 172.21.15.29:0/1314240977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d7fb2afd0 con 0x560d7f7674d0 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f484a0be640 1 --2- 172.21.15.29:0/1314240977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7674d0 0x560d7fb2be10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f483c00e380 tx=0x7f483c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1328a76bd76cfe86 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.483+0000 7f482f7fe640 1 -- 172.21.15.29:0/1314240977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c004ca0 con 0x560d7f7674d0 2024-09-21T02:54:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.484+0000 7f482f7fe640 1 -- 172.21.15.29:0/1314240977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f483c010c60 con 0x560d7f7674d0 2024-09-21T02:54:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.484+0000 7f482f7fe640 1 -- 172.21.15.29:0/1314240977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c019410 con 0x560d7f7674d0 2024-09-21T02:54:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.484+0000 7f4852a88740 1 -- 172.21.15.29:0/1314240977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7674d0 msgr2=0x560d7fb2be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.484+0000 7f4852a88740 1 --2- 172.21.15.29:0/1314240977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7674d0 0x560d7fb2be10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f483c00e380 tx=0x7f483c000f40 comp rx=0 tx=0).stop 2024-09-21T02:54:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 -- 172.21.15.29:0/1314240977 shutdown_connections 2024-09-21T02:54:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 --2- 172.21.15.29:0/1314240977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f7674d0 0x560d7fb2be10 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 --2- 172.21.15.29:0/1314240977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f75f2f0 0x560d7fb265c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 --2- 172.21.15.29:0/1314240977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f746d80 0x560d7fb2b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 -- 172.21.15.29:0/1314240977 >> 172.21.15.29:0/1314240977 conn(0x560d7f77c300 msgr2=0x560d7fb40860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 -- 172.21.15.29:0/1314240977 shutdown_connections 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 -- 172.21.15.29:0/1314240977 wait complete. 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 -- start start 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 Processor -- start 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f746d80 0x560d7fb47370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb47a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7674d0 0x560d7fb484e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f48441ab800 con 0x560d7f75f2f0 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f48441ab130 con 0x560d7f7674d0 2024-09-21T02:54:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.485+0000 7f4852a88740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48441a8370 con 0x560d7f746d80 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f4852a88740 10 client.0 _ll_register_callbacks cb 0x7f483c009380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb47a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb47a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58200/0 (socket says 172.21.15.29:58200) 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 -- 172.21.15.29:0/193216927 learned_addr learned my addr 172.21.15.29:0/193216927 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f484a0be640 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f746d80 0x560d7fb47370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f484a0be640 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7674d0 0x560d7fb484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 -- 172.21.15.29:0/193216927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f746d80 msgr2=0x560d7fb47370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f746d80 0x560d7fb47370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 -- 172.21.15.29:0/193216927 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7674d0 msgr2=0x560d7fb484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7674d0 0x560d7fb484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 -- 172.21.15.29:0/193216927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d7fb48c40 con 0x560d7f75f2f0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f48490bc640 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb47a30 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f48340057c0 tx=0x7f4834031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dad1283a4824cd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f482d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f482d7fa640 1 -- 172.21.15.29:0/193216927 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834035040 con 0x560d7f75f2f0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f482d7fa640 1 -- 172.21.15.29:0/193216927 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f483403c440 con 0x560d7f75f2f0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.486+0000 7f4852a88740 1 -- 172.21.15.29:0/193216927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f483c010c60 con 0x560d7f75f2f0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f482d7fa640 1 -- 172.21.15.29:0/193216927 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f483403c640 con 0x560d7f75f2f0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f4852a88740 10 client.17559 fetch_fsmap learned FSMap version 33 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f4852a88740 1 -- 172.21.15.29:0/193216927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560d7fb4a000 con 0x560d7f75f2f0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f482d7fa640 1 -- 172.21.15.29:0/193216927 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f483c010c60 con 0x560d7f75f2f0 2024-09-21T02:54:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f482d7fa640 1 -- 172.21.15.29:0/193216927 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f483403b820 con 0x560d7f75f2f0 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f4852a88740 10 client.17559 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f4852a88740 10 client.17559 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f4852a88740 20 client.17559 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f4852a88740 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560d7f7881b0 0x560d7fbbc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f4852a88740 4 client.17559 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f4852a88740 1 -- 172.21.15.29:0/193216927 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560d7faa4640 con 0x560d7f7881b0 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.487+0000 7f484a0be640 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560d7f7881b0 0x560d7fbbc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.488+0000 7f482d7fa640 10 client.17559 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.488+0000 7f484a0be640 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560d7f7881b0 0x560d7fbbc470 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2b7c650339e0e3a server_cookie=7463791e8bbd5df2 in_seq=0 out_seq=0 2024-09-21T02:54:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.706+0000 7f482d7fa640 1 -- 172.21.15.29:0/193216927 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560d7faa4640 con 0x560d7f7881b0 2024-09-21T02:54:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.706+0000 7f482d7fa640 10 client.17559 handle_command_reply: tid=0 2024-09-21T02:54:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 client.17559 shutdown 2024-09-21T02:54:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 -- 172.21.15.29:0/193216927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 msgr2=0x560d7fb47a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb47a30 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f48340057c0 tx=0x7f4834031a90 comp rx=0 tx=0).stop 2024-09-21T02:54:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 -- 172.21.15.29:0/193216927 shutdown_connections 2024-09-21T02:54:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560d7f7674d0 0x560d7fb484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560d7f7881b0 0x560d7fbbc470 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f482d7fa640 0 client.17559 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560d7f75f2f0 0x560d7fb47a30 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 --2- 172.21.15.29:0/193216927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d7f746d80 0x560d7fb47370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 -- 172.21.15.29:0/193216927 >> 172.21.15.29:0/193216927 conn(0x560d7f77c300 msgr2=0x560d7fb2b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 -- 172.21.15.29:0/193216927 shutdown_connections 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 1 -- 172.21.15.29:0/193216927 wait complete. 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.707+0000 7f4852a88740 20 client.17559 trim_cache size 0 max 16384 2024-09-21T02:54:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:54:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:54:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:54:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:54:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 -- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48300b6610 msgr2=0x7f48300b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48300b6610 0x7f48300b8b20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f48400ad530 tx=0x7f48400a9f70 comp rx=0 tx=0).stop 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 -- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48441025a0 msgr2=0x7f4844199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48441025a0 0x7f4844199e80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f484000e9f0 tx=0x7f484000eeb0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 -- 172.21.15.29:0/4229780282 shutdown_connections 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48300b6610 0x7f48300b8b20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4844103fc0 0x7f484419b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4844103620 0x7f484419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 --2- 172.21.15.29:0/4229780282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48441025a0 0x7f4844199e80 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 -- 172.21.15.29:0/4229780282 >> 172.21.15.29:0/4229780282 conn(0x7f48440fdee0 msgr2=0x7f4844106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 -- 172.21.15.29:0/4229780282 shutdown_connections 2024-09-21T02:54:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:15.708+0000 7f484bb21640 1 -- 172.21.15.29:0/4229780282 wait complete. 2024-09-21T02:54:15.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11577 7.11 MB/sec warmup 112 sec latency 269.056 ms 2024-09-21T02:54:15.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:15 smithi066 ceph-mon[33436]: pgmap v843: 129 pgs: 129 active+clean; 249 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-09-21T02:54:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:15 smithi190 ceph-mon[33871]: pgmap v843: 129 pgs: 129 active+clean; 249 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-09-21T02:54:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:15 smithi029 ceph-mon[31006]: pgmap v843: 129 pgs: 129 active+clean; 249 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-09-21T02:54:16.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11715 7.13 MB/sec warmup 113 sec latency 253.953 ms 2024-09-21T02:54:17.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11822 7.11 MB/sec warmup 114 sec latency 101.227 ms 2024-09-21T02:54:17.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:17 smithi066 ceph-mon[33436]: pgmap v844: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.9 MiB/s wr, 258 op/s 2024-09-21T02:54:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:17 smithi190 ceph-mon[33871]: pgmap v844: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.9 MiB/s wr, 258 op/s 2024-09-21T02:54:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:17 smithi029 ceph-mon[31006]: pgmap v844: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.9 MiB/s wr, 258 op/s 2024-09-21T02:54:18.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11932 7.07 MB/sec warmup 115 sec latency 102.948 ms 2024-09-21T02:54:19.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12034 7.06 MB/sec warmup 116 sec latency 297.998 ms 2024-09-21T02:54:19.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:19 smithi066 ceph-mon[33436]: pgmap v845: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.9 MiB/s wr, 255 op/s 2024-09-21T02:54:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:19 smithi190 ceph-mon[33871]: pgmap v845: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.9 MiB/s wr, 255 op/s 2024-09-21T02:54:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:19 smithi029 ceph-mon[31006]: pgmap v845: 129 pgs: 129 active+clean; 256 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.9 MiB/s wr, 255 op/s 2024-09-21T02:54:20.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12211 7.16 MB/sec warmup 117 sec latency 118.561 ms 2024-09-21T02:54:21.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:21 smithi190 ceph-mon[33871]: pgmap v846: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 3.9 MiB/s wr, 251 op/s 2024-09-21T02:54:21.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12381 7.21 MB/sec warmup 118 sec latency 107.341 ms 2024-09-21T02:54:21.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:21 smithi066 ceph-mon[33436]: pgmap v846: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 3.9 MiB/s wr, 251 op/s 2024-09-21T02:54:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:21 smithi029 ceph-mon[31006]: pgmap v846: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 3.9 MiB/s wr, 251 op/s 2024-09-21T02:54:22.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12524 7.23 MB/sec warmup 119 sec latency 99.384 ms 2024-09-21T02:54:23.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:23 smithi066 ceph-mon[33436]: pgmap v847: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-09-21T02:54:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:23 smithi190 ceph-mon[33871]: pgmap v847: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-09-21T02:54:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:23 smithi029 ceph-mon[31006]: pgmap v847: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-09-21T02:54:24.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12744 4.91 MB/sec execute 1 sec latency 184.536 ms 2024-09-21T02:54:25.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12849 3.24 MB/sec execute 2 sec latency 181.398 ms 2024-09-21T02:54:25.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:25 smithi066 ceph-mon[33436]: pgmap v848: 129 pgs: 129 active+clean; 264 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 4.2 MiB/s wr, 250 op/s 2024-09-21T02:54:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:25 smithi190 ceph-mon[33871]: pgmap v848: 129 pgs: 129 active+clean; 264 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 4.2 MiB/s wr, 250 op/s 2024-09-21T02:54:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:25 smithi029 ceph-mon[31006]: pgmap v848: 129 pgs: 129 active+clean; 264 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 4.2 MiB/s wr, 250 op/s 2024-09-21T02:54:26.712 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12981 3.62 MB/sec execute 3 sec latency 99.625 ms 2024-09-21T02:54:27.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13117 3.58 MB/sec execute 4 sec latency 119.936 ms 2024-09-21T02:54:27.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:27 smithi066 ceph-mon[33436]: pgmap v849: 129 pgs: 129 active+clean; 268 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 4.5 MiB/s wr, 260 op/s 2024-09-21T02:54:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:27 smithi190 ceph-mon[33871]: pgmap v849: 129 pgs: 129 active+clean; 268 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 4.5 MiB/s wr, 260 op/s 2024-09-21T02:54:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:27 smithi029 ceph-mon[31006]: pgmap v849: 129 pgs: 129 active+clean; 268 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 4.5 MiB/s wr, 260 op/s 2024-09-21T02:54:28.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13241 3.45 MB/sec execute 5 sec latency 128.592 ms 2024-09-21T02:54:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 -- 172.21.15.29:0/2659204952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 msgr2=0x7fad68103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 --2- 172.21.15.29:0/2659204952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 0x7fad68103a60 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fad58009900 tx=0x7fad5802a410 comp rx=0 tx=0).stop 2024-09-21T02:54:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 -- 172.21.15.29:0/2659204952 shutdown_connections 2024-09-21T02:54:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 --2- 172.21.15.29:0/2659204952 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad68103ff0 0x7fad6810a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 --2- 172.21.15.29:0/2659204952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 0x7fad68103a60 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 --2- 172.21.15.29:0/2659204952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad681025d0 0x7fad68102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 -- 172.21.15.29:0/2659204952 >> 172.21.15.29:0/2659204952 conn(0x7fad680fdf30 msgr2=0x7fad681003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 -- 172.21.15.29:0/2659204952 shutdown_connections 2024-09-21T02:54:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.846+0000 7fad70362640 1 -- 172.21.15.29:0/2659204952 wait complete. 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad70362640 1 Processor -- start 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad70362640 1 -- start start 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad70362640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad681025d0 0x7fad68199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad70362640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 0x7fad6819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad70362640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad68103ff0 0x7fad6819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad70362640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fad6810e0b0 con 0x7fad68103650 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad70362640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fad6810df30 con 0x7fad681025d0 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad70362640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fad6810e230 con 0x7fad68103ff0 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.847+0000 7fad6d8fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 0x7fad6819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6d8fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 0x7fad6819a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51890/0 (socket says 172.21.15.29:51890) 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2731109375 learned_addr learned my addr 172.21.15.29:0/2731109375 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad68103ff0 0x7fad6819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad681025d0 0x7fad68199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2731109375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad68103ff0 msgr2=0x7fad6819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad68103ff0 0x7fad6819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2731109375 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad681025d0 msgr2=0x7fad68199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad681025d0 0x7fad68199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2731109375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6819b920 con 0x7fad68103650 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 0x7fad6819a550 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fad58009230 tx=0x7fad58004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad5f7fe640 1 -- 172.21.15.29:0/2731109375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5803f030 con 0x7fad68103650 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.849+0000 7fad70362640 1 -- 172.21.15.29:0/2731109375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6819bbb0 con 0x7fad68103650 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.848+0000 7fad5f7fe640 1 -- 172.21.15.29:0/2731109375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad58004440 con 0x7fad68103650 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.849+0000 7fad5f7fe640 1 -- 172.21.15.29:0/2731109375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58035070 con 0x7fad68103650 2024-09-21T02:54:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.849+0000 7fad70362640 1 -- 172.21.15.29:0/2731109375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad681a2300 con 0x7fad68103650 2024-09-21T02:54:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.852+0000 7fad5f7fe640 1 -- 172.21.15.29:0/2731109375 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fad58042030 con 0x7fad68103650 2024-09-21T02:54:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.853+0000 7fad772d2740 1 Processor -- start 2024-09-21T02:54:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.853+0000 7fad772d2740 1 -- start start 2024-09-21T02:54:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.853+0000 7fad772d2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec4f1b0 0x561ebeffb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.853+0000 7fad772d2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeffb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad772d2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec2e4d0 0x561ebeffc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6e8ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec4f1b0 0x561ebeffb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6e0fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeffb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6e0fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeffb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:38878/0 (socket says 172.21.15.29:38878) 2024-09-21T02:54:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6e8ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec4f1b0 0x561ebeffb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51894/0 (socket says 172.21.15.29:51894) 2024-09-21T02:54:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6e0fe640 1 -- 172.21.15.29:0/2973115068 learned_addr learned my addr 172.21.15.29:0/2973115068 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad772d2740 1 -- 172.21.15.29:0/2973115068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561ebef696f0 con 0x561ebec4f1b0 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad772d2740 1 -- 172.21.15.29:0/2973115068 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561ebef58b20 con 0x561ebec262f0 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad772d2740 1 -- 172.21.15.29:0/2973115068 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561ebef5c930 con 0x561ebec2e4d0 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec2e4d0 0x561ebeffc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad5f7fe640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fad540b6590 0x7fad540b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2973115068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 msgr2=0x561ebeffb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeffb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2973115068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec4f1b0 msgr2=0x561ebeffb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec4f1b0 0x561ebeffb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.854+0000 7fad5f7fe640 1 -- 172.21.15.29:0/2731109375 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fad58107ab0 con 0x7fad68103650 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.855+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2973115068 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ebef1e530 con 0x561ebec2e4d0 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.855+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeffb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.855+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec4f1b0 0x561ebeffb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.855+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec2e4d0 0x561ebeffc420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad58105200 tx=0x7fad58009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33086b760f6d29c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.855+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fad540b6590 0x7fad540b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.855+0000 7fad5cbfe640 1 -- 172.21.15.29:0/2973115068 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58049070 con 0x561ebec2e4d0 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.855+0000 7fad5cbfe640 1 -- 172.21.15.29:0/2973115068 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad580cd3d0 con 0x561ebec2e4d0 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.855+0000 7fad5cbfe640 1 -- 172.21.15.29:0/2973115068 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad580d23a0 con 0x561ebec2e4d0 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.856+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fad540b6590 0x7fad540b8aa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fad60009fd0 tx=0x7fad60004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.856+0000 7fad772d2740 1 -- 172.21.15.29:0/2973115068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec2e4d0 msgr2=0x561ebeffc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.856+0000 7fad772d2740 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec2e4d0 0x561ebeffc420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad58105200 tx=0x7fad58009260 comp rx=0 tx=0).stop 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.856+0000 7fad772d2740 1 -- 172.21.15.29:0/2973115068 shutdown_connections 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.856+0000 7fad772d2740 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec2e4d0 0x561ebeffc420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.857+0000 7fad772d2740 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeffb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.857+0000 7fad772d2740 1 --2- 172.21.15.29:0/2973115068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec4f1b0 0x561ebeffb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.857+0000 7fad772d2740 1 -- 172.21.15.29:0/2973115068 >> 172.21.15.29:0/2973115068 conn(0x561ebec43300 msgr2=0x561ebeff6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.857+0000 7fad772d2740 1 -- 172.21.15.29:0/2973115068 shutdown_connections 2024-09-21T02:54:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.857+0000 7fad772d2740 1 -- 172.21.15.29:0/2973115068 wait complete. 2024-09-21T02:54:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.857+0000 7fad772d2740 1 -- start start 2024-09-21T02:54:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.857+0000 7fad772d2740 1 Processor -- start 2024-09-21T02:54:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad772d2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebf004640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad772d2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec2e4d0 0x561ebf004d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6d8fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec2e4d0 0x561ebf004d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6d8fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec2e4d0 0x561ebf004d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51900/0 (socket says 172.21.15.29:51900) 2024-09-21T02:54:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2341294504 learned_addr learned my addr 172.21.15.29:0/2341294504 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad772d2740 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec4f1b0 0x561ebf005800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561ebee90ab0 con 0x561ebec2e4d0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561ebeefe730 con 0x561ebec262f0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561ebef5c930 con 0x561ebec4f1b0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebf004640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad772d2740 10 client.0 _ll_register_callbacks cb 0x561ebef526b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec4f1b0 0x561ebf005800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec4f1b0 msgr2=0x561ebf005800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec4f1b0 0x561ebf005800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2341294504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 msgr2=0x561ebf004640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebf004640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.858+0000 7fad6d8fd640 1 -- 172.21.15.29:0/2341294504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ebecba790 con 0x561ebec2e4d0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.859+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec2e4d0 0x561ebf004d00 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fad58105280 tx=0x7fad580c9dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac39c84cda360b1c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.859+0000 7fad51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:54:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.859+0000 7fad51ffb640 1 -- 172.21.15.29:0/2341294504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58049070 con 0x561ebec2e4d0 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.859+0000 7fad51ffb640 1 -- 172.21.15.29:0/2341294504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fad580d7050 con 0x561ebec2e4d0 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.859+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad580cd3d0 con 0x561ebec2e4d0 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.859+0000 7fad51ffb640 1 -- 172.21.15.29:0/2341294504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fad580d72e0 con 0x561ebec2e4d0 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.859+0000 7fad772d2740 10 client.17571 fetch_fsmap learned FSMap version 33 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.859+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561ebed33620 con 0x561ebec2e4d0 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad51ffb640 1 -- 172.21.15.29:0/2341294504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad580cd3d0 con 0x561ebec2e4d0 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad51ffb640 1 -- 172.21.15.29:0/2341294504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7fad580cccb0 con 0x561ebec2e4d0 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad772d2740 10 client.17571 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad772d2740 10 client.17571 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad772d2740 20 client.17571 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad772d2740 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561ebec0dd80 0x561ebf011a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad772d2740 4 client.17571 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ebef6b640 con 0x561ebec0dd80 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.860+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561ebec0dd80 0x561ebf011a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.861+0000 7fad51ffb640 10 client.17571 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.861+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561ebec0dd80 0x561ebf011a20 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4582697c53ae3fbd server_cookie=676d2709c088d794 in_seq=0 out_seq=0 2024-09-21T02:54:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.862+0000 7fad51ffb640 1 -- 172.21.15.29:0/2341294504 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561ebef6b640 con 0x561ebec0dd80 2024-09-21T02:54:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.862+0000 7fad51ffb640 10 client.17571 handle_command_reply: tid=0 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.862+0000 7fad772d2740 1 client.17571 shutdown 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec2e4d0 msgr2=0x561ebf004d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec2e4d0 0x561ebf004d00 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fad58105280 tx=0x7fad580c9dc0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 shutdown_connections 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec4f1b0 0x561ebf005800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561ebec0dd80 0x561ebf011a20 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad51ffb640 0 client.17571 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec2e4d0 0x561ebf004d00 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 --2- 172.21.15.29:0/2341294504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebf004640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 >> 172.21.15.29:0/2341294504 conn(0x561ebec43300 msgr2=0x561ebeff0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 shutdown_connections 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 1 -- 172.21.15.29:0/2341294504 wait complete. 2024-09-21T02:54:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.863+0000 7fad772d2740 20 client.17571 trim_cache size 0 max 16384 2024-09-21T02:54:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.890+0000 7fad772d2740 1 Processor -- start 2024-09-21T02:54:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.890+0000 7fad772d2740 1 -- start start 2024-09-21T02:54:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad772d2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec43300 0x561ebf003590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad772d2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec0dd80 0x561ebf00c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad772d2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeff0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad772d2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fad681aa3c0 con 0x561ebec0dd80 2024-09-21T02:54:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad772d2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fad681abfc0 con 0x561ebec262f0 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad772d2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fad681aaa90 con 0x561ebec43300 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e8ff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeff0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e8ff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeff0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:38896/0 (socket says 172.21.15.29:38896) 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e8ff640 1 -- 172.21.15.29:0/1678397092 learned_addr learned my addr 172.21.15.29:0/1678397092 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec0dd80 0x561ebf00c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec43300 0x561ebf003590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e8ff640 1 -- 172.21.15.29:0/1678397092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec43300 msgr2=0x561ebf003590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec43300 0x561ebf003590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e8ff640 1 -- 172.21.15.29:0/1678397092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec0dd80 msgr2=0x561ebf00c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec0dd80 0x561ebf00c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.891+0000 7fad6e8ff640 1 -- 172.21.15.29:0/1678397092 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ebf005b70 con 0x561ebec262f0 2024-09-21T02:54:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.892+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeff0ea0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fad6000f920 tx=0x7fad6001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=653355f105d2dab7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.892+0000 7fad53fff640 1 -- 172.21.15.29:0/1678397092 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6001d960 con 0x561ebec262f0 2024-09-21T02:54:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.892+0000 7fad53fff640 1 -- 172.21.15.29:0/1678397092 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad6002e070 con 0x561ebec262f0 2024-09-21T02:54:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.892+0000 7fad53fff640 1 -- 172.21.15.29:0/1678397092 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60029cc0 con 0x561ebec262f0 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 -- 172.21.15.29:0/1678397092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 msgr2=0x561ebeff0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeff0ea0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fad6000f920 tx=0x7fad6001a000 comp rx=0 tx=0).stop 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 -- 172.21.15.29:0/1678397092 shutdown_connections 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebeff0ea0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec0dd80 0x561ebf00c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 --2- 172.21.15.29:0/1678397092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec43300 0x561ebf003590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 -- 172.21.15.29:0/1678397092 >> 172.21.15.29:0/1678397092 conn(0x561ebeffba00 msgr2=0x561ebeff6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 -- 172.21.15.29:0/1678397092 shutdown_connections 2024-09-21T02:54:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 -- 172.21.15.29:0/1678397092 wait complete. 2024-09-21T02:54:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.893+0000 7fad772d2740 1 -- start start 2024-09-21T02:54:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad772d2740 1 Processor -- start 2024-09-21T02:54:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad772d2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec0dd80 0x561ebf00b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad772d2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebf00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad772d2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec43300 0x561ebf00bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad772d2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fad681ab830 con 0x561ebec43300 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad772d2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fad681ab160 con 0x561ebec262f0 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad772d2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fad681a83a0 con 0x561ebec0dd80 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad772d2740 10 client.0 _ll_register_callbacks cb 0x7fad580f1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad6e8ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec43300 0x561ebf00bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad6e0fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec0dd80 0x561ebf00b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad6e0fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec0dd80 0x561ebf00b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49394/0 (socket says 172.21.15.29:49394) 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad6e8ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec43300 0x561ebf00bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51914/0 (socket says 172.21.15.29:51914) 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad6e0fe640 1 -- 172.21.15.29:0/1918545389 learned_addr learned my addr 172.21.15.29:0/1918545389 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebf00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad6e0fe640 1 -- 172.21.15.29:0/1918545389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 msgr2=0x561ebf00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.894+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebf00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad6e0fe640 1 -- 172.21.15.29:0/1918545389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec43300 msgr2=0x561ebf00bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec43300 0x561ebf00bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad6e0fe640 1 -- 172.21.15.29:0/1918545389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ebeff6170 con 0x561ebec0dd80 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad6e8ff640 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec43300 0x561ebf00bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad6e0fe640 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec0dd80 0x561ebf00b450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad64004700 tx=0x7fad640042b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8516c1311da5475 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad51ffb640 1 -- 172.21.15.29:0/1918545389 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad64020410 con 0x561ebec0dd80 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad51ffb640 1 -- 172.21.15.29:0/1918545389 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fad640205b0 con 0x561ebec0dd80 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad772d2740 1 -- 172.21.15.29:0/1918545389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad6002e070 con 0x561ebec0dd80 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad51ffb640 1 -- 172.21.15.29:0/1918545389 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fad64032050 con 0x561ebec0dd80 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad772d2740 10 client.27076 fetch_fsmap learned FSMap version 33 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.895+0000 7fad772d2740 1 -- 172.21.15.29:0/1918545389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561ebf080c50 con 0x561ebec0dd80 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad51ffb640 1 -- 172.21.15.29:0/1918545389 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad6002e070 con 0x561ebec0dd80 2024-09-21T02:54:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad51ffb640 1 -- 172.21.15.29:0/1918545389 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7fad64010710 con 0x561ebec0dd80 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad772d2740 10 client.27076 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad772d2740 10 client.27076 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad772d2740 20 client.27076 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad772d2740 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561ebec4f1b0 0x561ebf083760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad772d2740 4 client.27076 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad772d2740 1 -- 172.21.15.29:0/1918545389 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561ebef670f0 con 0x561ebec4f1b0 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.896+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561ebec4f1b0 0x561ebf083760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.897+0000 7fad6d8fd640 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561ebec4f1b0 0x561ebf083760 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=318641448435a166 server_cookie=397c702bcb5faefc in_seq=0 out_seq=0 2024-09-21T02:54:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:28.897+0000 7fad51ffb640 10 client.27076 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad51ffb640 1 -- 172.21.15.29:0/1918545389 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561ebef670f0 con 0x561ebec4f1b0 2024-09-21T02:54:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad51ffb640 10 client.27076 handle_command_reply: tid=0 2024-09-21T02:54:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 client.27076 shutdown 2024-09-21T02:54:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 -- 172.21.15.29:0/1918545389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec0dd80 msgr2=0x561ebf00b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec0dd80 0x561ebf00b450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad64004700 tx=0x7fad640042b0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 -- 172.21.15.29:0/1918545389 shutdown_connections 2024-09-21T02:54:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561ebec4f1b0 0x561ebf083760 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ebec43300 0x561ebf00bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561ebec262f0 0x561ebf00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 --2- 172.21.15.29:0/1918545389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ebec0dd80 0x561ebf00b450 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.039+0000 7fad772d2740 1 -- 172.21.15.29:0/1918545389 >> 172.21.15.29:0/1918545389 conn(0x561ebeffba00 msgr2=0x561ebeff6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.040+0000 7fad51ffb640 0 client.27076 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.040+0000 7fad772d2740 1 -- 172.21.15.29:0/1918545389 shutdown_connections 2024-09-21T02:54:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.040+0000 7fad772d2740 1 -- 172.21.15.29:0/1918545389 wait complete. 2024-09-21T02:54:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.040+0000 7fad772d2740 20 client.27076 trim_cache size 0 max 16384 2024-09-21T02:54:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:54:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:54:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:54:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:54:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.042+0000 7fad70362640 1 -- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fad540b6590 msgr2=0x7fad540b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.042+0000 7fad70362640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fad540b6590 0x7fad540b8aa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fad60009fd0 tx=0x7fad60004a40 comp rx=0 tx=0).stop 2024-09-21T02:54:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.042+0000 7fad70362640 1 -- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 msgr2=0x7fad6819a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.042+0000 7fad70362640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 0x7fad6819a550 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fad58009230 tx=0x7fad58004040 comp rx=0 tx=0).stop 2024-09-21T02:54:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.043+0000 7fad70362640 1 -- 172.21.15.29:0/2731109375 shutdown_connections 2024-09-21T02:54:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.043+0000 7fad70362640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fad540b6590 0x7fad540b8aa0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.043+0000 7fad70362640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad68103ff0 0x7fad6819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.043+0000 7fad70362640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad68103650 0x7fad6819a550 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.043+0000 7fad70362640 1 --2- 172.21.15.29:0/2731109375 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad681025d0 0x7fad68199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.043+0000 7fad70362640 1 -- 172.21.15.29:0/2731109375 >> 172.21.15.29:0/2731109375 conn(0x7fad680fdf30 msgr2=0x7fad680ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.043+0000 7fad70362640 1 -- 172.21.15.29:0/2731109375 shutdown_connections 2024-09-21T02:54:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:29.043+0000 7fad70362640 1 -- 172.21.15.29:0/2731109375 wait complete. 2024-09-21T02:54:29.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13359 3.97 MB/sec execute 6 sec latency 157.754 ms 2024-09-21T02:54:29.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:29 smithi066 ceph-mon[33436]: pgmap v850: 129 pgs: 129 active+clean; 268 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.6 MiB/s wr, 251 op/s 2024-09-21T02:54:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:29 smithi190 ceph-mon[33871]: pgmap v850: 129 pgs: 129 active+clean; 268 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.6 MiB/s wr, 251 op/s 2024-09-21T02:54:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:29 smithi029 ceph-mon[31006]: pgmap v850: 129 pgs: 129 active+clean; 268 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.6 MiB/s wr, 251 op/s 2024-09-21T02:54:30.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13511 4.43 MB/sec execute 7 sec latency 95.246 ms 2024-09-21T02:54:31.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13683 5.52 MB/sec execute 8 sec latency 100.258 ms 2024-09-21T02:54:31.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:31 smithi066 ceph-mon[33436]: pgmap v851: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 3.6 MiB/s wr, 252 op/s 2024-09-21T02:54:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:31 smithi190 ceph-mon[33871]: pgmap v851: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 3.6 MiB/s wr, 252 op/s 2024-09-21T02:54:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:31 smithi029 ceph-mon[31006]: pgmap v851: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 3.6 MiB/s wr, 252 op/s 2024-09-21T02:54:32.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13826 5.48 MB/sec execute 9 sec latency 109.329 ms 2024-09-21T02:54:32.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:32 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:32 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:32 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:33.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13971 5.64 MB/sec execute 10 sec latency 179.047 ms 2024-09-21T02:54:33.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:33 smithi066 ceph-mon[33436]: pgmap v852: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:54:33.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:33 smithi029 ceph-mon[31006]: pgmap v852: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:54:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:33 smithi190 ceph-mon[33871]: pgmap v852: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:54:34.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14049 5.53 MB/sec execute 11 sec latency 538.544 ms 2024-09-21T02:54:35.713 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14203 5.32 MB/sec execute 12 sec latency 540.127 ms 2024-09-21T02:54:35.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:35 smithi066 ceph-mon[33436]: pgmap v853: 129 pgs: 129 active+clean; 249 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-09-21T02:54:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:35 smithi190 ceph-mon[33871]: pgmap v853: 129 pgs: 129 active+clean; 249 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-09-21T02:54:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:35 smithi029 ceph-mon[31006]: pgmap v853: 129 pgs: 129 active+clean; 249 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 3.6 MiB/s wr, 292 op/s 2024-09-21T02:54:36.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14389 5.33 MB/sec execute 13 sec latency 183.941 ms 2024-09-21T02:54:37.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14563 5.43 MB/sec execute 14 sec latency 91.108 ms 2024-09-21T02:54:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:37 smithi066 ceph-mon[33436]: pgmap v854: 129 pgs: 129 active+clean; 249 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 3.4 MiB/s wr, 307 op/s 2024-09-21T02:54:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:37 smithi190 ceph-mon[33871]: pgmap v854: 129 pgs: 129 active+clean; 249 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 3.4 MiB/s wr, 307 op/s 2024-09-21T02:54:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:37 smithi029 ceph-mon[31006]: pgmap v854: 129 pgs: 129 active+clean; 249 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 3.4 MiB/s wr, 307 op/s 2024-09-21T02:54:38.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14709 5.42 MB/sec execute 15 sec latency 86.816 ms 2024-09-21T02:54:39.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14850 5.57 MB/sec execute 16 sec latency 149.704 ms 2024-09-21T02:54:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:39 smithi066 ceph-mon[33436]: pgmap v855: 129 pgs: 129 active+clean; 250 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.8 MiB/s wr, 310 op/s 2024-09-21T02:54:39.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:39 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:54:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:39 smithi190 ceph-mon[33871]: pgmap v855: 129 pgs: 129 active+clean; 250 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.8 MiB/s wr, 310 op/s 2024-09-21T02:54:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:39 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:54:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:39 smithi029 ceph-mon[31006]: pgmap v855: 129 pgs: 129 active+clean; 250 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.8 MiB/s wr, 310 op/s 2024-09-21T02:54:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:39 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:54:40.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14985 5.64 MB/sec execute 17 sec latency 88.710 ms 2024-09-21T02:54:40.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:40 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:40.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:54:40.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:54:40.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:54:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:40 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:54:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:54:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:54:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:40 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:54:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:54:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:54:41.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 15143 5.79 MB/sec execute 18 sec latency 83.085 ms 2024-09-21T02:54:41.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:41 smithi066 ceph-mon[33436]: pgmap v856: 129 pgs: 129 active+clean; 250 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-09-21T02:54:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:41 smithi190 ceph-mon[33871]: pgmap v856: 129 pgs: 129 active+clean; 250 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-09-21T02:54:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:41 smithi029 ceph-mon[31006]: pgmap v856: 129 pgs: 129 active+clean; 250 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-09-21T02:54:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 -- 172.21.15.29:0/4264249190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e0103fe0 msgr2=0x7fa9e010a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/4264249190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e0103fe0 0x7fa9e010a980 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009ca0 tx=0x7fa9d802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 -- 172.21.15.29:0/4264249190 shutdown_connections 2024-09-21T02:54:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/4264249190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e0103fe0 0x7fa9e010a980 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/4264249190 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9e0103640 0x7fa9e0103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/4264249190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9e01025c0 0x7fa9e0102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 -- 172.21.15.29:0/4264249190 >> 172.21.15.29:0/4264249190 conn(0x7fa9e00fdf00 msgr2=0x7fa9e01003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 -- 172.21.15.29:0/4264249190 shutdown_connections 2024-09-21T02:54:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.177+0000 7fa9e4e69640 1 -- 172.21.15.29:0/4264249190 wait complete. 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.178+0000 7fa9e4e69640 1 Processor -- start 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.178+0000 7fa9e4e69640 1 -- start start 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.178+0000 7fa9e4e69640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e01025c0 0x7fa9e0199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.178+0000 7fa9e4e69640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9e0103640 0x7fa9e019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.178+0000 7fa9e4e69640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9e0103fe0 0x7fa9e019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.178+0000 7fa9e4e69640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e010e060 con 0x7fa9e01025c0 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.178+0000 7fa9e4e69640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e010dee0 con 0x7fa9e0103fe0 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.178+0000 7fa9e4e69640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e010e1e0 con 0x7fa9e0103640 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e01025c0 0x7fa9e0199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e01025c0 0x7fa9e0199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55224/0 (socket says 172.21.15.29:55224) 2024-09-21T02:54:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 -- 172.21.15.29:0/429869208 learned_addr learned my addr 172.21.15.29:0/429869208 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9e0103fe0 0x7fa9e019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9ddd9b640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9e0103640 0x7fa9e019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 -- 172.21.15.29:0/429869208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9e0103640 msgr2=0x7fa9e019a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9e0103640 0x7fa9e019a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 -- 172.21.15.29:0/429869208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9e0103fe0 msgr2=0x7fa9e019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9e0103fe0 0x7fa9e019b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 -- 172.21.15.29:0/429869208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e019b860 con 0x7fa9e01025c0 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9ddd9b640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9e0103640 0x7fa9e019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9de59c640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e01025c0 0x7fa9e0199e20 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000ea60 tx=0x7fa9d000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.179+0000 7fa9cf7fe640 1 -- 172.21.15.29:0/429869208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d000cde0 con 0x7fa9e01025c0 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.180+0000 7fa9e4e69640 1 -- 172.21.15.29:0/429869208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e019bb50 con 0x7fa9e01025c0 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.180+0000 7fa9cf7fe640 1 -- 172.21.15.29:0/429869208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9d00035f0 con 0x7fa9e01025c0 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.180+0000 7fa9cf7fe640 1 -- 172.21.15.29:0/429869208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0010690 con 0x7fa9e01025c0 2024-09-21T02:54:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.180+0000 7fa9e4e69640 1 -- 172.21.15.29:0/429869208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e01a2240 con 0x7fa9e01025c0 2024-09-21T02:54:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.183+0000 7fa9ebdd8740 1 Processor -- start 2024-09-21T02:54:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.183+0000 7fa9ebdd8740 1 -- start start 2024-09-21T02:54:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56dc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 0x5635d56dc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56dd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5635d564a6f0 con 0x5635d530f4d0 2024-09-21T02:54:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5635d5639b20 con 0x5635d53072f0 2024-09-21T02:54:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5635d563d930 con 0x5635d53301b0 2024-09-21T02:54:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ded9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56dc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56dd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ded9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56dc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58758/0 (socket says 172.21.15.29:58758) 2024-09-21T02:54:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ded9d640 1 -- 172.21.15.29:0/1568578457 learned_addr learned my addr 172.21.15.29:0/1568578457 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ded9d640 1 -- 172.21.15.29:0/1568578457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 msgr2=0x5635d56dc8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:54:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/1568578457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 0x5635d56dc8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ded9d640 1 -- 172.21.15.29:0/1568578457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 msgr2=0x5635d56dd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/1568578457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56dd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9ded9d640 1 -- 172.21.15.29:0/1568578457 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d55ff530 con 0x5635d53301b0 2024-09-21T02:54:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.184+0000 7fa9cf7fe640 1 -- 172.21.15.29:0/429869208 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fa9d0010830 con 0x7fa9e01025c0 2024-09-21T02:54:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.185+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/1568578457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56dc260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8004ac0 tx=0x7fa9d8009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3461713d014feed server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.185+0000 7fa9ccff9640 1 -- 172.21.15.29:0/1568578457 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8031a80 con 0x5635d53301b0 2024-09-21T02:54:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.185+0000 7fa9ccff9640 1 -- 172.21.15.29:0/1568578457 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9d8036070 con 0x5635d53301b0 2024-09-21T02:54:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.185+0000 7fa9ccff9640 1 -- 172.21.15.29:0/1568578457 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8004340 con 0x5635d53301b0 2024-09-21T02:54:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9cf7fe640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9c80b6510 0x7fa9c80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9de59c640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9c80b6510 0x7fa9c80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9cf7fe640 1 -- 172.21.15.29:0/429869208 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa9d00e7920 con 0x7fa9e01025c0 2024-09-21T02:54:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1568578457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 msgr2=0x5635d56dc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/1568578457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56dc260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8004ac0 tx=0x7fa9d8009800 comp rx=0 tx=0).stop 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1568578457 shutdown_connections 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/1568578457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56dd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/1568578457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 0x5635d56dc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/1568578457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56dc260 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1568578457 >> 172.21.15.29:0/1568578457 conn(0x5635d5324300 msgr2=0x5635d56d38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1568578457 shutdown_connections 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.186+0000 7fa9de59c640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9c80b6510 0x7fa9c80b8a20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00aa9a0 tx=0x7fa9d0014000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.187+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1568578457 wait complete. 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.187+0000 7fa9ebdd8740 1 -- start start 2024-09-21T02:54:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.187+0000 7fa9ebdd8740 1 Processor -- start 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56e5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d530f4d0 0x5635d56e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5635d5571ab0 con 0x5635d53072f0 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5635d55df730 con 0x5635d530f4d0 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5635d56ddd60 con 0x5635d53301b0 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56e5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ddd9b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d530f4d0 0x5635d56e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56e5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55236/0 (socket says 172.21.15.29:55236) 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ded9d640 1 -- 172.21.15.29:0/897189885 learned_addr learned my addr 172.21.15.29:0/897189885 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ebdd8740 10 client.0 _ll_register_callbacks cb 0x5635d56336b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ddd9b640 1 -- 172.21.15.29:0/897189885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 msgr2=0x5635d56e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ddd9b640 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ddd9b640 1 -- 172.21.15.29:0/897189885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 msgr2=0x5635d56e5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ddd9b640 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56e5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ddd9b640 1 -- 172.21.15.29:0/897189885 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d539b790 con 0x5635d530f4d0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56e5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9ddd9b640 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d530f4d0 0x5635d56e5f10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4012d00 tx=0x7fa9d400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbc5f4ac93700ecf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.188+0000 7fa9c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.189+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/897189885 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4009d90 con 0x5635d530f4d0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.189+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/897189885 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa9d4012110 con 0x5635d530f4d0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.189+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/897189885 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d8036070 con 0x5635d530f4d0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.189+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/897189885 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa9d4025050 con 0x5635d530f4d0 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.189+0000 7fa9ebdd8740 10 client.26948 fetch_fsmap learned FSMap version 35 2024-09-21T02:54:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.189+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/897189885 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5635d5414620 con 0x5635d530f4d0 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.189+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/897189885 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9d8036070 con 0x5635d530f4d0 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.189+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/897189885 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 35) ==== 8378+0+0 (secure 0 0 0) 0x7fa9d4003b90 con 0x5635d530f4d0 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9ebdd8740 10 client.26948 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9ebdd8740 10 client.26948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9ebdd8740 20 client.26948 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5635d52eed80 0x5635d56f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9ebdd8740 4 client.26948 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/897189885 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635d564c640 con 0x5635d52eed80 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5635d52eed80 0x5635d56f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9c5ffb640 10 client.26948 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.190+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5635d52eed80 0x5635d56f30e0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3fe50f90af58862 server_cookie=d17423c979e5dc15 in_seq=0 out_seq=0 2024-09-21T02:54:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.191+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/897189885 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5635d564c640 con 0x5635d52eed80 2024-09-21T02:54:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.191+0000 7fa9c5ffb640 10 client.26948 handle_command_reply: tid=0 2024-09-21T02:54:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.191+0000 7fa9ebdd8740 1 client.26948 shutdown 2024-09-21T02:54:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/897189885 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d530f4d0 msgr2=0x5635d56e5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d530f4d0 0x5635d56e5f10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4012d00 tx=0x7fa9d400f790 comp rx=0 tx=0).stop 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/897189885 shutdown_connections 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d530f4d0 0x5635d56e5f10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5635d52eed80 0x5635d56f30e0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/897189885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56e5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9c5ffb640 0 client.26948 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/897189885 >> 172.21.15.29:0/897189885 conn(0x5635d5324300 msgr2=0x5635d56d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/897189885 shutdown_connections 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/897189885 wait complete. 2024-09-21T02:54:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.192+0000 7fa9ebdd8740 20 client.26948 trim_cache size 0 max 16384 2024-09-21T02:54:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ebdd8740 1 Processor -- start 2024-09-21T02:54:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ebdd8740 1 -- start start 2024-09-21T02:54:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d530f4d0 0x5635d56e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53301b0 0x5635d56ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e01aa300 con 0x5635d53072f0 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e01abea0 con 0x5635d53301b0 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e01aa9d0 con 0x5635d530f4d0 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.210+0000 7fa9ded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55246/0 (socket says 172.21.15.29:55246) 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ddd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d530f4d0 0x5635d56e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 -- 172.21.15.29:0/1319283319 learned_addr learned my addr 172.21.15.29:0/1319283319 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/1319283319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53301b0 0x5635d56ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 -- 172.21.15.29:0/1319283319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d530f4d0 msgr2=0x5635d56e4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/1319283319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d530f4d0 0x5635d56e4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 -- 172.21.15.29:0/1319283319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53301b0 msgr2=0x5635d56ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/1319283319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53301b0 0x5635d56ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 -- 172.21.15.29:0/1319283319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d56de160 con 0x5635d53072f0 2024-09-21T02:54:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/1319283319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56d3220 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa9d802ade0 tx=0x7fa9d8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57d2193de5ce613 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9c7fff640 1 -- 172.21.15.29:0/1319283319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8051040 con 0x5635d53072f0 2024-09-21T02:54:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9c7fff640 1 -- 172.21.15.29:0/1319283319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9d8036040 con 0x5635d53072f0 2024-09-21T02:54:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.211+0000 7fa9c7fff640 1 -- 172.21.15.29:0/1319283319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d80046d0 con 0x5635d53072f0 2024-09-21T02:54:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1319283319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 msgr2=0x5635d56d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/1319283319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56d3220 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa9d802ade0 tx=0x7fa9d8031000 comp rx=0 tx=0).stop 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1319283319 shutdown_connections 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/1319283319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53301b0 0x5635d56ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/1319283319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d530f4d0 0x5635d56e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/1319283319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d53072f0 0x5635d56d3220 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1319283319 >> 172.21.15.29:0/1319283319 conn(0x5635d5324300 msgr2=0x5635d56dd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1319283319 shutdown_connections 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/1319283319 wait complete. 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 -- start start 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.213+0000 7fa9ebdd8740 1 Processor -- start 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 0x5635d56ea400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ebdd8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56eb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e01ab770 con 0x5635d530f4d0 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e01ab0a0 con 0x5635d53072f0 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ebdd8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e01a82e0 con 0x5635d53301b0 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ebdd8740 10 client.0 _ll_register_callbacks cb 0x7fa9d4009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ded9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 0x5635d56ea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56eaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55248/0 (socket says 172.21.15.29:55248) 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ddd9b640 1 -- 172.21.15.29:0/3787032837 learned_addr learned my addr 172.21.15.29:0/3787032837 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ddd9b640 1 -- 172.21.15.29:0/3787032837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 msgr2=0x5635d56eb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ddd9b640 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56eb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ddd9b640 1 -- 172.21.15.29:0/3787032837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 msgr2=0x5635d56ea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ddd9b640 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 0x5635d56ea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ddd9b640 1 -- 172.21.15.29:0/3787032837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d56ebd90 con 0x5635d530f4d0 2024-09-21T02:54:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.214+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 0x5635d56ea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:54:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9ddd9b640 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56eaac0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4012d00 tx=0x7fa9d40123c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64c48d8bb56e1e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:54:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/3787032837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4010be0 con 0x5635d530f4d0 2024-09-21T02:54:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/3787032837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa9d4022040 con 0x5635d530f4d0 2024-09-21T02:54:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/3787032837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d8036040 con 0x5635d530f4d0 2024-09-21T02:54:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/3787032837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa9d4010d80 con 0x5635d530f4d0 2024-09-21T02:54:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9ebdd8740 10 client.17595 fetch_fsmap learned FSMap version 35 2024-09-21T02:54:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/3787032837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5635d56ed120 con 0x5635d530f4d0 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.215+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/3787032837 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9d8036040 con 0x5635d530f4d0 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/3787032837 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 35) ==== 8378+0+0 (secure 0 0 0) 0x7fa9d40038f0 con 0x5635d530f4d0 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9ebdd8740 10 client.17595 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9ebdd8740 10 client.17595 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9ebdd8740 20 client.17595 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5635d52c41c0 0x5635d575fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9ebdd8740 4 client.17595 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/3787032837 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635d56480f0 con 0x5635d52c41c0 2024-09-21T02:54:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5635d52c41c0 0x5635d575fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9c5ffb640 10 client.17595 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.216+0000 7fa9ded9d640 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5635d52c41c0 0x5635d575fab0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=303a039a209d8d48 server_cookie=fa6241c12810d967 in_seq=0 out_seq=0 2024-09-21T02:54:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.514+0000 7fa9c5ffb640 1 -- 172.21.15.29:0/3787032837 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635d56480f0 con 0x5635d52c41c0 2024-09-21T02:54:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.514+0000 7fa9c5ffb640 10 client.17595 handle_command_reply: tid=0 2024-09-21T02:54:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.514+0000 7fa9ebdd8740 1 client.17595 shutdown 2024-09-21T02:54:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/3787032837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 msgr2=0x5635d56eaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56eaac0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4012d00 tx=0x7fa9d40123c0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/3787032837 shutdown_connections 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5635d53301b0 0x5635d56eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5635d52c41c0 0x5635d575fab0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5635d530f4d0 0x5635d56eaac0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 --2- 172.21.15.29:0/3787032837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5635d53072f0 0x5635d56ea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/3787032837 >> 172.21.15.29:0/3787032837 conn(0x5635d5324300 msgr2=0x5635d56dd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9c5ffb640 0 client.17595 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/3787032837 shutdown_connections 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 1 -- 172.21.15.29:0/3787032837 wait complete. 2024-09-21T02:54:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.515+0000 7fa9ebdd8740 20 client.17595 trim_cache size 0 max 16384 2024-09-21T02:54:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:54:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:54:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:54:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:54:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 -- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9c80b6510 msgr2=0x7fa9c80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9c80b6510 0x7fa9c80b8a20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00aa9a0 tx=0x7fa9d0014000 comp rx=0 tx=0).stop 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 -- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e01025c0 msgr2=0x7fa9e0199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e01025c0 0x7fa9e0199e20 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000ea60 tx=0x7fa9d000ef20 comp rx=0 tx=0).stop 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 -- 172.21.15.29:0/429869208 shutdown_connections 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9c80b6510 0x7fa9c80b8a20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9e0103fe0 0x7fa9e019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9e0103640 0x7fa9e019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 --2- 172.21.15.29:0/429869208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9e01025c0 0x7fa9e0199e20 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 -- 172.21.15.29:0/429869208 >> 172.21.15.29:0/429869208 conn(0x7fa9e00fdf00 msgr2=0x7fa9e00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 -- 172.21.15.29:0/429869208 shutdown_connections 2024-09-21T02:54:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:42.517+0000 7fa9e4e69640 1 -- 172.21.15.29:0/429869208 wait complete. 2024-09-21T02:54:42.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 15348 6.60 MB/sec execute 19 sec latency 215.182 ms 2024-09-21T02:54:43.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 15571 7.03 MB/sec execute 20 sec latency 103.309 ms 2024-09-21T02:54:43.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:43 smithi066 ceph-mon[33436]: pgmap v857: 129 pgs: 129 active+clean; 256 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 2.9 MiB/s wr, 278 op/s 2024-09-21T02:54:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:43 smithi190 ceph-mon[33871]: pgmap v857: 129 pgs: 129 active+clean; 256 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 2.9 MiB/s wr, 278 op/s 2024-09-21T02:54:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:43 smithi029 ceph-mon[31006]: pgmap v857: 129 pgs: 129 active+clean; 256 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 2.9 MiB/s wr, 278 op/s 2024-09-21T02:54:44.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 15740 7.35 MB/sec execute 21 sec latency 210.098 ms 2024-09-21T02:54:45.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 15879 7.41 MB/sec execute 22 sec latency 92.777 ms 2024-09-21T02:54:45.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:45 smithi066 ceph-mon[33436]: pgmap v858: 129 pgs: 129 active+clean; 257 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.0 MiB/s wr, 263 op/s 2024-09-21T02:54:45.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:45 smithi029 ceph-mon[31006]: pgmap v858: 129 pgs: 129 active+clean; 257 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.0 MiB/s wr, 263 op/s 2024-09-21T02:54:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:45 smithi190 ceph-mon[33871]: pgmap v858: 129 pgs: 129 active+clean; 257 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.0 MiB/s wr, 263 op/s 2024-09-21T02:54:46.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 15953 7.13 MB/sec execute 23 sec latency 151.257 ms 2024-09-21T02:54:47.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16063 7.01 MB/sec execute 24 sec latency 98.781 ms 2024-09-21T02:54:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:47 smithi066 ceph-mon[33436]: pgmap v859: 129 pgs: 129 active+clean; 261 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 3.6 MiB/s wr, 269 op/s 2024-09-21T02:54:47.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:47 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:47 smithi190 ceph-mon[33871]: pgmap v859: 129 pgs: 129 active+clean; 261 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 3.6 MiB/s wr, 269 op/s 2024-09-21T02:54:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:47 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:47 smithi029 ceph-mon[31006]: pgmap v859: 129 pgs: 129 active+clean; 261 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 3.6 MiB/s wr, 269 op/s 2024-09-21T02:54:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:47 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:48.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16227 7.47 MB/sec execute 25 sec latency 118.022 ms 2024-09-21T02:54:49.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16419 7.87 MB/sec execute 26 sec latency 120.709 ms 2024-09-21T02:54:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:49 smithi066 ceph-mon[33436]: pgmap v860: 129 pgs: 129 active+clean; 265 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:54:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:49 smithi190 ceph-mon[33871]: pgmap v860: 129 pgs: 129 active+clean; 265 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:54:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:49 smithi029 ceph-mon[31006]: pgmap v860: 129 pgs: 129 active+clean; 265 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:54:50.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16561 7.78 MB/sec execute 27 sec latency 186.545 ms 2024-09-21T02:54:51.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16663 7.73 MB/sec execute 28 sec latency 109.437 ms 2024-09-21T02:54:51.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:51 smithi066 ceph-mon[33436]: pgmap v861: 129 pgs: 129 active+clean; 267 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 4.2 MiB/s wr, 268 op/s 2024-09-21T02:54:51.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:51 smithi190 ceph-mon[33871]: pgmap v861: 129 pgs: 129 active+clean; 267 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 4.2 MiB/s wr, 268 op/s 2024-09-21T02:54:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:51 smithi029 ceph-mon[31006]: pgmap v861: 129 pgs: 129 active+clean; 267 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 4.2 MiB/s wr, 268 op/s 2024-09-21T02:54:52.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16779 7.60 MB/sec execute 29 sec latency 107.733 ms 2024-09-21T02:54:53.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16907 7.51 MB/sec execute 30 sec latency 132.934 ms 2024-09-21T02:54:53.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:53 smithi066 ceph-mon[33436]: pgmap v862: 129 pgs: 129 active+clean; 270 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-09-21T02:54:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:53 smithi190 ceph-mon[33871]: pgmap v862: 129 pgs: 129 active+clean; 270 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-09-21T02:54:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:53 smithi029 ceph-mon[31006]: pgmap v862: 129 pgs: 129 active+clean; 270 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-09-21T02:54:54.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17026 7.32 MB/sec execute 31 sec latency 161.775 ms 2024-09-21T02:54:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.671+0000 7f73b394a640 1 -- 172.21.15.29:0/1592312608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac102ee0 msgr2=0x7f73ac107850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.671+0000 7f73b394a640 1 --2- 172.21.15.29:0/1592312608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac102ee0 0x7f73ac107850 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009be0 tx=0x7f73a402a6c0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.672+0000 7f73b394a640 1 -- 172.21.15.29:0/1592312608 shutdown_connections 2024-09-21T02:54:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.672+0000 7f73b394a640 1 --2- 172.21.15.29:0/1592312608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac102ee0 0x7f73ac107850 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.672+0000 7f73b394a640 1 --2- 172.21.15.29:0/1592312608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73ac102540 0x7f73ac102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.672+0000 7f73b394a640 1 --2- 172.21.15.29:0/1592312608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac101650 0x7f73ac101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.672+0000 7f73b394a640 1 -- 172.21.15.29:0/1592312608 >> 172.21.15.29:0/1592312608 conn(0x7f73ac0fd270 msgr2=0x7f73ac0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.672+0000 7f73b394a640 1 -- 172.21.15.29:0/1592312608 shutdown_connections 2024-09-21T02:54:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.673+0000 7f73b394a640 1 -- 172.21.15.29:0/1592312608 wait complete. 2024-09-21T02:54:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.673+0000 7f73b394a640 1 Processor -- start 2024-09-21T02:54:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b394a640 1 -- start start 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b394a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac101650 0x7f73ac1a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b394a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73ac102540 0x7f73ac1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b394a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac102ee0 0x7f73ac1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b394a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1157f0 con 0x7f73ac102ee0 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b394a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac115670 con 0x7f73ac102540 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b394a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac115970 con 0x7f73ac101650 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b16e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac101650 0x7f73ac1a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b16e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac101650 0x7f73ac1a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40152/0 (socket says 172.21.15.29:40152) 2024-09-21T02:54:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b16e6640 1 -- 172.21.15.29:0/4061740055 learned_addr learned my addr 172.21.15.29:0/4061740055 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.674+0000 7f73b0ee5640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73ac102540 0x7f73ac1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.675+0000 7f73b1ee7640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac102ee0 0x7f73ac1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.675+0000 7f73b16e6640 1 -- 172.21.15.29:0/4061740055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73ac102540 msgr2=0x7f73ac1a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.675+0000 7f73b16e6640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73ac102540 0x7f73ac1a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.675+0000 7f73b16e6640 1 -- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac102ee0 msgr2=0x7f73ac1a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.675+0000 7f73b16e6640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac102ee0 0x7f73ac1a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.675+0000 7f73b16e6640 1 -- 172.21.15.29:0/4061740055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac1a1ed0 con 0x7f73ac101650 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.676+0000 7f73b16e6640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac101650 0x7f73ac1a0490 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f73a800e9f0 tx=0x7f73a800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.676+0000 7f73a27fc640 1 -- 172.21.15.29:0/4061740055 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a800ccc0 con 0x7f73ac101650 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.676+0000 7f73a27fc640 1 -- 172.21.15.29:0/4061740055 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73a80054b0 con 0x7f73ac101650 2024-09-21T02:54:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.677+0000 7f73a27fc640 1 -- 172.21.15.29:0/4061740055 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8010430 con 0x7f73ac101650 2024-09-21T02:54:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.677+0000 7f73b394a640 1 -- 172.21.15.29:0/4061740055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ac1a21c0 con 0x7f73ac101650 2024-09-21T02:54:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.677+0000 7f73b394a640 1 -- 172.21.15.29:0/4061740055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ac1a88b0 con 0x7f73ac101650 2024-09-21T02:54:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.681+0000 7f73ba8b4740 1 Processor -- start 2024-09-21T02:54:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.681+0000 7f73ba8b4740 1 -- start start 2024-09-21T02:54:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.683+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c5521e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.683+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c55224c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.683+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c5523010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.683+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b5c54946f0 con 0x55b5c51512f0 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.683+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b5c5483b20 con 0x55b5c51594d0 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.683+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b5c5487930 con 0x55b5c517a1b0 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b0ee5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c5523010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b0ee5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c5523010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:59346/0 (socket says 172.21.15.29:59346) 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b0ee5640 1 -- 172.21.15.29:0/2278574262 learned_addr learned my addr 172.21.15.29:0/2278574262 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b1ee7640 1 --2- 172.21.15.29:0/2278574262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c5521e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73a27fc640 1 -- 172.21.15.29:0/4061740055 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f73a80046c0 con 0x7f73ac101650 2024-09-21T02:54:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b0ee5640 1 -- 172.21.15.29:0/2278574262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 msgr2=0x55b5c5521e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b0ee5640 1 --2- 172.21.15.29:0/2278574262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c5521e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b0ee5640 1 -- 172.21.15.29:0/2278574262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 msgr2=0x55b5c55224c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b0ee5640 1 --2- 172.21.15.29:0/2278574262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c55224c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.684+0000 7f73b0ee5640 1 -- 172.21.15.29:0/2278574262 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5c5449530 con 0x55b5c51594d0 2024-09-21T02:54:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.685+0000 7f73b0ee5640 1 --2- 172.21.15.29:0/2278574262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c5523010 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f739c009cf0 tx=0x7f739c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14750b3b4e950856 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.685+0000 7f73a27fc640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73980b6720 0x7f73980b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.686+0000 7f73a27fc640 1 -- 172.21.15.29:0/4061740055 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f73a80e7980 con 0x7f73ac101650 2024-09-21T02:54:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.686+0000 7f73b1ee7640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73980b6720 0x7f73980b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.686+0000 7f73977fe640 1 -- 172.21.15.29:0/2278574262 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c004230 con 0x55b5c51594d0 2024-09-21T02:54:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.686+0000 7f73977fe640 1 -- 172.21.15.29:0/2278574262 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f739c0043d0 con 0x55b5c51594d0 2024-09-21T02:54:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.687+0000 7f73977fe640 1 -- 172.21.15.29:0/2278574262 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c019750 con 0x55b5c51594d0 2024-09-21T02:54:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2278574262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 msgr2=0x55b5c5523010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2278574262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c5523010 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f739c009cf0 tx=0x7f739c011810 comp rx=0 tx=0).stop 2024-09-21T02:54:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73b1ee7640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73980b6720 0x7f73980b8c30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f73a4030800 tx=0x7f73a4009320 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2278574262 shutdown_connections 2024-09-21T02:54:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2278574262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c5523010 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2278574262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c55224c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2278574262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c5521e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2278574262 >> 172.21.15.29:0/2278574262 conn(0x55b5c516e300 msgr2=0x55b5c551d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2278574262 shutdown_connections 2024-09-21T02:54:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.688+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2278574262 wait complete. 2024-09-21T02:54:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.689+0000 7f73ba8b4740 1 -- start start 2024-09-21T02:54:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.689+0000 7f73ba8b4740 1 Processor -- start 2024-09-21T02:54:55.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.690+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51512f0 0x55b5c552b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.690+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51594d0 0x55b5c552bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.690+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c552c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.690+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b5c53bbab0 con 0x55b5c51594d0 2024-09-21T02:54:55.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.690+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b5c5429730 con 0x55b5c51512f0 2024-09-21T02:54:55.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.690+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b5c5523b80 con 0x55b5c517a1b0 2024-09-21T02:54:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.690+0000 7f73ba8b4740 10 client.0 _ll_register_callbacks cb 0x55b5c547d6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b1ee7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c552c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b1ee7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c552c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40164/0 (socket says 172.21.15.29:40164) 2024-09-21T02:54:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b1ee7640 1 -- 172.21.15.29:0/251264382 learned_addr learned my addr 172.21.15.29:0/251264382 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b0ee5640 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51594d0 0x55b5c552bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b16e6640 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51512f0 0x55b5c552b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b1ee7640 1 -- 172.21.15.29:0/251264382 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51512f0 msgr2=0x55b5c552b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b1ee7640 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51512f0 0x55b5c552b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b1ee7640 1 -- 172.21.15.29:0/251264382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51594d0 msgr2=0x55b5c552bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b1ee7640 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51594d0 0x55b5c552bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.691+0000 7f73b1ee7640 1 -- 172.21.15.29:0/251264382 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5c51e5790 con 0x55b5c517a1b0 2024-09-21T02:54:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.692+0000 7f73b1ee7640 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c552c8c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f73a4033000 tx=0x7f73a402a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cde655f062924cb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.692+0000 7f7394ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:54:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.693+0000 7f7394ff9640 1 -- 172.21.15.29:0/251264382 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4044070 con 0x55b5c517a1b0 2024-09-21T02:54:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.693+0000 7f7394ff9640 1 -- 172.21.15.29:0/251264382 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f73a4004240 con 0x55b5c517a1b0 2024-09-21T02:54:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.693+0000 7f73ba8b4740 1 -- 172.21.15.29:0/251264382 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f739c0043d0 con 0x55b5c517a1b0 2024-09-21T02:54:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.693+0000 7f7394ff9640 1 -- 172.21.15.29:0/251264382 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f73a4059050 con 0x55b5c517a1b0 2024-09-21T02:54:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.693+0000 7f73ba8b4740 10 client.27106 fetch_fsmap learned FSMap version 36 2024-09-21T02:54:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.693+0000 7f73ba8b4740 1 -- 172.21.15.29:0/251264382 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b5c525e620 con 0x55b5c517a1b0 2024-09-21T02:54:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.694+0000 7f7394ff9640 1 -- 172.21.15.29:0/251264382 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f739c0043d0 con 0x55b5c517a1b0 2024-09-21T02:54:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.695+0000 7f7394ff9640 1 -- 172.21.15.29:0/251264382 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 36) ==== 8374+0+0 (secure 0 0 0) 0x7f73a403c040 con 0x55b5c517a1b0 2024-09-21T02:54:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.695+0000 7f73ba8b4740 10 client.27106 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T02:54:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.695+0000 7f73ba8b4740 10 client.27106 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.695+0000 7f73ba8b4740 20 client.27106 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.695+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5c5138d80 0x55b5c5539290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.695+0000 7f73ba8b4740 4 client.27106 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:54:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.695+0000 7f73ba8b4740 1 -- 172.21.15.29:0/251264382 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5c5496640 con 0x55b5c5138d80 2024-09-21T02:54:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.696+0000 7f73b16e6640 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5c5138d80 0x55b5c5539290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.697+0000 7f73b16e6640 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5c5138d80 0x55b5c5539290 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3a41b5c87739768 server_cookie=23ed719b63c5cff4 in_seq=0 out_seq=0 2024-09-21T02:54:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.697+0000 7f7394ff9640 10 client.27106 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f7394ff9640 1 -- 172.21.15.29:0/251264382 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b5c5496640 con 0x55b5c5138d80 2024-09-21T02:54:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f7394ff9640 10 client.27106 handle_command_reply: tid=0 2024-09-21T02:54:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 client.27106 shutdown 2024-09-21T02:54:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 -- 172.21.15.29:0/251264382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 msgr2=0x55b5c552c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c552c8c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f73a4033000 tx=0x7f73a402a6c0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 -- 172.21.15.29:0/251264382 shutdown_connections 2024-09-21T02:54:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c552c8c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5c5138d80 0x55b5c5539290 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51594d0 0x55b5c552bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/251264382 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51512f0 0x55b5c552b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f73ba8b4740 1 -- 172.21.15.29:0/251264382 >> 172.21.15.29:0/251264382 conn(0x55b5c516e300 msgr2=0x55b5c551bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.701+0000 7f7394ff9640 0 client.27106 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.702+0000 7f73ba8b4740 1 -- 172.21.15.29:0/251264382 shutdown_connections 2024-09-21T02:54:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.702+0000 7f73ba8b4740 1 -- 172.21.15.29:0/251264382 wait complete. 2024-09-21T02:54:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.703+0000 7f73ba8b4740 20 client.27106 trim_cache size 0 max 16384 2024-09-21T02:54:55.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17166 7.21 MB/sec execute 32 sec latency 114.925 ms 2024-09-21T02:54:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.737+0000 7f73ba8b4740 1 Processor -- start 2024-09-21T02:54:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.737+0000 7f73ba8b4740 1 -- start start 2024-09-21T02:54:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c552b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c552a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c55185c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1b2c20 con 0x55b5c51512f0 2024-09-21T02:54:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1b4790 con 0x55b5c51594d0 2024-09-21T02:54:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73ba8b4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1b32f0 con 0x55b5c517a1b0 2024-09-21T02:54:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b0ee5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c552a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b16e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c552b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b0ee5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c552a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:59366/0 (socket says 172.21.15.29:59366) 2024-09-21T02:54:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b0ee5640 1 -- 172.21.15.29:0/1586499500 learned_addr learned my addr 172.21.15.29:0/1586499500 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b1ee7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c55185c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b16e6640 1 -- 172.21.15.29:0/1586499500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 msgr2=0x55b5c55185c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b16e6640 1 --2- 172.21.15.29:0/1586499500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c55185c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b16e6640 1 -- 172.21.15.29:0/1586499500 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 msgr2=0x55b5c552a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b16e6640 1 --2- 172.21.15.29:0/1586499500 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c552a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.738+0000 7f73b16e6640 1 -- 172.21.15.29:0/1586499500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5c551e010 con 0x55b5c51512f0 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.739+0000 7f73b16e6640 1 --2- 172.21.15.29:0/1586499500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c552b670 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f73a800e9c0 tx=0x7f73a80049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9d049bdec0f129 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.739+0000 7f7396ffd640 1 -- 172.21.15.29:0/1586499500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a80a9d80 con 0x55b5c51512f0 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.739+0000 7f7396ffd640 1 -- 172.21.15.29:0/1586499500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73a80ad240 con 0x55b5c51512f0 2024-09-21T02:54:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.739+0000 7f7396ffd640 1 -- 172.21.15.29:0/1586499500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a80b6cc0 con 0x55b5c51512f0 2024-09-21T02:54:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 -- 172.21.15.29:0/1586499500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 msgr2=0x55b5c552b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/1586499500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c552b670 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f73a800e9c0 tx=0x7f73a80049b0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 -- 172.21.15.29:0/1586499500 shutdown_connections 2024-09-21T02:54:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/1586499500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c517a1b0 0x55b5c55185c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/1586499500 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c51594d0 0x55b5c552a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/1586499500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c552b670 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 -- 172.21.15.29:0/1586499500 >> 172.21.15.29:0/1586499500 conn(0x55b5c516e300 msgr2=0x55b5c5522db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 -- 172.21.15.29:0/1586499500 shutdown_connections 2024-09-21T02:54:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.740+0000 7f73ba8b4740 1 -- 172.21.15.29:0/1586499500 wait complete. 2024-09-21T02:54:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.741+0000 7f73ba8b4740 1 -- start start 2024-09-21T02:54:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.741+0000 7f73ba8b4740 1 Processor -- start 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c5530830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c51594d0 0x55b5c5530ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73ba8b4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c517a1b0 0x55b5c55319a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b16e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c5530830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b16e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c5530830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42286/0 (socket says 172.21.15.29:42286) 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b16e6640 1 -- 172.21.15.29:0/2236426199 learned_addr learned my addr 172.21.15.29:0/2236426199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b0ee5640 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c51594d0 0x55b5c5530ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1b4090 con 0x55b5c51512f0 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b1ee7640 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c517a1b0 0x55b5c55319a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1b39c0 con 0x55b5c517a1b0 2024-09-21T02:54:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1b0c00 con 0x55b5c51594d0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73ba8b4740 10 client.0 _ll_register_callbacks cb 0x7f73a4034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b16e6640 1 -- 172.21.15.29:0/2236426199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c51594d0 msgr2=0x55b5c5530ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b16e6640 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c51594d0 0x55b5c5530ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b16e6640 1 -- 172.21.15.29:0/2236426199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c517a1b0 msgr2=0x55b5c55319a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b16e6640 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c517a1b0 0x55b5c55319a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.742+0000 7f73b16e6640 1 -- 172.21.15.29:0/2236426199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5c5532100 con 0x55b5c51512f0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.743+0000 7f73b16e6640 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c5530830 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f73a80b7310 tx=0x7f73a80b71c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d085502c81d58c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.743+0000 7f7394ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.743+0000 7f7394ff9640 1 -- 172.21.15.29:0/2236426199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a80b2210 con 0x55b5c51512f0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.743+0000 7f7394ff9640 1 -- 172.21.15.29:0/2236426199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f73a80ac990 con 0x55b5c51512f0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.743+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73a80ad240 con 0x55b5c51512f0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.743+0000 7f7394ff9640 1 -- 172.21.15.29:0/2236426199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f73a80cd050 con 0x55b5c51512f0 2024-09-21T02:54:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.743+0000 7f73ba8b4740 10 client.17610 fetch_fsmap learned FSMap version 36 2024-09-21T02:54:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.743+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b5c5533460 con 0x55b5c51512f0 2024-09-21T02:54:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f7394ff9640 1 -- 172.21.15.29:0/2236426199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73a80c8070 con 0x55b5c51512f0 2024-09-21T02:54:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f7394ff9640 1 -- 172.21.15.29:0/2236426199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 36) ==== 8374+0+0 (secure 0 0 0) 0x7f73a80ac400 con 0x55b5c51512f0 2024-09-21T02:54:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f73ba8b4740 10 client.17610 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T02:54:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f73ba8b4740 10 client.17610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:54:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f73ba8b4740 20 client.17610 populate_metadata read hostname 'smithi029' 2024-09-21T02:54:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5c510e1c0 0x55b5c55a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:54:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f73ba8b4740 4 client.17610 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:54:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f73a80ad240 con 0x55b5c510e1c0 2024-09-21T02:54:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.744+0000 7f73b0ee5640 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5c510e1c0 0x55b5c55a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:54:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.745+0000 7f7394ff9640 10 client.17610 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.745+0000 7f73b0ee5640 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5c510e1c0 0x55b5c55a5e00 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad2effb91c74554f server_cookie=8e43c068cb0db6d3 in_seq=0 out_seq=0 2024-09-21T02:54:55.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:55 smithi066 ceph-mon[33436]: pgmap v863: 129 pgs: 129 active+clean; 273 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 4.7 MiB/s wr, 266 op/s 2024-09-21T02:54:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:55 smithi190 ceph-mon[33871]: pgmap v863: 129 pgs: 129 active+clean; 273 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 4.7 MiB/s wr, 266 op/s 2024-09-21T02:54:55.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:55 smithi029 ceph-mon[31006]: pgmap v863: 129 pgs: 129 active+clean; 273 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 4.7 MiB/s wr, 266 op/s 2024-09-21T02:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f7394ff9640 1 -- 172.21.15.29:0/2236426199 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f73a80ad240 con 0x55b5c510e1c0 2024-09-21T02:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f7394ff9640 10 client.17610 handle_command_reply: tid=0 2024-09-21T02:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 client.17610 shutdown 2024-09-21T02:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 msgr2=0x55b5c5530830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c5530830 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f73a80b7310 tx=0x7f73a80b71c0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 shutdown_connections 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5c517a1b0 0x55b5c55319a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c51594d0 0x55b5c5530ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5c510e1c0 0x55b5c55a5e00 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 --2- 172.21.15.29:0/2236426199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5c51512f0 0x55b5c5530830 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f7394ff9640 0 client.17610 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 >> 172.21.15.29:0/2236426199 conn(0x55b5c516e300 msgr2=0x55b5c552a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 shutdown_connections 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.951+0000 7f73ba8b4740 1 -- 172.21.15.29:0/2236426199 wait complete. 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73ba8b4740 20 client.17610 trim_cache size 0 max 16384 2024-09-21T02:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:54:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:54:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:54:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:54:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 -- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73980b6720 msgr2=0x7f73980b8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73980b6720 0x7f73980b8c30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f73a4030800 tx=0x7f73a4009320 comp rx=0 tx=0).stop 2024-09-21T02:54:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 -- 172.21.15.29:0/4061740055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac101650 msgr2=0x7f73ac1a0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac101650 0x7f73ac1a0490 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f73a800e9f0 tx=0x7f73a800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 -- 172.21.15.29:0/4061740055 shutdown_connections 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73980b6720 0x7f73980b8c30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac102ee0 0x7f73ac1a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73ac102540 0x7f73ac1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 --2- 172.21.15.29:0/4061740055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73ac101650 0x7f73ac1a0490 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 -- 172.21.15.29:0/4061740055 >> 172.21.15.29:0/4061740055 conn(0x7f73ac0fd270 msgr2=0x7f73ac103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.952+0000 7f73b394a640 1 -- 172.21.15.29:0/4061740055 shutdown_connections 2024-09-21T02:54:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:54:55.953+0000 7f73b394a640 1 -- 172.21.15.29:0/4061740055 wait complete. 2024-09-21T02:54:56.714 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17286 7.07 MB/sec execute 33 sec latency 243.152 ms 2024-09-21T02:54:57.715 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17421 7.18 MB/sec execute 34 sec latency 227.460 ms 2024-09-21T02:54:57.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:57 smithi066 ceph-mon[33436]: pgmap v864: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 4.4 MiB/s wr, 236 op/s 2024-09-21T02:54:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:57 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:57 smithi190 ceph-mon[33871]: pgmap v864: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 4.4 MiB/s wr, 236 op/s 2024-09-21T02:54:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:57 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:57 smithi029 ceph-mon[31006]: pgmap v864: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 4.4 MiB/s wr, 236 op/s 2024-09-21T02:54:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:57 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:54:58.715 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17581 7.11 MB/sec execute 35 sec latency 129.116 ms 2024-09-21T02:54:59.715 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17703 7.11 MB/sec execute 36 sec latency 265.158 ms 2024-09-21T02:54:59.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:54:59 smithi066 ceph-mon[33436]: pgmap v865: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-09-21T02:54:59.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:54:59 smithi190 ceph-mon[33871]: pgmap v865: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-09-21T02:54:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:54:59 smithi029 ceph-mon[31006]: pgmap v865: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-09-21T02:55:00.715 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17861 7.32 MB/sec execute 37 sec latency 275.041 ms 2024-09-21T02:55:01.715 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18033 7.40 MB/sec execute 38 sec latency 230.080 ms 2024-09-21T02:55:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:01 smithi066 ceph-mon[33436]: pgmap v866: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 3.6 MiB/s wr, 289 op/s 2024-09-21T02:55:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:01 smithi190 ceph-mon[33871]: pgmap v866: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 3.6 MiB/s wr, 289 op/s 2024-09-21T02:55:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:01 smithi029 ceph-mon[31006]: pgmap v866: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 3.6 MiB/s wr, 289 op/s 2024-09-21T02:55:02.715 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18190 7.30 MB/sec execute 39 sec latency 285.965 ms 2024-09-21T02:55:03.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18326 7.17 MB/sec execute 40 sec latency 91.392 ms 2024-09-21T02:55:03.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:03 smithi066 ceph-mon[33436]: pgmap v867: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.8 MiB/s wr, 327 op/s 2024-09-21T02:55:03.874 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.2 2024-09-21T02:55:03.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:55:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:03 smithi190 ceph-mon[33871]: pgmap v867: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.8 MiB/s wr, 327 op/s 2024-09-21T02:55:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:03 smithi029 ceph-mon[31006]: pgmap v867: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.8 MiB/s wr, 327 op/s 2024-09-21T02:55:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:55:04.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18491 7.14 MB/sec execute 41 sec latency 90.217 ms 2024-09-21T02:55:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 -- 172.21.15.29:0/1540361876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95081083c0 msgr2=0x7f95081087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 --2- 172.21.15.29:0/1540361876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95081083c0 0x7f95081087d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009900 tx=0x7f94f402a410 comp rx=0 tx=0).stop 2024-09-21T02:55:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 -- 172.21.15.29:0/1540361876 shutdown_connections 2024-09-21T02:55:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 --2- 172.21.15.29:0/1540361876 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9508108d10 0x7f950810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 --2- 172.21.15.29:0/1540361876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95081083c0 0x7f95081087d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 --2- 172.21.15.29:0/1540361876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9508102ad0 0x7f9508102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 -- 172.21.15.29:0/1540361876 >> 172.21.15.29:0/1540361876 conn(0x7f95080796d0 msgr2=0x7f9508079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 -- 172.21.15.29:0/1540361876 shutdown_connections 2024-09-21T02:55:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.823+0000 7f950eb2e640 1 -- 172.21.15.29:0/1540361876 wait complete. 2024-09-21T02:55:04.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.824+0000 7f950eb2e640 1 Processor -- start 2024-09-21T02:55:04.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.824+0000 7f950eb2e640 1 -- start start 2024-09-21T02:55:04.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950eb2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9508102ad0 0x7f95081a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:04.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950eb2e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95081083c0 0x7f95081a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:04.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950eb2e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9508108d10 0x7f95081a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:04.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950eb2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112880 con 0x7f9508102ad0 2024-09-21T02:55:04.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950eb2e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112700 con 0x7f95081083c0 2024-09-21T02:55:04.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950eb2e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112a00 con 0x7f9508108d10 2024-09-21T02:55:04.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950c8ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9508102ad0 0x7f95081a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:04.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950d0cb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9508108d10 0x7f95081a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:04.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950c8ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9508102ad0 0x7f95081a09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45002/0 (socket says 172.21.15.29:45002) 2024-09-21T02:55:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950c8ca640 1 -- 172.21.15.29:0/168134855 learned_addr learned my addr 172.21.15.29:0/168134855 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950d0cb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9508108d10 0x7f95081a6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38118/0 (socket says 172.21.15.29:38118) 2024-09-21T02:55:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f9507fff640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95081083c0 0x7f95081a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950c8ca640 1 -- 172.21.15.29:0/168134855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9508108d10 msgr2=0x7f95081a6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950c8ca640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9508108d10 0x7f95081a6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950c8ca640 1 -- 172.21.15.29:0/168134855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95081083c0 msgr2=0x7f95081a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950c8ca640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95081083c0 0x7f95081a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.825+0000 7f950c8ca640 1 -- 172.21.15.29:0/168134855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95081a6940 con 0x7f9508102ad0 2024-09-21T02:55:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.826+0000 7f950d0cb640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9508108d10 0x7f95081a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:55:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.826+0000 7f950c8ca640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9508102ad0 0x7f95081a09d0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00e9f0 tx=0x7f94fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.826+0000 7f9505ffb640 1 -- 172.21.15.29:0/168134855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc00ccc0 con 0x7f9508102ad0 2024-09-21T02:55:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.826+0000 7f9505ffb640 1 -- 172.21.15.29:0/168134855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94fc00ce60 con 0x7f9508102ad0 2024-09-21T02:55:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.826+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95081a6c30 con 0x7f9508102ad0 2024-09-21T02:55:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.826+0000 7f9505ffb640 1 -- 172.21.15.29:0/168134855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc010430 con 0x7f9508102ad0 2024-09-21T02:55:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.826+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081ad390 con 0x7f9508102ad0 2024-09-21T02:55:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.828+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9508103d00 con 0x7f9508102ad0 2024-09-21T02:55:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.830+0000 7f9505ffb640 1 -- 172.21.15.29:0/168134855 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f94fc0046c0 con 0x7f9508102ad0 2024-09-21T02:55:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.830+0000 7f9505ffb640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94f80b6590 0x7f94f80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.831+0000 7f9507fff640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94f80b6590 0x7f94f80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.831+0000 7f9505ffb640 1 -- 172.21.15.29:0/168134855 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f94fc0e79e0 con 0x7f9508102ad0 2024-09-21T02:55:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.831+0000 7f9507fff640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94f80b6590 0x7f94f80b8aa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f94f4005940 tx=0x7f94f40058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:04.837+0000 7f9505ffb640 1 -- 172.21.15.29:0/168134855 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f94fc0adef0 con 0x7f9508102ad0 2024-09-21T02:55:05.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.192+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95081ad5f0 con 0x7f9508102ad0 2024-09-21T02:55:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.192+0000 7f9505ffb640 1 -- 172.21.15.29:0/168134855 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f94fc0b22d0 con 0x7f9508102ad0 2024-09-21T02:55:05.194 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:55:05.194 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:55:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:55:05.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94f80b6590 msgr2=0x7f94f80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:05.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94f80b6590 0x7f94f80b8aa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f94f4005940 tx=0x7f94f40058d0 comp rx=0 tx=0).stop 2024-09-21T02:55:05.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9508102ad0 msgr2=0x7f95081a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:05.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9508102ad0 0x7f95081a09d0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00e9f0 tx=0x7f94fc00edf0 comp rx=0 tx=0).stop 2024-09-21T02:55:05.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 shutdown_connections 2024-09-21T02:55:05.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94f80b6590 0x7f94f80b8aa0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:05.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9508108d10 0x7f95081a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:05.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95081083c0 0x7f95081a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:05.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 --2- 172.21.15.29:0/168134855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9508102ad0 0x7f95081a09d0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:05.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.196+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 >> 172.21.15.29:0/168134855 conn(0x7f95080796d0 msgr2=0x7f95081064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:05.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.197+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 shutdown_connections 2024-09-21T02:55:05.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:05.197+0000 7f950eb2e640 1 -- 172.21.15.29:0/168134855 wait complete. 2024-09-21T02:55:05.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18654 7.05 MB/sec execute 42 sec latency 96.433 ms 2024-09-21T02:55:05.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:05 smithi066 ceph-mon[33436]: pgmap v868: 129 pgs: 129 active+clean; 257 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-09-21T02:55:05.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:05 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/168134855' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:55:05.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:05 smithi190 ceph-mon[33871]: pgmap v868: 129 pgs: 129 active+clean; 257 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-09-21T02:55:05.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:05 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/168134855' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:55:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:05 smithi029 ceph-mon[31006]: pgmap v868: 129 pgs: 129 active+clean; 257 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-09-21T02:55:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:05 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/168134855' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:55:06.018 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 143.2 --await-for 45.0' 2024-09-21T02:55:06.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 143.2 --await-for 45.0 2024-09-21T02:55:06.431 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:55:06.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18812 7.11 MB/sec execute 43 sec latency 89.313 ms 2024-09-21T02:55:07.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 -- 172.21.15.29:0/412727902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 msgr2=0x7f9da410b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:07.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/412727902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 0x7f9da410b060 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009860 tx=0x7f9d9402a340 comp rx=0 tx=0).stop 2024-09-21T02:55:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 -- 172.21.15.29:0/412727902 shutdown_connections 2024-09-21T02:55:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/412727902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da410b5a0 0x7f9da4117410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/412727902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 0x7f9da410b060 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/412727902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9da410f550 0x7f9da410f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 -- 172.21.15.29:0/412727902 >> 172.21.15.29:0/412727902 conn(0x7f9da40712b0 msgr2=0x7f9da4073e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 -- 172.21.15.29:0/412727902 shutdown_connections 2024-09-21T02:55:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.026+0000 7f9dac5ca640 1 -- 172.21.15.29:0/412727902 wait complete. 2024-09-21T02:55:07.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.028+0000 7f9dac5ca640 1 Processor -- start 2024-09-21T02:55:07.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.028+0000 7f9dac5ca640 1 -- start start 2024-09-21T02:55:07.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9dac5ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 0x7f9da41a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9dac5ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9da410b5a0 0x7f9da41a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9dac5ca640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da410f550 0x7f9da41a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9dac5ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9da4119f80 con 0x7f9da410b5a0 2024-09-21T02:55:07.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9dac5ca640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9da4119e00 con 0x7f9da410f550 2024-09-21T02:55:07.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9dac5ca640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9da411a100 con 0x7f9da410ac50 2024-09-21T02:55:07.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9daa366640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 0x7f9da41a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9daa366640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 0x7f9da41a09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38132/0 (socket says 172.21.15.29:38132) 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.029+0000 7f9daa366640 1 -- 172.21.15.29:0/1174074229 learned_addr learned my addr 172.21.15.29:0/1174074229 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.030+0000 7f9da9b65640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9da410b5a0 0x7f9da41a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.030+0000 7f9daa366640 1 -- 172.21.15.29:0/1174074229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da410f550 msgr2=0x7f9da41a6210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.030+0000 7f9daa366640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da410f550 0x7f9da41a6210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.030+0000 7f9daa366640 1 -- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9da410b5a0 msgr2=0x7f9da41a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.030+0000 7f9daa366640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9da410b5a0 0x7f9da41a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.030+0000 7f9daa366640 1 -- 172.21.15.29:0/1174074229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da41a6940 con 0x7f9da410ac50 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.030+0000 7f9da9b65640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9da410b5a0 0x7f9da41a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.030+0000 7f9daa366640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 0x7f9da41a09d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9da0005c80 tx=0x7f9da000c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.031+0000 7f9d9b7fe640 1 -- 172.21.15.29:0/1174074229 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da0004230 con 0x7f9da410ac50 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.031+0000 7f9d9b7fe640 1 -- 172.21.15.29:0/1174074229 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9da00043d0 con 0x7f9da410ac50 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.031+0000 7f9d9b7fe640 1 -- 172.21.15.29:0/1174074229 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da000b970 con 0x7f9da410ac50 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.031+0000 7f9dac5ca640 1 -- 172.21.15.29:0/1174074229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da41a6c30 con 0x7f9da410ac50 2024-09-21T02:55:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.031+0000 7f9dac5ca640 1 -- 172.21.15.29:0/1174074229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da41ad390 con 0x7f9da410ac50 2024-09-21T02:55:07.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.034+0000 7f9db2b3e740 1 Processor -- start 2024-09-21T02:55:07.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.034+0000 7f9db2b3e740 1 -- start start 2024-09-21T02:55:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409b0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409b1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b640926b90 con 0x55b6405e0980 2024-09-21T02:55:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b640915fc0 con 0x55b640609840 2024-09-21T02:55:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b640919dd0 con 0x55b6405e8b60 2024-09-21T02:55:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9daab67640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9daab67640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42490/0 (socket says 172.21.15.29:42490) 2024-09-21T02:55:07.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9daab67640 1 -- 172.21.15.29:0/3799388253 learned_addr learned my addr 172.21.15.29:0/3799388253 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:07.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.035+0000 7f9d9b7fe640 1 -- 172.21.15.29:0/1174074229 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9da00035f0 con 0x7f9da410ac50 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9daab67640 1 -- 172.21.15.29:0/3799388253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 msgr2=0x55b6409b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9daa366640 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9da9b65640 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9daab67640 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9daab67640 1 -- 172.21.15.29:0/3799388253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 msgr2=0x55b6409b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9daab67640 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9daab67640 1 -- 172.21.15.29:0/3799388253 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b64089d060 con 0x55b640609840 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9da9b65640 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9daa366640 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.036+0000 7f9daab67640 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409b0600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009cd0 tx=0x7f9d9c00e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a35c51b5768233b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.037+0000 7f9d98ff9640 1 -- 172.21.15.29:0/3799388253 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c016070 con 0x55b640609840 2024-09-21T02:55:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.037+0000 7f9d98ff9640 1 -- 172.21.15.29:0/3799388253 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d9c005020 con 0x55b640609840 2024-09-21T02:55:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.037+0000 7f9d9b7fe640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9d900b6590 0x7f9d900b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.037+0000 7f9daa366640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9d900b6590 0x7f9d900b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.037+0000 7f9d9b7fe640 1 -- 172.21.15.29:0/1174074229 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9da00e7c40 con 0x7f9da410ac50 2024-09-21T02:55:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.038+0000 7f9d98ff9640 1 -- 172.21.15.29:0/3799388253 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c0116d0 con 0x55b640609840 2024-09-21T02:55:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.039+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3799388253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 msgr2=0x55b6409b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.039+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409b0600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009cd0 tx=0x7f9d9c00e6c0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.039+0000 7f9daa366640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9d900b6590 0x7f9d900b8aa0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55b6409af280 tx=0x7f9da000a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.039+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3799388253 shutdown_connections 2024-09-21T02:55:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.039+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.039+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.039+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/3799388253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409b0600 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.039+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3799388253 >> 172.21.15.29:0/3799388253 conn(0x55b6405fd990 msgr2=0x55b6409ae4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.040+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3799388253 shutdown_connections 2024-09-21T02:55:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.040+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3799388253 wait complete. 2024-09-21T02:55:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.040+0000 7f9db2b3e740 1 -- start start 2024-09-21T02:55:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.040+0000 7f9db2b3e740 1 Processor -- start 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409c4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b6407777a0 con 0x55b6405e0980 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b6408bb0a0 con 0x55b640609840 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b6409c75b0 con 0x55b6405e8b60 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9db2b3e740 10 client.0 _ll_register_callbacks cb 0x55b6408983c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9daab67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9daab67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409c4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45036/0 (socket says 172.21.15.29:45036) 2024-09-21T02:55:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9daab67640 1 -- 172.21.15.29:0/1113378978 learned_addr learned my addr 172.21.15.29:0/1113378978 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.041+0000 7f9da9b65640 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9daab67640 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9daab67640 1 -- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 msgr2=0x55b6409c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9daab67640 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9daab67640 1 -- 172.21.15.29:0/1113378978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 msgr2=0x55b6409d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9daab67640 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9daab67640 1 -- 172.21.15.29:0/1113378978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6408a9880 con 0x55b6405e0980 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9da9b65640 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9daab67640 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409c4920 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c005a40 tx=0x7f9d9c00efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4e725fd134e393 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9d8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9d8dffb640 1 -- 172.21.15.29:0/1113378978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c016070 con 0x55b6405e0980 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9db2b3e740 1 -- 172.21.15.29:0/1113378978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d9c005020 con 0x55b6405e0980 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9d8dffb640 1 -- 172.21.15.29:0/1113378978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9d9c003b70 con 0x55b6405e0980 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.042+0000 7f9d8dffb640 1 -- 172.21.15.29:0/1113378978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9d9c030050 con 0x55b6405e0980 2024-09-21T02:55:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9db2b3e740 10 client.17625 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9db2b3e740 1 -- 172.21.15.29:0/1113378978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b640683b90 con 0x55b6405e0980 2024-09-21T02:55:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9d8dffb640 1 -- 172.21.15.29:0/1113378978 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d9c005020 con 0x55b6405e0980 2024-09-21T02:55:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9d8dffb640 1 -- 172.21.15.29:0/1113378978 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f9d9c00f040 con 0x55b6405e0980 2024-09-21T02:55:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9db2b3e740 10 client.17625 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9db2b3e740 10 client.17625 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9db2b3e740 20 client.17625 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b6405c81c0 0x55b6409d65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9db2b3e740 4 client.17625 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9db2b3e740 1 -- 172.21.15.29:0/1113378978 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b640928ae0 con 0x55b6405c81c0 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.043+0000 7f9da9b65640 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b6405c81c0 0x55b6409d65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.044+0000 7f9d8dffb640 10 client.17625 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.044+0000 7f9da9b65640 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b6405c81c0 0x55b6409d65f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1af312bb2cd9b9ff server_cookie=df8506b12ee43993 in_seq=0 out_seq=0 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.045+0000 7f9d8dffb640 1 -- 172.21.15.29:0/1113378978 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b640928ae0 con 0x55b6405c81c0 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.045+0000 7f9d8dffb640 10 client.17625 handle_command_reply: tid=0 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.045+0000 7f9db2b3e740 1 client.17625 shutdown 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.045+0000 7f9db2b3e740 1 -- 172.21.15.29:0/1113378978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 msgr2=0x55b6409c4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.045+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409c4920 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c005a40 tx=0x7f9d9c00efb0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 1 -- 172.21.15.29:0/1113378978 shutdown_connections 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b640609840 0x55b6409d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e8b60 0x55b6409c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b6405c81c0 0x55b6409d65f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/1113378978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e0980 0x55b6409c4920 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9d8dffb640 0 client.17625 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 1 -- 172.21.15.29:0/1113378978 >> 172.21.15.29:0/1113378978 conn(0x55b6405fd990 msgr2=0x55b6409b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 1 -- 172.21.15.29:0/1113378978 shutdown_connections 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 1 -- 172.21.15.29:0/1113378978 wait complete. 2024-09-21T02:55:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.046+0000 7f9db2b3e740 20 client.17625 trim_cache size 0 max 16384 2024-09-21T02:55:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.071+0000 7f9db2b3e740 1 Processor -- start 2024-09-21T02:55:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.071+0000 7f9db2b3e740 1 -- start start 2024-09-21T02:55:07.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.071+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405c81c0 0x55b6409c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.071+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405e0980 0x55b6409b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.071+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 0x55b6409d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.071+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9da41b7090 con 0x55b6405e8b60 2024-09-21T02:55:07.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.071+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9da41b8bd0 con 0x55b6405e0980 2024-09-21T02:55:07.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.071+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9da41b7760 con 0x55b6405c81c0 2024-09-21T02:55:07.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405c81c0 0x55b6409c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405c81c0 0x55b6409c62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:38144/0 (socket says 172.21.15.29:38144) 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 -- 172.21.15.29:0/3376861506 learned_addr learned my addr 172.21.15.29:0/3376861506 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 0x55b6409d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9da9b65640 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405e0980 0x55b6409b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 -- 172.21.15.29:0/3376861506 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405e0980 msgr2=0x55b6409b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405e0980 0x55b6409b3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 -- 172.21.15.29:0/3376861506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 msgr2=0x55b6409d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 0x55b6409d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 -- 172.21.15.29:0/3376861506 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6409b44d0 con 0x55b6405c81c0 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9da9b65640 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405e0980 0x55b6409b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.072+0000 7f9daab67640 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405c81c0 0x55b6409c62b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c005d70 tx=0x7f9d9c00e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=417dbb9c455a58de server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.073+0000 7f9d8ffff640 1 -- 172.21.15.29:0/3376861506 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c016070 con 0x55b6405c81c0 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.073+0000 7f9d8ffff640 1 -- 172.21.15.29:0/3376861506 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d9c0097e0 con 0x55b6405c81c0 2024-09-21T02:55:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.073+0000 7f9d8ffff640 1 -- 172.21.15.29:0/3376861506 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c023040 con 0x55b6405c81c0 2024-09-21T02:55:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3376861506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405c81c0 msgr2=0x55b6409c62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405c81c0 0x55b6409c62b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c005d70 tx=0x7f9d9c00e6c0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3376861506 shutdown_connections 2024-09-21T02:55:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 0x55b6409d0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405e0980 0x55b6409b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/3376861506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405c81c0 0x55b6409c62b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3376861506 >> 172.21.15.29:0/3376861506 conn(0x55b6405fd990 msgr2=0x55b6409b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3376861506 shutdown_connections 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 -- 172.21.15.29:0/3376861506 wait complete. 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.074+0000 7f9db2b3e740 1 -- start start 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9db2b3e740 1 Processor -- start 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405c81c0 0x55b6409c9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e0980 0x55b6409c9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9db2b3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 0x55b640a39f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9da41b8500 con 0x55b6405e8b60 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9da41b7e30 con 0x55b6405c81c0 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9db2b3e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9da41b5070 con 0x55b6405e0980 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9db2b3e740 10 client.0 _ll_register_callbacks cb 0x7f9d9c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9daab67640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405c81c0 0x55b6409c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9daab67640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405c81c0 0x55b6409c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42520/0 (socket says 172.21.15.29:42520) 2024-09-21T02:55:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9daab67640 1 -- 172.21.15.29:0/4112549939 learned_addr learned my addr 172.21.15.29:0/4112549939 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9da9b65640 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e0980 0x55b6409c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.075+0000 7f9daab67640 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 0x55b640a39f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.076+0000 7f9daab67640 1 -- 172.21.15.29:0/4112549939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e0980 msgr2=0x55b6409c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.076+0000 7f9daab67640 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e0980 0x55b6409c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.076+0000 7f9daab67640 1 -- 172.21.15.29:0/4112549939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 msgr2=0x55b640a39f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.076+0000 7f9daab67640 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 0x55b640a39f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.076+0000 7f9daab67640 1 -- 172.21.15.29:0/4112549939 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b640a3a5b0 con 0x55b6405c81c0 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.076+0000 7f9daab67640 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405c81c0 0x55b6409c9340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c005a20 tx=0x7f9d9c02a900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bee751e78dd479 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.076+0000 7f9d8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.076+0000 7f9d8dffb640 1 -- 172.21.15.29:0/4112549939 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c016070 con 0x55b6405c81c0 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.077+0000 7f9d8dffb640 1 -- 172.21.15.29:0/4112549939 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9d9c004e80 con 0x55b6405c81c0 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.077+0000 7f9db2b3e740 1 -- 172.21.15.29:0/4112549939 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d9c0097e0 con 0x55b6405c81c0 2024-09-21T02:55:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.077+0000 7f9d8dffb640 1 -- 172.21.15.29:0/4112549939 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9d9c00ebc0 con 0x55b6405c81c0 2024-09-21T02:55:07.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.077+0000 7f9db2b3e740 10 client.26993 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:07.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.077+0000 7f9db2b3e740 1 -- 172.21.15.29:0/4112549939 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b640a41960 con 0x55b6405c81c0 2024-09-21T02:55:07.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.077+0000 7f9d8dffb640 1 -- 172.21.15.29:0/4112549939 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d9c02ae40 con 0x55b6405c81c0 2024-09-21T02:55:07.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.077+0000 7f9d8dffb640 1 -- 172.21.15.29:0/4112549939 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f9d9c004030 con 0x55b6405c81c0 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9db2b3e740 10 client.26993 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9db2b3e740 10 client.26993 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9db2b3e740 20 client.26993 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b640609840 0x55b640a44150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9db2b3e740 4 client.26993 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.2, "await_for": 45.0}] 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9db2b3e740 1 -- 172.21.15.29:0/4112549939 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.2, "await_for": 45.0}) -- 0x7f9d9c0097e0 con 0x55b640609840 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9da9b65640 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b640609840 0x55b640a44150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9da9b65640 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b640609840 0x55b640a44150 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1229586d6833ec6f server_cookie=fd9f51ba56e3e5b3 in_seq=0 out_seq=0 2024-09-21T02:55:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.078+0000 7f9d8dffb640 10 client.26993 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9d8dffb640 1 -- 172.21.15.29:0/4112549939 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f9d9c0097e0 con 0x55b640609840 2024-09-21T02:55:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9d8dffb640 10 client.26993 handle_command_reply: tid=0 2024-09-21T02:55:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 client.26993 shutdown 2024-09-21T02:55:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 -- 172.21.15.29:0/4112549939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405c81c0 msgr2=0x55b6409c9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405c81c0 0x55b6409c9340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c005a20 tx=0x7f9d9c02a900 comp rx=0 tx=0).stop 2024-09-21T02:55:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 -- 172.21.15.29:0/4112549939 shutdown_connections 2024-09-21T02:55:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b640609840 0x55b640a44150 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6405e8b60 0x55b640a39f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9d8dffb640 0 client.26993 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b6405e0980 0x55b6409c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 --2- 172.21.15.29:0/4112549939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b6405c81c0 0x55b6409c9340 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 -- 172.21.15.29:0/4112549939 >> 172.21.15.29:0/4112549939 conn(0x55b6405fd990 msgr2=0x55b6409aca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 -- 172.21.15.29:0/4112549939 shutdown_connections 2024-09-21T02:55:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.621+0000 7f9db2b3e740 1 -- 172.21.15.29:0/4112549939 wait complete. 2024-09-21T02:55:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9db2b3e740 20 client.26993 trim_cache size 0 max 16384 2024-09-21T02:55:07.627 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:55:07.627 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:55:07.627 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:55:07.627 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 62, 2024-09-21T02:55:07.627 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:55:07.628 INFO:teuthology.orchestra.run.smithi029.stdout: "adcecebd": { 2024-09-21T02:55:07.628 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 62, 2024-09-21T02:55:07.628 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:55:07.628 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:55:07.628 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:55:07.628 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:55:07.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:55:07.629 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:55:07.629 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 143.2, 2024-09-21T02:55:07.629 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:55:07.629 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:55:07.629 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:55:07.629 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:55:07.630 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:55:07.630 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:55:07.630 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:07.630 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:07.630 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:07.630 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:07.630 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:07.631 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:55:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 -- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9d900b6590 msgr2=0x7f9d900b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9d900b6590 0x7f9d900b8aa0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55b6409af280 tx=0x7f9da000a250 comp rx=0 tx=0).stop 2024-09-21T02:55:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 -- 172.21.15.29:0/1174074229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 msgr2=0x7f9da41a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 0x7f9da41a09d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9da0005c80 tx=0x7f9da000c7a0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 -- 172.21.15.29:0/1174074229 shutdown_connections 2024-09-21T02:55:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9d900b6590 0x7f9d900b8aa0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da410f550 0x7f9da41a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9da410b5a0 0x7f9da41a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 --2- 172.21.15.29:0/1174074229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9da410ac50 0x7f9da41a09d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.622+0000 7f9dac5ca640 1 -- 172.21.15.29:0/1174074229 >> 172.21.15.29:0/1174074229 conn(0x7f9da40712b0 msgr2=0x7f9da410e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.623+0000 7f9dac5ca640 1 -- 172.21.15.29:0/1174074229 shutdown_connections 2024-09-21T02:55:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:07.623+0000 7f9dac5ca640 1 -- 172.21.15.29:0/1174074229 wait complete. 2024-09-21T02:55:07.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18862 6.98 MB/sec execute 44 sec latency 558.905 ms 2024-09-21T02:55:07.764 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:07 smithi066 ceph-mon[33436]: pgmap v869: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-09-21T02:55:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:07 smithi190 ceph-mon[33871]: pgmap v869: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-09-21T02:55:07.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:07 smithi029 ceph-mon[31006]: pgmap v869: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-09-21T02:55:08.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 6.83 MB/sec execute 45 sec latency 1558.994 ms 2024-09-21T02:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 -- 172.21.15.29:0/918005085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33441095a0 msgr2=0x7f334410b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 --2- 172.21.15.29:0/918005085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33441095a0 0x7f334410b9c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f333c00b0a0 tx=0x7f333c02a710 comp rx=0 tx=0).stop 2024-09-21T02:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 -- 172.21.15.29:0/918005085 shutdown_connections 2024-09-21T02:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 --2- 172.21.15.29:0/918005085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33441095a0 0x7f334410b9c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 --2- 172.21.15.29:0/918005085 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3344106c40 0x7f3344109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 --2- 172.21.15.29:0/918005085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3344101af0 0x7f3344101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 -- 172.21.15.29:0/918005085 >> 172.21.15.29:0/918005085 conn(0x7f33440fd690 msgr2=0x7f33440ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 -- 172.21.15.29:0/918005085 shutdown_connections 2024-09-21T02:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.068+0000 7f334a36e640 1 -- 172.21.15.29:0/918005085 wait complete. 2024-09-21T02:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.069+0000 7f334a36e640 1 Processor -- start 2024-09-21T02:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.069+0000 7f334a36e640 1 -- start start 2024-09-21T02:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.069+0000 7f334a36e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3344101af0 0x7f3344199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f334a36e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3344106c40 0x7f334419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f334a36e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f33441095a0 0x7f334419b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f334a36e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f334410f1e0 con 0x7f3344106c40 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f334a36e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f334410f060 con 0x7f33441095a0 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f334a36e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f334410f360 con 0x7f3344101af0 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3344106c40 0x7f334419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3344106c40 0x7f334419a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45068/0 (socket says 172.21.15.29:45068) 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 -- 172.21.15.29:0/1110019751 learned_addr learned my addr 172.21.15.29:0/1110019751 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f3343fff640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3344101af0 0x7f3344199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f334890b640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f33441095a0 0x7f334419b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 -- 172.21.15.29:0/1110019751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3344101af0 msgr2=0x7f3344199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3344101af0 0x7f3344199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 -- 172.21.15.29:0/1110019751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f33441095a0 msgr2=0x7f334419b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f33441095a0 0x7f334419b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 -- 172.21.15.29:0/1110019751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334419b8a0 con 0x7f3344106c40 2024-09-21T02:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f334890b640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f33441095a0 0x7f334419b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33437fe640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3344106c40 0x7f334419a4d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f333800d950 tx=0x7f333800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.070+0000 7f33417fa640 1 -- 172.21.15.29:0/1110019751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3338017030 con 0x7f3344106c40 2024-09-21T02:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.071+0000 7f33417fa640 1 -- 172.21.15.29:0/1110019751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3338004020 con 0x7f3344106c40 2024-09-21T02:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.071+0000 7f33417fa640 1 -- 172.21.15.29:0/1110019751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33380055d0 con 0x7f3344106c40 2024-09-21T02:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.071+0000 7f334a36e640 1 -- 172.21.15.29:0/1110019751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f334419bb90 con 0x7f3344106c40 2024-09-21T02:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.071+0000 7f334a36e640 1 -- 172.21.15.29:0/1110019751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33441a2280 con 0x7f3344106c40 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.073+0000 7f33513bd740 1 Processor -- start 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.073+0000 7f33513bd740 1 -- start start 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.073+0000 7f33417fa640 1 -- 172.21.15.29:0/1110019751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f3338009a30 con 0x7f3344106c40 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f33513bd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b96ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f33513bd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b59a2f0 0x55861b96b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f33513bd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 0x55861b96c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f33513bd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55861b8dd6f0 con 0x55861b59a2f0 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f33513bd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55861b8ccb20 con 0x55861b5c31b0 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f33513bd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55861b8d0930 con 0x55861b5a24d0 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b59a2f0 0x55861b96b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b59a2f0 0x55861b96b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45084/0 (socket says 172.21.15.29:45084) 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 -- 172.21.15.29:0/2413197850 learned_addr learned my addr 172.21.15.29:0/2413197850 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f334890b640 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b96ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f33437fe640 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 0x55861b96c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 -- 172.21.15.29:0/2413197850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 msgr2=0x55861b96c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 0x55861b96c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 -- 172.21.15.29:0/2413197850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 msgr2=0x55861b96ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b96ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 -- 172.21.15.29:0/2413197850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55861b892530 con 0x55861b59a2f0 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f3343fff640 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b59a2f0 0x55861b96b4c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f333400b760 tx=0x7f333400bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf4c051132a0cdc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.074+0000 7f33417fa640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f332c0b6510 0x7f332c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.075+0000 7f332a7fc640 1 -- 172.21.15.29:0/2413197850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3334003dd0 con 0x55861b59a2f0 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.075+0000 7f334890b640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f332c0b6510 0x7f332c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.075+0000 7f332a7fc640 1 -- 172.21.15.29:0/2413197850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3334015070 con 0x55861b59a2f0 2024-09-21T02:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.075+0000 7f33417fa640 1 -- 172.21.15.29:0/1110019751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f33380e7070 con 0x7f3344106c40 2024-09-21T02:55:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.075+0000 7f332a7fc640 1 -- 172.21.15.29:0/2413197850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3334004c80 con 0x55861b59a2f0 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 -- 172.21.15.29:0/2413197850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b59a2f0 msgr2=0x55861b96b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b59a2f0 0x55861b96b4c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f333400b760 tx=0x7f333400bc20 comp rx=0 tx=0).stop 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f334890b640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f332c0b6510 0x7f332c0b8a20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f333c006130 tx=0x7f333c0060a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 -- 172.21.15.29:0/2413197850 shutdown_connections 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 0x55861b96c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b59a2f0 0x55861b96b4c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 --2- 172.21.15.29:0/2413197850 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b96ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 -- 172.21.15.29:0/2413197850 >> 172.21.15.29:0/2413197850 conn(0x55861b5b7300 msgr2=0x55861b9668e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 -- 172.21.15.29:0/2413197850 shutdown_connections 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 -- 172.21.15.29:0/2413197850 wait complete. 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 -- start start 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.076+0000 7f33513bd740 1 Processor -- start 2024-09-21T02:55:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33513bd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b59a2f0 0x55861b9745d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33513bd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b974c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33513bd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b975800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33513bd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55861b804ab0 con 0x55861b5a24d0 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33513bd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55861b872730 con 0x55861b5c31b0 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33513bd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55861b96cb30 con 0x55861b59a2f0 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33513bd740 10 client.0 _ll_register_callbacks cb 0x55861b8c66b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b974c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b974c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45096/0 (socket says 172.21.15.29:45096) 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 -- 172.21.15.29:0/3038141761 learned_addr learned my addr 172.21.15.29:0/3038141761 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f334890b640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b975800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f3343fff640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b59a2f0 0x55861b9745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 -- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b59a2f0 msgr2=0x55861b9745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b59a2f0 0x55861b9745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 -- 172.21.15.29:0/3038141761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 msgr2=0x55861b975800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b975800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 -- 172.21.15.29:0/3038141761 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55861b62e790 con 0x55861b5a24d0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f3343fff640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b59a2f0 0x55861b9745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f334890b640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b975800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33437fe640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b974c90 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3338005fd0 tx=0x7f33380a95f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06eb70707aebf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.077+0000 7f33237fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33237fe640 1 -- 172.21.15.29:0/3038141761 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33380ac5f0 con 0x55861b5a24d0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33237fe640 1 -- 172.21.15.29:0/3038141761 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f33380b1600 con 0x55861b5a24d0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 1 -- 172.21.15.29:0/3038141761 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3334015070 con 0x55861b5a24d0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33237fe640 1 -- 172.21.15.29:0/3038141761 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f33380ac990 con 0x55861b5a24d0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 10 client.17643 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 1 -- 172.21.15.29:0/3038141761 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55861b6a7620 con 0x55861b5a24d0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33237fe640 1 -- 172.21.15.29:0/3038141761 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3334015070 con 0x55861b5a24d0 2024-09-21T02:55:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33237fe640 1 -- 172.21.15.29:0/3038141761 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f33380b4ce0 con 0x55861b5a24d0 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 10 client.17643 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 10 client.17643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 20 client.17643 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55861b581d80 0x55861b981d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 4 client.17643 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.078+0000 7f33513bd740 1 -- 172.21.15.29:0/3038141761 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55861b8df640 con 0x55861b581d80 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.079+0000 7f3343fff640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55861b581d80 0x55861b981d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.079+0000 7f33237fe640 10 client.17643 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.079+0000 7f3343fff640 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55861b581d80 0x55861b981d50 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bf58d093c290cc4 server_cookie=76dcb950cea3c695 in_seq=0 out_seq=0 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33237fe640 1 -- 172.21.15.29:0/3038141761 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55861b8df640 con 0x55861b581d80 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33237fe640 10 client.17643 handle_command_reply: tid=0 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 client.17643 shutdown 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 -- 172.21.15.29:0/3038141761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 msgr2=0x55861b974c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b974c90 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3338005fd0 tx=0x7f33380a95f0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 -- 172.21.15.29:0/3038141761 shutdown_connections 2024-09-21T02:55:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b5c31b0 0x55861b975800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55861b581d80 0x55861b981d50 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b974c90 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 --2- 172.21.15.29:0/3038141761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b59a2f0 0x55861b9745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33237fe640 0 client.17643 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 -- 172.21.15.29:0/3038141761 >> 172.21.15.29:0/3038141761 conn(0x55861b5b7300 msgr2=0x55861b964ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 -- 172.21.15.29:0/3038141761 shutdown_connections 2024-09-21T02:55:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.080+0000 7f33513bd740 1 -- 172.21.15.29:0/3038141761 wait complete. 2024-09-21T02:55:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.081+0000 7f33513bd740 20 client.17643 trim_cache size 0 max 16384 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.095+0000 7f33513bd740 1 Processor -- start 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.095+0000 7f33513bd740 1 -- start start 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.095+0000 7f33513bd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b9731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.095+0000 7f33513bd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b96adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.095+0000 7f33513bd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5c31b0 0x55861b96b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.095+0000 7f33513bd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33441ac5c0 con 0x55861b5a24d0 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33513bd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f33441ae100 con 0x55861b59a2f0 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33513bd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f33441acc90 con 0x55861b5c31b0 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b96adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b96adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45110/0 (socket says 172.21.15.29:45110) 2024-09-21T02:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 -- 172.21.15.29:0/3615031015 learned_addr learned my addr 172.21.15.29:0/3615031015 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f3343fff640 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b9731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 -- 172.21.15.29:0/3615031015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5c31b0 msgr2=0x55861b96b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f334890b640 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5c31b0 0x55861b96b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5c31b0 0x55861b96b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 -- 172.21.15.29:0/3615031015 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 msgr2=0x55861b9731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b9731f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 -- 172.21.15.29:0/3615031015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55861b960b40 con 0x55861b5a24d0 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f3343fff640 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b9731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f33437fe640 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b96adf0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3338096040 tx=0x7f3338004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbe373a254152bc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f3329ffb640 1 -- 172.21.15.29:0/3615031015 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33380ac5f0 con 0x55861b5a24d0 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f3329ffb640 1 -- 172.21.15.29:0/3615031015 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33380ad580 con 0x55861b5a24d0 2024-09-21T02:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.096+0000 7f3329ffb640 1 -- 172.21.15.29:0/3615031015 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33380ea3f0 con 0x55861b5a24d0 2024-09-21T02:55:09.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 -- 172.21.15.29:0/3615031015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 msgr2=0x55861b96adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b96adf0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3338096040 tx=0x7f3338004530 comp rx=0 tx=0).stop 2024-09-21T02:55:09.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 -- 172.21.15.29:0/3615031015 shutdown_connections 2024-09-21T02:55:09.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5c31b0 0x55861b96b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5a24d0 0x55861b96adf0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 --2- 172.21.15.29:0/3615031015 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b9731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 -- 172.21.15.29:0/3615031015 >> 172.21.15.29:0/3615031015 conn(0x55861b5b7300 msgr2=0x55861b972ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 -- 172.21.15.29:0/3615031015 shutdown_connections 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 -- 172.21.15.29:0/3615031015 wait complete. 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.097+0000 7f33513bd740 1 -- start start 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33513bd740 1 Processor -- start 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33513bd740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b97c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33513bd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 0x55861b97b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33513bd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5c31b0 0x55861b97b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33513bd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33441ada30 con 0x55861b5c31b0 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33513bd740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f33441ad360 con 0x55861b59a2f0 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33513bd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f33441aa5a0 con 0x55861b5a24d0 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33513bd740 10 client.0 _ll_register_callbacks cb 0x7f33380a94d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f334890b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5c31b0 0x55861b97b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f334890b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5c31b0 0x55861b97b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45126/0 (socket says 172.21.15.29:45126) 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f334890b640 1 -- 172.21.15.29:0/1073813045 learned_addr learned my addr 172.21.15.29:0/1073813045 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f3343fff640 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b97c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.098+0000 7f33437fe640 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 0x55861b97b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f334890b640 1 -- 172.21.15.29:0/1073813045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 msgr2=0x55861b97b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f334890b640 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 0x55861b97b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f334890b640 1 -- 172.21.15.29:0/1073813045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 msgr2=0x55861b97c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f334890b640 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b97c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f334890b640 1 -- 172.21.15.29:0/1073813045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55861b97be60 con 0x55861b5c31b0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f334890b640 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5c31b0 0x55861b97b6d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f333c009fd0 tx=0x7f333c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77449cb6960872d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f33237fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f33237fe640 1 -- 172.21.15.29:0/1073813045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333c04e070 con 0x55861b5c31b0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f33237fe640 1 -- 172.21.15.29:0/1073813045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f333c008d20 con 0x55861b5c31b0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f33513bd740 1 -- 172.21.15.29:0/1073813045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33380ad580 con 0x55861b5c31b0 2024-09-21T02:55:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f33237fe640 1 -- 172.21.15.29:0/1073813045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f333c041040 con 0x55861b5c31b0 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f33513bd740 10 client.17649 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.099+0000 7f33513bd740 1 -- 172.21.15.29:0/1073813045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55861b9ec820 con 0x55861b5c31b0 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f33237fe640 1 -- 172.21.15.29:0/1073813045 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33380ad580 con 0x55861b5c31b0 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f33237fe640 1 -- 172.21.15.29:0/1073813045 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f333c004040 con 0x55861b5c31b0 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f33513bd740 10 client.17649 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f33513bd740 10 client.17649 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f33513bd740 20 client.17649 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f33513bd740 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55861b5571c0 0x55861b9ef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f33513bd740 4 client.17649 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f33513bd740 1 -- 172.21.15.29:0/1073813045 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55861b8db0f0 con 0x55861b5571c0 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.100+0000 7f3343fff640 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55861b5571c0 0x55861b9ef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.101+0000 7f33237fe640 10 client.17649 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.101+0000 7f3343fff640 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55861b5571c0 0x55861b9ef1f0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6755170a69ae13 server_cookie=b21dccd43c70b6d2 in_seq=0 out_seq=0 2024-09-21T02:55:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.290+0000 7f33237fe640 1 -- 172.21.15.29:0/1073813045 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55861b8db0f0 con 0x55861b5571c0 2024-09-21T02:55:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.290+0000 7f33237fe640 10 client.17649 handle_command_reply: tid=0 2024-09-21T02:55:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.290+0000 7f33513bd740 1 client.17649 shutdown 2024-09-21T02:55:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 -- 172.21.15.29:0/1073813045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5c31b0 msgr2=0x55861b97b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5c31b0 0x55861b97b6d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f333c009fd0 tx=0x7f333c02a710 comp rx=0 tx=0).stop 2024-09-21T02:55:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 -- 172.21.15.29:0/1073813045 shutdown_connections 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55861b5571c0 0x55861b9ef1f0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55861b5c31b0 0x55861b97b6d0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55861b5a24d0 0x55861b97b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33237fe640 0 client.17649 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 --2- 172.21.15.29:0/1073813045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55861b59a2f0 0x55861b97c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 -- 172.21.15.29:0/1073813045 >> 172.21.15.29:0/1073813045 conn(0x55861b5b7300 msgr2=0x55861b96ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 -- 172.21.15.29:0/1073813045 shutdown_connections 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 1 -- 172.21.15.29:0/1073813045 wait complete. 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f33513bd740 20 client.17649 trim_cache size 0 max 16384 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:55:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:55:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:55:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f334a36e640 1 -- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f332c0b6510 msgr2=0x7f332c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f334a36e640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f332c0b6510 0x7f332c0b8a20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f333c006130 tx=0x7f333c0060a0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f334a36e640 1 -- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3344106c40 msgr2=0x7f334419a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.291+0000 7f334a36e640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3344106c40 0x7f334419a4d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f333800d950 tx=0x7f333800de10 comp rx=0 tx=0).stop 2024-09-21T02:55:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.292+0000 7f334a36e640 1 -- 172.21.15.29:0/1110019751 shutdown_connections 2024-09-21T02:55:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.292+0000 7f334a36e640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f332c0b6510 0x7f332c0b8a20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.292+0000 7f334a36e640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f33441095a0 0x7f334419b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.292+0000 7f334a36e640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3344106c40 0x7f334419a4d0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.292+0000 7f334a36e640 1 --2- 172.21.15.29:0/1110019751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3344101af0 0x7f3344199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.292+0000 7f334a36e640 1 -- 172.21.15.29:0/1110019751 >> 172.21.15.29:0/1110019751 conn(0x7f33440fd690 msgr2=0x7f3344107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.292+0000 7f334a36e640 1 -- 172.21.15.29:0/1110019751 shutdown_connections 2024-09-21T02:55:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:09.292+0000 7f334a36e640 1 -- 172.21.15.29:0/1110019751 wait complete. 2024-09-21T02:55:09.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 6.68 MB/sec execute 46 sec latency 2559.102 ms 2024-09-21T02:55:09.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:09 smithi066 ceph-mon[33436]: pgmap v870: 129 pgs: 129 active+clean; 261 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 3.1 MiB/s wr, 348 op/s 2024-09-21T02:55:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:09 smithi190 ceph-mon[33871]: pgmap v870: 129 pgs: 129 active+clean; 261 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 3.1 MiB/s wr, 348 op/s 2024-09-21T02:55:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:09 smithi029 ceph-mon[31006]: pgmap v870: 129 pgs: 129 active+clean; 261 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 3.1 MiB/s wr, 348 op/s 2024-09-21T02:55:10.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 6.54 MB/sec execute 47 sec latency 3559.159 ms 2024-09-21T02:55:11.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 6.40 MB/sec execute 48 sec latency 4559.246 ms 2024-09-21T02:55:11.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:11 smithi066 ceph-mon[33436]: pgmap v871: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-09-21T02:55:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:11 smithi190 ceph-mon[33871]: pgmap v871: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-09-21T02:55:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:11 smithi029 ceph-mon[31006]: pgmap v871: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-09-21T02:55:12.547 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'adcecebd', quiesce took 8.7 seconds. Will release after: 14.5 2024-09-21T02:55:12.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 6.27 MB/sec execute 49 sec latency 5559.329 ms 2024-09-21T02:55:13.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 6.14 MB/sec execute 50 sec latency 6559.415 ms 2024-09-21T02:55:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:13 smithi066 ceph-mon[33436]: pgmap v872: 129 pgs: 129 active+clean; 259 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 2.4 MiB/s wr, 256 op/s 2024-09-21T02:55:13.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:13 smithi190 ceph-mon[33871]: pgmap v872: 129 pgs: 129 active+clean; 259 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 2.4 MiB/s wr, 256 op/s 2024-09-21T02:55:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:13 smithi029 ceph-mon[31006]: pgmap v872: 129 pgs: 129 active+clean; 259 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 2.4 MiB/s wr, 256 op/s 2024-09-21T02:55:14.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 6.02 MB/sec execute 51 sec latency 7559.511 ms 2024-09-21T02:55:15.716 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.91 MB/sec execute 52 sec latency 8559.619 ms 2024-09-21T02:55:15.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:15 smithi066 ceph-mon[33436]: pgmap v873: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-09-21T02:55:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:15 smithi190 ceph-mon[33871]: pgmap v873: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-09-21T02:55:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:15 smithi029 ceph-mon[31006]: pgmap v873: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-09-21T02:55:16.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.80 MB/sec execute 53 sec latency 9559.727 ms 2024-09-21T02:55:17.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.69 MB/sec execute 54 sec latency 10559.832 ms 2024-09-21T02:55:17.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:17 smithi066 ceph-mon[33436]: pgmap v874: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-09-21T02:55:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:17 smithi190 ceph-mon[33871]: pgmap v874: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-09-21T02:55:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:17 smithi029 ceph-mon[31006]: pgmap v874: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-09-21T02:55:18.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.59 MB/sec execute 55 sec latency 11559.945 ms 2024-09-21T02:55:19.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.49 MB/sec execute 56 sec latency 12560.061 ms 2024-09-21T02:55:19.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:19 smithi066 ceph-mon[33436]: pgmap v875: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 905 KiB/s wr, 135 op/s 2024-09-21T02:55:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:19 smithi190 ceph-mon[33871]: pgmap v875: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 905 KiB/s wr, 135 op/s 2024-09-21T02:55:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:19 smithi029 ceph-mon[31006]: pgmap v875: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 905 KiB/s wr, 135 op/s 2024-09-21T02:55:20.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.39 MB/sec execute 57 sec latency 13560.166 ms 2024-09-21T02:55:21.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.30 MB/sec execute 58 sec latency 14560.316 ms 2024-09-21T02:55:21.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:21 smithi066 ceph-mon[33436]: pgmap v876: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 567 KiB/s wr, 103 op/s 2024-09-21T02:55:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:21 smithi190 ceph-mon[33871]: pgmap v876: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 567 KiB/s wr, 103 op/s 2024-09-21T02:55:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:21 smithi029 ceph-mon[31006]: pgmap v876: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 567 KiB/s wr, 103 op/s 2024-09-21T02:55:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 -- 172.21.15.29:0/270545659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 msgr2=0x7f1b90103b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 --2- 172.21.15.29:0/270545659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 0x7f1b90103b50 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009ca0 tx=0x7f1b8c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 -- 172.21.15.29:0/270545659 shutdown_connections 2024-09-21T02:55:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 --2- 172.21.15.29:0/270545659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b90105100 0x7f1b9010baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 --2- 172.21.15.29:0/270545659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b90104760 0x7f1b90104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 --2- 172.21.15.29:0/270545659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 0x7f1b90103b50 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 -- 172.21.15.29:0/270545659 >> 172.21.15.29:0/270545659 conn(0x7f1b900ff020 msgr2=0x7f1b901014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 -- 172.21.15.29:0/270545659 shutdown_connections 2024-09-21T02:55:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 -- 172.21.15.29:0/270545659 wait complete. 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 Processor -- start 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.404+0000 7f1b981ed640 1 -- start start 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b981ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 0x7f1b90199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b981ed640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b90104760 0x7f1b9019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b981ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b90105100 0x7f1b9019b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b981ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9010f160 con 0x7f1b901036e0 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b981ed640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9010efe0 con 0x7f1b90104760 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b981ed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9010f2e0 con 0x7f1b90105100 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95f89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 0x7f1b90199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95f89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 0x7f1b90199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53208/0 (socket says 172.21.15.29:53208) 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95f89640 1 -- 172.21.15.29:0/555528423 learned_addr learned my addr 172.21.15.29:0/555528423 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95788640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b90104760 0x7f1b9019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b9678a640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b90105100 0x7f1b9019b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95f89640 1 -- 172.21.15.29:0/555528423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b90105100 msgr2=0x7f1b9019b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95f89640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b90105100 0x7f1b9019b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95f89640 1 -- 172.21.15.29:0/555528423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b90104760 msgr2=0x7f1b9019a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95f89640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b90104760 0x7f1b9019a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.405+0000 7f1b95f89640 1 -- 172.21.15.29:0/555528423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9019b8e0 con 0x7f1b901036e0 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.406+0000 7f1b95f89640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 0x7f1b90199ea0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c02ade0 tx=0x7f1b8c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.406+0000 7f1b86ffd640 1 -- 172.21.15.29:0/555528423 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c004040 con 0x7f1b901036e0 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.406+0000 7f1b86ffd640 1 -- 172.21.15.29:0/555528423 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b8c004670 con 0x7f1b901036e0 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.406+0000 7f1b981ed640 1 -- 172.21.15.29:0/555528423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b9019bb70 con 0x7f1b901036e0 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.406+0000 7f1b86ffd640 1 -- 172.21.15.29:0/555528423 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c030d50 con 0x7f1b901036e0 2024-09-21T02:55:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.406+0000 7f1b981ed640 1 -- 172.21.15.29:0/555528423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b901a22c0 con 0x7f1b901036e0 2024-09-21T02:55:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.408+0000 7f1b9f236740 1 Processor -- start 2024-09-21T02:55:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9f236740 1 -- start start 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b86ffd640 1 -- 172.21.15.29:0/555528423 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f1b8c0305c0 con 0x7f1b901036e0 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae8c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 0x55a2eae8c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaabf4d0 0x55a2eae8d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9f236740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2eadfa6f0 con 0x55a2eaae01b0 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9f236740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a2eade9b20 con 0x55a2eaabf4d0 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9f236740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a2eaded930 con 0x55a2eaab72f0 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9678a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae8c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9678a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae8c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53210/0 (socket says 172.21.15.29:53210) 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9678a640 1 -- 172.21.15.29:0/2672507364 learned_addr learned my addr 172.21.15.29:0/2672507364 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b95f89640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 0x55a2eae8c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b95788640 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaabf4d0 0x55a2eae8d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.409+0000 7f1b9678a640 1 -- 172.21.15.29:0/2672507364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 msgr2=0x55a2eae8c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b9678a640 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 0x55a2eae8c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b9678a640 1 -- 172.21.15.29:0/2672507364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaabf4d0 msgr2=0x55a2eae8d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b9678a640 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaabf4d0 0x55a2eae8d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b9678a640 1 -- 172.21.15.29:0/2672507364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2eadaf530 con 0x55a2eaae01b0 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b95788640 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaabf4d0 0x55a2eae8d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b9678a640 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae8c260 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800b700 tx=0x7f1b8800bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4330b30257af1018 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b86ffd640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b6510 0x7f1b7c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b7bfff640 1 -- 172.21.15.29:0/2672507364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b88003dd0 con 0x55a2eaae01b0 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b95f89640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b6510 0x7f1b7c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b7bfff640 1 -- 172.21.15.29:0/2672507364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b88015070 con 0x55a2eaae01b0 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b7bfff640 1 -- 172.21.15.29:0/2672507364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b88011e60 con 0x55a2eaae01b0 2024-09-21T02:55:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.410+0000 7f1b86ffd640 1 -- 172.21.15.29:0/555528423 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1b8c108aa0 con 0x7f1b901036e0 2024-09-21T02:55:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 -- 172.21.15.29:0/2672507364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 msgr2=0x55a2eae8c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b95f89640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b6510 0x7f1b7c0b8a20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0049d0 tx=0x7f1b8c10c580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae8c260 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800b700 tx=0x7f1b8800bbc0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 -- 172.21.15.29:0/2672507364 shutdown_connections 2024-09-21T02:55:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaabf4d0 0x55a2eae8d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 0x55a2eae8c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 --2- 172.21.15.29:0/2672507364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae8c260 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 -- 172.21.15.29:0/2672507364 >> 172.21.15.29:0/2672507364 conn(0x55a2eaad4300 msgr2=0x55a2eae838e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 -- 172.21.15.29:0/2672507364 shutdown_connections 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.411+0000 7f1b9f236740 1 -- 172.21.15.29:0/2672507364 wait complete. 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 1 -- start start 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 1 Processor -- start 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 0x55a2eae95a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae96110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaae01b0 0x55a2eae96c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2ead21ab0 con 0x55a2eaabf4d0 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a2ead8f730 con 0x55a2eaae01b0 2024-09-21T02:55:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a2eae8df40 con 0x55a2eaab72f0 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9f236740 10 client.0 _ll_register_callbacks cb 0x55a2eade36b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b95788640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae96110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b95788640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae96110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53224/0 (socket says 172.21.15.29:53224) 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b95788640 1 -- 172.21.15.29:0/1186673446 learned_addr learned my addr 172.21.15.29:0/1186673446 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9678a640 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 0x55a2eae95a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.412+0000 7f1b9678a640 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaae01b0 0x55a2eae96c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b95788640 1 -- 172.21.15.29:0/1186673446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 msgr2=0x55a2eae95a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b95788640 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 0x55a2eae95a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b95788640 1 -- 172.21.15.29:0/1186673446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaae01b0 msgr2=0x55a2eae96c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b95788640 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaae01b0 0x55a2eae96c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b95788640 1 -- 172.21.15.29:0/1186673446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2eab4b790 con 0x55a2eaabf4d0 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b95788640 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae96110 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f1b80012700 tx=0x7f1b80012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65af1498fc27efc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b797fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:55:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b797fa640 1 -- 172.21.15.29:0/1186673446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b80004050 con 0x55a2eaabf4d0 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b797fa640 1 -- 172.21.15.29:0/1186673446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1b800041f0 con 0x55a2eaabf4d0 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b9f236740 1 -- 172.21.15.29:0/1186673446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b88015070 con 0x55a2eaabf4d0 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b797fa640 1 -- 172.21.15.29:0/1186673446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1b800043f0 con 0x55a2eaabf4d0 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b9f236740 10 client.17661 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.413+0000 7f1b9f236740 1 -- 172.21.15.29:0/1186673446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a2eabc4620 con 0x55a2eaabf4d0 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b797fa640 1 -- 172.21.15.29:0/1186673446 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b88015070 con 0x55a2eaabf4d0 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b797fa640 1 -- 172.21.15.29:0/1186673446 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f1b80003ab0 con 0x55a2eaabf4d0 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b9f236740 10 client.17661 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b9f236740 10 client.17661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b9f236740 20 client.17661 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b9f236740 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2eaa9ed80 0x55a2eaea3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b9f236740 4 client.17661 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:55:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b9f236740 1 -- 172.21.15.29:0/1186673446 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2eadfc640 con 0x55a2eaa9ed80 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.414+0000 7f1b9678a640 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2eaa9ed80 0x55a2eaea3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.415+0000 7f1b797fa640 10 client.17661 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.415+0000 7f1b9678a640 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2eaa9ed80 0x55a2eaea3200 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f512b5f76a8b978 server_cookie=263d8a478b8d39b1 in_seq=0 out_seq=0 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.415+0000 7f1b797fa640 1 -- 172.21.15.29:0/1186673446 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a2eadfc640 con 0x55a2eaa9ed80 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.415+0000 7f1b797fa640 10 client.17661 handle_command_reply: tid=0 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 client.17661 shutdown 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 -- 172.21.15.29:0/1186673446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 msgr2=0x55a2eae96110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae96110 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f1b80012700 tx=0x7f1b80012bc0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 -- 172.21.15.29:0/1186673446 shutdown_connections 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaae01b0 0x55a2eae96c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2eaa9ed80 0x55a2eaea3200 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae96110 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 --2- 172.21.15.29:0/1186673446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaab72f0 0x55a2eae95a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 -- 172.21.15.29:0/1186673446 >> 172.21.15.29:0/1186673446 conn(0x55a2eaad4300 msgr2=0x55a2eae81ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b797fa640 0 client.17661 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 -- 172.21.15.29:0/1186673446 shutdown_connections 2024-09-21T02:55:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 1 -- 172.21.15.29:0/1186673446 wait complete. 2024-09-21T02:55:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.416+0000 7f1b9f236740 20 client.17661 trim_cache size 0 max 16384 2024-09-21T02:55:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9f236740 1 Processor -- start 2024-09-21T02:55:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9f236740 1 -- start start 2024-09-21T02:55:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 0x55a2eae95a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaabf4d0 0x55a2eae95f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae93f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9f236740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1b901aa3d0 con 0x55a2eaae01b0 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9f236740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1b901abf10 con 0x55a2eaab72f0 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9f236740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1b901aaaa0 con 0x55a2eaabf4d0 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9678a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae93f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9678a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae93f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53234/0 (socket says 172.21.15.29:53234) 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b9678a640 1 -- 172.21.15.29:0/4160743696 learned_addr learned my addr 172.21.15.29:0/4160743696 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.431+0000 7f1b95788640 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaabf4d0 0x55a2eae95f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 0x55a2eae95a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b9678a640 1 -- 172.21.15.29:0/4160743696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaabf4d0 msgr2=0x55a2eae95f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaabf4d0 0x55a2eae95f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b9678a640 1 -- 172.21.15.29:0/4160743696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 msgr2=0x55a2eae95a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 0x55a2eae95a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b9678a640 1 -- 172.21.15.29:0/4160743696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2eae9df20 con 0x55a2eaae01b0 2024-09-21T02:55:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae93f60 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800b700 tx=0x7f1b88000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df436742077c8be server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b7b7fe640 1 -- 172.21.15.29:0/4160743696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b88023040 con 0x55a2eaae01b0 2024-09-21T02:55:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b7b7fe640 1 -- 172.21.15.29:0/4160743696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b88015070 con 0x55a2eaae01b0 2024-09-21T02:55:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.432+0000 7f1b7b7fe640 1 -- 172.21.15.29:0/4160743696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b88011a80 con 0x55a2eaae01b0 2024-09-21T02:55:22.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 -- 172.21.15.29:0/4160743696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 msgr2=0x55a2eae93f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae93f60 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800b700 tx=0x7f1b88000bc0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 -- 172.21.15.29:0/4160743696 shutdown_connections 2024-09-21T02:55:22.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaae01b0 0x55a2eae93f60 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaabf4d0 0x55a2eae95f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4160743696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 0x55a2eae95a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 -- 172.21.15.29:0/4160743696 >> 172.21.15.29:0/4160743696 conn(0x55a2eaad4300 msgr2=0x55a2eae83220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 -- 172.21.15.29:0/4160743696 shutdown_connections 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 -- 172.21.15.29:0/4160743696 wait complete. 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 -- start start 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.433+0000 7f1b9f236740 1 Processor -- start 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 0x55a2eae9d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae9c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9f236740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaae01b0 0x55a2eae9c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9f236740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1b901ab840 con 0x55a2eaabf4d0 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9f236740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1b901ab170 con 0x55a2eaab72f0 2024-09-21T02:55:22.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9f236740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1b901a83b0 con 0x55a2eaae01b0 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9f236740 10 client.0 _ll_register_callbacks cb 0x7f1b800097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae9c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae9c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53242/0 (socket says 172.21.15.29:53242) 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 -- 172.21.15.29:0/4074524302 learned_addr learned my addr 172.21.15.29:0/4074524302 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaae01b0 0x55a2eae9c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 0x55a2eae9d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 -- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaae01b0 msgr2=0x55a2eae9c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaae01b0 0x55a2eae9c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 -- 172.21.15.29:0/4074524302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 msgr2=0x55a2eae9d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 0x55a2eae9d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 -- 172.21.15.29:0/4074524302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2eae9cde0 con 0x55a2eaabf4d0 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaae01b0 0x55a2eae9c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b95788640 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae9c230 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1b800126d0 tx=0x7f1b80004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40c898196005d85 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:22.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b797fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b797fa640 1 -- 172.21.15.29:0/4074524302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8000fb30 con 0x55a2eaabf4d0 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b797fa640 1 -- 172.21.15.29:0/4074524302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1b8000fcd0 con 0x55a2eaabf4d0 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.434+0000 7f1b9f236740 1 -- 172.21.15.29:0/4074524302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b88015070 con 0x55a2eaabf4d0 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b797fa640 1 -- 172.21.15.29:0/4074524302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1b8003a050 con 0x55a2eaabf4d0 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9f236740 10 client.17667 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9f236740 1 -- 172.21.15.29:0/4074524302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a2eaf0dbc0 con 0x55a2eaabf4d0 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b797fa640 1 -- 172.21.15.29:0/4074524302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b88015070 con 0x55a2eaabf4d0 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b797fa640 1 -- 172.21.15.29:0/4074524302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f1b80003820 con 0x55a2eaabf4d0 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9f236740 10 client.17667 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9f236740 10 client.17667 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9f236740 20 client.17667 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2eaa741c0 0x55a2eaf10440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9f236740 4 client.17667 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:55:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9f236740 1 -- 172.21.15.29:0/4074524302 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a2eadf80f0 con 0x55a2eaa741c0 2024-09-21T02:55:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.435+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2eaa741c0 0x55a2eaf10440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.436+0000 7f1b9678a640 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2eaa741c0 0x55a2eaf10440 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fbc9cffb0ded165 server_cookie=69cb74c731934b49 in_seq=0 out_seq=0 2024-09-21T02:55:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.436+0000 7f1b797fa640 10 client.17667 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:22.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.463+0000 7f1b797fa640 1 -- 172.21.15.29:0/4074524302 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a2eadf80f0 con 0x55a2eaa741c0 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.463+0000 7f1b797fa640 10 client.17667 handle_command_reply: tid=0 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.463+0000 7f1b9f236740 1 client.17667 shutdown 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.463+0000 7f1b9f236740 1 -- 172.21.15.29:0/4074524302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 msgr2=0x55a2eae9c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.463+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae9c230 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1b800126d0 tx=0x7f1b80004040 comp rx=0 tx=0).stop 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 1 -- 172.21.15.29:0/4074524302 shutdown_connections 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2eaae01b0 0x55a2eae9c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a2eaa741c0 0x55a2eaf10440 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2eaabf4d0 0x55a2eae9c230 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 1 --2- 172.21.15.29:0/4074524302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a2eaab72f0 0x55a2eae9d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b797fa640 0 client.17667 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 1 -- 172.21.15.29:0/4074524302 >> 172.21.15.29:0/4074524302 conn(0x55a2eaad4300 msgr2=0x55a2eae90370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 1 -- 172.21.15.29:0/4074524302 shutdown_connections 2024-09-21T02:55:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 1 -- 172.21.15.29:0/4074524302 wait complete. 2024-09-21T02:55:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b9f236740 20 client.17667 trim_cache size 0 max 16384 2024-09-21T02:55:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:55:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:55:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:55:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 -- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b6510 msgr2=0x7f1b7c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b6510 0x7f1b7c0b8a20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0049d0 tx=0x7f1b8c10c580 comp rx=0 tx=0).stop 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 -- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 msgr2=0x7f1b90199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 0x7f1b90199ea0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c02ade0 tx=0x7f1b8c003ba0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 -- 172.21.15.29:0/555528423 shutdown_connections 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1b7c0b6510 0x7f1b7c0b8a20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b90105100 0x7f1b9019b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b90104760 0x7f1b9019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 --2- 172.21.15.29:0/555528423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b901036e0 0x7f1b90199ea0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.464+0000 7f1b981ed640 1 -- 172.21.15.29:0/555528423 >> 172.21.15.29:0/555528423 conn(0x7f1b900ff020 msgr2=0x7f1b90107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.465+0000 7f1b981ed640 1 -- 172.21.15.29:0/555528423 shutdown_connections 2024-09-21T02:55:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:22.465+0000 7f1b981ed640 1 -- 172.21.15.29:0/555528423 wait complete. 2024-09-21T02:55:22.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.21 MB/sec execute 59 sec latency 15560.415 ms 2024-09-21T02:55:23.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.12 MB/sec execute 60 sec latency 16560.519 ms 2024-09-21T02:55:23.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:23 smithi066 ceph-mon[33436]: pgmap v877: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 397 KiB/s wr, 89 op/s 2024-09-21T02:55:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:23 smithi190 ceph-mon[33871]: pgmap v877: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 397 KiB/s wr, 89 op/s 2024-09-21T02:55:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:23 smithi029 ceph-mon[31006]: pgmap v877: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 397 KiB/s wr, 89 op/s 2024-09-21T02:55:24.717 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 5.04 MB/sec execute 61 sec latency 17560.668 ms 2024-09-21T02:55:25.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 4.96 MB/sec execute 62 sec latency 18560.794 ms 2024-09-21T02:55:25.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:25 smithi066 ceph-mon[33436]: pgmap v878: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 175 KiB/s wr, 38 op/s 2024-09-21T02:55:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:25 smithi190 ceph-mon[33871]: pgmap v878: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 175 KiB/s wr, 38 op/s 2024-09-21T02:55:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:25 smithi029 ceph-mon[31006]: pgmap v878: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 175 KiB/s wr, 38 op/s 2024-09-21T02:55:26.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 4.88 MB/sec execute 63 sec latency 19560.926 ms 2024-09-21T02:55:27.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:55:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:55:27.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 4.80 MB/sec execute 64 sec latency 20561.049 ms 2024-09-21T02:55:27.733 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:27 smithi029 ceph-mon[31006]: pgmap v879: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 20 op/s 2024-09-21T02:55:27.762 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:27 smithi066 ceph-mon[33436]: pgmap v879: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 20 op/s 2024-09-21T02:55:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 -- 172.21.15.29:0/3924977382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 msgr2=0x7f02a01098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 --2- 172.21.15.29:0/3924977382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 0x7f02a01098d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0294009900 tx=0x7f029402a410 comp rx=0 tx=0).stop 2024-09-21T02:55:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 -- 172.21.15.29:0/3924977382 shutdown_connections 2024-09-21T02:55:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 --2- 172.21.15.29:0/3924977382 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02a0109e10 0x7f02a0110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 --2- 172.21.15.29:0/3924977382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 0x7f02a01098d0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 --2- 172.21.15.29:0/3924977382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02a0103b80 0x7f02a0103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 -- 172.21.15.29:0/3924977382 >> 172.21.15.29:0/3924977382 conn(0x7f02a00ff610 msgr2=0x7f02a0101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 -- 172.21.15.29:0/3924977382 shutdown_connections 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.824+0000 7f02a7481640 1 -- 172.21.15.29:0/3924977382 wait complete. 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a7481640 1 Processor -- start 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a7481640 1 -- start start 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a7481640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02a0103b80 0x7f02a019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a7481640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 0x7f02a019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a7481640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02a0109e10 0x7f02a019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a7481640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0113980 con 0x7f02a01094c0 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a7481640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0113800 con 0x7f02a0103b80 2024-09-21T02:55:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a7481640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0113b00 con 0x7f02a0109e10 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a4a1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 0x7f02a019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a4a1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 0x7f02a019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43764/0 (socket says 172.21.15.29:43764) 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a4a1c640 1 -- 172.21.15.29:0/48682307 learned_addr learned my addr 172.21.15.29:0/48682307 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.825+0000 7f02a521d640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02a0103b80 0x7f02a019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a5a1e640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02a0109e10 0x7f02a019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a4a1c640 1 -- 172.21.15.29:0/48682307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02a0109e10 msgr2=0x7f02a019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a4a1c640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02a0109e10 0x7f02a019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a4a1c640 1 -- 172.21.15.29:0/48682307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02a0103b80 msgr2=0x7f02a019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a4a1c640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02a0103b80 0x7f02a019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a4a1c640 1 -- 172.21.15.29:0/48682307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a01a0280 con 0x7f02a01094c0 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a5a1e640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02a0109e10 0x7f02a019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a4a1c640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 0x7f02a019a970 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0294004770 tx=0x7f0294004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f029e7fc640 1 -- 172.21.15.29:0/48682307 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029403f030 con 0x7f02a01094c0 2024-09-21T02:55:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a01a0510 con 0x7f02a01094c0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f029e7fc640 1 -- 172.21.15.29:0/48682307 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0294004440 con 0x7f02a01094c0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f029e7fc640 1 -- 172.21.15.29:0/48682307 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0294035070 con 0x7f02a01094c0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.826+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a01a0a40 con 0x7f02a01094c0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.827+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a0104e00 con 0x7f02a01094c0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.829+0000 7f029e7fc640 1 -- 172.21.15.29:0/48682307 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f0294042030 con 0x7f02a01094c0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.829+0000 7f029e7fc640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02880b6540 0x7f02880b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.830+0000 7f02a521d640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02880b6540 0x7f02880b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.830+0000 7f029e7fc640 1 -- 172.21.15.29:0/48682307 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0294107a10 con 0x7f02a01094c0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.830+0000 7f02a521d640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02880b6540 0x7f02880b8a50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0290007ec0 tx=0x7f02900075b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:27.834+0000 7f029e7fc640 1 -- 172.21.15.29:0/48682307 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f02a01a0a40 con 0x7f02a01094c0 2024-09-21T02:55:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:27 smithi190 ceph-mon[33871]: pgmap v879: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 20 op/s 2024-09-21T02:55:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.061+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02a0103f90 con 0x7f02a01094c0 2024-09-21T02:55:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.062+0000 7f029e7fc640 1 -- 172.21.15.29:0/48682307 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f02940cdf20 con 0x7f02a01094c0 2024-09-21T02:55:28.064 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:55:28.064 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:55:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02880b6540 msgr2=0x7f02880b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02880b6540 0x7f02880b8a50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0290007ec0 tx=0x7f02900075b0 comp rx=0 tx=0).stop 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 msgr2=0x7f02a019a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 0x7f02a019a970 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0294004770 tx=0x7f0294004040 comp rx=0 tx=0).stop 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 shutdown_connections 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02880b6540 0x7f02880b8a50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02a0109e10 0x7f02a019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02a01094c0 0x7f02a019a970 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 --2- 172.21.15.29:0/48682307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02a0103b80 0x7f02a019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.064+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 >> 172.21.15.29:0/48682307 conn(0x7f02a00ff610 msgr2=0x7f02a0101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.065+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 shutdown_connections 2024-09-21T02:55:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:28.065+0000 7f02a7481640 1 -- 172.21.15.29:0/48682307 wait complete. 2024-09-21T02:55:28.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18863 4.73 MB/sec execute 65 sec latency 21561.426 ms 2024-09-21T02:55:28.788 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:28 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/48682307' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:55:28.819 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id adcecebd --release --await' 2024-09-21T02:55:28.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id adcecebd --release --await 2024-09-21T02:55:28.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:28 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/48682307' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:55:29.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:28 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/48682307' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:55:29.084 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:55:29.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 -- 172.21.15.29:0/2569016174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 msgr2=0x7feba410f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 --2- 172.21.15.29:0/2569016174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 0x7feba410f5a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009ca0 tx=0x7feb9c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 -- 172.21.15.29:0/2569016174 shutdown_connections 2024-09-21T02:55:29.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 --2- 172.21.15.29:0/2569016174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 0x7feba410f5a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 --2- 172.21.15.29:0/2569016174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba4108420 0x7feba4108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 --2- 172.21.15.29:0/2569016174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feba4102ae0 0x7feba4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 -- 172.21.15.29:0/2569016174 >> 172.21.15.29:0/2569016174 conn(0x7feba4079770 msgr2=0x7feba4079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 -- 172.21.15.29:0/2569016174 shutdown_connections 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.605+0000 7febaa3f1640 1 -- 172.21.15.29:0/2569016174 wait complete. 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.606+0000 7febaa3f1640 1 Processor -- start 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.606+0000 7febaa3f1640 1 -- start start 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.606+0000 7febaa3f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba4102ae0 0x7feba4199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7febaa3f1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feba4108420 0x7feba41998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7febaa3f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 0x7feba419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7febaa3f1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feba4112930 con 0x7feba4108d70 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7febaa3f1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feba41127b0 con 0x7feba4108420 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7febaa3f1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feba4112ab0 con 0x7feba4102ae0 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 0x7feba419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 0x7feba419eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43768/0 (socket says 172.21.15.29:43768) 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 -- 172.21.15.29:0/3366960165 learned_addr learned my addr 172.21.15.29:0/3366960165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba3fff640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba4102ae0 0x7feba4199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba37fe640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feba4108420 0x7feba41998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 -- 172.21.15.29:0/3366960165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba4102ae0 msgr2=0x7feba4199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba4102ae0 0x7feba4199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 -- 172.21.15.29:0/3366960165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feba4108420 msgr2=0x7feba41998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feba4108420 0x7feba41998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 -- 172.21.15.29:0/3366960165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba419f270 con 0x7feba4108d70 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba37fe640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feba4108420 0x7feba41998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba898e640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 0x7feba419eb40 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7feb9c005e30 tx=0x7feb9c0093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba17fa640 1 -- 172.21.15.29:0/3366960165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c032d10 con 0x7feba4108d70 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7febaa3f1640 1 -- 172.21.15.29:0/3366960165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba419f500 con 0x7feba4108d70 2024-09-21T02:55:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.607+0000 7feba17fa640 1 -- 172.21.15.29:0/3366960165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb9c032eb0 con 0x7feba4108d70 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.608+0000 7feba17fa640 1 -- 172.21.15.29:0/3366960165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c030b20 con 0x7feba4108d70 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.608+0000 7febaa3f1640 1 -- 172.21.15.29:0/3366960165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba41a5cc0 con 0x7feba4108d70 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.610+0000 7feba17fa640 1 -- 172.21.15.29:0/3366960165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7feb9c030ce0 con 0x7feba4108d70 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.610+0000 7febb0965740 1 Processor -- start 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.610+0000 7febb0965740 1 -- start start 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7febb0965740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c35f520 0x55b52c705510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7febb0965740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c336660 0x55b52c705a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7febb0965740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c705f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7febb0965740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b52c5f0dc0 con 0x55b52c33e840 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7febb0965740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b52c67d1f0 con 0x55b52c35f520 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7febb0965740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b52c66c620 con 0x55b52c336660 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c705f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c336660 0x55b52c705a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c705f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43776/0 (socket says 172.21.15.29:43776) 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba37fe640 1 -- 172.21.15.29:0/2892508437 learned_addr learned my addr 172.21.15.29:0/2892508437 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba17fa640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feb8c0b6510 0x7feb8c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba37fe640 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c35f520 0x55b52c705510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba3fff640 1 -- 172.21.15.29:0/2892508437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c35f520 msgr2=0x55b52c705510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba3fff640 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c35f520 0x55b52c705510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba3fff640 1 -- 172.21.15.29:0/2892508437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 msgr2=0x55b52c705f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba3fff640 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c705f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba3fff640 1 -- 172.21.15.29:0/2892508437 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b52c634fd0 con 0x55b52c336660 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba37fe640 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c705f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba898e640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feb8c0b6510 0x7feb8c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba17fa640 1 -- 172.21.15.29:0/3366960165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7feb9c1088c0 con 0x7feba4108d70 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.611+0000 7feba3fff640 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c336660 0x55b52c705a50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7feb94005fd0 tx=0x7feb940107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e97afd13ed122569 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.612+0000 7feba898e640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feb8c0b6510 0x7feb8c0b8a20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0cb940 tx=0x7feb9c0c2f80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.612+0000 7feb8a7fc640 1 -- 172.21.15.29:0/2892508437 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94015a50 con 0x55b52c336660 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.612+0000 7feb8a7fc640 1 -- 172.21.15.29:0/2892508437 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb9400be20 con 0x55b52c336660 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.612+0000 7feb8a7fc640 1 -- 172.21.15.29:0/2892508437 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9401e380 con 0x55b52c336660 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 -- 172.21.15.29:0/2892508437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c336660 msgr2=0x55b52c705a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c336660 0x55b52c705a50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7feb94005fd0 tx=0x7feb940107a0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 -- 172.21.15.29:0/2892508437 shutdown_connections 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c705f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c336660 0x55b52c705a50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 --2- 172.21.15.29:0/2892508437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c35f520 0x55b52c705510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 -- 172.21.15.29:0/2892508437 >> 172.21.15.29:0/2892508437 conn(0x55b52c353910 msgr2=0x55b52c703400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 -- 172.21.15.29:0/2892508437 shutdown_connections 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 -- 172.21.15.29:0/2892508437 wait complete. 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 -- start start 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.613+0000 7febb0965740 1 Processor -- start 2024-09-21T02:55:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7febb0965740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 0x55b52c710ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7febb0965740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c33e840 0x55b52c711400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7febb0965740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c35f520 0x55b52c71cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7febb0965740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b52c607300 con 0x55b52c35f520 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7febb0965740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b52c670430 con 0x55b52c336660 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7febb0965740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b52c4cdbb0 con 0x55b52c33e840 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7febb0965740 10 client.0 _ll_register_callbacks cb 0x55b52c621120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c35f520 0x55b52c71cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c33e840 0x55b52c711400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c35f520 0x55b52c71cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43778/0 (socket says 172.21.15.29:43778) 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 -- 172.21.15.29:0/481940443 learned_addr learned my addr 172.21.15.29:0/481940443 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 0x55b52c710ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 -- 172.21.15.29:0/481940443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c33e840 msgr2=0x55b52c711400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c33e840 0x55b52c711400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 -- 172.21.15.29:0/481940443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 msgr2=0x55b52c710ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 0x55b52c710ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 -- 172.21.15.29:0/481940443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b52c5f6780 con 0x55b52c35f520 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feba37fe640 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c35f520 0x55b52c71cc50 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7feb98009aa0 tx=0x7feb98015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf5ac91ec534898 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feb837fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feb837fe640 1 -- 172.21.15.29:0/481940443 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98005020 con 0x55b52c35f520 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7feb837fe640 1 -- 172.21.15.29:0/481940443 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7feb980051c0 con 0x55b52c35f520 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.614+0000 7febb0965740 1 -- 172.21.15.29:0/481940443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb9400be20 con 0x55b52c35f520 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7feb837fe640 1 -- 172.21.15.29:0/481940443 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7feb98029050 con 0x55b52c35f520 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7febb0965740 10 client.17685 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7febb0965740 1 -- 172.21.15.29:0/481940443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b52c3cab00 con 0x55b52c35f520 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7feb837fe640 1 -- 172.21.15.29:0/481940443 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb9400be20 con 0x55b52c35f520 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7feb837fe640 1 -- 172.21.15.29:0/481940443 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7feb9801e400 con 0x55b52c35f520 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7febb0965740 10 client.17685 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7febb0965740 10 client.17685 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7febb0965740 20 client.17685 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7febb0965740 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b52c31dea0 0x55b52c722e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7febb0965740 4 client.17685 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7febb0965740 1 -- 172.21.15.29:0/481940443 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b52c67f140 con 0x55b52c31dea0 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.615+0000 7feba3fff640 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b52c31dea0 0x55b52c722e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.616+0000 7feb837fe640 10 client.17685 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.616+0000 7feba3fff640 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b52c31dea0 0x55b52c722e90 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf040b61c9014174 server_cookie=42bc066b01a3e6fb in_seq=0 out_seq=0 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7feb837fe640 1 -- 172.21.15.29:0/481940443 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b52c67f140 con 0x55b52c31dea0 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7feb837fe640 10 client.17685 handle_command_reply: tid=0 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 client.17685 shutdown 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 -- 172.21.15.29:0/481940443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c35f520 msgr2=0x55b52c71cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c35f520 0x55b52c71cc50 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7feb98009aa0 tx=0x7feb98015040 comp rx=0 tx=0).stop 2024-09-21T02:55:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 -- 172.21.15.29:0/481940443 shutdown_connections 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b52c31dea0 0x55b52c722e90 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c35f520 0x55b52c71cc50 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c33e840 0x55b52c711400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 --2- 172.21.15.29:0/481940443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 0x55b52c710ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7feb837fe640 0 client.17685 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 -- 172.21.15.29:0/481940443 >> 172.21.15.29:0/481940443 conn(0x55b52c353910 msgr2=0x55b52c706f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 -- 172.21.15.29:0/481940443 shutdown_connections 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 1 -- 172.21.15.29:0/481940443 wait complete. 2024-09-21T02:55:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.617+0000 7febb0965740 20 client.17685 trim_cache size 0 max 16384 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7febb0965740 1 Processor -- start 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7febb0965740 1 -- start start 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7febb0965740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 0x55b52c713e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7febb0965740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c70e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7febb0965740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 0x55b52c70e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7febb0965740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feba41ae750 con 0x55b52c33e840 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7febb0965740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feba41b0290 con 0x55b52c336660 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7febb0965740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feba41aee20 con 0x55b52c35f520 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c70e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c70e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43788/0 (socket says 172.21.15.29:43788) 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba3fff640 1 -- 172.21.15.29:0/2482016130 learned_addr learned my addr 172.21.15.29:0/2482016130 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba37fe640 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 0x55b52c70e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba37fe640 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 0x55b52c713e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba3fff640 1 -- 172.21.15.29:0/2482016130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 msgr2=0x55b52c70e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba3fff640 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 0x55b52c70e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba3fff640 1 -- 172.21.15.29:0/2482016130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 msgr2=0x55b52c713e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba3fff640 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 0x55b52c713e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.635+0000 7feba3fff640 1 -- 172.21.15.29:0/2482016130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b52c441960 con 0x55b52c33e840 2024-09-21T02:55:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.636+0000 7feba3fff640 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c70e0b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7feb94005d00 tx=0x7feb94022040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625469001e23071b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.636+0000 7feb89ffb640 1 -- 172.21.15.29:0/2482016130 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94038070 con 0x55b52c33e840 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.636+0000 7feb89ffb640 1 -- 172.21.15.29:0/2482016130 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb94015ce0 con 0x55b52c33e840 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.636+0000 7feb89ffb640 1 -- 172.21.15.29:0/2482016130 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb940054a0 con 0x55b52c33e840 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 -- 172.21.15.29:0/2482016130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 msgr2=0x55b52c70e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c70e0b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7feb94005d00 tx=0x7feb94022040 comp rx=0 tx=0).stop 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 -- 172.21.15.29:0/2482016130 shutdown_connections 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 0x55b52c70e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c33e840 0x55b52c70e0b0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 --2- 172.21.15.29:0/2482016130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c336660 0x55b52c713e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 -- 172.21.15.29:0/2482016130 >> 172.21.15.29:0/2482016130 conn(0x55b52c353910 msgr2=0x55b52c7112e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 -- 172.21.15.29:0/2482016130 shutdown_connections 2024-09-21T02:55:29.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 -- 172.21.15.29:0/2482016130 wait complete. 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 -- start start 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.637+0000 7febb0965740 1 Processor -- start 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7febb0965740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c336660 0x55b52c71af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7febb0965740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c33e840 0x55b52c71b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7febb0965740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 0x55b52c715c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7febb0965740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feba41afbc0 con 0x55b52c336660 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7febb0965740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feba41af4f0 con 0x55b52c33e840 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7febb0965740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feba41ac730 con 0x55b52c35f520 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7febb0965740 10 client.0 _ll_register_callbacks cb 0x7feb98013620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c336660 0x55b52c71af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c336660 0x55b52c71af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43800/0 (socket says 172.21.15.29:43800) 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 -- 172.21.15.29:0/2294130042 learned_addr learned my addr 172.21.15.29:0/2294130042 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:29.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba3fff640 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c33e840 0x55b52c71b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 -- 172.21.15.29:0/2294130042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 msgr2=0x55b52c715c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 0x55b52c715c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 -- 172.21.15.29:0/2294130042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c33e840 msgr2=0x55b52c71b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c33e840 0x55b52c71b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 -- 172.21.15.29:0/2294130042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b52c716420 con 0x55b52c336660 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba3fff640 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c33e840 0x55b52c71b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feba37fe640 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c336660 0x55b52c71af80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7feb9801e990 tx=0x7feb980118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab60bc078a30eda6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.638+0000 7feb837fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7feb837fe640 1 -- 172.21.15.29:0/2294130042 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9801ebc0 con 0x55b52c336660 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7feb837fe640 1 -- 172.21.15.29:0/2294130042 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7feb9801ed60 con 0x55b52c336660 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 1 -- 172.21.15.29:0/2294130042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb94015ce0 con 0x55b52c336660 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7feb837fe640 1 -- 172.21.15.29:0/2294130042 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7feb98041050 con 0x55b52c336660 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 10 client.17691 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:29.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 1 -- 172.21.15.29:0/2294130042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b52c78e310 con 0x55b52c336660 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7feb837fe640 1 -- 172.21.15.29:0/2294130042 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb94015ce0 con 0x55b52c336660 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7feb837fe640 1 -- 172.21.15.29:0/2294130042 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7feb98005300 con 0x55b52c336660 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 10 client.17691 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 10 client.17691 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 20 client.17691 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b52c2f3250 0x55b52c790a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 4 client.17691 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "adcecebd", "release": true, "await": true}] 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.639+0000 7febb0965740 1 -- 172.21.15.29:0/2294130042 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "adcecebd", "release": true, "await": true}) -- 0x55b52c70ae60 con 0x55b52c2f3250 2024-09-21T02:55:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.640+0000 7feba3fff640 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b52c2f3250 0x55b52c790a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:29.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.640+0000 7feba3fff640 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b52c2f3250 0x55b52c790a20 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db3099503a83f418 server_cookie=900a76217d7ba97a in_seq=0 out_seq=0 2024-09-21T02:55:29.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.640+0000 7feb837fe640 10 client.17691 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:29.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.700+0000 7feb837fe640 1 -- 172.21.15.29:0/2294130042 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b52c70ae60 con 0x55b52c2f3250 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.700+0000 7feb837fe640 10 client.17691 handle_command_reply: tid=0 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 client.17691 shutdown 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 -- 172.21.15.29:0/2294130042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c336660 msgr2=0x55b52c71af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c336660 0x55b52c71af80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7feb9801e990 tx=0x7feb980118f0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 -- 172.21.15.29:0/2294130042 shutdown_connections 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b52c35f520 0x55b52c715c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b52c33e840 0x55b52c71b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b52c2f3250 0x55b52c790a20 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 --2- 172.21.15.29:0/2294130042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b52c336660 0x55b52c71af80 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 -- 172.21.15.29:0/2294130042 >> 172.21.15.29:0/2294130042 conn(0x55b52c353910 msgr2=0x55b52c713c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7feb837fe640 0 client.17691 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 -- 172.21.15.29:0/2294130042 shutdown_connections 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 1 -- 172.21.15.29:0/2294130042 wait complete. 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.701+0000 7febb0965740 20 client.17691 trim_cache size 0 max 16384 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 64, 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout: "adcecebd": { 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 64, 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:55:29.704 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 143.2, 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:29.705 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:29.706 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:55:29.706 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:55:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 -- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feb8c0b6510 msgr2=0x7feb8c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feb8c0b6510 0x7feb8c0b8a20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0cb940 tx=0x7feb9c0c2f80 comp rx=0 tx=0).stop 2024-09-21T02:55:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 -- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 msgr2=0x7feba419eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 0x7feba419eb40 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7feb9c005e30 tx=0x7feb9c0093d0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 -- 172.21.15.29:0/3366960165 shutdown_connections 2024-09-21T02:55:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feb8c0b6510 0x7feb8c0b8a20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba4108d70 0x7feba419eb40 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feba4108420 0x7feba41998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 --2- 172.21.15.29:0/3366960165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba4102ae0 0x7feba4199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 -- 172.21.15.29:0/3366960165 >> 172.21.15.29:0/3366960165 conn(0x7feba4079770 msgr2=0x7feba4106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 -- 172.21.15.29:0/3366960165 shutdown_connections 2024-09-21T02:55:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:55:29.702+0000 7febaa3f1640 1 -- 172.21.15.29:0/3366960165 wait complete. 2024-09-21T02:55:29.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18867 4.66 MB/sec execute 66 sec latency 22497.327 ms 2024-09-21T02:55:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:29 smithi190 ceph-mon[33871]: pgmap v880: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-09-21T02:55:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:29 smithi029 ceph-mon[31006]: pgmap v880: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-09-21T02:55:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:29 smithi066 ceph-mon[33436]: pgmap v880: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-09-21T02:55:30.576 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'adcecebd', total seconds elapsed: 26.7 2024-09-21T02:55:30.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19013 4.72 MB/sec execute 67 sec latency 127.390 ms 2024-09-21T02:55:31.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19157 4.77 MB/sec execute 68 sec latency 84.232 ms 2024-09-21T02:55:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:31 smithi190 ceph-mon[33871]: pgmap v881: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-21T02:55:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:31 smithi029 ceph-mon[31006]: pgmap v881: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-21T02:55:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:31 smithi066 ceph-mon[33436]: pgmap v881: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-21T02:55:32.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19357 4.96 MB/sec execute 69 sec latency 116.235 ms 2024-09-21T02:55:33.718 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19577 5.04 MB/sec execute 70 sec latency 109.599 ms 2024-09-21T02:55:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:33 smithi190 ceph-mon[33871]: pgmap v882: 129 pgs: 129 active+clean; 256 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 187 KiB/s wr, 30 op/s 2024-09-21T02:55:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:33 smithi029 ceph-mon[31006]: pgmap v882: 129 pgs: 129 active+clean; 256 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 187 KiB/s wr, 30 op/s 2024-09-21T02:55:34.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:33 smithi066 ceph-mon[33436]: pgmap v882: 129 pgs: 129 active+clean; 256 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 187 KiB/s wr, 30 op/s 2024-09-21T02:55:34.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19768 5.16 MB/sec execute 71 sec latency 120.253 ms 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 -- 172.21.15.29:0/1232156718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22d01025a0 msgr2=0x7f22d0102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/1232156718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22d01025a0 0x7f22d0102a10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009860 tx=0x7f22c002a340 comp rx=0 tx=0).stop 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 -- 172.21.15.29:0/1232156718 shutdown_connections 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/1232156718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d0103fc0 0x7f22d010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/1232156718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22d0103620 0x7f22d0103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/1232156718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22d01025a0 0x7f22d0102a10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 -- 172.21.15.29:0/1232156718 >> 172.21.15.29:0/1232156718 conn(0x7f22d00fdee0 msgr2=0x7f22d0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 -- 172.21.15.29:0/1232156718 shutdown_connections 2024-09-21T02:55:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.603+0000 7f22d4f5f640 1 -- 172.21.15.29:0/1232156718 wait complete. 2024-09-21T02:55:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.604+0000 7f22d4f5f640 1 Processor -- start 2024-09-21T02:55:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22d4f5f640 1 -- start start 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22d4f5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d01025a0 0x7f22d006b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22d4f5f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22d0103620 0x7f22d019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22d4f5f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22d0103fc0 0x7f22d01a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22d4f5f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f22d010e0a0 con 0x7f22d01025a0 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22d4f5f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f22d010df20 con 0x7f22d0103fc0 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22d4f5f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f22d010e220 con 0x7f22d0103620 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22ce59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d01025a0 0x7f22d006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22ce59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d01025a0 0x7f22d006b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56190/0 (socket says 172.21.15.29:56190) 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22ce59c640 1 -- 172.21.15.29:0/2932459294 learned_addr learned my addr 172.21.15.29:0/2932459294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22ced9d640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22d0103fc0 0x7f22d01a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.605+0000 7f22cdd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22d0103620 0x7f22d019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22ced9d640 1 -- 172.21.15.29:0/2932459294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22d0103620 msgr2=0x7f22d019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22ced9d640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22d0103620 0x7f22d019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22ced9d640 1 -- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d01025a0 msgr2=0x7f22d006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22ced9d640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d01025a0 0x7f22d006b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22ced9d640 1 -- 172.21.15.29:0/2932459294 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d01a1950 con 0x7f22d0103fc0 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22ced9d640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22d0103fc0 0x7f22d01a1150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009fa0 tx=0x7f22c800c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22bf7fe640 1 -- 172.21.15.29:0/2932459294 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8004d90 con 0x7f22d0103fc0 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22d4f5f640 1 -- 172.21.15.29:0/2932459294 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d01a1be0 con 0x7f22d0103fc0 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22bf7fe640 1 -- 172.21.15.29:0/2932459294 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22c800fae0 con 0x7f22d0103fc0 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22bf7fe640 1 -- 172.21.15.29:0/2932459294 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8017b40 con 0x7f22d0103fc0 2024-09-21T02:55:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.606+0000 7f22d4f5f640 1 -- 172.21.15.29:0/2932459294 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d01a2130 con 0x7f22d0103fc0 2024-09-21T02:55:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.609+0000 7f22bf7fe640 1 -- 172.21.15.29:0/2932459294 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f22c8017ce0 con 0x7f22d0103fc0 2024-09-21T02:55:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.610+0000 7f22bf7fe640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22b80b6510 0x7f22b80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.610+0000 7f22dbec9740 1 Processor -- start 2024-09-21T02:55:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22bf7fe640 1 -- 172.21.15.29:0/2932459294 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f22c80e7c80 con 0x7f22d0103fc0 2024-09-21T02:55:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22dbec9740 1 -- start start 2024-09-21T02:55:35.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22ce59c640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22b80b6510 0x7f22b80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59c41b0 0x55abb5d70260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb599b2f0 0x55abb5d708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d71420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22dbec9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55abb5cde6f0 con 0x55abb59c41b0 2024-09-21T02:55:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22dbec9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55abb5ccdb20 con 0x55abb59a34d0 2024-09-21T02:55:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22dbec9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55abb5cd1930 con 0x55abb599b2f0 2024-09-21T02:55:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22ced9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d71420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22ced9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d71420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46620/0 (socket says 172.21.15.29:46620) 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.611+0000 7f22ced9d640 1 -- 172.21.15.29:0/65511116 learned_addr learned my addr 172.21.15.29:0/65511116 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22cdd9b640 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb599b2f0 0x55abb5d708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22ced9d640 1 -- 172.21.15.29:0/65511116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb599b2f0 msgr2=0x55abb5d708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22ce59c640 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59c41b0 0x55abb5d70260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22ced9d640 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb599b2f0 0x55abb5d708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22ced9d640 1 -- 172.21.15.29:0/65511116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59c41b0 msgr2=0x55abb5d70260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22ced9d640 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59c41b0 0x55abb5d70260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22ced9d640 1 -- 172.21.15.29:0/65511116 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb5c93530 con 0x55abb59a34d0 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22cdd9b640 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb599b2f0 0x55abb5d708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22ce59c640 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59c41b0 0x55abb5d70260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.612+0000 7f22ce59c640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22b80b6510 0x7f22b80b8a20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f22c0005e00 tx=0x7f22c00058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.613+0000 7f22ced9d640 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d71420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f22c80e7fa0 tx=0x7f22c80ad490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de01df3421c5d6e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.613+0000 7f22bcff9640 1 -- 172.21.15.29:0/65511116 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c80ad650 con 0x55abb59a34d0 2024-09-21T02:55:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.613+0000 7f22bcff9640 1 -- 172.21.15.29:0/65511116 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22c80b2570 con 0x55abb59a34d0 2024-09-21T02:55:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.613+0000 7f22bcff9640 1 -- 172.21.15.29:0/65511116 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c80a9490 con 0x55abb59a34d0 2024-09-21T02:55:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 -- 172.21.15.29:0/65511116 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 msgr2=0x55abb5d71420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d71420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f22c80e7fa0 tx=0x7f22c80ad490 comp rx=0 tx=0).stop 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 -- 172.21.15.29:0/65511116 shutdown_connections 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d71420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb599b2f0 0x55abb5d708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 --2- 172.21.15.29:0/65511116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59c41b0 0x55abb5d70260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 -- 172.21.15.29:0/65511116 >> 172.21.15.29:0/65511116 conn(0x55abb59b8300 msgr2=0x55abb5d678e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 -- 172.21.15.29:0/65511116 shutdown_connections 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 -- 172.21.15.29:0/65511116 wait complete. 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.614+0000 7f22dbec9740 1 -- start start 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22dbec9740 1 Processor -- start 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb599b2f0 0x55abb5d799a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d7a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb59c41b0 0x55abb5d7ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22dbec9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55abb5c05ab0 con 0x55abb599b2f0 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22dbec9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55abb5c73730 con 0x55abb59a34d0 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22dbec9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55abb5d71e80 con 0x55abb59c41b0 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22ce59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb599b2f0 0x55abb5d799a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22ce59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb599b2f0 0x55abb5d799a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56206/0 (socket says 172.21.15.29:56206) 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22dbec9740 10 client.0 _ll_register_callbacks cb 0x55abb5cc76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22ce59c640 1 -- 172.21.15.29:0/1628524728 learned_addr learned my addr 172.21.15.29:0/1628524728 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22cdd9b640 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d7a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.615+0000 7f22ced9d640 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb59c41b0 0x55abb5d7ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22ced9d640 1 -- 172.21.15.29:0/1628524728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 msgr2=0x55abb5d7a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22ced9d640 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d7a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22ced9d640 1 -- 172.21.15.29:0/1628524728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb599b2f0 msgr2=0x55abb5d799a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22ced9d640 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb599b2f0 0x55abb5d799a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22ced9d640 1 -- 172.21.15.29:0/1628524728 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb5a2f790 con 0x55abb59c41b0 2024-09-21T02:55:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22ced9d640 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb59c41b0 0x55abb5d7ab60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22c80b7670 tx=0x7f22c80a9e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acda8331c84eff4d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22b5ffb640 1 -- 172.21.15.29:0/1628524728 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c80b74b0 con 0x55abb59c41b0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22dbec9740 1 -- 172.21.15.29:0/1628524728 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c80b2570 con 0x55abb59c41b0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22b5ffb640 1 -- 172.21.15.29:0/1628524728 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f22c80bee90 con 0x55abb59c41b0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22b5ffb640 1 -- 172.21.15.29:0/1628524728 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f22c80acda0 con 0x55abb59c41b0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22dbec9740 10 client.27193 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.616+0000 7f22dbec9740 1 -- 172.21.15.29:0/1628524728 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55abb5aa8620 con 0x55abb59c41b0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22b5ffb640 1 -- 172.21.15.29:0/1628524728 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22c80b2570 con 0x55abb59c41b0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22b5ffb640 1 -- 172.21.15.29:0/1628524728 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f22c80ac810 con 0x55abb59c41b0 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22dbec9740 10 client.27193 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22dbec9740 10 client.27193 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22dbec9740 20 client.27193 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55abb5982d80 0x55abb5d87090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22dbec9740 4 client.27193 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:55:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22dbec9740 1 -- 172.21.15.29:0/1628524728 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abb5ce0640 con 0x55abb5982d80 2024-09-21T02:55:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.617+0000 7f22ce59c640 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55abb5982d80 0x55abb5d87090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.618+0000 7f22b5ffb640 10 client.27193 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.618+0000 7f22ce59c640 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55abb5982d80 0x55abb5d87090 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79b9129de40800c6 server_cookie=a1888b6b90d30275 in_seq=0 out_seq=0 2024-09-21T02:55:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.619+0000 7f22b5ffb640 1 -- 172.21.15.29:0/1628524728 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55abb5ce0640 con 0x55abb5982d80 2024-09-21T02:55:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.619+0000 7f22b5ffb640 10 client.27193 handle_command_reply: tid=0 2024-09-21T02:55:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 client.27193 shutdown 2024-09-21T02:55:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 -- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb59c41b0 msgr2=0x55abb5d7ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb59c41b0 0x55abb5d7ab60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22c80b7670 tx=0x7f22c80a9e50 comp rx=0 tx=0).stop 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 -- 172.21.15.29:0/1628524728 shutdown_connections 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb59c41b0 0x55abb5d7ab60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59a34d0 0x55abb5d7a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55abb5982d80 0x55abb5d87090 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1628524728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb599b2f0 0x55abb5d799a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22b5ffb640 0 client.27193 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 -- 172.21.15.29:0/1628524728 >> 172.21.15.29:0/1628524728 conn(0x55abb59b8300 msgr2=0x55abb5d65ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 -- 172.21.15.29:0/1628524728 shutdown_connections 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 1 -- 172.21.15.29:0/1628524728 wait complete. 2024-09-21T02:55:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.620+0000 7f22dbec9740 20 client.27193 trim_cache size 0 max 16384 2024-09-21T02:55:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.651+0000 7f22dbec9740 1 Processor -- start 2024-09-21T02:55:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.651+0000 7f22dbec9740 1 -- start start 2024-09-21T02:55:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.651+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb599b2f0 0x55abb5d625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d77f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22dbec9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01aa2e0 con 0x55abb59a34d0 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22dbec9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01abe20 con 0x55abb599b2f0 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22dbec9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01aa9b0 con 0x55abb5982d80 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d77f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d77f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56212/0 (socket says 172.21.15.29:56212) 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22ced9d640 1 -- 172.21.15.29:0/1151486932 learned_addr learned my addr 172.21.15.29:0/1151486932 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22ced9d640 1 -- 172.21.15.29:0/1151486932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 msgr2=0x55abb5d6fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22ced9d640 1 --2- 172.21.15.29:0/1151486932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22ced9d640 1 -- 172.21.15.29:0/1151486932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb599b2f0 msgr2=0x55abb5d625c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22ced9d640 1 --2- 172.21.15.29:0/1151486932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb599b2f0 0x55abb5d625c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.652+0000 7f22ced9d640 1 -- 172.21.15.29:0/1151486932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb5d77bb0 con 0x55abb59a34d0 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.653+0000 7f22ced9d640 1 --2- 172.21.15.29:0/1151486932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d77f30 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f22c80aa300 tx=0x7f22c8004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18c1f290676c6c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.653+0000 7f22b7fff640 1 -- 172.21.15.29:0/1151486932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c80b74b0 con 0x55abb59a34d0 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.653+0000 7f22b7fff640 1 -- 172.21.15.29:0/1151486932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22c80ad650 con 0x55abb59a34d0 2024-09-21T02:55:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.653+0000 7f22b7fff640 1 -- 172.21.15.29:0/1151486932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c80b21d0 con 0x55abb59a34d0 2024-09-21T02:55:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.654+0000 7f22dbec9740 1 -- 172.21.15.29:0/1151486932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 msgr2=0x55abb5d77f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.654+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1151486932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d77f30 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f22c80aa300 tx=0x7f22c8004a90 comp rx=0 tx=0).stop 2024-09-21T02:55:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.655+0000 7f22dbec9740 1 -- 172.21.15.29:0/1151486932 shutdown_connections 2024-09-21T02:55:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.655+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1151486932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d77f30 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.655+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1151486932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb599b2f0 0x55abb5d625c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.655+0000 7f22dbec9740 1 --2- 172.21.15.29:0/1151486932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.655+0000 7f22dbec9740 1 -- 172.21.15.29:0/1151486932 >> 172.21.15.29:0/1151486932 conn(0x55abb59b8300 msgr2=0x55abb5d67290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.656+0000 7f22dbec9740 1 -- 172.21.15.29:0/1151486932 shutdown_connections 2024-09-21T02:55:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.656+0000 7f22dbec9740 1 -- 172.21.15.29:0/1151486932 wait complete. 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.656+0000 7f22dbec9740 1 -- start start 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.656+0000 7f22dbec9740 1 Processor -- start 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.657+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.657+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d6f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.657+0000 7f22dbec9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59c41b0 0x55abb5d81880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.657+0000 7f22dbec9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01ab750 con 0x55abb59a34d0 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.657+0000 7f22dbec9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01ab080 con 0x55abb59c41b0 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.657+0000 7f22dbec9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01a82c0 con 0x55abb5982d80 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.657+0000 7f22cdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d6f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.657+0000 7f22ce59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22dbec9740 10 client.0 _ll_register_callbacks cb 0x7f22c80a9a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22ce59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55576/0 (socket says 172.21.15.29:55576) 2024-09-21T02:55:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22ce59c640 1 -- 172.21.15.29:0/3653250002 learned_addr learned my addr 172.21.15.29:0/3653250002 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22ced9d640 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59c41b0 0x55abb5d81880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22ce59c640 1 -- 172.21.15.29:0/3653250002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59c41b0 msgr2=0x55abb5d81880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22ce59c640 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59c41b0 0x55abb5d81880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22ce59c640 1 -- 172.21.15.29:0/3653250002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 msgr2=0x55abb5d6f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22ce59c640 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d6f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.658+0000 7f22ce59c640 1 -- 172.21.15.29:0/3653250002 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb5deba60 con 0x55abb5982d80 2024-09-21T02:55:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.659+0000 7f22ce59c640 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6ebe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009830 tx=0x7f22c002a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4add3b2ed93d7bfa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.659+0000 7f22b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:55:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.659+0000 7f22b5ffb640 1 -- 172.21.15.29:0/3653250002 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0037040 con 0x55abb5982d80 2024-09-21T02:55:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.659+0000 7f22b5ffb640 1 -- 172.21.15.29:0/3653250002 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f22c0004240 con 0x55abb5982d80 2024-09-21T02:55:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.660+0000 7f22dbec9740 1 -- 172.21.15.29:0/3653250002 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c80ad650 con 0x55abb5982d80 2024-09-21T02:55:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.660+0000 7f22b5ffb640 1 -- 172.21.15.29:0/3653250002 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f22c0004440 con 0x55abb5982d80 2024-09-21T02:55:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.660+0000 7f22dbec9740 10 client.27196 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.660+0000 7f22dbec9740 1 -- 172.21.15.29:0/3653250002 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55abb5decd40 con 0x55abb5982d80 2024-09-21T02:55:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.662+0000 7f22b5ffb640 1 -- 172.21.15.29:0/3653250002 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22c0044040 con 0x55abb5982d80 2024-09-21T02:55:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.662+0000 7f22b5ffb640 1 -- 172.21.15.29:0/3653250002 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f22c0049040 con 0x55abb5982d80 2024-09-21T02:55:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.662+0000 7f22dbec9740 10 client.27196 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.662+0000 7f22dbec9740 10 client.27196 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:55:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.662+0000 7f22dbec9740 20 client.27196 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.662+0000 7f22dbec9740 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55abb59581c0 0x55abb5def710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.662+0000 7f22dbec9740 4 client.27196 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:55:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.662+0000 7f22dbec9740 1 -- 172.21.15.29:0/3653250002 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f22c80ad650 con 0x55abb59581c0 2024-09-21T02:55:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.664+0000 7f22cdd9b640 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55abb59581c0 0x55abb5def710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.667+0000 7f22cdd9b640 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55abb59581c0 0x55abb5def710 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21aaadcc92468872 server_cookie=a12fcf6b8ef8aa1a in_seq=0 out_seq=0 2024-09-21T02:55:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.667+0000 7f22b5ffb640 10 client.27196 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:35.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19879 5.18 MB/sec execute 72 sec latency 112.551 ms 2024-09-21T02:55:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22b5ffb640 1 -- 172.21.15.29:0/3653250002 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f22c80ad650 con 0x55abb59581c0 2024-09-21T02:55:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22b5ffb640 10 client.27196 handle_command_reply: tid=0 2024-09-21T02:55:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 client.27196 shutdown 2024-09-21T02:55:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 -- 172.21.15.29:0/3653250002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 msgr2=0x55abb5d6ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6ebe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009830 tx=0x7f22c002a340 comp rx=0 tx=0).stop 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 -- 172.21.15.29:0/3653250002 shutdown_connections 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55abb59c41b0 0x55abb5d81880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55abb59581c0 0x55abb5def710 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22b5ffb640 0 client.27196 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abb59a34d0 0x55abb5d6f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 --2- 172.21.15.29:0/3653250002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55abb5982d80 0x55abb5d6ebe0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.812+0000 7f22dbec9740 1 -- 172.21.15.29:0/3653250002 >> 172.21.15.29:0/3653250002 conn(0x55abb59b8300 msgr2=0x55abb5d70200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22dbec9740 1 -- 172.21.15.29:0/3653250002 shutdown_connections 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22dbec9740 1 -- 172.21.15.29:0/3653250002 wait complete. 2024-09-21T02:55:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22dbec9740 20 client.27196 trim_cache size 0 max 16384 2024-09-21T02:55:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:55:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:55:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:55:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:55:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22d4f5f640 1 -- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22b80b6510 msgr2=0x7f22b80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22b80b6510 0x7f22b80b8a20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f22c0005e00 tx=0x7f22c00058b0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22d4f5f640 1 -- 172.21.15.29:0/2932459294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22d0103fc0 msgr2=0x7f22d01a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22d0103fc0 0x7f22d01a1150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009fa0 tx=0x7f22c800c7e0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22d4f5f640 1 -- 172.21.15.29:0/2932459294 shutdown_connections 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f22b80b6510 0x7f22b80b8a20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22d0103fc0 0x7f22d01a1150 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.813+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22d0103620 0x7f22d019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.814+0000 7f22d4f5f640 1 --2- 172.21.15.29:0/2932459294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d01025a0 0x7f22d006b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.814+0000 7f22d4f5f640 1 -- 172.21.15.29:0/2932459294 >> 172.21.15.29:0/2932459294 conn(0x7f22d00fdee0 msgr2=0x7f22d0106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.814+0000 7f22d4f5f640 1 -- 172.21.15.29:0/2932459294 shutdown_connections 2024-09-21T02:55:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:35.814+0000 7f22d4f5f640 1 -- 172.21.15.29:0/2932459294 wait complete. 2024-09-21T02:55:35.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:35 smithi190 ceph-mon[33871]: pgmap v883: 129 pgs: 129 active+clean; 263 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-09-21T02:55:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:35 smithi029 ceph-mon[31006]: pgmap v883: 129 pgs: 129 active+clean; 263 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-09-21T02:55:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:35 smithi066 ceph-mon[33436]: pgmap v883: 129 pgs: 129 active+clean; 263 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-09-21T02:55:36.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20000 5.22 MB/sec execute 73 sec latency 201.377 ms 2024-09-21T02:55:37.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20128 5.25 MB/sec execute 74 sec latency 97.279 ms 2024-09-21T02:55:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:37 smithi190 ceph-mon[33871]: pgmap v884: 129 pgs: 129 active+clean; 269 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T02:55:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:37 smithi029 ceph-mon[31006]: pgmap v884: 129 pgs: 129 active+clean; 269 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T02:55:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:37 smithi066 ceph-mon[33436]: pgmap v884: 129 pgs: 129 active+clean; 269 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T02:55:38.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20314 5.42 MB/sec execute 75 sec latency 91.526 ms 2024-09-21T02:55:39.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20458 5.47 MB/sec execute 76 sec latency 92.743 ms 2024-09-21T02:55:39.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:39 smithi190 ceph-mon[33871]: pgmap v885: 129 pgs: 129 active+clean; 274 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-09-21T02:55:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:39 smithi029 ceph-mon[31006]: pgmap v885: 129 pgs: 129 active+clean; 274 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-09-21T02:55:40.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:39 smithi066 ceph-mon[33436]: pgmap v885: 129 pgs: 129 active+clean; 274 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-09-21T02:55:40.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20611 5.57 MB/sec execute 77 sec latency 93.841 ms 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:40 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:40 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:40 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20760 5.62 MB/sec execute 78 sec latency 120.877 ms 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: pgmap v886: 129 pgs: 129 active+clean; 275 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi066 to 176.0M 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: Unable to set osd_memory_target on smithi066 to 184555212: error parsing value: Value '184555212' is below minimum 939524096 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi190 to 176.0M 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: Unable to set osd_memory_target on smithi190 to 184583884: error parsing value: Value '184583884' is below minimum 939524096 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: Adjusting osd_memory_target on smithi029 to 177.0M 2024-09-21T02:55:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: Unable to set osd_memory_target on smithi029 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T02:55:41.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:55:41.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:55:41.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:55:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: pgmap v886: 129 pgs: 129 active+clean; 275 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-09-21T02:55:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi066 to 176.0M 2024-09-21T02:55:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: Unable to set osd_memory_target on smithi066 to 184555212: error parsing value: Value '184555212' is below minimum 939524096 2024-09-21T02:55:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi190 to 176.0M 2024-09-21T02:55:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: Unable to set osd_memory_target on smithi190 to 184583884: error parsing value: Value '184583884' is below minimum 939524096 2024-09-21T02:55:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:41.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: Adjusting osd_memory_target on smithi029 to 177.0M 2024-09-21T02:55:41.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: Unable to set osd_memory_target on smithi029 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T02:55:41.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:55:41.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:55:41.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: pgmap v886: 129 pgs: 129 active+clean; 275 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi066 to 176.0M 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: Unable to set osd_memory_target on smithi066 to 184555212: error parsing value: Value '184555212' is below minimum 939524096 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi190 to 176.0M 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: Unable to set osd_memory_target on smithi190 to 184583884: error parsing value: Value '184583884' is below minimum 939524096 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: Adjusting osd_memory_target on smithi029 to 177.0M 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: Unable to set osd_memory_target on smithi029 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:55:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:55:42.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20868 5.58 MB/sec execute 79 sec latency 92.907 ms 2024-09-21T02:55:43.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20981 5.56 MB/sec execute 80 sec latency 109.357 ms 2024-09-21T02:55:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:43 smithi190 ceph-mon[33871]: pgmap v887: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 3.7 MiB/s wr, 206 op/s 2024-09-21T02:55:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:43 smithi029 ceph-mon[31006]: pgmap v887: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 3.7 MiB/s wr, 206 op/s 2024-09-21T02:55:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:43 smithi066 ceph-mon[33436]: pgmap v887: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 3.7 MiB/s wr, 206 op/s 2024-09-21T02:55:44.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21104 5.52 MB/sec execute 81 sec latency 129.314 ms 2024-09-21T02:55:45.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21233 5.52 MB/sec execute 82 sec latency 180.779 ms 2024-09-21T02:55:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:45 smithi190 ceph-mon[33871]: pgmap v888: 129 pgs: 129 active+clean; 280 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-09-21T02:55:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:45 smithi029 ceph-mon[31006]: pgmap v888: 129 pgs: 129 active+clean; 280 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-09-21T02:55:46.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:45 smithi066 ceph-mon[33436]: pgmap v888: 129 pgs: 129 active+clean; 280 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-09-21T02:55:46.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21362 5.50 MB/sec execute 83 sec latency 210.795 ms 2024-09-21T02:55:47.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21501 5.51 MB/sec execute 84 sec latency 108.999 ms 2024-09-21T02:55:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:47 smithi190 ceph-mon[33871]: pgmap v889: 129 pgs: 129 active+clean; 281 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 4.3 MiB/s wr, 247 op/s 2024-09-21T02:55:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:47 smithi029 ceph-mon[31006]: pgmap v889: 129 pgs: 129 active+clean; 281 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 4.3 MiB/s wr, 247 op/s 2024-09-21T02:55:48.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:47 smithi066 ceph-mon[33436]: pgmap v889: 129 pgs: 129 active+clean; 281 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 4.3 MiB/s wr, 247 op/s 2024-09-21T02:55:48.719 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21639 5.53 MB/sec execute 85 sec latency 117.936 ms 2024-09-21T02:55:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 -- 172.21.15.29:0/581480046 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e9c103640 msgr2=0x7f9e9c103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 --2- 172.21.15.29:0/581480046 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e9c103640 0x7f9e9c103a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009860 tx=0x7f9e8c02a340 comp rx=0 tx=0).stop 2024-09-21T02:55:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 -- 172.21.15.29:0/581480046 shutdown_connections 2024-09-21T02:55:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 --2- 172.21.15.29:0/581480046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e9c103fe0 0x7f9e9c10a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 --2- 172.21.15.29:0/581480046 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e9c103640 0x7f9e9c103a50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 --2- 172.21.15.29:0/581480046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e9c1025c0 0x7f9e9c102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 -- 172.21.15.29:0/581480046 >> 172.21.15.29:0/581480046 conn(0x7f9e9c0fdf00 msgr2=0x7f9e9c1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:48.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 -- 172.21.15.29:0/581480046 shutdown_connections 2024-09-21T02:55:48.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.948+0000 7f9ea3951640 1 -- 172.21.15.29:0/581480046 wait complete. 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.949+0000 7f9ea3951640 1 Processor -- start 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.949+0000 7f9ea3951640 1 -- start start 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.949+0000 7f9ea3951640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e9c1025c0 0x7f9e9c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea3951640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e9c103640 0x7f9e9c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea3951640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e9c103fe0 0x7f9e9c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea3951640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c10e0a0 con 0x7f9e9c103640 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea3951640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c10df20 con 0x7f9e9c1025c0 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea3951640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c10e220 con 0x7f9e9c103fe0 2024-09-21T02:55:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea0eec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e9c103640 0x7f9e9c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea16ed640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e9c1025c0 0x7f9e9c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea0eec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e9c103640 0x7f9e9c19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55486/0 (socket says 172.21.15.29:55486) 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea0eec640 1 -- 172.21.15.29:0/469387284 learned_addr learned my addr 172.21.15.29:0/469387284 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e9c103fe0 0x7f9e9c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea1eee640 1 -- 172.21.15.29:0/469387284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e9c1025c0 msgr2=0x7f9e9c06c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e9c1025c0 0x7f9e9c06c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea1eee640 1 -- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e9c103640 msgr2=0x7f9e9c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e9c103640 0x7f9e9c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea1eee640 1 -- 172.21.15.29:0/469387284 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c1a1940 con 0x7f9e9c103fe0 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e9c103640 0x7f9e9c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.950+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e9c103fe0 0x7f9e9c1a1120 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009fd0 tx=0x7f9e9400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.951+0000 7f9e927fc640 1 -- 172.21.15.29:0/469387284 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400ce20 con 0x7f9e9c103fe0 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.951+0000 7f9e927fc640 1 -- 172.21.15.29:0/469387284 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9e94005020 con 0x7f9e9c103fe0 2024-09-21T02:55:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.951+0000 7f9e927fc640 1 -- 172.21.15.29:0/469387284 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94010690 con 0x7f9e9c103fe0 2024-09-21T02:55:48.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.951+0000 7f9ea3951640 1 -- 172.21.15.29:0/469387284 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9c1a1c30 con 0x7f9e9c103fe0 2024-09-21T02:55:48.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.951+0000 7f9ea3951640 1 -- 172.21.15.29:0/469387284 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9c1a2180 con 0x7f9e9c103fe0 2024-09-21T02:55:48.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.954+0000 7f9eaa8b8740 1 Processor -- start 2024-09-21T02:55:48.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9e927fc640 1 -- 172.21.15.29:0/469387284 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9e94004a40 con 0x7f9e9c103fe0 2024-09-21T02:55:48.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9eaa8b8740 1 -- start start 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a8b1b0 0x5578e8e30cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a622f0 0x5578e8e31230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e42d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5578e8da56f0 con 0x5578e8a8b1b0 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5578e8d94b20 con 0x5578e8a6a4d0 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5578e8d98930 con 0x5578e8a622f0 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9ea16ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a622f0 0x5578e8e31230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9ea0eec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a8b1b0 0x5578e8e30cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9ea16ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a622f0 0x5578e8e31230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32834/0 (socket says 172.21.15.29:32834) 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.955+0000 7f9ea16ed640 1 -- 172.21.15.29:0/4134917132 learned_addr learned my addr 172.21.15.29:0/4134917132 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/4134917132 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e42d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9ea16ed640 1 -- 172.21.15.29:0/4134917132 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 msgr2=0x5578e8e42d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/4134917132 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e42d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9ea16ed640 1 -- 172.21.15.29:0/4134917132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a8b1b0 msgr2=0x5578e8e30cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/4134917132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a8b1b0 0x5578e8e30cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9ea16ed640 1 -- 172.21.15.29:0/4134917132 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578e8d5a530 con 0x5578e8a622f0 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9e927fc640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9e880b6590 0x7f9e880b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/4134917132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a622f0 0x5578e8e31230 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009cc0 tx=0x7f9e98011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5b227abefcfe768 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9e880b6590 0x7f9e880b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9e877fe640 1 -- 172.21.15.29:0/4134917132 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98005220 con 0x5578e8a622f0 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.956+0000 7f9e927fc640 1 -- 172.21.15.29:0/469387284 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9e940e7ce0 con 0x7f9e9c103fe0 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.957+0000 7f9e877fe640 1 -- 172.21.15.29:0/4134917132 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9e98004c30 con 0x5578e8a622f0 2024-09-21T02:55:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.957+0000 7f9e877fe640 1 -- 172.21.15.29:0/4134917132 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98019670 con 0x5578e8a622f0 2024-09-21T02:55:48.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/4134917132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a622f0 msgr2=0x5578e8e31230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:48.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/4134917132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a622f0 0x5578e8e31230 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009cc0 tx=0x7f9e98011810 comp rx=0 tx=0).stop 2024-09-21T02:55:48.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9e880b6590 0x7f9e880b8aa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c005e00 tx=0x7f9e8c0058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:48.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/4134917132 shutdown_connections 2024-09-21T02:55:48.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/4134917132 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e42d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/4134917132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a622f0 0x5578e8e31230 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/4134917132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a8b1b0 0x5578e8e30cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/4134917132 >> 172.21.15.29:0/4134917132 conn(0x5578e8a7f300 msgr2=0x5578e8e2e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/4134917132 shutdown_connections 2024-09-21T02:55:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/4134917132 wait complete. 2024-09-21T02:55:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.958+0000 7f9eaa8b8740 1 -- start start 2024-09-21T02:55:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.959+0000 7f9eaa8b8740 1 Processor -- start 2024-09-21T02:55:48.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.959+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e44e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.959+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e454d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.959+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e45fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5578e8cccab0 con 0x5578e8a622f0 2024-09-21T02:55:48.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5578e8d3a730 con 0x5578e8a6a4d0 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5578e8e433d0 con 0x5578e8a8b1b0 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9eaa8b8740 10 client.0 _ll_register_callbacks cb 0x5578e8d8e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea16ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e44e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea16ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e44e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55496/0 (socket says 172.21.15.29:55496) 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea16ed640 1 -- 172.21.15.29:0/3033341742 learned_addr learned my addr 172.21.15.29:0/3033341742 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e45fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea1eee640 1 -- 172.21.15.29:0/3033341742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 msgr2=0x5578e8e454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e454d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea1eee640 1 -- 172.21.15.29:0/3033341742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 msgr2=0x5578e8e44e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e44e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea1eee640 1 -- 172.21.15.29:0/3033341742 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578e8af6790 con 0x5578e8a8b1b0 2024-09-21T02:55:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e45fd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9e940ad7f0 tx=0x7f9e940ad320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fb3ed773c893a67 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9e84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.960+0000 7f9e84ff9640 1 -- 172.21.15.29:0/3033341742 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e940ae1f0 con 0x5578e8a8b1b0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9e84ff9640 1 -- 172.21.15.29:0/3033341742 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9e940051d0 con 0x5578e8a8b1b0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3033341742 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e98004c30 con 0x5578e8a8b1b0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9e84ff9640 1 -- 172.21.15.29:0/3033341742 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9e940b25d0 con 0x5578e8a8b1b0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 10 client.27208 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3033341742 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5578e8b6f620 con 0x5578e8a8b1b0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9e84ff9640 1 -- 172.21.15.29:0/3033341742 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9e98004c30 con 0x5578e8a8b1b0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9e84ff9640 1 -- 172.21.15.29:0/3033341742 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f9e940cfb00 con 0x5578e8a8b1b0 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 10 client.27208 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 10 client.27208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 20 client.27208 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5578e8a49d80 0x5578e8e52b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 4 client.27208 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:55:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.961+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3033341742 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578e8da7640 con 0x5578e8a49d80 2024-09-21T02:55:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.962+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5578e8a49d80 0x5578e8e52b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.962+0000 7f9e84ff9640 10 client.27208 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.962+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5578e8a49d80 0x5578e8e52b00 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1ec268b25ffff0e server_cookie=1365671b7f1335ff in_seq=0 out_seq=0 2024-09-21T02:55:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.964+0000 7f9e84ff9640 1 -- 172.21.15.29:0/3033341742 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5578e8da7640 con 0x5578e8a49d80 2024-09-21T02:55:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.964+0000 7f9e84ff9640 10 client.27208 handle_command_reply: tid=0 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.964+0000 7f9eaa8b8740 1 client.27208 shutdown 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.964+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 msgr2=0x5578e8e45fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.964+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e45fd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9e940ad7f0 tx=0x7f9e940ad320 comp rx=0 tx=0).stop 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3033341742 shutdown_connections 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e45fd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5578e8a49d80 0x5578e8e52b00 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3033341742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e44e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3033341742 >> 172.21.15.29:0/3033341742 conn(0x5578e8a7f300 msgr2=0x5578e8e2f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9e84ff9640 0 client.27208 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3033341742 shutdown_connections 2024-09-21T02:55:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.965+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3033341742 wait complete. 2024-09-21T02:55:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.966+0000 7f9eaa8b8740 20 client.27208 trim_cache size 0 max 16384 2024-09-21T02:55:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9eaa8b8740 1 Processor -- start 2024-09-21T02:55:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9eaa8b8740 1 -- start start 2024-09-21T02:55:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e52a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e2e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e2e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1aa280 con 0x5578e8a622f0 2024-09-21T02:55:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1abdc0 con 0x5578e8a6a4d0 2024-09-21T02:55:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1aa950 con 0x5578e8a8b1b0 2024-09-21T02:55:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9ea16ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e52a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9ea16ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e52a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55502/0 (socket says 172.21.15.29:55502) 2024-09-21T02:55:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9ea16ed640 1 -- 172.21.15.29:0/3823256869 learned_addr learned my addr 172.21.15.29:0/3823256869 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e2e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.998+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e2e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9ea16ed640 1 -- 172.21.15.29:0/3823256869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 msgr2=0x5578e8e2e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e2e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9ea16ed640 1 -- 172.21.15.29:0/3823256869 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 msgr2=0x5578e8e2e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e2e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9ea16ed640 1 -- 172.21.15.29:0/3823256869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578e8e43e00 con 0x5578e8a622f0 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e2e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e2e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e52a30 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009400 tx=0x7f9e98009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fef8c957065f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9e86ffd640 1 -- 172.21.15.29:0/3823256869 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98004220 con 0x5578e8a622f0 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9e86ffd640 1 -- 172.21.15.29:0/3823256869 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9e980043c0 con 0x5578e8a622f0 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:48.999+0000 7f9e86ffd640 1 -- 172.21.15.29:0/3823256869 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98019650 con 0x5578e8a622f0 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3823256869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 msgr2=0x5578e8e52a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e52a30 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009400 tx=0x7f9e98009f10 comp rx=0 tx=0).stop 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3823256869 shutdown_connections 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a8b1b0 0x5578e8e2e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a6a4d0 0x5578e8e2e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/3823256869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e52a30 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3823256869 >> 172.21.15.29:0/3823256869 conn(0x5578e8a7f300 msgr2=0x5578e8e295c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3823256869 shutdown_connections 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.000+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/3823256869 wait complete. 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.001+0000 7f9eaa8b8740 1 -- start start 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.001+0000 7f9eaa8b8740 1 Processor -- start 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.001+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e49760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.001+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a6a4d0 0x5578e8e49e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.001+0000 7f9eaa8b8740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a8b1b0 0x5578e8e4a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1ab6f0 con 0x5578e8a622f0 2024-09-21T02:55:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1ab020 con 0x5578e8a8b1b0 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9eaa8b8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1a8260 con 0x5578e8a6a4d0 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9eaa8b8740 10 client.0 _ll_register_callbacks cb 0x7f9e94005bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea0eec640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a6a4d0 0x5578e8e49e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea16ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e49760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea0eec640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a6a4d0 0x5578e8e49e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32860/0 (socket says 172.21.15.29:32860) 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea0eec640 1 -- 172.21.15.29:0/527065061 learned_addr learned my addr 172.21.15.29:0/527065061 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea1eee640 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a8b1b0 0x5578e8e4a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea16ed640 1 -- 172.21.15.29:0/527065061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a6a4d0 msgr2=0x5578e8e49e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a6a4d0 0x5578e8e49e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea16ed640 1 -- 172.21.15.29:0/527065061 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a8b1b0 msgr2=0x5578e8e4a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a8b1b0 0x5578e8e4a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea16ed640 1 -- 172.21.15.29:0/527065061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578e8e4b080 con 0x5578e8a622f0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a6a4d0 0x5578e8e49e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9ea16ed640 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e49760 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9e98006010 tx=0x7f9e98009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c9b719f304501c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.002+0000 7f9e84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9e84ff9640 1 -- 172.21.15.29:0/527065061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e980035f0 con 0x5578e8a622f0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9e84ff9640 1 -- 172.21.15.29:0/527065061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9e98003790 con 0x5578e8a622f0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/527065061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e980043c0 con 0x5578e8a622f0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9e84ff9640 1 -- 172.21.15.29:0/527065061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9e98003990 con 0x5578e8a622f0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9eaa8b8740 10 client.17724 fetch_fsmap learned FSMap version 37 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/527065061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5578e8e4c370 con 0x5578e8a622f0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9e84ff9640 1 -- 172.21.15.29:0/527065061 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9e980043c0 con 0x5578e8a622f0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9e84ff9640 1 -- 172.21.15.29:0/527065061 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f9e98019430 con 0x5578e8a622f0 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.003+0000 7f9eaa8b8740 10 client.17724 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:55:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.004+0000 7f9eaa8b8740 10 client.17724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:55:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.004+0000 7f9eaa8b8740 20 client.17724 populate_metadata read hostname 'smithi029' 2024-09-21T02:55:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.004+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5578e8a1f1c0 0x5578e8e4cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:55:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.004+0000 7f9eaa8b8740 4 client.17724 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:55:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.004+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/527065061 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5578e8da30f0 con 0x5578e8a1f1c0 2024-09-21T02:55:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.004+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5578e8a1f1c0 0x5578e8e4cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:55:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.004+0000 7f9e84ff9640 10 client.17724 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.004+0000 7f9ea0eec640 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5578e8a1f1c0 0x5578e8e4cb10 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c63fb01fe58c6372 server_cookie=d5e7193516b6d438 in_seq=0 out_seq=0 2024-09-21T02:55:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.150+0000 7f9e84ff9640 1 -- 172.21.15.29:0/527065061 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5578e8da30f0 con 0x5578e8a1f1c0 2024-09-21T02:55:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.150+0000 7f9e84ff9640 10 client.17724 handle_command_reply: tid=0 2024-09-21T02:55:49.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.150+0000 7f9eaa8b8740 1 client.17724 shutdown 2024-09-21T02:55:49.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.150+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/527065061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 msgr2=0x5578e8e49760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:49.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.150+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e49760 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9e98006010 tx=0x7f9e98009590 comp rx=0 tx=0).stop 2024-09-21T02:55:49.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/527065061 shutdown_connections 2024-09-21T02:55:49.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5578e8a8b1b0 0x5578e8e4a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5578e8a6a4d0 0x5578e8e49e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5578e8a1f1c0 0x5578e8e4cb10 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 1 --2- 172.21.15.29:0/527065061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5578e8a622f0 0x5578e8e49760 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9e84ff9640 0 client.17724 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:55:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/527065061 >> 172.21.15.29:0/527065061 conn(0x5578e8a7f300 msgr2=0x5578e8e42940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/527065061 shutdown_connections 2024-09-21T02:55:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 1 -- 172.21.15.29:0/527065061 wait complete. 2024-09-21T02:55:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.151+0000 7f9eaa8b8740 20 client.17724 trim_cache size 0 max 16384 2024-09-21T02:55:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:55:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:55:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:55:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.153+0000 7f9ea3951640 1 -- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9e880b6590 msgr2=0x7f9e880b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.153+0000 7f9ea3951640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9e880b6590 0x7f9e880b8aa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c005e00 tx=0x7f9e8c0058b0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.153+0000 7f9ea3951640 1 -- 172.21.15.29:0/469387284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e9c103fe0 msgr2=0x7f9e9c1a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.153+0000 7f9ea3951640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e9c103fe0 0x7f9e9c1a1120 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009fd0 tx=0x7f9e9400ef10 comp rx=0 tx=0).stop 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.154+0000 7f9ea3951640 1 -- 172.21.15.29:0/469387284 shutdown_connections 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.154+0000 7f9ea3951640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9e880b6590 0x7f9e880b8aa0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.154+0000 7f9ea3951640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e9c103fe0 0x7f9e9c1a1120 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.154+0000 7f9ea3951640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e9c103640 0x7f9e9c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.154+0000 7f9ea3951640 1 --2- 172.21.15.29:0/469387284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e9c1025c0 0x7f9e9c06c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.154+0000 7f9ea3951640 1 -- 172.21.15.29:0/469387284 >> 172.21.15.29:0/469387284 conn(0x7f9e9c0fdf00 msgr2=0x7f9e9c106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.154+0000 7f9ea3951640 1 -- 172.21.15.29:0/469387284 shutdown_connections 2024-09-21T02:55:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:55:49.154+0000 7f9ea3951640 1 -- 172.21.15.29:0/469387284 wait complete. 2024-09-21T02:55:49.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21790 5.63 MB/sec execute 86 sec latency 206.245 ms 2024-09-21T02:55:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:49 smithi190 ceph-mon[33871]: pgmap v890: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 4.2 MiB/s wr, 258 op/s 2024-09-21T02:55:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:49 smithi029 ceph-mon[31006]: pgmap v890: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 4.2 MiB/s wr, 258 op/s 2024-09-21T02:55:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:49 smithi066 ceph-mon[33436]: pgmap v890: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 4.2 MiB/s wr, 258 op/s 2024-09-21T02:55:50.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21922 5.61 MB/sec execute 87 sec latency 190.211 ms 2024-09-21T02:55:51.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 22067 5.59 MB/sec execute 88 sec latency 191.103 ms 2024-09-21T02:55:51.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:51 smithi190 ceph-mon[33871]: pgmap v891: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 4.0 MiB/s wr, 228 op/s 2024-09-21T02:55:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:51 smithi029 ceph-mon[31006]: pgmap v891: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 4.0 MiB/s wr, 228 op/s 2024-09-21T02:55:52.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:51 smithi066 ceph-mon[33436]: pgmap v891: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 4.0 MiB/s wr, 228 op/s 2024-09-21T02:55:52.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 22248 5.61 MB/sec execute 89 sec latency 90.249 ms 2024-09-21T02:55:53.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 22393 5.58 MB/sec execute 90 sec latency 94.517 ms 2024-09-21T02:55:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:53 smithi190 ceph-mon[33871]: pgmap v892: 129 pgs: 129 active+clean; 266 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 4.3 MiB/s wr, 297 op/s 2024-09-21T02:55:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:53 smithi029 ceph-mon[31006]: pgmap v892: 129 pgs: 129 active+clean; 266 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 4.3 MiB/s wr, 297 op/s 2024-09-21T02:55:54.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:53 smithi066 ceph-mon[33436]: pgmap v892: 129 pgs: 129 active+clean; 266 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 4.3 MiB/s wr, 297 op/s 2024-09-21T02:55:54.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 22551 5.56 MB/sec execute 91 sec latency 92.195 ms 2024-09-21T02:55:55.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 22708 5.58 MB/sec execute 92 sec latency 145.815 ms 2024-09-21T02:55:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:55 smithi190 ceph-mon[33871]: pgmap v893: 129 pgs: 129 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 3.6 MiB/s wr, 318 op/s 2024-09-21T02:55:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:55 smithi029 ceph-mon[31006]: pgmap v893: 129 pgs: 129 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 3.6 MiB/s wr, 318 op/s 2024-09-21T02:55:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:55 smithi066 ceph-mon[33436]: pgmap v893: 129 pgs: 129 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 3.6 MiB/s wr, 318 op/s 2024-09-21T02:55:56.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 22884 5.63 MB/sec execute 93 sec latency 343.347 ms 2024-09-21T02:55:57.720 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23028 5.66 MB/sec execute 94 sec latency 85.044 ms 2024-09-21T02:55:57.812 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:57 smithi066 ceph-mon[33436]: pgmap v894: 129 pgs: 129 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 2.6 MiB/s wr, 308 op/s 2024-09-21T02:55:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:57 smithi190 ceph-mon[33871]: pgmap v894: 129 pgs: 129 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 2.6 MiB/s wr, 308 op/s 2024-09-21T02:55:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:57 smithi029 ceph-mon[31006]: pgmap v894: 129 pgs: 129 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 2.6 MiB/s wr, 308 op/s 2024-09-21T02:55:58.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23174 5.72 MB/sec execute 95 sec latency 100.898 ms 2024-09-21T02:55:59.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23314 5.74 MB/sec execute 96 sec latency 134.806 ms 2024-09-21T02:55:59.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:55:59 smithi190 ceph-mon[33871]: pgmap v895: 129 pgs: 129 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.7 MiB/s wr, 308 op/s 2024-09-21T02:55:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:55:59 smithi029 ceph-mon[31006]: pgmap v895: 129 pgs: 129 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.7 MiB/s wr, 308 op/s 2024-09-21T02:56:00.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:55:59 smithi066 ceph-mon[33436]: pgmap v895: 129 pgs: 129 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.7 MiB/s wr, 308 op/s 2024-09-21T02:56:00.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23512 5.86 MB/sec execute 97 sec latency 86.447 ms 2024-09-21T02:56:01.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23701 5.89 MB/sec execute 98 sec latency 185.925 ms 2024-09-21T02:56:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:01 smithi190 ceph-mon[33871]: pgmap v896: 129 pgs: 129 active+clean; 263 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-09-21T02:56:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:01 smithi029 ceph-mon[31006]: pgmap v896: 129 pgs: 129 active+clean; 263 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-09-21T02:56:02.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:01 smithi066 ceph-mon[33436]: pgmap v896: 129 pgs: 129 active+clean; 263 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-09-21T02:56:02.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 -- 172.21.15.29:0/3937995925 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 msgr2=0x7f0f14102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3937995925 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 0x7f0f14102a10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009900 tx=0x7f0f0802a410 comp rx=0 tx=0).stop 2024-09-21T02:56:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 -- 172.21.15.29:0/3937995925 shutdown_connections 2024-09-21T02:56:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3937995925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14103fc0 0x7f0f1410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3937995925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f14103620 0x7f0f14103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3937995925 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 0x7f0f14102a10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 -- 172.21.15.29:0/3937995925 >> 172.21.15.29:0/3937995925 conn(0x7f0f140fdee0 msgr2=0x7f0f14100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 -- 172.21.15.29:0/3937995925 shutdown_connections 2024-09-21T02:56:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.296+0000 7f0f1b132640 1 -- 172.21.15.29:0/3937995925 wait complete. 2024-09-21T02:56:02.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f1b132640 1 Processor -- start 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f1b132640 1 -- start start 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f1b132640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 0x7f0f14199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f1b132640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14103620 0x7f0f1419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f1b132640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f14103fc0 0x7f0f1419b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f1b132640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1410e0e0 con 0x7f0f14103620 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f1b132640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1410df60 con 0x7f0f141025a0 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f1b132640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1410e260 con 0x7f0f14103fc0 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.297+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14103620 0x7f0f1419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 0x7f0f14199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 0x7f0f14199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46678/0 (socket says 172.21.15.29:46678) 2024-09-21T02:56:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f196cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f14103fc0 0x7f0f1419b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 -- 172.21.15.29:0/3532801816 learned_addr learned my addr 172.21.15.29:0/3532801816 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 -- 172.21.15.29:0/3532801816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f14103fc0 msgr2=0x7f0f1419b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f14103fc0 0x7f0f1419b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 -- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14103620 msgr2=0x7f0f1419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14103620 0x7f0f1419a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 -- 172.21.15.29:0/3532801816 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1419b8b0 con 0x7f0f141025a0 2024-09-21T02:56:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.298+0000 7f0f18ece640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 0x7f0f14199e70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f08005760 tx=0x7f0f08005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.299+0000 7f0f11ffb640 1 -- 172.21.15.29:0/3532801816 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f08035030 con 0x7f0f141025a0 2024-09-21T02:56:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.299+0000 7f0f11ffb640 1 -- 172.21.15.29:0/3532801816 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f08041070 con 0x7f0f141025a0 2024-09-21T02:56:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.299+0000 7f0f11ffb640 1 -- 172.21.15.29:0/3532801816 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f08031d00 con 0x7f0f141025a0 2024-09-21T02:56:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.299+0000 7f0f1b132640 1 -- 172.21.15.29:0/3532801816 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1419bb40 con 0x7f0f141025a0 2024-09-21T02:56:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.299+0000 7f0f1b132640 1 -- 172.21.15.29:0/3532801816 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f14104b10 con 0x7f0f141025a0 2024-09-21T02:56:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.302+0000 7f0f2209e740 1 Processor -- start 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.302+0000 7f0f2209e740 1 -- start start 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f11ffb640 1 -- 172.21.15.29:0/3532801816 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f0f08046470 con 0x7f0f141025a0 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddae2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7092f0 0x55b2ddae2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddae3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f2209e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b2dda4c6f0 con 0x55b2dd7114d0 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f2209e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b2dda3bb20 con 0x55b2dd7092f0 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f2209e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b2dda3f930 con 0x55b2dd7321b0 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f196cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddae2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f196cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddae2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:33408/0 (socket says 172.21.15.29:33408) 2024-09-21T02:56:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f196cf640 1 -- 172.21.15.29:0/1828359723 learned_addr learned my addr 172.21.15.29:0/1828359723 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f18ece640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7092f0 0x55b2ddae2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.303+0000 7f0f13fff640 1 --2- 172.21.15.29:0/1828359723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddae3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0f196cf640 1 -- 172.21.15.29:0/1828359723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7092f0 msgr2=0x55b2ddae2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0f196cf640 1 --2- 172.21.15.29:0/1828359723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7092f0 0x55b2ddae2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0f196cf640 1 -- 172.21.15.29:0/1828359723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 msgr2=0x55b2ddae3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0f196cf640 1 --2- 172.21.15.29:0/1828359723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddae3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0f196cf640 1 -- 172.21.15.29:0/1828359723 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2dda01530 con 0x55b2dd7321b0 2024-09-21T02:56:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0f196cf640 1 --2- 172.21.15.29:0/1828359723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddae2660 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c011470 tx=0x7f0f0c011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65f092ce92abcc00 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0efeffd640 1 -- 172.21.15.29:0/1828359723 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c01e070 con 0x55b2dd7321b0 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0efeffd640 1 -- 172.21.15.29:0/1828359723 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f0c00f9d0 con 0x55b2dd7321b0 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.304+0000 7f0efeffd640 1 -- 172.21.15.29:0/1828359723 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c019b40 con 0x55b2dd7321b0 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.305+0000 7f0f11ffb640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f040b6590 0x7f0f040b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.305+0000 7f0f18ece640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f040b6590 0x7f0f040b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.305+0000 7f0f11ffb640 1 -- 172.21.15.29:0/3532801816 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0f08107b80 con 0x7f0f141025a0 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 -- 172.21.15.29:0/1828359723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 msgr2=0x55b2ddae2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 --2- 172.21.15.29:0/1828359723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddae2660 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c011470 tx=0x7f0f0c011930 comp rx=0 tx=0).stop 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 -- 172.21.15.29:0/1828359723 shutdown_connections 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 --2- 172.21.15.29:0/1828359723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddae3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 --2- 172.21.15.29:0/1828359723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7092f0 0x55b2ddae2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 --2- 172.21.15.29:0/1828359723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddae2660 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 -- 172.21.15.29:0/1828359723 >> 172.21.15.29:0/1828359723 conn(0x55b2dd726300 msgr2=0x55b2ddad58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 -- 172.21.15.29:0/1828359723 shutdown_connections 2024-09-21T02:56:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f18ece640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f040b6590 0x7f0f040b8aa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0f080ce360 tx=0x7f0f080f5040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 -- 172.21.15.29:0/1828359723 wait complete. 2024-09-21T02:56:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 -- start start 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.306+0000 7f0f2209e740 1 Processor -- start 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 0x55b2ddaebe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaec4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7321b0 0x55b2ddaed050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f2209e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b2dd973ab0 con 0x55b2dd7321b0 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f2209e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b2dd9e1730 con 0x55b2dd7114d0 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f2209e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b2ddae4340 con 0x55b2dd7092f0 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f196cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 0x55b2ddaebe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaec4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaec4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46710/0 (socket says 172.21.15.29:46710) 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f13fff640 1 -- 172.21.15.29:0/2832575468 learned_addr learned my addr 172.21.15.29:0/2832575468 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f196cf640 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7321b0 0x55b2ddaed050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.307+0000 7f0f2209e740 10 client.0 _ll_register_callbacks cb 0x55b2dda356b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0f13fff640 1 -- 172.21.15.29:0/2832575468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 msgr2=0x55b2ddaebe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0f13fff640 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 0x55b2ddaebe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0f13fff640 1 -- 172.21.15.29:0/2832575468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7321b0 msgr2=0x55b2ddaed050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0f13fff640 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7321b0 0x55b2ddaed050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0f13fff640 1 -- 172.21.15.29:0/2832575468 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2dd79d790 con 0x55b2dd7114d0 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0f13fff640 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaec4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f00012740 tx=0x7f0f00012c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a9c825dc7f4261f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0ef3fff640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0ef3fff640 1 -- 172.21.15.29:0/2832575468 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0000f920 con 0x55b2dd7114d0 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0ef3fff640 1 -- 172.21.15.29:0/2832575468 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0f0000fac0 con 0x55b2dd7114d0 2024-09-21T02:56:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0f2209e740 1 -- 172.21.15.29:0/2832575468 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f0c00f9d0 con 0x55b2dd7114d0 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0ef3fff640 1 -- 172.21.15.29:0/2832575468 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0f0000fd00 con 0x55b2dd7114d0 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.308+0000 7f0f2209e740 10 client.27086 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0f2209e740 1 -- 172.21.15.29:0/2832575468 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b2dd816620 con 0x55b2dd7114d0 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0ef3fff640 1 -- 172.21.15.29:0/2832575468 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f0c00f9d0 con 0x55b2dd7114d0 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0ef3fff640 1 -- 172.21.15.29:0/2832575468 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f0f0001c420 con 0x55b2dd7114d0 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0f2209e740 10 client.27086 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0f2209e740 10 client.27086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0f2209e740 20 client.27086 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0f2209e740 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b2dd6f0d80 0x55b2ddaf9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0f2209e740 4 client.27086 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0f2209e740 1 -- 172.21.15.29:0/2832575468 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2dda4e640 con 0x55b2dd6f0d80 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.309+0000 7f0f196cf640 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b2dd6f0d80 0x55b2ddaf9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.310+0000 7f0f196cf640 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b2dd6f0d80 0x55b2ddaf9610 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f08d1aab3d72ff4f server_cookie=7fed9e5212b5c6d1 in_seq=0 out_seq=0 2024-09-21T02:56:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.310+0000 7f0ef3fff640 10 client.27086 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.311+0000 7f0ef3fff640 1 -- 172.21.15.29:0/2832575468 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b2dda4e640 con 0x55b2dd6f0d80 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.311+0000 7f0ef3fff640 10 client.27086 handle_command_reply: tid=0 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 client.27086 shutdown 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 -- 172.21.15.29:0/2832575468 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 msgr2=0x55b2ddaec4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaec4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f00012740 tx=0x7f0f00012c00 comp rx=0 tx=0).stop 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 -- 172.21.15.29:0/2832575468 shutdown_connections 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b2dd6f0d80 0x55b2ddaf9610 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7321b0 0x55b2ddaed050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaec4e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 --2- 172.21.15.29:0/2832575468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 0x55b2ddaebe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 -- 172.21.15.29:0/2832575468 >> 172.21.15.29:0/2832575468 conn(0x55b2dd726300 msgr2=0x55b2ddae36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0ef3fff640 0 client.27086 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 -- 172.21.15.29:0/2832575468 shutdown_connections 2024-09-21T02:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 1 -- 172.21.15.29:0/2832575468 wait complete. 2024-09-21T02:56:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.312+0000 7f0f2209e740 20 client.27086 trim_cache size 0 max 16384 2024-09-21T02:56:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.338+0000 7f0f2209e740 1 Processor -- start 2024-09-21T02:56:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f2209e740 1 -- start start 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7092f0 0x55b2ddaec530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaed050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddaed590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f2209e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f141aa300 con 0x55b2dd7092f0 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f2209e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0f141abe40 con 0x55b2dd7114d0 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f2209e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f141aa9d0 con 0x55b2dd7321b0 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f196cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7092f0 0x55b2ddaec530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f196cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7092f0 0x55b2ddaec530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41120/0 (socket says 172.21.15.29:41120) 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f196cf640 1 -- 172.21.15.29:0/1301922922 learned_addr learned my addr 172.21.15.29:0/1301922922 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f13fff640 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaed050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.339+0000 7f0f196cf640 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddaed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0f196cf640 1 -- 172.21.15.29:0/1301922922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 msgr2=0x55b2ddaed590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0f196cf640 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddaed590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0f196cf640 1 -- 172.21.15.29:0/1301922922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 msgr2=0x55b2ddaed050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0f196cf640 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaed050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0f196cf640 1 -- 172.21.15.29:0/1301922922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2dd6f0fb0 con 0x55b2dd7092f0 2024-09-21T02:56:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0f196cf640 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7092f0 0x55b2ddaec530 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c011440 tx=0x7f0f0c009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c5e67ef8285c45 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0efe7fc640 1 -- 172.21.15.29:0/1301922922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c030020 con 0x55b2dd7092f0 2024-09-21T02:56:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0efe7fc640 1 -- 172.21.15.29:0/1301922922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f0c003e50 con 0x55b2dd7092f0 2024-09-21T02:56:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.340+0000 7f0efe7fc640 1 -- 172.21.15.29:0/1301922922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c01e040 con 0x55b2dd7092f0 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.341+0000 7f0f2209e740 1 -- 172.21.15.29:0/1301922922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7092f0 msgr2=0x55b2ddaec530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.341+0000 7f0f2209e740 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7092f0 0x55b2ddaec530 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c011440 tx=0x7f0f0c009d20 comp rx=0 tx=0).stop 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.341+0000 7f0f2209e740 1 -- 172.21.15.29:0/1301922922 shutdown_connections 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.341+0000 7f0f2209e740 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7321b0 0x55b2ddaed590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.341+0000 7f0f2209e740 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7114d0 0x55b2ddaed050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.341+0000 7f0f2209e740 1 --2- 172.21.15.29:0/1301922922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7092f0 0x55b2ddaec530 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.341+0000 7f0f2209e740 1 -- 172.21.15.29:0/1301922922 >> 172.21.15.29:0/1301922922 conn(0x55b2dd726300 msgr2=0x55b2ddaebe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.342+0000 7f0f2209e740 1 -- 172.21.15.29:0/1301922922 shutdown_connections 2024-09-21T02:56:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.342+0000 7f0f2209e740 1 -- 172.21.15.29:0/1301922922 wait complete. 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.342+0000 7f0f2209e740 1 -- start start 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.342+0000 7f0f2209e740 1 Processor -- start 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 0x55b2ddaf0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddaf10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f2209e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7321b0 0x55b2ddaf1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f2209e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f141ab770 con 0x55b2dd7114d0 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f2209e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f0f141ab0a0 con 0x55b2dd7321b0 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f2209e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f141a82e0 con 0x55b2dd7092f0 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f2209e740 10 client.0 _ll_register_callbacks cb 0x7f0f00009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddaf10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f196cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 0x55b2ddaf0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddaf10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41126/0 (socket says 172.21.15.29:41126) 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 -- 172.21.15.29:0/40324917 learned_addr learned my addr 172.21.15.29:0/40324917 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f196cf640 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7321b0 0x55b2ddaf1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 -- 172.21.15.29:0/40324917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 msgr2=0x55b2ddaf0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 0x55b2ddaf0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 -- 172.21.15.29:0/40324917 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7321b0 msgr2=0x55b2ddaf1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7321b0 0x55b2ddaf1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 -- 172.21.15.29:0/40324917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2ddaf2340 con 0x55b2dd7114d0 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0f13fff640 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddaf10c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f0f0001c890 tx=0x7f0f0001c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43acaec3ed59db06 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.343+0000 7f0ef3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0ef3fff640 1 -- 172.21.15.29:0/40324917 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f00009e90 con 0x55b2dd7114d0 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0ef3fff640 1 -- 172.21.15.29:0/40324917 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f0f0001cd10 con 0x55b2dd7114d0 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0f2209e740 1 -- 172.21.15.29:0/40324917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f0c003e50 con 0x55b2dd7114d0 2024-09-21T02:56:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0ef3fff640 1 -- 172.21.15.29:0/40324917 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0f000050d0 con 0x55b2dd7114d0 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0f2209e740 10 client.17742 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0f2209e740 1 -- 172.21.15.29:0/40324917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b2ddaf3630 con 0x55b2dd7114d0 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0ef3fff640 1 -- 172.21.15.29:0/40324917 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0f00012e50 con 0x55b2dd7114d0 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0ef3fff640 1 -- 172.21.15.29:0/40324917 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f0f000038f0 con 0x55b2dd7114d0 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0f2209e740 10 client.17742 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0f2209e740 10 client.17742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0f2209e740 20 client.17742 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.344+0000 7f0f2209e740 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b2dd6c61c0 0x55b2ddb66020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.345+0000 7f0f2209e740 4 client.17742 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.345+0000 7f0f2209e740 1 -- 172.21.15.29:0/40324917 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f0f0c003e50 con 0x55b2dd6c61c0 2024-09-21T02:56:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.345+0000 7f0f196cf640 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b2dd6c61c0 0x55b2ddb66020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.345+0000 7f0ef3fff640 10 client.17742 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.345+0000 7f0f196cf640 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b2dd6c61c0 0x55b2ddb66020 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6db8d6194d27b07 server_cookie=6391b458576baa0 in_seq=0 out_seq=0 2024-09-21T02:56:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0ef3fff640 1 -- 172.21.15.29:0/40324917 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f0f0c003e50 con 0x55b2dd6c61c0 2024-09-21T02:56:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0ef3fff640 10 client.17742 handle_command_reply: tid=0 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 client.17742 shutdown 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 -- 172.21.15.29:0/40324917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 msgr2=0x55b2ddaf10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddaf10c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f0f0001c890 tx=0x7f0f0001c6d0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 -- 172.21.15.29:0/40324917 shutdown_connections 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b2dd7321b0 0x55b2ddaf1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b2dd6c61c0 0x55b2ddb66020 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2dd7114d0 0x55b2ddaf10c0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 --2- 172.21.15.29:0/40324917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2dd7092f0 0x55b2ddaf0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0ef3fff640 0 client.17742 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 -- 172.21.15.29:0/40324917 >> 172.21.15.29:0/40324917 conn(0x55b2dd726300 msgr2=0x55b2ddaea1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 -- 172.21.15.29:0/40324917 shutdown_connections 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 1 -- 172.21.15.29:0/40324917 wait complete. 2024-09-21T02:56:02.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.537+0000 7f0f2209e740 20 client.17742 trim_cache size 0 max 16384 2024-09-21T02:56:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:56:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:56:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:56:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 -- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f040b6590 msgr2=0x7f0f040b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f040b6590 0x7f0f040b8aa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0f080ce360 tx=0x7f0f080f5040 comp rx=0 tx=0).stop 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 -- 172.21.15.29:0/3532801816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 msgr2=0x7f0f14199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 0x7f0f14199e70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f08005760 tx=0x7f0f08005e50 comp rx=0 tx=0).stop 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 -- 172.21.15.29:0/3532801816 shutdown_connections 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f0f040b6590 0x7f0f040b8aa0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f14103fc0 0x7f0f1419b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14103620 0x7f0f1419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 --2- 172.21.15.29:0/3532801816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f141025a0 0x7f0f14199e70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 -- 172.21.15.29:0/3532801816 >> 172.21.15.29:0/3532801816 conn(0x7f0f140fdee0 msgr2=0x7f0f1406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 -- 172.21.15.29:0/3532801816 shutdown_connections 2024-09-21T02:56:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:02.538+0000 7f0f1b132640 1 -- 172.21.15.29:0/3532801816 wait complete. 2024-09-21T02:56:02.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23864 5.95 MB/sec execute 99 sec latency 222.651 ms 2024-09-21T02:56:03.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23985 5.95 MB/sec execute 100 sec latency 97.446 ms 2024-09-21T02:56:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:03 smithi190 ceph-mon[33871]: pgmap v897: 129 pgs: 129 active+clean; 269 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 3.3 MiB/s wr, 287 op/s 2024-09-21T02:56:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:03 smithi029 ceph-mon[31006]: pgmap v897: 129 pgs: 129 active+clean; 269 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 3.3 MiB/s wr, 287 op/s 2024-09-21T02:56:04.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:03 smithi066 ceph-mon[33436]: pgmap v897: 129 pgs: 129 active+clean; 269 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 3.3 MiB/s wr, 287 op/s 2024-09-21T02:56:04.723 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24124 5.99 MB/sec execute 101 sec latency 90.037 ms 2024-09-21T02:56:05.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24210 5.94 MB/sec execute 102 sec latency 107.041 ms 2024-09-21T02:56:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:05 smithi190 ceph-mon[33871]: pgmap v898: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.6 MiB/s wr, 277 op/s 2024-09-21T02:56:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:05 smithi029 ceph-mon[31006]: pgmap v898: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.6 MiB/s wr, 277 op/s 2024-09-21T02:56:06.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:05 smithi066 ceph-mon[33436]: pgmap v898: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.6 MiB/s wr, 277 op/s 2024-09-21T02:56:06.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24383 6.04 MB/sec execute 103 sec latency 103.655 ms 2024-09-21T02:56:07.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24531 6.07 MB/sec execute 104 sec latency 100.175 ms 2024-09-21T02:56:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:07 smithi190 ceph-mon[33871]: pgmap v899: 129 pgs: 129 active+clean; 280 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 3.8 MiB/s wr, 270 op/s 2024-09-21T02:56:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:07 smithi029 ceph-mon[31006]: pgmap v899: 129 pgs: 129 active+clean; 280 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 3.8 MiB/s wr, 270 op/s 2024-09-21T02:56:08.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:07 smithi066 ceph-mon[33436]: pgmap v899: 129 pgs: 129 active+clean; 280 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 3.8 MiB/s wr, 270 op/s 2024-09-21T02:56:08.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24691 6.13 MB/sec execute 105 sec latency 98.529 ms 2024-09-21T02:56:09.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24855 6.15 MB/sec execute 106 sec latency 100.335 ms 2024-09-21T02:56:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:09 smithi190 ceph-mon[33871]: pgmap v900: 129 pgs: 129 active+clean; 277 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 4.3 MiB/s wr, 296 op/s 2024-09-21T02:56:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:09 smithi029 ceph-mon[31006]: pgmap v900: 129 pgs: 129 active+clean; 277 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 4.3 MiB/s wr, 296 op/s 2024-09-21T02:56:10.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:09 smithi066 ceph-mon[33436]: pgmap v900: 129 pgs: 129 active+clean; 277 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 4.3 MiB/s wr, 296 op/s 2024-09-21T02:56:10.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24992 6.16 MB/sec execute 107 sec latency 89.548 ms 2024-09-21T02:56:11.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25083 6.12 MB/sec execute 108 sec latency 130.855 ms 2024-09-21T02:56:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:11 smithi190 ceph-mon[33871]: pgmap v901: 129 pgs: 129 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 4.4 MiB/s wr, 266 op/s 2024-09-21T02:56:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:11 smithi029 ceph-mon[31006]: pgmap v901: 129 pgs: 129 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 4.4 MiB/s wr, 266 op/s 2024-09-21T02:56:12.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:11 smithi066 ceph-mon[33436]: pgmap v901: 129 pgs: 129 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 4.4 MiB/s wr, 266 op/s 2024-09-21T02:56:12.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25234 6.14 MB/sec execute 109 sec latency 98.544 ms 2024-09-21T02:56:13.663 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-09-21T02:56:13.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:56:13.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25374 6.13 MB/sec execute 110 sec latency 133.353 ms 2024-09-21T02:56:13.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:13 smithi190 ceph-mon[33871]: pgmap v902: 129 pgs: 129 active+clean; 282 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-09-21T02:56:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:13 smithi029 ceph-mon[31006]: pgmap v902: 129 pgs: 129 active+clean; 282 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-09-21T02:56:14.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:13 smithi066 ceph-mon[33436]: pgmap v902: 129 pgs: 129 active+clean; 282 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-09-21T02:56:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:56:14.721 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25480 6.11 MB/sec execute 111 sec latency 165.294 ms 2024-09-21T02:56:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 -- 172.21.15.29:0/1337727396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c1056d0 msgr2=0x7f304c107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 --2- 172.21.15.29:0/1337727396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c1056d0 0x7f304c107bb0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f304000d910 tx=0x7f304002c380 comp rx=0 tx=0).stop 2024-09-21T02:56:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 -- 172.21.15.29:0/1337727396 shutdown_connections 2024-09-21T02:56:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 --2- 172.21.15.29:0/1337727396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f304c1082f0 0x7f304c10a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 --2- 172.21.15.29:0/1337727396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c1056d0 0x7f304c107bb0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 --2- 172.21.15.29:0/1337727396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f304c102c90 0x7f304c1050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 -- 172.21.15.29:0/1337727396 >> 172.21.15.29:0/1337727396 conn(0x7f304c0fd680 msgr2=0x7f304c0ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 -- 172.21.15.29:0/1337727396 shutdown_connections 2024-09-21T02:56:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.660+0000 7f305237b640 1 -- 172.21.15.29:0/1337727396 wait complete. 2024-09-21T02:56:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.661+0000 7f305237b640 1 Processor -- start 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.661+0000 7f305237b640 1 -- start start 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f305237b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c102c90 0x7f304c197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f305237b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f304c1056d0 0x7f304c198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f305237b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f304c1082f0 0x7f304c198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f305237b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f304c10d000 con 0x7f304c102c90 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f305237b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f304c10ce80 con 0x7f304c1056d0 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f305237b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f304c10d180 con 0x7f304c1082f0 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c102c90 0x7f304c197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c102c90 0x7f304c197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40168/0 (socket says 172.21.15.29:40168) 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f304c1056d0 0x7f304c198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304bfff640 1 -- 172.21.15.29:0/1826079805 learned_addr learned my addr 172.21.15.29:0/1826079805 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f3050918640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f304c1082f0 0x7f304c198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304bfff640 1 -- 172.21.15.29:0/1826079805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f304c1082f0 msgr2=0x7f304c198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304bfff640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f304c1082f0 0x7f304c198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304bfff640 1 -- 172.21.15.29:0/1826079805 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f304c1056d0 msgr2=0x7f304c198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304bfff640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f304c1056d0 0x7f304c198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304bfff640 1 -- 172.21.15.29:0/1826079805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304c1996e0 con 0x7f304c102c90 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.662+0000 7f304b7fe640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f304c1056d0 0x7f304c198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.663+0000 7f304bfff640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c102c90 0x7f304c197ca0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f303c00ea60 tx=0x7f303c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.663+0000 7f30497fa640 1 -- 172.21.15.29:0/1826079805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c00cd20 con 0x7f304c102c90 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.663+0000 7f305237b640 1 -- 172.21.15.29:0/1826079805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304c1999d0 con 0x7f304c102c90 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.663+0000 7f30497fa640 1 -- 172.21.15.29:0/1826079805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f303c0035f0 con 0x7f304c102c90 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.663+0000 7f30497fa640 1 -- 172.21.15.29:0/1826079805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c010690 con 0x7f304c102c90 2024-09-21T02:56:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.663+0000 7f305237b640 1 -- 172.21.15.29:0/1826079805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304c1a00c0 con 0x7f304c102c90 2024-09-21T02:56:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.666+0000 7f30593c4740 1 Processor -- start 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.667+0000 7f30497fa640 1 -- 172.21.15.29:0/1826079805 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f303c004000 con 0x7f304c102c90 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.667+0000 7f30593c4740 1 -- start start 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.667+0000 7f30593c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d2681e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.667+0000 7f30593c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26824c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.667+0000 7f30593c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d2683010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.667+0000 7f30593c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f2d25f46f0 con 0x55f2d22da1b0 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.667+0000 7f30593c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f2d25e3b20 con 0x55f2d22b12f0 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.667+0000 7f30593c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2d25e7930 con 0x55f2d22b94d0 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.668+0000 7f3050918640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d2681e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.668+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d2683010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.668+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d2683010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49806/0 (socket says 172.21.15.29:49806) 2024-09-21T02:56:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.668+0000 7f3050918640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d2681e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40172/0 (socket says 172.21.15.29:40172) 2024-09-21T02:56:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.668+0000 7f304b7fe640 1 -- 172.21.15.29:0/1815224906 learned_addr learned my addr 172.21.15.29:0/1815224906 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.668+0000 7f304bfff640 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26824c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.668+0000 7f3050918640 1 -- 172.21.15.29:0/1815224906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 msgr2=0x55f2d2683010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f30497fa640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30340ba920 0x7f30340bce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f3050918640 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d2683010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f3050918640 1 -- 172.21.15.29:0/1815224906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 msgr2=0x55f2d26824c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f3050918640 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26824c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f3050918640 1 -- 172.21.15.29:0/1815224906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d25a9530 con 0x55f2d22da1b0 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f304bfff640 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26824c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f304b7fe640 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d2683010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f30497fa640 1 -- 172.21.15.29:0/1826079805 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f303c0e7910 con 0x7f304c102c90 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f3050918640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30340ba920 0x7f30340bce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f3050918640 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d2681e50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f3044009f10 tx=0x7f3044011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4936ca023360caa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.669+0000 7f30327fc640 1 -- 172.21.15.29:0/1815224906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3044004030 con 0x55f2d22da1b0 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.670+0000 7f30327fc640 1 -- 172.21.15.29:0/1815224906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30440047e0 con 0x55f2d22da1b0 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.670+0000 7f30327fc640 1 -- 172.21.15.29:0/1815224906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304401c400 con 0x55f2d22da1b0 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 -- 172.21.15.29:0/1815224906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 msgr2=0x55f2d2681e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f3050918640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30340ba920 0x7f30340bce30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3044020000 tx=0x7f3044021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d2681e50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f3044009f10 tx=0x7f3044011810 comp rx=0 tx=0).stop 2024-09-21T02:56:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 -- 172.21.15.29:0/1815224906 shutdown_connections 2024-09-21T02:56:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d2683010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26824c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 --2- 172.21.15.29:0/1815224906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d2681e50 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 -- 172.21.15.29:0/1815224906 >> 172.21.15.29:0/1815224906 conn(0x55f2d22ce300 msgr2=0x55f2d267d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 -- 172.21.15.29:0/1815224906 shutdown_connections 2024-09-21T02:56:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.671+0000 7f30593c4740 1 -- 172.21.15.29:0/1815224906 wait complete. 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.672+0000 7f30593c4740 1 -- start start 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.672+0000 7f30593c4740 1 Processor -- start 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f30593c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d269c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f30593c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d269cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f30593c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d269d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f30593c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f2d251bab0 con 0x55f2d22da1b0 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f30593c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f2d2589730 con 0x55f2d22b12f0 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f30593c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2d2683b30 con 0x55f2d22b94d0 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f30593c4740 10 client.0 _ll_register_callbacks cb 0x55f2d25dd6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f3050918640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d269d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f3050918640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d269d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40184/0 (socket says 172.21.15.29:40184) 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f3050918640 1 -- 172.21.15.29:0/3528144619 learned_addr learned my addr 172.21.15.29:0/3528144619 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.673+0000 7f304b7fe640 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d269cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.674+0000 7f304bfff640 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d269c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.674+0000 7f304b7fe640 1 -- 172.21.15.29:0/3528144619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 msgr2=0x55f2d269c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.674+0000 7f304b7fe640 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d269c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.674+0000 7f304b7fe640 1 -- 172.21.15.29:0/3528144619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 msgr2=0x55f2d269d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.674+0000 7f304b7fe640 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d269d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.674+0000 7f304b7fe640 1 -- 172.21.15.29:0/3528144619 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d2345790 con 0x55f2d22b94d0 2024-09-21T02:56:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.674+0000 7f304b7fe640 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d269cca0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3040009230 tx=0x7f3040004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a216bbc87cac94e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.674+0000 7f302b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.675+0000 7f302b7fe640 1 -- 172.21.15.29:0/3528144619 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040043020 con 0x55f2d22b94d0 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.675+0000 7f30593c4740 1 -- 172.21.15.29:0/3528144619 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30440047e0 con 0x55f2d22b94d0 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.675+0000 7f302b7fe640 1 -- 172.21.15.29:0/3528144619 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f30400038c0 con 0x55f2d22b94d0 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.675+0000 7f302b7fe640 1 -- 172.21.15.29:0/3528144619 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f304004e050 con 0x55f2d22b94d0 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.676+0000 7f30593c4740 10 client.27244 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.676+0000 7f30593c4740 1 -- 172.21.15.29:0/3528144619 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f2d23be620 con 0x55f2d22b94d0 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.676+0000 7f302b7fe640 1 -- 172.21.15.29:0/3528144619 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30440047e0 con 0x55f2d22b94d0 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.676+0000 7f302b7fe640 1 -- 172.21.15.29:0/3528144619 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f304003fc30 con 0x55f2d22b94d0 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.676+0000 7f30593c4740 10 client.27244 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.676+0000 7f30593c4740 10 client.27244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.677+0000 7f30593c4740 20 client.27244 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.677+0000 7f30593c4740 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2d2298d80 0x55f2d26a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.677+0000 7f30593c4740 4 client.27244 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:56:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.677+0000 7f30593c4740 1 -- 172.21.15.29:0/3528144619 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2d25f6640 con 0x55f2d2298d80 2024-09-21T02:56:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.677+0000 7f304bfff640 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2d2298d80 0x55f2d26a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.678+0000 7f304bfff640 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2d2298d80 0x55f2d26a9e10 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd40841bd843e71f server_cookie=8f35a4ccd8d8e8b3 in_seq=0 out_seq=0 2024-09-21T02:56:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.678+0000 7f302b7fe640 10 client.27244 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.679+0000 7f302b7fe640 1 -- 172.21.15.29:0/3528144619 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f2d25f6640 con 0x55f2d2298d80 2024-09-21T02:56:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.679+0000 7f302b7fe640 10 client.27244 handle_command_reply: tid=0 2024-09-21T02:56:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.679+0000 7f30593c4740 1 client.27244 shutdown 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 -- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 msgr2=0x55f2d269cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d269cca0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3040009230 tx=0x7f3040004740 comp rx=0 tx=0).stop 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 -- 172.21.15.29:0/3528144619 shutdown_connections 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2d2298d80 0x55f2d26a9e10 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d269d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d269cca0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 --2- 172.21.15.29:0/3528144619 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d269c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 -- 172.21.15.29:0/3528144619 >> 172.21.15.29:0/3528144619 conn(0x55f2d22ce300 msgr2=0x55f2d267bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f302b7fe640 0 client.27244 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 -- 172.21.15.29:0/3528144619 shutdown_connections 2024-09-21T02:56:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 1 -- 172.21.15.29:0/3528144619 wait complete. 2024-09-21T02:56:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.680+0000 7f30593c4740 20 client.27244 trim_cache size 0 max 16384 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.709+0000 7f30593c4740 1 Processor -- start 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.709+0000 7f30593c4740 1 -- start start 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f30593c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f30593c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22b94d0 0x55f2d267d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f30593c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22da1b0 0x55f2d269b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f30593c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f304c1a1b50 con 0x55f2d22b94d0 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f30593c4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f304c1a3690 con 0x55f2d22b12f0 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f30593c4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f304c1a2220 con 0x55f2d22da1b0 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22b94d0 0x55f2d267d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22b94d0 0x55f2d267d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40194/0 (socket says 172.21.15.29:40194) 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304b7fe640 1 -- 172.21.15.29:0/2998958770 learned_addr learned my addr 172.21.15.29:0/2998958770 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f3050918640 1 --2- 172.21.15.29:0/2998958770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22da1b0 0x55f2d269b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304b7fe640 1 -- 172.21.15.29:0/2998958770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22da1b0 msgr2=0x55f2d269b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304b7fe640 1 --2- 172.21.15.29:0/2998958770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22da1b0 0x55f2d269b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304b7fe640 1 -- 172.21.15.29:0/2998958770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 msgr2=0x55f2d26a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304b7fe640 1 --2- 172.21.15.29:0/2998958770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.710+0000 7f304b7fe640 1 -- 172.21.15.29:0/2998958770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d26790a0 con 0x55f2d22b94d0 2024-09-21T02:56:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.711+0000 7f304b7fe640 1 --2- 172.21.15.29:0/2998958770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22b94d0 0x55f2d267d220 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f3040004650 tx=0x7f304002c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f783eea212ec77 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.711+0000 7f3031ffb640 1 -- 172.21.15.29:0/2998958770 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040043040 con 0x55f2d22b94d0 2024-09-21T02:56:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.711+0000 7f3031ffb640 1 -- 172.21.15.29:0/2998958770 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3040057070 con 0x55f2d22b94d0 2024-09-21T02:56:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.711+0000 7f3031ffb640 1 -- 172.21.15.29:0/2998958770 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040003b20 con 0x55f2d22b94d0 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.712+0000 7f30593c4740 1 -- 172.21.15.29:0/2998958770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22b94d0 msgr2=0x55f2d267d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.712+0000 7f30593c4740 1 --2- 172.21.15.29:0/2998958770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22b94d0 0x55f2d267d220 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f3040004650 tx=0x7f304002c380 comp rx=0 tx=0).stop 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.712+0000 7f30593c4740 1 -- 172.21.15.29:0/2998958770 shutdown_connections 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.712+0000 7f30593c4740 1 --2- 172.21.15.29:0/2998958770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22da1b0 0x55f2d269b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.712+0000 7f30593c4740 1 --2- 172.21.15.29:0/2998958770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22b94d0 0x55f2d267d220 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.712+0000 7f30593c4740 1 --2- 172.21.15.29:0/2998958770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.712+0000 7f30593c4740 1 -- 172.21.15.29:0/2998958770 >> 172.21.15.29:0/2998958770 conn(0x55f2d22ce300 msgr2=0x55f2d269afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.713+0000 7f30593c4740 1 -- 172.21.15.29:0/2998958770 shutdown_connections 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.713+0000 7f30593c4740 1 -- 172.21.15.29:0/2998958770 wait complete. 2024-09-21T02:56:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.713+0000 7f30593c4740 1 -- start start 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.714+0000 7f30593c4740 1 Processor -- start 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.714+0000 7f30593c4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.714+0000 7f30593c4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d26a2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.714+0000 7f304bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.714+0000 7f304bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:37914/0 (socket says 172.21.15.29:37914) 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.715+0000 7f304bfff640 1 -- 172.21.15.29:0/2885814315 learned_addr learned my addr 172.21.15.29:0/2885814315 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.715+0000 7f30593c4740 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d26a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.715+0000 7f304b7fe640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d26a2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.715+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f304c1a2fc0 con 0x55f2d22da1b0 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.715+0000 7f3050918640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d26a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.715+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f304c1a28f0 con 0x55f2d22b12f0 2024-09-21T02:56:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.715+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f304c074e80 con 0x55f2d22b94d0 2024-09-21T02:56:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.716+0000 7f30593c4740 10 client.0 _ll_register_callbacks cb 0x7f304000d790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.716+0000 7f304bfff640 1 -- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 msgr2=0x55f2d26a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.716+0000 7f304bfff640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d26a2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.716+0000 7f304bfff640 1 -- 172.21.15.29:0/2885814315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 msgr2=0x55f2d26a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.716+0000 7f304bfff640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d26a2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.716+0000 7f304bfff640 1 -- 172.21.15.29:0/2885814315 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d26a3350 con 0x55f2d22b12f0 2024-09-21T02:56:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.716+0000 7f304b7fe640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d26a2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:15.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.716+0000 7f3050918640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d26a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.717+0000 7f304bfff640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a19c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f303c0a9cf0 tx=0x7f303c0e7d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd1e400f6589e648 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.717+0000 7f302b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:56:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.717+0000 7f302b7fe640 1 -- 172.21.15.29:0/2885814315 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0ad070 con 0x55f2d22b12f0 2024-09-21T02:56:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.717+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3040057070 con 0x55f2d22b12f0 2024-09-21T02:56:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.718+0000 7f302b7fe640 1 -- 172.21.15.29:0/2885814315 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f303c0a9990 con 0x55f2d22b12f0 2024-09-21T02:56:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.718+0000 7f302b7fe640 1 -- 172.21.15.29:0/2885814315 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f303c0042f0 con 0x55f2d22b12f0 2024-09-21T02:56:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.718+0000 7f30593c4740 10 client.27110 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.718+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f2d26a46b0 con 0x55f2d22b12f0 2024-09-21T02:56:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.718+0000 7f302b7fe640 1 -- 172.21.15.29:0/2885814315 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3040057070 con 0x55f2d22b12f0 2024-09-21T02:56:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.719+0000 7f302b7fe640 1 -- 172.21.15.29:0/2885814315 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f303c0ac3f0 con 0x55f2d22b12f0 2024-09-21T02:56:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.719+0000 7f30593c4740 10 client.27110 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.719+0000 7f30593c4740 10 client.27110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.719+0000 7f30593c4740 20 client.27110 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.719+0000 7f30593c4740 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2d226e1c0 0x55f2d2716ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.719+0000 7f30593c4740 4 client.27110 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:56:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.719+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2d25f20f0 con 0x55f2d226e1c0 2024-09-21T02:56:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.719+0000 7f304b7fe640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2d226e1c0 0x55f2d2716ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.720+0000 7f302b7fe640 10 client.27110 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.720+0000 7f304b7fe640 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2d226e1c0 0x55f2d2716ba0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc9a4b7f793fe699 server_cookie=9c7606691f684308 in_seq=0 out_seq=0 2024-09-21T02:56:15.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25596 6.11 MB/sec execute 112 sec latency 248.643 ms 2024-09-21T02:56:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:15 smithi190 ceph-mon[33871]: pgmap v903: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 4.6 MiB/s wr, 287 op/s 2024-09-21T02:56:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:15 smithi029 ceph-mon[31006]: pgmap v903: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 4.6 MiB/s wr, 287 op/s 2024-09-21T02:56:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.891+0000 7f302b7fe640 1 -- 172.21.15.29:0/2885814315 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f2d25f20f0 con 0x55f2d226e1c0 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.891+0000 7f302b7fe640 10 client.27110 handle_command_reply: tid=0 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.891+0000 7f30593c4740 1 client.27110 shutdown 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 msgr2=0x55f2d26a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a19c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f303c0a9cf0 tx=0x7f303c0e7d00 comp rx=0 tx=0).stop 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 shutdown_connections 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f2d226e1c0 0x55f2d2716ba0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2d22da1b0 0x55f2d26a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2d22b94d0 0x55f2d26a2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f302b7fe640 0 client.27110 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 --2- 172.21.15.29:0/2885814315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f2d22b12f0 0x55f2d26a19c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 >> 172.21.15.29:0/2885814315 conn(0x55f2d22ce300 msgr2=0x55f2d267d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 shutdown_connections 2024-09-21T02:56:15.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 1 -- 172.21.15.29:0/2885814315 wait complete. 2024-09-21T02:56:15.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.892+0000 7f30593c4740 20 client.27110 trim_cache size 0 max 16384 2024-09-21T02:56:15.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:56:15.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:56:15.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:56:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:56:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.893+0000 7f305237b640 1 -- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30340ba920 msgr2=0x7f30340bce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.893+0000 7f305237b640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30340ba920 0x7f30340bce30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3044020000 tx=0x7f3044021000 comp rx=0 tx=0).stop 2024-09-21T02:56:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.893+0000 7f305237b640 1 -- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c102c90 msgr2=0x7f304c197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.893+0000 7f305237b640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c102c90 0x7f304c197ca0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f303c00ea60 tx=0x7f303c00ee60 comp rx=0 tx=0).stop 2024-09-21T02:56:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.894+0000 7f305237b640 1 -- 172.21.15.29:0/1826079805 shutdown_connections 2024-09-21T02:56:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.894+0000 7f305237b640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30340ba920 0x7f30340bce30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.894+0000 7f305237b640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f304c1082f0 0x7f304c198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.894+0000 7f305237b640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f304c1056d0 0x7f304c198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.894+0000 7f305237b640 1 --2- 172.21.15.29:0/1826079805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c102c90 0x7f304c197ca0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.894+0000 7f305237b640 1 -- 172.21.15.29:0/1826079805 >> 172.21.15.29:0/1826079805 conn(0x7f304c0fd680 msgr2=0x7f304c103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.894+0000 7f305237b640 1 -- 172.21.15.29:0/1826079805 shutdown_connections 2024-09-21T02:56:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:15.894+0000 7f305237b640 1 -- 172.21.15.29:0/1826079805 wait complete. 2024-09-21T02:56:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:15 smithi066 ceph-mon[33436]: pgmap v903: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 4.6 MiB/s wr, 287 op/s 2024-09-21T02:56:16.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25722 6.11 MB/sec execute 113 sec latency 215.321 ms 2024-09-21T02:56:17.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25812 6.08 MB/sec execute 114 sec latency 258.549 ms 2024-09-21T02:56:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:17 smithi190 ceph-mon[33871]: pgmap v904: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 4.0 MiB/s wr, 258 op/s 2024-09-21T02:56:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:17 smithi029 ceph-mon[31006]: pgmap v904: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 4.0 MiB/s wr, 258 op/s 2024-09-21T02:56:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:17 smithi066 ceph-mon[33436]: pgmap v904: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 4.0 MiB/s wr, 258 op/s 2024-09-21T02:56:18.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25939 6.16 MB/sec execute 115 sec latency 287.211 ms 2024-09-21T02:56:19.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26019 6.14 MB/sec execute 116 sec latency 358.622 ms 2024-09-21T02:56:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:19 smithi190 ceph-mon[33871]: pgmap v905: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 4.0 MiB/s wr, 268 op/s 2024-09-21T02:56:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:19 smithi029 ceph-mon[31006]: pgmap v905: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 4.0 MiB/s wr, 268 op/s 2024-09-21T02:56:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:19 smithi066 ceph-mon[33436]: pgmap v905: 129 pgs: 129 active+clean; 275 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 4.0 MiB/s wr, 268 op/s 2024-09-21T02:56:20.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26094 6.09 MB/sec execute 117 sec latency 408.349 ms 2024-09-21T02:56:21.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26186 6.07 MB/sec execute 118 sec latency 318.912 ms 2024-09-21T02:56:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:21 smithi190 ceph-mon[33871]: pgmap v906: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 3.8 MiB/s wr, 266 op/s 2024-09-21T02:56:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:21 smithi029 ceph-mon[31006]: pgmap v906: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 3.8 MiB/s wr, 266 op/s 2024-09-21T02:56:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:21 smithi066 ceph-mon[33436]: pgmap v906: 129 pgs: 129 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 3.8 MiB/s wr, 266 op/s 2024-09-21T02:56:22.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26294 6.08 MB/sec execute 119 sec latency 351.663 ms 2024-09-21T02:56:22.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:22 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3602100968' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:56:22.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:22 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3602100968' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:56:23.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:22 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3602100968' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:56:23.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26432 6.04 MB/sec execute 120 sec latency 144.264 ms 2024-09-21T02:56:23.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:56:23.723 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:56:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 -- 172.21.15.29:0/916082443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb564102a60 msgr2=0x7fb564102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 --2- 172.21.15.29:0/916082443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb564102a60 0x7fb564102e70 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009ca0 tx=0x7fb55c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 -- 172.21.15.29:0/916082443 shutdown_connections 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 --2- 172.21.15.29:0/916082443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb564108cf0 0x7fb56410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 --2- 172.21.15.29:0/916082443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb5641083a0 0x7fb5641087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 --2- 172.21.15.29:0/916082443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb564102a60 0x7fb564102e70 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 -- 172.21.15.29:0/916082443 >> 172.21.15.29:0/916082443 conn(0x7fb564079730 msgr2=0x7fb564079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 -- 172.21.15.29:0/916082443 shutdown_connections 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.364+0000 7fb56aac4640 1 -- 172.21.15.29:0/916082443 wait complete. 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.365+0000 7fb56aac4640 1 Processor -- start 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.365+0000 7fb56aac4640 1 -- start start 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.365+0000 7fb56aac4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb564102a60 0x7fb56419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb56aac4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5641083a0 0x7fb56419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb56aac4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb564108cf0 0x7fb56419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb56aac4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb564112860 con 0x7fb5641083a0 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb56aac4640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641126e0 con 0x7fb564108cf0 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb56aac4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641129e0 con 0x7fb564102a60 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5641083a0 0x7fb56419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb568860640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb564102a60 0x7fb56419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5641083a0 0x7fb56419a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40216/0 (socket says 172.21.15.29:40216) 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 -- 172.21.15.29:0/3602100968 learned_addr learned my addr 172.21.15.29:0/3602100968 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb569061640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb564108cf0 0x7fb56419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 -- 172.21.15.29:0/3602100968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb564102a60 msgr2=0x7fb56419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb564102a60 0x7fb56419a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 -- 172.21.15.29:0/3602100968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb564108cf0 msgr2=0x7fb56419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb564108cf0 0x7fb56419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 -- 172.21.15.29:0/3602100968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5641a02d0 con 0x7fb5641083a0 2024-09-21T02:56:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb563fff640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5641083a0 0x7fb56419a900 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb55400d770 tx=0x7fb55400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.366+0000 7fb561ffb640 1 -- 172.21.15.29:0/3602100968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb554017070 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.367+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5641a05c0 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.367+0000 7fb561ffb640 1 -- 172.21.15.29:0/3602100968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb554009c30 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.367+0000 7fb561ffb640 1 -- 172.21.15.29:0/3602100968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55400f620 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.367+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5641a6d20 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.368+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb564102e70 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.370+0000 7fb561ffb640 1 -- 172.21.15.29:0/3602100968 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb55401a070 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.370+0000 7fb561ffb640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb5500b6510 0x7fb5500b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.370+0000 7fb561ffb640 1 -- 172.21.15.29:0/3602100968 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb5540e70d0 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.370+0000 7fb568860640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb5500b6510 0x7fb5500b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.371+0000 7fb568860640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb5500b6510 0x7fb5500b8a20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb55c000f80 tx=0x7fb55c031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.374+0000 7fb561ffb640 1 -- 172.21.15.29:0/3602100968 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb554009de0 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.638+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb56419b490 con 0x7fb5641083a0 2024-09-21T02:56:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.639+0000 7fb561ffb640 1 -- 172.21.15.29:0/3602100968 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7fb5540b19c0 con 0x7fb5641083a0 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb5500b6510 msgr2=0x7fb5500b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb5500b6510 0x7fb5500b8a20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb55c000f80 tx=0x7fb55c031000 comp rx=0 tx=0).stop 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5641083a0 msgr2=0x7fb56419a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5641083a0 0x7fb56419a900 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb55400d770 tx=0x7fb55400dc30 comp rx=0 tx=0).stop 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 shutdown_connections 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb5500b6510 0x7fb5500b8a20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb564108cf0 0x7fb56419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5641083a0 0x7fb56419a900 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 --2- 172.21.15.29:0/3602100968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb564102a60 0x7fb56419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 >> 172.21.15.29:0/3602100968 conn(0x7fb564079730 msgr2=0x7fb5641019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 shutdown_connections 2024-09-21T02:56:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:21.642+0000 7fb56aac4640 1 -- 172.21.15.29:0/3602100968 wait complete. 2024-09-21T02:56:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:23 smithi190 ceph-mon[33871]: pgmap v907: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 3.4 MiB/s wr, 286 op/s 2024-09-21T02:56:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:23 smithi029 ceph-mon[31006]: pgmap v907: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 3.4 MiB/s wr, 286 op/s 2024-09-21T02:56:24.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:23 smithi066 ceph-mon[33436]: pgmap v907: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 3.4 MiB/s wr, 286 op/s 2024-09-21T02:56:24.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26568 6.01 MB/sec execute 121 sec latency 244.130 ms 2024-09-21T02:56:25.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26681 5.98 MB/sec execute 122 sec latency 243.011 ms 2024-09-21T02:56:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:25 smithi190 ceph-mon[33871]: pgmap v908: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 3.0 MiB/s wr, 308 op/s 2024-09-21T02:56:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:25 smithi029 ceph-mon[31006]: pgmap v908: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 3.0 MiB/s wr, 308 op/s 2024-09-21T02:56:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:25 smithi066 ceph-mon[33436]: pgmap v908: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 3.0 MiB/s wr, 308 op/s 2024-09-21T02:56:26.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26754 5.94 MB/sec execute 123 sec latency 335.396 ms 2024-09-21T02:56:27.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26903 5.98 MB/sec execute 124 sec latency 368.740 ms 2024-09-21T02:56:27.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:27 smithi190 ceph-mon[33871]: pgmap v909: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.2 MiB/s wr, 288 op/s 2024-09-21T02:56:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:27 smithi029 ceph-mon[31006]: pgmap v909: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.2 MiB/s wr, 288 op/s 2024-09-21T02:56:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:27 smithi066 ceph-mon[33436]: pgmap v909: 129 pgs: 129 active+clean; 271 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.2 MiB/s wr, 288 op/s 2024-09-21T02:56:28.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27037 5.97 MB/sec execute 125 sec latency 261.698 ms 2024-09-21T02:56:28.827 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-09-21T02:56:28.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-09-21T02:56:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.034+0000 7fc87c320640 1 -- 172.21.15.29:0/3306573854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874102dd0 msgr2=0x7fc8741031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.034+0000 7fc87c320640 1 --2- 172.21.15.29:0/3306573854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874102dd0 0x7fc8741031e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fc86400aa10 tx=0x7fc86402b720 comp rx=0 tx=0).stop 2024-09-21T02:56:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.034+0000 7fc87c320640 1 -- 172.21.15.29:0/3306573854 shutdown_connections 2024-09-21T02:56:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.034+0000 7fc87c320640 1 --2- 172.21.15.29:0/3306573854 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874103770 0x7fc87410a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.034+0000 7fc87c320640 1 --2- 172.21.15.29:0/3306573854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874102dd0 0x7fc8741031e0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.034+0000 7fc87c320640 1 --2- 172.21.15.29:0/3306573854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc874101d50 0x7fc8741021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.034+0000 7fc87c320640 1 -- 172.21.15.29:0/3306573854 >> 172.21.15.29:0/3306573854 conn(0x7fc8740fd690 msgr2=0x7fc8740ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.035+0000 7fc87c320640 1 -- 172.21.15.29:0/3306573854 shutdown_connections 2024-09-21T02:56:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.035+0000 7fc87c320640 1 -- 172.21.15.29:0/3306573854 wait complete. 2024-09-21T02:56:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.035+0000 7fc87c320640 1 Processor -- start 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.035+0000 7fc87c320640 1 -- start start 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87c320640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874101d50 0x7fc874199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87c320640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc874102dd0 0x7fc87419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87c320640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874103770 0x7fc87419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87c320640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc87410cff0 con 0x7fc874103770 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87c320640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc87410ce70 con 0x7fc874102dd0 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87c320640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc87410d170 con 0x7fc874101d50 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a8bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874103770 0x7fc87419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a0bc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874101d50 0x7fc874199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a8bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874103770 0x7fc87419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44076/0 (socket says 172.21.15.29:44076) 2024-09-21T02:56:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a0bc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874101d50 0x7fc874199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48116/0 (socket says 172.21.15.29:48116) 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a8bd640 1 -- 172.21.15.29:0/528450409 learned_addr learned my addr 172.21.15.29:0/528450409 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc8798bb640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc874102dd0 0x7fc87419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a0bc640 1 -- 172.21.15.29:0/528450409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc874102dd0 msgr2=0x7fc87419a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc874102dd0 0x7fc87419a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a0bc640 1 -- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874103770 msgr2=0x7fc87419b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874103770 0x7fc87419b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.036+0000 7fc87a0bc640 1 -- 172.21.15.29:0/528450409 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87419b8f0 con 0x7fc874101d50 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.037+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874103770 0x7fc87419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.037+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874101d50 0x7fc874199eb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc87000e9f0 tx=0x7fc87000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.037+0000 7fc86b7fe640 1 -- 172.21.15.29:0/528450409 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87000ccc0 con 0x7fc874101d50 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.037+0000 7fc86b7fe640 1 -- 172.21.15.29:0/528450409 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8700108c0 con 0x7fc874101d50 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.037+0000 7fc87c320640 1 -- 172.21.15.29:0/528450409 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87419bbe0 con 0x7fc874101d50 2024-09-21T02:56:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.037+0000 7fc86b7fe640 1 -- 172.21.15.29:0/528450409 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8700187e0 con 0x7fc874101d50 2024-09-21T02:56:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.037+0000 7fc87c320640 1 -- 172.21.15.29:0/528450409 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8741a22d0 con 0x7fc874101d50 2024-09-21T02:56:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.040+0000 7fc88328d740 1 Processor -- start 2024-09-21T02:56:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc88328d740 1 -- start start 2024-09-21T02:56:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc88328d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b32111f1b0 0x55b3214c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc88328d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc88328d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210fe4d0 0x55b3214c8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc88328d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b3214396f0 con 0x55b3210f62f0 2024-09-21T02:56:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc88328d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b321428b20 con 0x55b32111f1b0 2024-09-21T02:56:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc88328d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b32142c930 con 0x55b3210fe4d0 2024-09-21T02:56:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc8798bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210fe4d0 0x55b3214c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc8798bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210fe4d0 0x55b3214c8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48124/0 (socket says 172.21.15.29:48124) 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc8798bb640 1 -- 172.21.15.29:0/2874111470 learned_addr learned my addr 172.21.15.29:0/2874111470 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/2874111470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b32111f1b0 0x55b3214c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc8798bb640 1 -- 172.21.15.29:0/2874111470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b32111f1b0 msgr2=0x55b3214c6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc8798bb640 1 --2- 172.21.15.29:0/2874111470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b32111f1b0 0x55b3214c6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc8798bb640 1 -- 172.21.15.29:0/2874111470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 msgr2=0x55b3214c74c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc8798bb640 1 --2- 172.21.15.29:0/2874111470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c74c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.041+0000 7fc8798bb640 1 -- 172.21.15.29:0/2874111470 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3213ee530 con 0x55b3210fe4d0 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.042+0000 7fc8798bb640 1 --2- 172.21.15.29:0/2874111470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210fe4d0 0x55b3214c8010 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc86402bd10 tx=0x7fc864004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9851dc73eb7b7d51 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.042+0000 7fc868bfe640 1 -- 172.21.15.29:0/2874111470 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864036040 con 0x55b3210fe4d0 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.042+0000 7fc868bfe640 1 -- 172.21.15.29:0/2874111470 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc86403ebf0 con 0x55b3210fe4d0 2024-09-21T02:56:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.042+0000 7fc868bfe640 1 -- 172.21.15.29:0/2874111470 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864003ca0 con 0x55b3210fe4d0 2024-09-21T02:56:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.042+0000 7fc86b7fe640 1 -- 172.21.15.29:0/528450409 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc870010430 con 0x7fc874101d50 2024-09-21T02:56:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.043+0000 7fc88328d740 1 -- 172.21.15.29:0/2874111470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210fe4d0 msgr2=0x55b3214c8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.043+0000 7fc88328d740 1 --2- 172.21.15.29:0/2874111470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210fe4d0 0x55b3214c8010 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc86402bd10 tx=0x7fc864004040 comp rx=0 tx=0).stop 2024-09-21T02:56:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.043+0000 7fc86b7fe640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8600b6510 0x7fc8600b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.043+0000 7fc88328d740 1 -- 172.21.15.29:0/2874111470 shutdown_connections 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.043+0000 7fc88328d740 1 --2- 172.21.15.29:0/2874111470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210fe4d0 0x55b3214c8010 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.043+0000 7fc88328d740 1 --2- 172.21.15.29:0/2874111470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.043+0000 7fc88328d740 1 --2- 172.21.15.29:0/2874111470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b32111f1b0 0x55b3214c6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.043+0000 7fc88328d740 1 -- 172.21.15.29:0/2874111470 >> 172.21.15.29:0/2874111470 conn(0x55b321113300 msgr2=0x55b3214c28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.044+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8600b6510 0x7fc8600b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.044+0000 7fc86b7fe640 1 -- 172.21.15.29:0/528450409 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc8700e79b0 con 0x7fc874101d50 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.044+0000 7fc88328d740 1 -- 172.21.15.29:0/2874111470 shutdown_connections 2024-09-21T02:56:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.044+0000 7fc88328d740 1 -- 172.21.15.29:0/2874111470 wait complete. 2024-09-21T02:56:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.044+0000 7fc88328d740 1 -- start start 2024-09-21T02:56:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8600b6510 0x7fc8600b8a20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc86c005c40 tx=0x7fc86c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc88328d740 1 Processor -- start 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc88328d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210f62f0 0x55b3214d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc88328d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc88328d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b32111f1b0 0x55b3214d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc88328d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b321360ab0 con 0x55b32111f1b0 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc8798bb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc8798bb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:57524/0 (socket says 172.21.15.29:57524) 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc8798bb640 1 -- 172.21.15.29:0/2248360866 learned_addr learned my addr 172.21.15.29:0/2248360866 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b3213ce730 con 0x55b3210fe4d0 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b3214c8ad0 con 0x55b3210f62f0 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b32111f1b0 0x55b3214d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.045+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210f62f0 0x55b3214d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc88328d740 10 client.0 _ll_register_callbacks cb 0x55b3214226b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc87a0bc640 1 -- 172.21.15.29:0/2248360866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 msgr2=0x55b3214d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc87a0bc640 1 -- 172.21.15.29:0/2248360866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b32111f1b0 msgr2=0x55b3214d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b32111f1b0 0x55b3214d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc87a0bc640 1 -- 172.21.15.29:0/2248360866 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b32118a790 con 0x55b3210f62f0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b32111f1b0 0x55b3214d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210f62f0 0x55b3214d0640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc8700e54a0 tx=0x7fc870010720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=431e465a64c1af75 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc85dffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.046+0000 7fc85dffb640 1 -- 172.21.15.29:0/2248360866 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8700b22a0 con 0x55b3210f62f0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc85dffb640 1 -- 172.21.15.29:0/2248360866 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc8700aeaf0 con 0x55b3210f62f0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc86403ebf0 con 0x55b3210f62f0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc85dffb640 1 -- 172.21.15.29:0/2248360866 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc8700b71e0 con 0x55b3210f62f0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 10 client.27268 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b321203620 con 0x55b3210f62f0 2024-09-21T02:56:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc85dffb640 1 -- 172.21.15.29:0/2248360866 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc86403ebf0 con 0x55b3210f62f0 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc85dffb640 1 -- 172.21.15.29:0/2248360866 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fc8700b6720 con 0x55b3210f62f0 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 10 client.27268 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 10 client.27268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 20 client.27268 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3210ddd80 0x55b3214ddd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 4 client.27268 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b32143b640 con 0x55b3210ddd80 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.047+0000 7fc8798bb640 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3210ddd80 0x55b3214ddd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.048+0000 7fc85dffb640 10 client.27268 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.048+0000 7fc8798bb640 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3210ddd80 0x55b3214ddd10 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93d910c150543fa2 server_cookie=1a80b73c06f9082e in_seq=0 out_seq=0 2024-09-21T02:56:29.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.049+0000 7fc85dffb640 1 -- 172.21.15.29:0/2248360866 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b32143b640 con 0x55b3210ddd80 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.049+0000 7fc85dffb640 10 client.27268 handle_command_reply: tid=0 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.049+0000 7fc88328d740 1 client.27268 shutdown 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210f62f0 msgr2=0x55b3214d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210f62f0 0x55b3214d0640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc8700e54a0 tx=0x7fc870010720 comp rx=0 tx=0).stop 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 shutdown_connections 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b3210ddd80 0x55b3214ddd10 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc85dffb640 0 client.27268 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b32111f1b0 0x55b3214d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 --2- 172.21.15.29:0/2248360866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210f62f0 0x55b3214d0640 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 >> 172.21.15.29:0/2248360866 conn(0x55b321113300 msgr2=0x55b3214c0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 shutdown_connections 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 1 -- 172.21.15.29:0/2248360866 wait complete. 2024-09-21T02:56:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.050+0000 7fc88328d740 20 client.27268 trim_cache size 0 max 16384 2024-09-21T02:56:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.077+0000 7fc88328d740 1 Processor -- start 2024-09-21T02:56:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.077+0000 7fc88328d740 1 -- start start 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.077+0000 7fc88328d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214c0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.077+0000 7fc88328d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc88328d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc88328d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741aa300 con 0x55b3210f62f0 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44106/0 (socket says 172.21.15.29:44106) 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc88328d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741abe40 con 0x55b3210fe4d0 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 -- 172.21.15.29:0/2642594806 learned_addr learned my addr 172.21.15.29:0/2642594806 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc88328d740 1 -- 172.21.15.29:0/2642594806 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741aa9d0 con 0x55b3210ddd80 2024-09-21T02:56:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214c0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 -- 172.21.15.29:0/2642594806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 msgr2=0x55b3214c0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214c0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 -- 172.21.15.29:0/2642594806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 msgr2=0x55b3214cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 -- 172.21.15.29:0/2642594806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3214d2430 con 0x55b3210f62f0 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.078+0000 7fc8798bb640 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c2220 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc86400aa10 tx=0x7fc86404a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ada42719318ff28 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.079+0000 7fc85ffff640 1 -- 172.21.15.29:0/2642594806 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864036040 con 0x55b3210f62f0 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.079+0000 7fc85ffff640 1 -- 172.21.15.29:0/2642594806 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc864003670 con 0x55b3210f62f0 2024-09-21T02:56:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.079+0000 7fc85ffff640 1 -- 172.21.15.29:0/2642594806 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864030870 con 0x55b3210f62f0 2024-09-21T02:56:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 -- 172.21.15.29:0/2642594806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 msgr2=0x55b3214c2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c2220 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc86400aa10 tx=0x7fc86404a040 comp rx=0 tx=0).stop 2024-09-21T02:56:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 -- 172.21.15.29:0/2642594806 shutdown_connections 2024-09-21T02:56:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214c2220 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 --2- 172.21.15.29:0/2642594806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214c0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 -- 172.21.15.29:0/2642594806 >> 172.21.15.29:0/2642594806 conn(0x55b321113300 msgr2=0x55b3214c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 -- 172.21.15.29:0/2642594806 shutdown_connections 2024-09-21T02:56:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.080+0000 7fc88328d740 1 -- 172.21.15.29:0/2642594806 wait complete. 2024-09-21T02:56:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.081+0000 7fc88328d740 1 -- start start 2024-09-21T02:56:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.081+0000 7fc88328d740 1 Processor -- start 2024-09-21T02:56:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc88328d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214d5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc88328d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214d5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc88328d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc88328d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741ab770 con 0x55b3210f62f0 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc88328d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741ab0a0 con 0x55b3210fe4d0 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc87a0bc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214d5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc88328d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741a82e0 con 0x55b3210ddd80 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc87a0bc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214d5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48146/0 (socket says 172.21.15.29:48146) 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc87a0bc640 1 -- 172.21.15.29:0/3236027608 learned_addr learned my addr 172.21.15.29:0/3236027608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc8798bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214d5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc88328d740 10 client.0 _ll_register_callbacks cb 0x7fc8700a9ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.082+0000 7fc87a0bc640 1 -- 172.21.15.29:0/3236027608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 msgr2=0x55b3214d6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc87a0bc640 1 -- 172.21.15.29:0/3236027608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 msgr2=0x55b3214d5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214d5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc87a0bc640 1 -- 172.21.15.29:0/3236027608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3214d6da0 con 0x55b3210ddd80 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc87a8bd640 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc8798bb640 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214d5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc87a0bc640 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214d5410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc870000c00 tx=0x7fc8700b6950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fec3299482b4d9e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc85dffb640 10 client.27274 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc88328d740 1 -- 172.21.15.29:0/3236027608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc864003670 con 0x55b3210ddd80 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc85dffb640 1 -- 172.21.15.29:0/3236027608 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8700ad640 con 0x55b3210ddd80 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.083+0000 7fc85dffb640 1 -- 172.21.15.29:0/3236027608 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fc8700ac540 con 0x55b3210ddd80 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc85dffb640 1 -- 172.21.15.29:0/3236027608 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc8700ad1c0 con 0x55b3210ddd80 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc88328d740 10 client.27274 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc88328d740 1 -- 172.21.15.29:0/3236027608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=37}) -- 0x55b3214d7fe0 con 0x55b3210ddd80 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc85dffb640 1 -- 172.21.15.29:0/3236027608 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fc8700ac740 con 0x55b3210ddd80 2024-09-21T02:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc88328d740 10 client.27274 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc88328d740 10 client.27274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc88328d740 20 client.27274 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc88328d740 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b32111f1b0 0x55b3214d8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc88328d740 4 client.27274 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:56:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc88328d740 1 -- 172.21.15.29:0/3236027608 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3214370f0 con 0x55b32111f1b0 2024-09-21T02:56:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.084+0000 7fc8798bb640 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b32111f1b0 0x55b3214d8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.085+0000 7fc85dffb640 10 client.27274 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.085+0000 7fc8798bb640 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b32111f1b0 0x55b3214d8740 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d382a05cfbde96b server_cookie=727e0982974e44e8 in_seq=0 out_seq=0 2024-09-21T02:56:29.238 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:56:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc85dffb640 1 -- 172.21.15.29:0/3236027608 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b3214370f0 con 0x55b32111f1b0 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc85dffb640 10 client.27274 handle_command_reply: tid=0 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 client.27274 shutdown 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 -- 172.21.15.29:0/3236027608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 msgr2=0x55b3214d5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214d5410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc870000c00 tx=0x7fc8700b6950 comp rx=0 tx=0).stop 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 -- 172.21.15.29:0/3236027608 shutdown_connections 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b3210fe4d0 0x55b3214d6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b32111f1b0 0x55b3214d8740 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3210f62f0 0x55b3214d5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 --2- 172.21.15.29:0/3236027608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3210ddd80 0x55b3214d5410 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 -- 172.21.15.29:0/3236027608 >> 172.21.15.29:0/3236027608 conn(0x55b321113300 msgr2=0x55b3214cf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc85dffb640 0 client.27274 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 -- 172.21.15.29:0/3236027608 shutdown_connections 2024-09-21T02:56:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 1 -- 172.21.15.29:0/3236027608 wait complete. 2024-09-21T02:56:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.250+0000 7fc88328d740 20 client.27274 trim_cache size 0 max 16384 2024-09-21T02:56:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:56:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:56:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:56:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:56:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 -- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8600b6510 msgr2=0x7fc8600b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8600b6510 0x7fc8600b8a20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc86c005c40 tx=0x7fc86c009500 comp rx=0 tx=0).stop 2024-09-21T02:56:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 -- 172.21.15.29:0/528450409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874101d50 msgr2=0x7fc874199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874101d50 0x7fc874199eb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc87000e9f0 tx=0x7fc87000eeb0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 -- 172.21.15.29:0/528450409 shutdown_connections 2024-09-21T02:56:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc8600b6510 0x7fc8600b8a20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874103770 0x7fc87419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc874102dd0 0x7fc87419a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 --2- 172.21.15.29:0/528450409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874101d50 0x7fc874199eb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 -- 172.21.15.29:0/528450409 >> 172.21.15.29:0/528450409 conn(0x7fc8740fd690 msgr2=0x7fc874108300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 -- 172.21.15.29:0/528450409 shutdown_connections 2024-09-21T02:56:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:29.252+0000 7fc87c320640 1 -- 172.21.15.29:0/528450409 wait complete. 2024-09-21T02:56:29.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:29 smithi029 ceph-mon[31006]: pgmap v910: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.0 MiB/s wr, 262 op/s 2024-09-21T02:56:29.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27148 5.96 MB/sec execute 126 sec latency 201.272 ms 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.834+0000 7f4afc4d7640 1 -- 172.21.15.29:0/1352917901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af41083c0 msgr2=0x7f4af41087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.834+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/1352917901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af41083c0 0x7f4af41087d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009860 tx=0x7f4ae402a340 comp rx=0 tx=0).stop 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.834+0000 7f4afc4d7640 1 -- 172.21.15.29:0/1352917901 shutdown_connections 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.834+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/1352917901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4af4108d10 0x7f4af410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.834+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/1352917901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af41083c0 0x7f4af41087d0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.834+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/1352917901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4af4102a80 0x7f4af4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.834+0000 7f4afc4d7640 1 -- 172.21.15.29:0/1352917901 >> 172.21.15.29:0/1352917901 conn(0x7f4af4079760 msgr2=0x7f4af4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.834+0000 7f4afc4d7640 1 -- 172.21.15.29:0/1352917901 shutdown_connections 2024-09-21T02:56:29.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.835+0000 7f4afc4d7640 1 -- 172.21.15.29:0/1352917901 wait complete. 2024-09-21T02:56:29.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.835+0000 7f4afc4d7640 1 Processor -- start 2024-09-21T02:56:29.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afc4d7640 1 -- start start 2024-09-21T02:56:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afc4d7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4af4102a80 0x7f4af419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afc4d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4af41083c0 0x7f4af419cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afc4d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4108d10 0x7f4af41a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afc4d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4115b00 con 0x7f4af4108d10 2024-09-21T02:56:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afc4d7640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4115980 con 0x7f4af4102a80 2024-09-21T02:56:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afc4d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4115c80 con 0x7f4af41083c0 2024-09-21T02:56:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afaa74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4108d10 0x7f4af41a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afa273640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4af4102a80 0x7f4af419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afa273640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4af4102a80 0x7f4af419c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:57552/0 (socket says 172.21.15.29:57552) 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afa273640 1 -- 172.21.15.29:0/4149968876 learned_addr learned my addr 172.21.15.29:0/4149968876 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4afaa74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4108d10 0x7f4af41a1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44134/0 (socket says 172.21.15.29:44134) 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.836+0000 7f4af9a72640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4af41083c0 0x7f4af419cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.837+0000 7f4af9a72640 1 -- 172.21.15.29:0/4149968876 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4af4102a80 msgr2=0x7f4af419c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.837+0000 7f4af9a72640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4af4102a80 0x7f4af419c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.837+0000 7f4af9a72640 1 -- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4108d10 msgr2=0x7f4af41a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.837+0000 7f4af9a72640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4108d10 0x7f4af41a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.837+0000 7f4af9a72640 1 -- 172.21.15.29:0/4149968876 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af41a2520 con 0x7f4af41083c0 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.837+0000 7f4afaa74640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4108d10 0x7f4af41a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.837+0000 7f4af9a72640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4af41083c0 0x7f4af419cb50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4ae402a860 tx=0x7f4ae402ae40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.837+0000 7f4aeb7fe640 1 -- 172.21.15.29:0/4149968876 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4040020 con 0x7f4af41083c0 2024-09-21T02:56:29.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.838+0000 7f4aeb7fe640 1 -- 172.21.15.29:0/4149968876 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae40044d0 con 0x7f4af41083c0 2024-09-21T02:56:29.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.838+0000 7f4afc4d7640 1 -- 172.21.15.29:0/4149968876 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af41a27b0 con 0x7f4af41083c0 2024-09-21T02:56:29.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.838+0000 7f4aeb7fe640 1 -- 172.21.15.29:0/4149968876 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4037040 con 0x7f4af41083c0 2024-09-21T02:56:29.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.838+0000 7f4afc4d7640 1 -- 172.21.15.29:0/4149968876 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af41a8f70 con 0x7f4af41083c0 2024-09-21T02:56:29.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.841+0000 7f4aeb7fe640 1 -- 172.21.15.29:0/4149968876 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4ae4043020 con 0x7f4af41083c0 2024-09-21T02:56:29.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.842+0000 7f4b02a4b740 1 Processor -- start 2024-09-21T02:56:29.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.842+0000 7f4b02a4b740 1 -- start start 2024-09-21T02:56:29.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.843+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f56840 0x55f0112fd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.843+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f0112fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.843+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f35b60 0x55f0112fe080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.843+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f011273b90 con 0x55f010f56840 2024-09-21T02:56:29.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.843+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f011262fc0 con 0x55f010f35b60 2024-09-21T02:56:29.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.843+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f011266dd0 con 0x55f010f2d980 2024-09-21T02:56:29.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.843+0000 7f4afaa74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f56840 0x55f0112fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.843+0000 7f4afa273640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f0112fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4afa273640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f0112fdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48160/0 (socket says 172.21.15.29:48160) 2024-09-21T02:56:29.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4afaa74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f56840 0x55f0112fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44146/0 (socket says 172.21.15.29:44146) 2024-09-21T02:56:29.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4afa273640 1 -- 172.21.15.29:0/3069031420 learned_addr learned my addr 172.21.15.29:0/3069031420 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4af9a72640 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f35b60 0x55f0112fe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4afaa74640 1 -- 172.21.15.29:0/3069031420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 msgr2=0x55f0112fdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4afaa74640 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f0112fdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4afaa74640 1 -- 172.21.15.29:0/3069031420 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f35b60 msgr2=0x55f0112fe080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4afaa74640 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f35b60 0x55f0112fe080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4afaa74640 1 -- 172.21.15.29:0/3069031420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0111ea060 con 0x55f010f56840 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.844+0000 7f4af9a72640 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f35b60 0x55f0112fe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.845+0000 7f4afaa74640 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f56840 0x55f0112fd600 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4aec005fd0 tx=0x7f4aec011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31a140afdce20ac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.845+0000 7f4aeb7fe640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ae00b6660 0x7f4ae00b8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.845+0000 7f4aeb7fe640 1 -- 172.21.15.29:0/4149968876 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4ae4107440 con 0x7f4af41083c0 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.845+0000 7f4afa273640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ae00b6660 0x7f4ae00b8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.845+0000 7f4ae8ff9640 1 -- 172.21.15.29:0/3069031420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec004d20 con 0x55f010f56840 2024-09-21T02:56:29.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.845+0000 7f4ae8ff9640 1 -- 172.21.15.29:0/3069031420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4aec00f9a0 con 0x55f010f56840 2024-09-21T02:56:29.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.845+0000 7f4ae8ff9640 1 -- 172.21.15.29:0/3069031420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec0043d0 con 0x55f010f56840 2024-09-21T02:56:29.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.846+0000 7f4afa273640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ae00b6660 0x7f4ae00b8b70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55f0112fc280 tx=0x7f4af0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.846+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3069031420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f56840 msgr2=0x55f0112fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.846+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f56840 0x55f0112fd600 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4aec005fd0 tx=0x7f4aec011bf0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.846+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3069031420 shutdown_connections 2024-09-21T02:56:29.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.846+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f35b60 0x55f0112fe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.846+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f0112fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.846+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/3069031420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f56840 0x55f0112fd600 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.846+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3069031420 >> 172.21.15.29:0/3069031420 conn(0x55f010f4a990 msgr2=0x55f0112fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.847+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3069031420 shutdown_connections 2024-09-21T02:56:29.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.847+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3069031420 wait complete. 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.847+0000 7f4b02a4b740 1 -- start start 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4b02a4b740 1 Processor -- start 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f011309210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f011309790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f01131d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f0110c47a0 con 0x55f010f35b60 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f0112080a0 con 0x55f010f2d980 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4af9a72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f011309790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4af9a72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f011309790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44156/0 (socket says 172.21.15.29:44156) 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f01130bde0 con 0x55f010f56840 2024-09-21T02:56:29.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4af9a72640 1 -- 172.21.15.29:0/2519080510 learned_addr learned my addr 172.21.15.29:0/2519080510 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4afa273640 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f011309210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4afaa74640 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f01131d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.848+0000 7f4b02a4b740 10 client.0 _ll_register_callbacks cb 0x55f0111e53c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4afa273640 1 -- 172.21.15.29:0/2519080510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 msgr2=0x55f01131d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4afa273640 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f01131d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4afa273640 1 -- 172.21.15.29:0/2519080510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 msgr2=0x55f011309790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4afa273640 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f011309790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4afa273640 1 -- 172.21.15.29:0/2519080510 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0111f6880 con 0x55f010f2d980 2024-09-21T02:56:29.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4af9a72640 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f011309790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4afa273640 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f011309210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4af0005a00 tx=0x7f4af0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4cdca873222fa62 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4addffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4addffb640 1 -- 172.21.15.29:0/2519080510 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af001de10 con 0x55f010f2d980 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4addffb640 1 -- 172.21.15.29:0/2519080510 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4af0018030 con 0x55f010f2d980 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4b02a4b740 1 -- 172.21.15.29:0/2519080510 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aec00f9a0 con 0x55f010f2d980 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4addffb640 1 -- 172.21.15.29:0/2519080510 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4af0033050 con 0x55f010f2d980 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.849+0000 7f4b02a4b740 10 client.27143 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4b02a4b740 1 -- 172.21.15.29:0/2519080510 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f010fd0b90 con 0x55f010f2d980 2024-09-21T02:56:29.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4addffb640 1 -- 172.21.15.29:0/2519080510 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4af00044c0 con 0x55f010f2d980 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4addffb640 1 -- 172.21.15.29:0/2519080510 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f4af0028410 con 0x55f010f2d980 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4b02a4b740 10 client.27143 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4b02a4b740 10 client.27143 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4b02a4b740 20 client.27143 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f010f151c0 0x55f011323990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4b02a4b740 4 client.27143 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4b02a4b740 1 -- 172.21.15.29:0/2519080510 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4aec00f9a0 con 0x55f010f151c0 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.850+0000 7f4af9a72640 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f010f151c0 0x55f011323990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.851+0000 7f4addffb640 10 client.27143 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.851+0000 7f4af9a72640 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f010f151c0 0x55f011323990 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fb15ca5ebb02955 server_cookie=ffd13fa68612b3d1 in_seq=0 out_seq=0 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4addffb640 1 -- 172.21.15.29:0/2519080510 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f4aec00f9a0 con 0x55f010f151c0 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4addffb640 10 client.27143 handle_command_reply: tid=0 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 client.27143 shutdown 2024-09-21T02:56:29.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 -- 172.21.15.29:0/2519080510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 msgr2=0x55f011309210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f011309210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4af0005a00 tx=0x7f4af0004730 comp rx=0 tx=0).stop 2024-09-21T02:56:29.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 -- 172.21.15.29:0/2519080510 shutdown_connections 2024-09-21T02:56:29.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f01131d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f010f151c0 0x55f011323990 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f011309790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/2519080510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f011309210 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 -- 172.21.15.29:0/2519080510 >> 172.21.15.29:0/2519080510 conn(0x55f010f4a990 msgr2=0x55f0113007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4addffb640 0 client.27143 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:29.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 -- 172.21.15.29:0/2519080510 shutdown_connections 2024-09-21T02:56:29.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 1 -- 172.21.15.29:0/2519080510 wait complete. 2024-09-21T02:56:29.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.852+0000 7f4b02a4b740 20 client.27143 trim_cache size 0 max 16384 2024-09-21T02:56:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:29 smithi190 ceph-mon[33871]: pgmap v910: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.0 MiB/s wr, 262 op/s 2024-09-21T02:56:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4b02a4b740 1 Processor -- start 2024-09-21T02:56:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4b02a4b740 1 -- start start 2024-09-21T02:56:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f01130ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0112faaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f0112fafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4af41ae720 con 0x55f010f35b60 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4af41b0260 con 0x55f010f2d980 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4af41aedf0 con 0x55f010f56840 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4afa273640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f01130ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4af9a72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0112faaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4afa273640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f01130ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:57574/0 (socket says 172.21.15.29:57574) 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4afa273640 1 -- 172.21.15.29:0/3898335896 learned_addr learned my addr 172.21.15.29:0/3898335896 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.877+0000 7f4afaa74640 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f0112fafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4afa273640 1 -- 172.21.15.29:0/3898335896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 msgr2=0x55f0112fafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4afa273640 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f0112fafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4afa273640 1 -- 172.21.15.29:0/3898335896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 msgr2=0x55f0112faaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4afa273640 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0112faaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4afa273640 1 -- 172.21.15.29:0/3898335896 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f01130a2d0 con 0x55f010f2d980 2024-09-21T02:56:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4afaa74640 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f0112fafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4afa273640 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f01130ac40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4af0005a70 tx=0x7f4af001de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9a103af0d716fc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4adffff640 1 -- 172.21.15.29:0/3898335896 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af003c070 con 0x55f010f2d980 2024-09-21T02:56:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4adffff640 1 -- 172.21.15.29:0/3898335896 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4af0004500 con 0x55f010f2d980 2024-09-21T02:56:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.878+0000 7f4adffff640 1 -- 172.21.15.29:0/3898335896 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0003850 con 0x55f010f2d980 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.879+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3898335896 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 msgr2=0x55f01130ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.879+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f01130ac40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4af0005a70 tx=0x7f4af001de10 comp rx=0 tx=0).stop 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.879+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3898335896 shutdown_connections 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.879+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f56840 0x55f0112fafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.879+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0112faaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.879+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/3898335896 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f2d980 0x55f01130ac40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.879+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3898335896 >> 172.21.15.29:0/3898335896 conn(0x55f010f4a990 msgr2=0x55f01130a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.880+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3898335896 shutdown_connections 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.880+0000 7f4b02a4b740 1 -- 172.21.15.29:0/3898335896 wait complete. 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.880+0000 7f4b02a4b740 1 -- start start 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.880+0000 7f4b02a4b740 1 Processor -- start 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.880+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f011316210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.880+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0113167f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.880+0000 7f4b02a4b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f56840 0x55f0113873d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4af41afb90 con 0x55f010f35b60 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4af41af4c0 con 0x55f010f56840 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4b02a4b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4af41ac700 con 0x55f010f2d980 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f011316210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4b02a4b740 10 client.0 _ll_register_callbacks cb 0x7f4af0012f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f011316210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48190/0 (socket says 172.21.15.29:48190) 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 -- 172.21.15.29:0/1501353608 learned_addr learned my addr 172.21.15.29:0/1501353608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afaa74640 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f56840 0x55f0113873d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4af9a72640 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0113167f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 -- 172.21.15.29:0/1501353608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f56840 msgr2=0x55f0113873d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f56840 0x55f0113873d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 -- 172.21.15.29:0/1501353608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 msgr2=0x55f0113167f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0113167f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 -- 172.21.15.29:0/1501353608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f011387a40 con 0x55f010f2d980 2024-09-21T02:56:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4af9a72640 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0113167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:56:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4afa273640 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f011316210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4af003a040 tx=0x7f4af00093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=756bd87a20f4c1f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.881+0000 7f4addffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:56:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4addffb640 1 -- 172.21.15.29:0/1501353608 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af003c070 con 0x55f010f2d980 2024-09-21T02:56:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4addffb640 1 -- 172.21.15.29:0/1501353608 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4af0028410 con 0x55f010f2d980 2024-09-21T02:56:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4b02a4b740 1 -- 172.21.15.29:0/1501353608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4af0004500 con 0x55f010f2d980 2024-09-21T02:56:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4addffb640 1 -- 172.21.15.29:0/1501353608 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4af004d050 con 0x55f010f2d980 2024-09-21T02:56:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4b02a4b740 10 client.27292 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4b02a4b740 1 -- 172.21.15.29:0/1501353608 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f01138ed40 con 0x55f010f2d980 2024-09-21T02:56:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4addffb640 1 -- 172.21.15.29:0/1501353608 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4af00054c0 con 0x55f010f2d980 2024-09-21T02:56:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4addffb640 1 -- 172.21.15.29:0/1501353608 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f4af0004cd0 con 0x55f010f2d980 2024-09-21T02:56:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4b02a4b740 10 client.27292 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4b02a4b740 10 client.27292 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:56:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4b02a4b740 20 client.27292 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.882+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f010eea570 0x55f011391450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.883+0000 7f4b02a4b740 4 client.27292 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-09-21T02:56:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.883+0000 7f4b02a4b740 1 -- 172.21.15.29:0/1501353608 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x7f4af0004500 con 0x55f010eea570 2024-09-21T02:56:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.883+0000 7f4af9a72640 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f010eea570 0x55f011391450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.883+0000 7f4addffb640 10 client.27292 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:29.883+0000 7f4af9a72640 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f010eea570 0x55f011391450 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef444faf9e899ecb server_cookie=a4c265921a2dc865 in_seq=0 out_seq=0 2024-09-21T02:56:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:29 smithi066 ceph-mon[33436]: pgmap v910: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.0 MiB/s wr, 262 op/s 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4addffb640 1 -- 172.21.15.29:0/1501353608 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f4af0004500 con 0x55f010eea570 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4addffb640 10 client.27292 handle_command_reply: tid=0 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 client.27292 shutdown 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 -- 172.21.15.29:0/1501353608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 msgr2=0x55f011316210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f011316210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4af003a040 tx=0x7f4af00093e0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 -- 172.21.15.29:0/1501353608 shutdown_connections 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f010f56840 0x55f0113873d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f010eea570 0x55f011391450 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f010f35b60 0x55f0113167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 --2- 172.21.15.29:0/1501353608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f010f2d980 0x55f011316210 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4addffb640 0 client.27292 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 -- 172.21.15.29:0/1501353608 >> 172.21.15.29:0/1501353608 conn(0x55f010f4a990 msgr2=0x55f01130a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 -- 172.21.15.29:0/1501353608 shutdown_connections 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 1 -- 172.21.15.29:0/1501353608 wait complete. 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.475+0000 7f4b02a4b740 20 client.27292 trim_cache size 0 max 16384 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 66, 2024-09-21T02:56:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "4d82b865": { 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 66, 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 130.0, 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:56:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:30.480 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:30.480 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:30.480 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:30.480 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:30.480 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:56:30.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 -- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ae00b6660 msgr2=0x7f4ae00b8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:30.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ae00b6660 0x7f4ae00b8b70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55f0112fc280 tx=0x7f4af0009260 comp rx=0 tx=0).stop 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 -- 172.21.15.29:0/4149968876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4af41083c0 msgr2=0x7f4af419cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4af41083c0 0x7f4af419cb50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4ae402a860 tx=0x7f4ae402ae40 comp rx=0 tx=0).stop 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 -- 172.21.15.29:0/4149968876 shutdown_connections 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ae00b6660 0x7f4ae00b8b70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4108d10 0x7f4af41a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4af41083c0 0x7f4af419cb50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 --2- 172.21.15.29:0/4149968876 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4af4102a80 0x7f4af419c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.476+0000 7f4afc4d7640 1 -- 172.21.15.29:0/4149968876 >> 172.21.15.29:0/4149968876 conn(0x7f4af4079760 msgr2=0x7f4af4105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.477+0000 7f4afc4d7640 1 -- 172.21.15.29:0/4149968876 shutdown_connections 2024-09-21T02:56:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:30.477+0000 7f4afc4d7640 1 -- 172.21.15.29:0/4149968876 wait complete. 2024-09-21T02:56:30.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27181 5.93 MB/sec execute 127 sec latency 802.781 ms 2024-09-21T02:56:31.236 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '4d82b865', quiesce took 17.6 seconds. Will release after: -7.600000000000001 2024-09-21T02:56:31.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:56:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:56:31.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27181 5.88 MB/sec execute 128 sec latency 1802.869 ms 2024-09-21T02:56:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:31 smithi029 ceph-mon[31006]: pgmap v911: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-09-21T02:56:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:31 smithi190 ceph-mon[33871]: pgmap v911: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-09-21T02:56:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:31 smithi066 ceph-mon[33436]: pgmap v911: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-09-21T02:56:32.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.071+0000 7fe46b5ab640 1 -- 172.21.15.29:0/1716246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 msgr2=0x7fe46410d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:32.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.071+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/1716246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 0x7fe46410d930 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fe454009900 tx=0x7fe45402a410 comp rx=0 tx=0).stop 2024-09-21T02:56:32.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.072+0000 7fe46b5ab640 1 -- 172.21.15.29:0/1716246696 shutdown_connections 2024-09-21T02:56:32.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.072+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/1716246696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe46410de70 0x7fe464110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.072+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/1716246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 0x7fe46410d930 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.072+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/1716246696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4641048c0 0x7fe464104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.072+0000 7fe46b5ab640 1 -- 172.21.15.29:0/1716246696 >> 172.21.15.29:0/1716246696 conn(0x7fe464079740 msgr2=0x7fe464079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.072+0000 7fe46b5ab640 1 -- 172.21.15.29:0/1716246696 shutdown_connections 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.072+0000 7fe46b5ab640 1 -- 172.21.15.29:0/1716246696 wait complete. 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe46b5ab640 1 Processor -- start 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe46b5ab640 1 -- start start 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe46b5ab640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4641048c0 0x7fe46419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe46b5ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 0x7fe46419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe46b5ab640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe46410de70 0x7fe46419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe46b5ab640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe464113930 con 0x7fe4641052f0 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe46b5ab640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641137b0 con 0x7fe4641048c0 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe46b5ab640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe464113ab0 con 0x7fe46410de70 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe468b46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 0x7fe46419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe468b46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 0x7fe46419a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44204/0 (socket says 172.21.15.29:44204) 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe468b46640 1 -- 172.21.15.29:0/745995669 learned_addr learned my addr 172.21.15.29:0/745995669 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe469b48640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe46410de70 0x7fe46419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe469347640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4641048c0 0x7fe46419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe468b46640 1 -- 172.21.15.29:0/745995669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe46410de70 msgr2=0x7fe46419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe468b46640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe46410de70 0x7fe46419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe468b46640 1 -- 172.21.15.29:0/745995669 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4641048c0 msgr2=0x7fe46419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe468b46640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4641048c0 0x7fe46419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe468b46640 1 -- 172.21.15.29:0/745995669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4641a0310 con 0x7fe4641052f0 2024-09-21T02:56:32.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.073+0000 7fe469347640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4641048c0 0x7fe46419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.074+0000 7fe468b46640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 0x7fe46419a940 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fe454005940 tx=0x7fe4540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.074+0000 7fe45a7fc640 1 -- 172.21.15.29:0/745995669 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45403f030 con 0x7fe4641052f0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.074+0000 7fe45a7fc640 1 -- 172.21.15.29:0/745995669 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe454003c90 con 0x7fe4641052f0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.074+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4641a05a0 con 0x7fe4641052f0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.074+0000 7fe45a7fc640 1 -- 172.21.15.29:0/745995669 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe454035070 con 0x7fe4641052f0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.074+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4641a6d60 con 0x7fe4641052f0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.075+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe46406c180 con 0x7fe4641052f0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.077+0000 7fe45a7fc640 1 -- 172.21.15.29:0/745995669 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fe454042030 con 0x7fe4641052f0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.077+0000 7fe45a7fc640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4500b6510 0x7fe4500b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.077+0000 7fe45a7fc640 1 -- 172.21.15.29:0/745995669 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fe454107fe0 con 0x7fe4641052f0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.077+0000 7fe469347640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4500b6510 0x7fe4500b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.078+0000 7fe469347640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4500b6510 0x7fe4500b8a20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe460005d00 tx=0x7fe460005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:32.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.081+0000 7fe45a7fc640 1 -- 172.21.15.29:0/745995669 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe4540ce4f0 con 0x7fe4641052f0 2024-09-21T02:56:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.310+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe464104cd0 con 0x7fe4641052f0 2024-09-21T02:56:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.311+0000 7fe45a7fc640 1 -- 172.21.15.29:0/745995669 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7fe4540d28d0 con 0x7fe4641052f0 2024-09-21T02:56:32.312 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:56:32.313 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:56:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4500b6510 msgr2=0x7fe4500b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4500b6510 0x7fe4500b8a20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe460005d00 tx=0x7fe460005ef0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 msgr2=0x7fe46419a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 0x7fe46419a940 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fe454005940 tx=0x7fe4540042f0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 shutdown_connections 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe4500b6510 0x7fe4500b8a20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe46410de70 0x7fe46419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4641052f0 0x7fe46419a940 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 --2- 172.21.15.29:0/745995669 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4641048c0 0x7fe46419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 >> 172.21.15.29:0/745995669 conn(0x7fe464079740 msgr2=0x7fe464100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:32.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 shutdown_connections 2024-09-21T02:56:32.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:32.313+0000 7fe46b5ab640 1 -- 172.21.15.29:0/745995669 wait complete. 2024-09-21T02:56:32.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27181 5.83 MB/sec execute 129 sec latency 2802.979 ms 2024-09-21T02:56:32.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:32 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/745995669' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:56:32.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:32 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/745995669' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:56:33.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:32 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/745995669' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:56:33.638 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 4d82b865 --release --await' 2024-09-21T02:56:33.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 4d82b865 --release --await 2024-09-21T02:56:33.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27181 5.79 MB/sec execute 130 sec latency 3803.045 ms 2024-09-21T02:56:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:33 smithi190 ceph-mon[33871]: pgmap v912: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 1.9 MiB/s wr, 220 op/s 2024-09-21T02:56:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:33 smithi029 ceph-mon[31006]: pgmap v912: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 1.9 MiB/s wr, 220 op/s 2024-09-21T02:56:33.909 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:56:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:33 smithi066 ceph-mon[33436]: pgmap v912: 129 pgs: 129 active+clean; 269 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 1.9 MiB/s wr, 220 op/s 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 -- 172.21.15.29:0/2464020045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810108480 msgr2=0x7fd81010ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 --2- 172.21.15.29:0/2464020045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810108480 0x7fd81010ecb0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd808009ca0 tx=0x7fd80802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 -- 172.21.15.29:0/2464020045 shutdown_connections 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 --2- 172.21.15.29:0/2464020045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810108480 0x7fd81010ecb0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 --2- 172.21.15.29:0/2464020045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810107b30 0x7fd810107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 --2- 172.21.15.29:0/2464020045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd8101021f0 0x7fd810102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 -- 172.21.15.29:0/2464020045 >> 172.21.15.29:0/2464020045 conn(0x7fd810077c60 msgr2=0x7fd8100780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 -- 172.21.15.29:0/2464020045 shutdown_connections 2024-09-21T02:56:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.438+0000 7fd81773a640 1 -- 172.21.15.29:0/2464020045 wait complete. 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd81773a640 1 Processor -- start 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd81773a640 1 -- start start 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd81773a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd8101021f0 0x7fd81019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd81773a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810107b30 0x7fd81019cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd81773a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810108480 0x7fd8101a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd81773a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd810112030 con 0x7fd810107b30 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd81773a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd810111eb0 con 0x7fd8101021f0 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd81773a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8101121b0 con 0x7fd810108480 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd814cd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810107b30 0x7fd81019cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd814cd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810107b30 0x7fd81019cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36824/0 (socket says 172.21.15.29:36824) 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd814cd5640 1 -- 172.21.15.29:0/1520435642 learned_addr learned my addr 172.21.15.29:0/1520435642 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd8154d6640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd8101021f0 0x7fd81019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.439+0000 7fd815cd7640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810108480 0x7fd8101a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd814cd5640 1 -- 172.21.15.29:0/1520435642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810108480 msgr2=0x7fd8101a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd814cd5640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810108480 0x7fd8101a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd814cd5640 1 -- 172.21.15.29:0/1520435642 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd8101021f0 msgr2=0x7fd81019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd814cd5640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd8101021f0 0x7fd81019c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd814cd5640 1 -- 172.21.15.29:0/1520435642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8101a24b0 con 0x7fd810107b30 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd815cd7640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810108480 0x7fd8101a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd814cd5640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810107b30 0x7fd81019cae0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd80000d740 tx=0x7fd80000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd8067fc640 1 -- 172.21.15.29:0/1520435642 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800017070 con 0x7fd810107b30 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd8067fc640 1 -- 172.21.15.29:0/1520435642 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd80000de50 con 0x7fd810107b30 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd81773a640 1 -- 172.21.15.29:0/1520435642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8101a27a0 con 0x7fd810107b30 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd8067fc640 1 -- 172.21.15.29:0/1520435642 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80000f620 con 0x7fd810107b30 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.440+0000 7fd81773a640 1 -- 172.21.15.29:0/1520435642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8101a8f00 con 0x7fd810107b30 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.443+0000 7fd8067fc640 1 -- 172.21.15.29:0/1520435642 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd80001a070 con 0x7fd810107b30 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.443+0000 7fd81dcae740 1 Processor -- start 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.443+0000 7fd81dcae740 1 -- start start 2024-09-21T02:56:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed0a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e93b660 0x56527ed0aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed0af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd81dcae740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56527ebf5dc0 con 0x56527e964520 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd81dcae740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56527ec821f0 con 0x56527e943840 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd81dcae740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56527ec71620 con 0x56527e93b660 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd815cd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed0a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd815cd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed0a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36836/0 (socket says 172.21.15.29:36836) 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd815cd7640 1 -- 172.21.15.29:0/2678359337 learned_addr learned my addr 172.21.15.29:0/2678359337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd8154d6640 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e93b660 0x56527ed0aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd814cd5640 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed0af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd815cd7640 1 -- 172.21.15.29:0/2678359337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e93b660 msgr2=0x56527ed0aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd815cd7640 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e93b660 0x56527ed0aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd8067fc640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7fc0b6510 0x7fd7fc0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd815cd7640 1 -- 172.21.15.29:0/2678359337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 msgr2=0x56527ed0af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd815cd7640 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed0af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.444+0000 7fd815cd7640 1 -- 172.21.15.29:0/2678359337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56527ec39fd0 con 0x56527e964520 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.445+0000 7fd815cd7640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7fc0b6510 0x7fd7fc0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.445+0000 7fd8067fc640 1 -- 172.21.15.29:0/1520435642 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd8000e70a0 con 0x7fd810107b30 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.445+0000 7fd815cd7640 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed0a510 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd808031500 tx=0x7fd808031530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb8417c4cc60193 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.445+0000 7fd7fb7fe640 1 -- 172.21.15.29:0/2678359337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80804c070 con 0x56527e964520 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.445+0000 7fd7fb7fe640 1 -- 172.21.15.29:0/2678359337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd808036070 con 0x56527e964520 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.445+0000 7fd7fb7fe640 1 -- 172.21.15.29:0/2678359337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808030800 con 0x56527e964520 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 -- 172.21.15.29:0/2678359337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 msgr2=0x56527ed0a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed0a510 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd808031500 tx=0x7fd808031530 comp rx=0 tx=0).stop 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd815cd7640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7fc0b6510 0x7fd7fc0b8a20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd808049000 tx=0x7fd80804a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 -- 172.21.15.29:0/2678359337 shutdown_connections 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed0af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e93b660 0x56527ed0aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 --2- 172.21.15.29:0/2678359337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed0a510 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 -- 172.21.15.29:0/2678359337 >> 172.21.15.29:0/2678359337 conn(0x56527e958910 msgr2=0x56527ed08400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 -- 172.21.15.29:0/2678359337 shutdown_connections 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 -- 172.21.15.29:0/2678359337 wait complete. 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.446+0000 7fd81dcae740 1 -- start start 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd81dcae740 1 Processor -- start 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed15bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed160f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed16630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd81dcae740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56527ec0c300 con 0x56527e93b660 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd81dcae740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56527ec75430 con 0x56527e943840 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd8154d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed15bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd81dcae740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56527ead2bb0 con 0x56527e964520 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd8154d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed15bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36840/0 (socket says 172.21.15.29:36840) 2024-09-21T02:56:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd8154d6640 1 -- 172.21.15.29:0/974701487 learned_addr learned my addr 172.21.15.29:0/974701487 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd81dcae740 10 client.0 _ll_register_callbacks cb 0x56527ec26120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.447+0000 7fd814cd5640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd815cd7640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed16630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd8154d6640 1 -- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 msgr2=0x56527ed16630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd8154d6640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed16630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd8154d6640 1 -- 172.21.15.29:0/974701487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 msgr2=0x56527ed160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd8154d6640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd8154d6640 1 -- 172.21.15.29:0/974701487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56527ebfb780 con 0x56527e93b660 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd815cd7640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed16630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd814cd5640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd8154d6640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed15bb0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00bad0 tx=0x7fd80c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cb7e42706f9030 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd7f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/974701487 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c005020 con 0x56527e93b660 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/974701487 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd80c0051c0 con 0x56527e93b660 2024-09-21T02:56:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.448+0000 7fd81dcae740 1 -- 172.21.15.29:0/974701487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd808036070 con 0x56527e93b660 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/974701487 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd80c024050 con 0x56527e93b660 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd81dcae740 10 client.17817 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd81dcae740 1 -- 172.21.15.29:0/974701487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56527e9cfb00 con 0x56527e93b660 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/974701487 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd808036070 con 0x56527e93b660 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/974701487 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fd80c01b470 con 0x56527e93b660 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd81dcae740 10 client.17817 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd81dcae740 10 client.17817 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd81dcae740 20 client.17817 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd81dcae740 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56527e922ea0 0x56527ed27f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd81dcae740 4 client.17817 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.449+0000 7fd81dcae740 1 -- 172.21.15.29:0/974701487 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56527ec84140 con 0x56527e922ea0 2024-09-21T02:56:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.450+0000 7fd814cd5640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56527e922ea0 0x56527ed27f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.450+0000 7fd7f8ff9640 10 client.17817 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.450+0000 7fd814cd5640 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56527e922ea0 0x56527ed27f30 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a21d4fc1dc68c03 server_cookie=1fc11cc07f89335e in_seq=0 out_seq=0 2024-09-21T02:56:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/974701487 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56527ec84140 con 0x56527e922ea0 2024-09-21T02:56:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd7f8ff9640 10 client.17817 handle_command_reply: tid=0 2024-09-21T02:56:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 client.17817 shutdown 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 -- 172.21.15.29:0/974701487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 msgr2=0x56527ed15bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed15bb0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00bad0 tx=0x7fd80c00bed0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 -- 172.21.15.29:0/974701487 shutdown_connections 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed16630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56527e922ea0 0x56527ed27f30 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 --2- 172.21.15.29:0/974701487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed15bb0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd7f8ff9640 0 client.17817 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 -- 172.21.15.29:0/974701487 >> 172.21.15.29:0/974701487 conn(0x56527e958910 msgr2=0x56527ed0d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 -- 172.21.15.29:0/974701487 shutdown_connections 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.451+0000 7fd81dcae740 1 -- 172.21.15.29:0/974701487 wait complete. 2024-09-21T02:56:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.452+0000 7fd81dcae740 20 client.17817 trim_cache size 0 max 16384 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd81dcae740 1 Processor -- start 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd81dcae740 1 -- start start 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed0b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed12da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed132e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd81dcae740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8101b2b80 con 0x56527e93b660 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd81dcae740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd8101b46c0 con 0x56527e943840 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd81dcae740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8101b3250 con 0x56527e964520 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd8154d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed0b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd8154d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed0b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36852/0 (socket says 172.21.15.29:36852) 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd8154d6640 1 -- 172.21.15.29:0/1802545679 learned_addr learned my addr 172.21.15.29:0/1802545679 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.469+0000 7fd814cd5640 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed12da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd815cd7640 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed132e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd8154d6640 1 -- 172.21.15.29:0/1802545679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 msgr2=0x56527ed132e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd8154d6640 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed132e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd8154d6640 1 -- 172.21.15.29:0/1802545679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 msgr2=0x56527ed12da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd8154d6640 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed12da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd8154d6640 1 -- 172.21.15.29:0/1802545679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56527ed0c450 con 0x56527e93b660 2024-09-21T02:56:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd815cd7640 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed132e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:56:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd8154d6640 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed0b540 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd80c01b890 tx=0x7fd80c0044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb8a4726986d1c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd7faffd640 1 -- 172.21.15.29:0/1802545679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c003e80 con 0x56527e93b660 2024-09-21T02:56:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd7faffd640 1 -- 172.21.15.29:0/1802545679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd80c00b400 con 0x56527e93b660 2024-09-21T02:56:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.470+0000 7fd7faffd640 1 -- 172.21.15.29:0/1802545679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c011760 con 0x56527e93b660 2024-09-21T02:56:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 -- 172.21.15.29:0/1802545679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 msgr2=0x56527ed0b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed0b540 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd80c01b890 tx=0x7fd80c0044e0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 -- 172.21.15.29:0/1802545679 shutdown_connections 2024-09-21T02:56:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e964520 0x56527ed132e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e943840 0x56527ed12da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 --2- 172.21.15.29:0/1802545679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e93b660 0x56527ed0b540 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 -- 172.21.15.29:0/1802545679 >> 172.21.15.29:0/1802545679 conn(0x56527e958910 msgr2=0x56527ed17a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 -- 172.21.15.29:0/1802545679 shutdown_connections 2024-09-21T02:56:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.471+0000 7fd81dcae740 1 -- 172.21.15.29:0/1802545679 wait complete. 2024-09-21T02:56:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.472+0000 7fd81dcae740 1 -- start start 2024-09-21T02:56:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.472+0000 7fd81dcae740 1 Processor -- start 2024-09-21T02:56:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.472+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e93b660 0x56527ed1a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.472+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e943840 0x56527ed1aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd81dcae740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed918f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd81dcae740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8101b3ff0 con 0x56527e964520 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd81dcae740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd8101b3920 con 0x56527e93b660 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd81dcae740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8101b0b60 con 0x56527e943840 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd81dcae740 10 client.0 _ll_register_callbacks cb 0x7fd80c0040a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed918f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36860/0 (socket says 172.21.15.29:36860) 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd8154d6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e93b660 0x56527ed1a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 -- 172.21.15.29:0/4238242436 learned_addr learned my addr 172.21.15.29:0/4238242436 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd814cd5640 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e943840 0x56527ed1aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 -- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e943840 msgr2=0x56527ed1aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e943840 0x56527ed1aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 -- 172.21.15.29:0/4238242436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e93b660 msgr2=0x56527ed1a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e93b660 0x56527ed1a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 -- 172.21.15.29:0/4238242436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56527ea46960 con 0x56527e964520 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd814cd5640 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e943840 0x56527ed1aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd815cd7640 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed918f0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd808031e90 tx=0x7fd80804f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab765fcf235455f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.473+0000 7fd7f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/4238242436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808060060 con 0x56527e964520 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/4238242436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd808004410 con 0x56527e964520 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 1 -- 172.21.15.29:0/4238242436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd80c00b400 con 0x56527e964520 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/4238242436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd808067050 con 0x56527e964520 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 10 client.17823 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 1 -- 172.21.15.29:0/4238242436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56527ed92f30 con 0x56527e964520 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/4238242436 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd80c00b400 con 0x56527e964520 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/4238242436 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fd808042070 con 0x56527e964520 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 10 client.17823 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 10 client.17823 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 20 client.17823 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56527e8f8250 0x56527ed952f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 4 client.17823 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "4d82b865", "release": true, "await": true}] 2024-09-21T02:56:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.474+0000 7fd81dcae740 1 -- 172.21.15.29:0/4238242436 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "4d82b865", "release": true, "await": true}) -- 0x56527ec7fbf0 con 0x56527e8f8250 2024-09-21T02:56:34.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.475+0000 7fd8154d6640 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56527e8f8250 0x56527ed952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:34.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.475+0000 7fd8154d6640 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56527e8f8250 0x56527ed952f0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63a8ce67b06c4ee3 server_cookie=2ff872210038950b in_seq=0 out_seq=0 2024-09-21T02:56:34.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.475+0000 7fd7f8ff9640 10 client.17823 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd7f8ff9640 1 -- 172.21.15.29:0/4238242436 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56527ec7fbf0 con 0x56527e8f8250 2024-09-21T02:56:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd7f8ff9640 10 client.17823 handle_command_reply: tid=0 2024-09-21T02:56:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 client.17823 shutdown 2024-09-21T02:56:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 -- 172.21.15.29:0/4238242436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 msgr2=0x56527ed918f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed918f0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd808031e90 tx=0x7fd80804f000 comp rx=0 tx=0).stop 2024-09-21T02:56:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 -- 172.21.15.29:0/4238242436 shutdown_connections 2024-09-21T02:56:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56527e8f8250 0x56527ed952f0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56527e964520 0x56527ed918f0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd7f8ff9640 0 client.17823 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56527e943840 0x56527ed1aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 --2- 172.21.15.29:0/4238242436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56527e93b660 0x56527ed1a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 -- 172.21.15.29:0/4238242436 >> 172.21.15.29:0/4238242436 conn(0x56527e958910 msgr2=0x56527ed10790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.536+0000 7fd81dcae740 1 -- 172.21.15.29:0/4238242436 shutdown_connections 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.537+0000 7fd81dcae740 1 -- 172.21.15.29:0/4238242436 wait complete. 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.537+0000 7fd81dcae740 20 client.17823 trim_cache size 0 max 16384 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 68, 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: "4d82b865": { 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 68, 2024-09-21T02:56:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 130.0, 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:34.541 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:34.541 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:56:34.541 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:56:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 -- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7fc0b6510 msgr2=0x7fd7fc0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7fc0b6510 0x7fd7fc0b8a20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd808049000 tx=0x7fd80804a000 comp rx=0 tx=0).stop 2024-09-21T02:56:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 -- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810107b30 msgr2=0x7fd81019cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810107b30 0x7fd81019cae0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd80000d740 tx=0x7fd80000dc00 comp rx=0 tx=0).stop 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 -- 172.21.15.29:0/1520435642 shutdown_connections 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7fc0b6510 0x7fd7fc0b8a20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd810108480 0x7fd8101a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd810107b30 0x7fd81019cae0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 --2- 172.21.15.29:0/1520435642 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd8101021f0 0x7fd81019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 -- 172.21.15.29:0/1520435642 >> 172.21.15.29:0/1520435642 conn(0x7fd810077c60 msgr2=0x7fd81006a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 -- 172.21.15.29:0/1520435642 shutdown_connections 2024-09-21T02:56:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:56:34.538+0000 7fd81773a640 1 -- 172.21.15.29:0/1520435642 wait complete. 2024-09-21T02:56:34.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27205 5.75 MB/sec execute 131 sec latency 4608.961 ms 2024-09-21T02:56:35.514 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '4d82b865', total seconds elapsed: 21.9 2024-09-21T02:56:35.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27371 5.82 MB/sec execute 132 sec latency 87.666 ms 2024-09-21T02:56:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:35 smithi190 ceph-mon[33871]: pgmap v913: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-21T02:56:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:35 smithi029 ceph-mon[31006]: pgmap v913: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-21T02:56:36.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:35 smithi066 ceph-mon[33436]: pgmap v913: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-21T02:56:36.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27568 5.90 MB/sec execute 133 sec latency 85.684 ms 2024-09-21T02:56:37.723 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27771 5.94 MB/sec execute 134 sec latency 82.765 ms 2024-09-21T02:56:37.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:37 smithi190 ceph-mon[33871]: pgmap v914: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-09-21T02:56:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:37 smithi029 ceph-mon[31006]: pgmap v914: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-09-21T02:56:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:37 smithi066 ceph-mon[33436]: pgmap v914: 129 pgs: 129 active+clean; 270 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-09-21T02:56:38.722 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27940 5.99 MB/sec execute 135 sec latency 91.710 ms 2024-09-21T02:56:39.723 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28102 6.02 MB/sec execute 136 sec latency 90.322 ms 2024-09-21T02:56:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:39 smithi190 ceph-mon[33871]: pgmap v915: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 2.2 MiB/s wr, 192 op/s 2024-09-21T02:56:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:39 smithi029 ceph-mon[31006]: pgmap v915: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 2.2 MiB/s wr, 192 op/s 2024-09-21T02:56:40.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:39 smithi066 ceph-mon[33436]: pgmap v915: 129 pgs: 129 active+clean; 277 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 2.2 MiB/s wr, 192 op/s 2024-09-21T02:56:40.724 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28218 6.01 MB/sec execute 137 sec latency 84.713 ms 2024-09-21T02:56:41.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28371 6.11 MB/sec execute 138 sec latency 105.169 ms 2024-09-21T02:56:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:41 smithi190 ceph-mon[33871]: pgmap v916: 129 pgs: 129 active+clean; 281 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 2.4 MiB/s wr, 193 op/s 2024-09-21T02:56:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:56:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:56:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:56:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:56:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:41 smithi029 ceph-mon[31006]: pgmap v916: 129 pgs: 129 active+clean; 281 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 2.4 MiB/s wr, 193 op/s 2024-09-21T02:56:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:56:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:56:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:56:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:56:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:41 smithi066 ceph-mon[33436]: pgmap v916: 129 pgs: 129 active+clean; 281 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 2.4 MiB/s wr, 193 op/s 2024-09-21T02:56:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:56:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:56:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:56:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:56:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.391+0000 7fac8eede640 1 -- 172.21.15.29:0/49814523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac881040d0 msgr2=0x7fac8810ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.391+0000 7fac8eede640 1 --2- 172.21.15.29:0/49814523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac881040d0 0x7fac8810ab60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fac80009ca0 tx=0x7fac8002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.391+0000 7fac8eede640 1 -- 172.21.15.29:0/49814523 shutdown_connections 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.391+0000 7fac8eede640 1 --2- 172.21.15.29:0/49814523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac881040d0 0x7fac8810ab60 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.392+0000 7fac8eede640 1 --2- 172.21.15.29:0/49814523 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac881034c0 0x7fac88103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.392+0000 7fac8eede640 1 --2- 172.21.15.29:0/49814523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac88102ae0 0x7fac88102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.392+0000 7fac8eede640 1 -- 172.21.15.29:0/49814523 >> 172.21.15.29:0/49814523 conn(0x7fac880fd690 msgr2=0x7fac880ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.392+0000 7fac8eede640 1 -- 172.21.15.29:0/49814523 shutdown_connections 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.392+0000 7fac8eede640 1 -- 172.21.15.29:0/49814523 wait complete. 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.392+0000 7fac8eede640 1 Processor -- start 2024-09-21T02:56:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.392+0000 7fac8eede640 1 -- start start 2024-09-21T02:56:42.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8eede640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac88102ae0 0x7fac88197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8eede640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac881034c0 0x7fac88198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8eede640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac881040d0 0x7fac88198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8eede640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fac8810d030 con 0x7fac88102ae0 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8eede640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fac8810ceb0 con 0x7fac881034c0 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8eede640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fac8810d1b0 con 0x7fac881040d0 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8cc7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac88102ae0 0x7fac88197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8cc7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac88102ae0 0x7fac88197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36864/0 (socket says 172.21.15.29:36864) 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8cc7a640 1 -- 172.21.15.29:0/2507777641 learned_addr learned my addr 172.21.15.29:0/2507777641 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac87fff640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac881034c0 0x7fac88198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8d47b640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac881040d0 0x7fac88198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8cc7a640 1 -- 172.21.15.29:0/2507777641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac881040d0 msgr2=0x7fac88198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac881040d0 0x7fac88198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8cc7a640 1 -- 172.21.15.29:0/2507777641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac881034c0 msgr2=0x7fac88198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac881034c0 0x7fac88198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.393+0000 7fac8cc7a640 1 -- 172.21.15.29:0/2507777641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac88199630 con 0x7fac88102ae0 2024-09-21T02:56:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.394+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac88102ae0 0x7fac88197bf0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009730 tx=0x7fac7c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.394+0000 7fac85ffb640 1 -- 172.21.15.29:0/2507777641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c007bb0 con 0x7fac88102ae0 2024-09-21T02:56:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.394+0000 7fac85ffb640 1 -- 172.21.15.29:0/2507777641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac7c01b070 con 0x7fac88102ae0 2024-09-21T02:56:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.394+0000 7fac85ffb640 1 -- 172.21.15.29:0/2507777641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c015380 con 0x7fac88102ae0 2024-09-21T02:56:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.394+0000 7fac8eede640 1 -- 172.21.15.29:0/2507777641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac88199920 con 0x7fac88102ae0 2024-09-21T02:56:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.394+0000 7fac8eede640 1 -- 172.21.15.29:0/2507777641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac881a0010 con 0x7fac88102ae0 2024-09-21T02:56:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.397+0000 7fac95e4b740 1 Processor -- start 2024-09-21T02:56:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.397+0000 7fac95e4b740 1 -- start start 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be29e1b0 0x5587be645e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 0x5587be6464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be647010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac95e4b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5587be5b86f0 con 0x5587be2752f0 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac95e4b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5587be5a7b20 con 0x5587be27d4d0 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac95e4b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5587be5ab930 con 0x5587be29e1b0 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac8d47b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be29e1b0 0x5587be645e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac8d47b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be29e1b0 0x5587be645e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43948/0 (socket says 172.21.15.29:43948) 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac8d47b640 1 -- 172.21.15.29:0/3600442118 learned_addr learned my addr 172.21.15.29:0/3600442118 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac87fff640 1 --2- 172.21.15.29:0/3600442118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be647010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac8d47b640 1 -- 172.21.15.29:0/3600442118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 msgr2=0x5587be647010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac8d47b640 1 --2- 172.21.15.29:0/3600442118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be647010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac8d47b640 1 -- 172.21.15.29:0/3600442118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 msgr2=0x5587be6464c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:56:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac8d47b640 1 --2- 172.21.15.29:0/3600442118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 0x5587be6464c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.398+0000 7fac8d47b640 1 -- 172.21.15.29:0/3600442118 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587be56d530 con 0x5587be29e1b0 2024-09-21T02:56:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.399+0000 7fac8d47b640 1 --2- 172.21.15.29:0/3600442118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be29e1b0 0x5587be645e50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fac80035000 tx=0x7fac80005c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffd9b4f75c8cce9e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.399+0000 7fac72ffd640 1 -- 172.21.15.29:0/3600442118 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8004c070 con 0x5587be29e1b0 2024-09-21T02:56:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.399+0000 7fac72ffd640 1 -- 172.21.15.29:0/3600442118 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac80036040 con 0x5587be29e1b0 2024-09-21T02:56:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.399+0000 7fac72ffd640 1 -- 172.21.15.29:0/3600442118 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8003c940 con 0x5587be29e1b0 2024-09-21T02:56:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.399+0000 7fac85ffb640 1 -- 172.21.15.29:0/2507777641 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fac7c016420 con 0x7fac88102ae0 2024-09-21T02:56:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 -- 172.21.15.29:0/3600442118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be29e1b0 msgr2=0x5587be645e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3600442118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be29e1b0 0x5587be645e50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fac80035000 tx=0x7fac80005c70 comp rx=0 tx=0).stop 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 -- 172.21.15.29:0/3600442118 shutdown_connections 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3600442118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be647010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3600442118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 0x5587be6464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3600442118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be29e1b0 0x5587be645e50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 -- 172.21.15.29:0/3600442118 >> 172.21.15.29:0/3600442118 conn(0x5587be292300 msgr2=0x5587be6418e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 -- 172.21.15.29:0/3600442118 shutdown_connections 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac95e4b740 1 -- 172.21.15.29:0/3600442118 wait complete. 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.400+0000 7fac85ffb640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fac780b6520 0x7fac780b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.401+0000 7fac85ffb640 1 -- 172.21.15.29:0/2507777641 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fac7c0e6bc0 con 0x7fac88102ae0 2024-09-21T02:56:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.401+0000 7fac95e4b740 1 -- start start 2024-09-21T02:56:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.401+0000 7fac87fff640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fac780b6520 0x7fac780b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.401+0000 7fac95e4b740 1 Processor -- start 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be657ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fac780b6520 0x7fac780b8a30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fac74009d40 tx=0x7fac74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be27d4d0 0x5587be658590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be29e1b0 0x5587be659100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac95e4b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5587be4dfab0 con 0x5587be29e1b0 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac95e4b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5587be54d730 con 0x5587be2752f0 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac95e4b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5587be647ad0 con 0x5587be27d4d0 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac8d47b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be29e1b0 0x5587be659100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be27d4d0 0x5587be658590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac95e4b740 10 client.0 _ll_register_callbacks cb 0x5587be5a16b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac8cc7a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be657ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be27d4d0 0x5587be658590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43964/0 (socket says 172.21.15.29:43964) 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac8d47b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be29e1b0 0x5587be659100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36866/0 (socket says 172.21.15.29:36866) 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 -- 172.21.15.29:0/80494845 learned_addr learned my addr 172.21.15.29:0/80494845 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac8cc7a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be657ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:57814/0 (socket says 172.21.15.29:57814) 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 -- 172.21.15.29:0/80494845 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 msgr2=0x5587be657ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be657ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 -- 172.21.15.29:0/80494845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be29e1b0 msgr2=0x5587be659100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be29e1b0 0x5587be659100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.402+0000 7fac87fff640 1 -- 172.21.15.29:0/80494845 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587be309790 con 0x5587be27d4d0 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.403+0000 7fac87fff640 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be27d4d0 0x5587be658590 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fac74004620 tx=0x7fac74004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa815470179bcd1d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.403+0000 7fac67fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.403+0000 7fac67fff640 1 -- 172.21.15.29:0/80494845 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7401f640 con 0x5587be27d4d0 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.403+0000 7fac67fff640 1 -- 172.21.15.29:0/80494845 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fac7401f7e0 con 0x5587be27d4d0 2024-09-21T02:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.403+0000 7fac95e4b740 1 -- 172.21.15.29:0/80494845 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac80036040 con 0x5587be27d4d0 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.403+0000 7fac67fff640 1 -- 172.21.15.29:0/80494845 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fac7401f9e0 con 0x5587be27d4d0 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.403+0000 7fac95e4b740 10 client.27325 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.403+0000 7fac95e4b740 1 -- 172.21.15.29:0/80494845 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5587be382620 con 0x5587be27d4d0 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac67fff640 1 -- 172.21.15.29:0/80494845 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac80036040 con 0x5587be27d4d0 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac67fff640 1 -- 172.21.15.29:0/80494845 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fac74027070 con 0x5587be27d4d0 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac95e4b740 10 client.27325 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac95e4b740 10 client.27325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac95e4b740 20 client.27325 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac95e4b740 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5587be25cd80 0x5587be665730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac95e4b740 4 client.27325 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac95e4b740 1 -- 172.21.15.29:0/80494845 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587be5ba640 con 0x5587be25cd80 2024-09-21T02:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.404+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5587be25cd80 0x5587be665730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.405+0000 7fac67fff640 10 client.27325 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.405+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5587be25cd80 0x5587be665730 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e97ddd98e6cb1ade server_cookie=10d4c2653cfa74c2 in_seq=0 out_seq=0 2024-09-21T02:56:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac67fff640 1 -- 172.21.15.29:0/80494845 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5587be5ba640 con 0x5587be25cd80 2024-09-21T02:56:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac67fff640 10 client.27325 handle_command_reply: tid=0 2024-09-21T02:56:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 client.27325 shutdown 2024-09-21T02:56:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 -- 172.21.15.29:0/80494845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be27d4d0 msgr2=0x5587be658590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be27d4d0 0x5587be658590 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fac74004620 tx=0x7fac74004730 comp rx=0 tx=0).stop 2024-09-21T02:56:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 -- 172.21.15.29:0/80494845 shutdown_connections 2024-09-21T02:56:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5587be25cd80 0x5587be665730 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac67fff640 0 client.27325 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be29e1b0 0x5587be659100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be27d4d0 0x5587be658590 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 --2- 172.21.15.29:0/80494845 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be657ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 -- 172.21.15.29:0/80494845 >> 172.21.15.29:0/80494845 conn(0x5587be292300 msgr2=0x5587be63fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 -- 172.21.15.29:0/80494845 shutdown_connections 2024-09-21T02:56:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 1 -- 172.21.15.29:0/80494845 wait complete. 2024-09-21T02:56:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.408+0000 7fac95e4b740 20 client.27325 trim_cache size 0 max 16384 2024-09-21T02:56:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac95e4b740 1 Processor -- start 2024-09-21T02:56:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac95e4b740 1 -- start start 2024-09-21T02:56:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be63c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be65ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be27d4d0 0x5587be641dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac95e4b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fac881aa2b0 con 0x5587be27d4d0 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac95e4b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fac881abdf0 con 0x5587be2752f0 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac95e4b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fac881aa980 con 0x5587be25cd80 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac8d47b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be27d4d0 0x5587be641dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac8d47b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be27d4d0 0x5587be641dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36876/0 (socket says 172.21.15.29:36876) 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac8cc7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be63c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac8d47b640 1 -- 172.21.15.29:0/1102169464 learned_addr learned my addr 172.21.15.29:0/1102169464 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.434+0000 7fac87fff640 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be65ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac8cc7a640 1 -- 172.21.15.29:0/1102169464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 msgr2=0x5587be65ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be65ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac8cc7a640 1 -- 172.21.15.29:0/1102169464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be27d4d0 msgr2=0x5587be641dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be27d4d0 0x5587be641dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac8cc7a640 1 -- 172.21.15.29:0/1102169464 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587be64d8a0 con 0x5587be25cd80 2024-09-21T02:56:42.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac8d47b640 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be27d4d0 0x5587be641dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:42.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be63c5c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0a9c70 tx=0x7fac7c007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53e5264a02d64c0b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:42.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac727fc640 1 -- 172.21.15.29:0/1102169464 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0a9050 con 0x5587be25cd80 2024-09-21T02:56:42.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac727fc640 1 -- 172.21.15.29:0/1102169464 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac7c0ac510 con 0x5587be25cd80 2024-09-21T02:56:42.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.435+0000 7fac727fc640 1 -- 172.21.15.29:0/1102169464 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0a8c70 con 0x5587be25cd80 2024-09-21T02:56:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 -- 172.21.15.29:0/1102169464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 msgr2=0x5587be63c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be63c5c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0a9c70 tx=0x7fac7c007d50 comp rx=0 tx=0).stop 2024-09-21T02:56:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 -- 172.21.15.29:0/1102169464 shutdown_connections 2024-09-21T02:56:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be27d4d0 0x5587be641dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be2752f0 0x5587be65ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 --2- 172.21.15.29:0/1102169464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be63c5c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 -- 172.21.15.29:0/1102169464 >> 172.21.15.29:0/1102169464 conn(0x5587be292300 msgr2=0x5587be641220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 -- 172.21.15.29:0/1102169464 shutdown_connections 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.436+0000 7fac95e4b740 1 -- 172.21.15.29:0/1102169464 wait complete. 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 1 -- start start 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 1 Processor -- start 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be65fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 0x5587be65e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be65eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fac881ab720 con 0x5587be2752f0 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fac881ab050 con 0x5587be27d4d0 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fac881a8290 con 0x5587be25cd80 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.437+0000 7fac95e4b740 10 client.0 _ll_register_callbacks cb 0x7fac740099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac87fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 0x5587be65e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be65fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be65fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43982/0 (socket says 172.21.15.29:43982) 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 -- 172.21.15.29:0/3792371592 learned_addr learned my addr 172.21.15.29:0/3792371592 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8d47b640 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be65eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 -- 172.21.15.29:0/3792371592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 msgr2=0x5587be65eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be65eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 -- 172.21.15.29:0/3792371592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 msgr2=0x5587be65e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 0x5587be65e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 -- 172.21.15.29:0/3792371592 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587be65f670 con 0x5587be25cd80 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac87fff640 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 0x5587be65e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8d47b640 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be65eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac8cc7a640 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be65fce0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0a8970 tx=0x7fac7c005b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60731c80c8dda9d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac67fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.438+0000 7fac67fff640 1 -- 172.21.15.29:0/3792371592 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0b14e0 con 0x5587be25cd80 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac67fff640 1 -- 172.21.15.29:0/3792371592 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fac7c016710 con 0x5587be25cd80 2024-09-21T02:56:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 1 -- 172.21.15.29:0/3792371592 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac7c0ac510 con 0x5587be25cd80 2024-09-21T02:56:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac67fff640 1 -- 172.21.15.29:0/3792371592 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fac7c0ad100 con 0x5587be25cd80 2024-09-21T02:56:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 10 client.27331 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 1 -- 172.21.15.29:0/3792371592 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5587be6d0130 con 0x5587be25cd80 2024-09-21T02:56:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac67fff640 1 -- 172.21.15.29:0/3792371592 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac7c0ac510 con 0x5587be25cd80 2024-09-21T02:56:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac67fff640 1 -- 172.21.15.29:0/3792371592 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fac7c0d1030 con 0x5587be25cd80 2024-09-21T02:56:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 10 client.27331 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 10 client.27331 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 20 client.27331 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5587be29e1b0 0x5587be6d2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 4 client.27331 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:56:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac95e4b740 1 -- 172.21.15.29:0/3792371592 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5587be5b60f0 con 0x5587be29e1b0 2024-09-21T02:56:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.439+0000 7fac87fff640 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5587be29e1b0 0x5587be6d2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.440+0000 7fac67fff640 10 client.27331 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.440+0000 7fac87fff640 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5587be29e1b0 0x5587be6d2ad0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71b1bcbb50f63d91 server_cookie=fa9709233959a1f1 in_seq=0 out_seq=0 2024-09-21T02:56:42.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.611+0000 7fac67fff640 1 -- 172.21.15.29:0/3792371592 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5587be5b60f0 con 0x5587be29e1b0 2024-09-21T02:56:42.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.611+0000 7fac67fff640 10 client.27331 handle_command_reply: tid=0 2024-09-21T02:56:42.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.611+0000 7fac95e4b740 1 client.27331 shutdown 2024-09-21T02:56:42.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.611+0000 7fac95e4b740 1 -- 172.21.15.29:0/3792371592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 msgr2=0x5587be65fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.611+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be65fce0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0a8970 tx=0x7fac7c005b40 comp rx=0 tx=0).stop 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 1 -- 172.21.15.29:0/3792371592 shutdown_connections 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5587be27d4d0 0x5587be65eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5587be29e1b0 0x5587be6d2ad0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587be2752f0 0x5587be65e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac67fff640 0 client.27331 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 1 --2- 172.21.15.29:0/3792371592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5587be25cd80 0x5587be65fce0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 1 -- 172.21.15.29:0/3792371592 >> 172.21.15.29:0/3792371592 conn(0x5587be292300 msgr2=0x5587be641220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 1 -- 172.21.15.29:0/3792371592 shutdown_connections 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 1 -- 172.21.15.29:0/3792371592 wait complete. 2024-09-21T02:56:42.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.612+0000 7fac95e4b740 20 client.27331 trim_cache size 0 max 16384 2024-09-21T02:56:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:56:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:56:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:56:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:56:42.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 -- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fac780b6520 msgr2=0x7fac780b8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fac780b6520 0x7fac780b8a30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fac74009d40 tx=0x7fac74005f90 comp rx=0 tx=0).stop 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 -- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac88102ae0 msgr2=0x7fac88197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac88102ae0 0x7fac88197bf0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009730 tx=0x7fac7c009bf0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 -- 172.21.15.29:0/2507777641 shutdown_connections 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fac780b6520 0x7fac780b8a30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac881040d0 0x7fac88198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac881034c0 0x7fac88198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 --2- 172.21.15.29:0/2507777641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac88102ae0 0x7fac88197bf0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 -- 172.21.15.29:0/2507777641 >> 172.21.15.29:0/2507777641 conn(0x7fac880fd690 msgr2=0x7fac88106b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.613+0000 7fac8eede640 1 -- 172.21.15.29:0/2507777641 shutdown_connections 2024-09-21T02:56:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:42.614+0000 7fac8eede640 1 -- 172.21.15.29:0/2507777641 wait complete. 2024-09-21T02:56:42.724 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28484 6.08 MB/sec execute 139 sec latency 215.768 ms 2024-09-21T02:56:43.724 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28613 6.10 MB/sec execute 140 sec latency 99.766 ms 2024-09-21T02:56:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:43 smithi190 ceph-mon[33871]: pgmap v917: 129 pgs: 129 active+clean; 287 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-09-21T02:56:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:43 smithi029 ceph-mon[31006]: pgmap v917: 129 pgs: 129 active+clean; 287 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-09-21T02:56:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:43 smithi066 ceph-mon[33436]: pgmap v917: 129 pgs: 129 active+clean; 287 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-09-21T02:56:44.724 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28752 6.12 MB/sec execute 141 sec latency 99.887 ms 2024-09-21T02:56:45.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28884 6.12 MB/sec execute 142 sec latency 91.634 ms 2024-09-21T02:56:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:45 smithi190 ceph-mon[33871]: pgmap v918: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-09-21T02:56:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:45 smithi029 ceph-mon[31006]: pgmap v918: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-09-21T02:56:46.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:45 smithi066 ceph-mon[33436]: pgmap v918: 129 pgs: 129 active+clean; 286 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-09-21T02:56:46.724 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29035 6.14 MB/sec execute 143 sec latency 92.155 ms 2024-09-21T02:56:47.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29189 6.14 MB/sec execute 144 sec latency 110.325 ms 2024-09-21T02:56:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:47 smithi190 ceph-mon[33871]: pgmap v919: 129 pgs: 129 active+clean; 287 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 3.5 MiB/s wr, 211 op/s 2024-09-21T02:56:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:47 smithi029 ceph-mon[31006]: pgmap v919: 129 pgs: 129 active+clean; 287 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 3.5 MiB/s wr, 211 op/s 2024-09-21T02:56:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:47 smithi066 ceph-mon[33436]: pgmap v919: 129 pgs: 129 active+clean; 287 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 3.5 MiB/s wr, 211 op/s 2024-09-21T02:56:48.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29288 6.11 MB/sec execute 145 sec latency 108.391 ms 2024-09-21T02:56:49.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29433 6.12 MB/sec execute 146 sec latency 109.849 ms 2024-09-21T02:56:49.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:49 smithi029 ceph-mon[31006]: pgmap v920: 129 pgs: 129 active+clean; 291 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 4.1 MiB/s wr, 242 op/s 2024-09-21T02:56:50.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:49 smithi066 ceph-mon[33436]: pgmap v920: 129 pgs: 129 active+clean; 291 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 4.1 MiB/s wr, 242 op/s 2024-09-21T02:56:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:49 smithi190 ceph-mon[33871]: pgmap v920: 129 pgs: 129 active+clean; 291 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 4.1 MiB/s wr, 242 op/s 2024-09-21T02:56:50.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29537 6.10 MB/sec execute 147 sec latency 164.322 ms 2024-09-21T02:56:51.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29652 6.09 MB/sec execute 148 sec latency 259.147 ms 2024-09-21T02:56:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:51 smithi029 ceph-mon[31006]: pgmap v921: 129 pgs: 129 active+clean; 293 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 4.0 MiB/s wr, 258 op/s 2024-09-21T02:56:51.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:51 smithi190 ceph-mon[33871]: pgmap v921: 129 pgs: 129 active+clean; 293 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 4.0 MiB/s wr, 258 op/s 2024-09-21T02:56:52.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:51 smithi066 ceph-mon[33436]: pgmap v921: 129 pgs: 129 active+clean; 293 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 4.0 MiB/s wr, 258 op/s 2024-09-21T02:56:52.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29806 6.08 MB/sec execute 149 sec latency 115.691 ms 2024-09-21T02:56:53.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29977 6.15 MB/sec execute 150 sec latency 122.691 ms 2024-09-21T02:56:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:53 smithi029 ceph-mon[31006]: pgmap v922: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.0 MiB/s wr, 253 op/s 2024-09-21T02:56:54.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:53 smithi066 ceph-mon[33436]: pgmap v922: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.0 MiB/s wr, 253 op/s 2024-09-21T02:56:54.086 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10.2 2024-09-21T02:56:54.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:56:54.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:53 smithi190 ceph-mon[33871]: pgmap v922: 129 pgs: 129 active+clean; 288 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.0 MiB/s wr, 253 op/s 2024-09-21T02:56:54.450 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:56:54.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30111 6.14 MB/sec execute 151 sec latency 80.374 ms 2024-09-21T02:56:55.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30240 6.15 MB/sec execute 152 sec latency 142.400 ms 2024-09-21T02:56:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 -- 172.21.15.29:0/3176612409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 msgr2=0x7fca1c103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 --2- 172.21.15.29:0/3176612409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 0x7fca1c103b30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fca18009900 tx=0x7fca1802a410 comp rx=0 tx=0).stop 2024-09-21T02:56:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 -- 172.21.15.29:0/3176612409 shutdown_connections 2024-09-21T02:56:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 --2- 172.21.15.29:0/3176612409 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca1c1050e0 0x7fca1c10ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 --2- 172.21.15.29:0/3176612409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca1c104740 0x7fca1c104b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 --2- 172.21.15.29:0/3176612409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 0x7fca1c103b30 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 -- 172.21.15.29:0/3176612409 >> 172.21.15.29:0/3176612409 conn(0x7fca1c0ff020 msgr2=0x7fca1c1014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 -- 172.21.15.29:0/3176612409 shutdown_connections 2024-09-21T02:56:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.738+0000 7fca23539640 1 -- 172.21.15.29:0/3176612409 wait complete. 2024-09-21T02:56:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca23539640 1 Processor -- start 2024-09-21T02:56:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca23539640 1 -- start start 2024-09-21T02:56:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca23539640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 0x7fca1c199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca23539640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca1c104740 0x7fca1c19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca23539640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca1c1050e0 0x7fca1c19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca23539640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c10f190 con 0x7fca1c1036c0 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca23539640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c10f010 con 0x7fca1c1050e0 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca23539640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c10f310 con 0x7fca1c104740 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.739+0000 7fca212d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 0x7fca1c199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca212d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 0x7fca1c199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51186/0 (socket says 172.21.15.29:51186) 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca212d5640 1 -- 172.21.15.29:0/3225409657 learned_addr learned my addr 172.21.15.29:0/3225409657 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca1c104740 0x7fca1c19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca21ad6640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca1c1050e0 0x7fca1c19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca212d5640 1 -- 172.21.15.29:0/3225409657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca1c104740 msgr2=0x7fca1c19a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca212d5640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca1c104740 0x7fca1c19a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca212d5640 1 -- 172.21.15.29:0/3225409657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca1c1050e0 msgr2=0x7fca1c19b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca212d5640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca1c1050e0 0x7fca1c19b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca212d5640 1 -- 172.21.15.29:0/3225409657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1c19b8e0 con 0x7fca1c1036c0 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca1c104740 0x7fca1c19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca212d5640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 0x7fca1c199ea0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fca18009230 tx=0x7fca18004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca127fc640 1 -- 172.21.15.29:0/3225409657 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1803f030 con 0x7fca1c1036c0 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.740+0000 7fca23539640 1 -- 172.21.15.29:0/3225409657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1c19bb70 con 0x7fca1c1036c0 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.741+0000 7fca127fc640 1 -- 172.21.15.29:0/3225409657 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca18004440 con 0x7fca1c1036c0 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.741+0000 7fca127fc640 1 -- 172.21.15.29:0/3225409657 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18035070 con 0x7fca1c1036c0 2024-09-21T02:56:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.741+0000 7fca23539640 1 -- 172.21.15.29:0/3225409657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca1c1a22c0 con 0x7fca1c1036c0 2024-09-21T02:56:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.744+0000 7fca2a4a7740 1 Processor -- start 2024-09-21T02:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.744+0000 7fca127fc640 1 -- 172.21.15.29:0/3225409657 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fca18042030 con 0x7fca1c1036c0 2024-09-21T02:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.744+0000 7fca2a4a7740 1 -- start start 2024-09-21T02:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba57be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba57c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba57d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562fba4ee6f0 con 0x562fba1d41b0 2024-09-21T02:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562fba4ddb20 con 0x562fba1ab2f0 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562fba4e1930 con 0x562fba1b34d0 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca20ad4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba57d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca21ad6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba57be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca20ad4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba57d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45630/0 (socket says 172.21.15.29:45630) 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca20ad4640 1 -- 172.21.15.29:0/2156871707 learned_addr learned my addr 172.21.15.29:0/2156871707 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca212d5640 1 --2- 172.21.15.29:0/2156871707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba57c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca21ad6640 1 -- 172.21.15.29:0/2156871707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 msgr2=0x562fba57d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca21ad6640 1 --2- 172.21.15.29:0/2156871707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba57d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca21ad6640 1 -- 172.21.15.29:0/2156871707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 msgr2=0x562fba57c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.745+0000 7fca21ad6640 1 --2- 172.21.15.29:0/2156871707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba57c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.746+0000 7fca21ad6640 1 -- 172.21.15.29:0/2156871707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fba4a3530 con 0x562fba1d41b0 2024-09-21T02:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.746+0000 7fca127fc640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fca080b6590 0x7fca080b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.746+0000 7fca21ad6640 1 --2- 172.21.15.29:0/2156871707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba57be50 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fca14009f00 tx=0x7fca14011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8f81808ec3e819 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.746+0000 7fca212d5640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fca080b6590 0x7fca080b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.746+0000 7fca127fc640 1 -- 172.21.15.29:0/3225409657 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fca18107ae0 con 0x7fca1c1036c0 2024-09-21T02:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.746+0000 7fca077fe640 1 -- 172.21.15.29:0/2156871707 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14004d20 con 0x562fba1d41b0 2024-09-21T02:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.746+0000 7fca077fe640 1 -- 172.21.15.29:0/2156871707 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca14004030 con 0x562fba1d41b0 2024-09-21T02:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.746+0000 7fca077fe640 1 -- 172.21.15.29:0/2156871707 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14005280 con 0x562fba1d41b0 2024-09-21T02:56:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.747+0000 7fca212d5640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fca080b6590 0x7fca080b8aa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fca180cac10 tx=0x7fca1810b5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.747+0000 7fca2a4a7740 1 -- 172.21.15.29:0/2156871707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 msgr2=0x562fba57be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.747+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/2156871707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba57be50 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fca14009f00 tx=0x7fca14011810 comp rx=0 tx=0).stop 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 -- 172.21.15.29:0/2156871707 shutdown_connections 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/2156871707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba57d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/2156871707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba57c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/2156871707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba57be50 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 -- 172.21.15.29:0/2156871707 >> 172.21.15.29:0/2156871707 conn(0x562fba1c8300 msgr2=0x562fba5778e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 -- 172.21.15.29:0/2156871707 shutdown_connections 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 -- 172.21.15.29:0/2156871707 wait complete. 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 -- start start 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.748+0000 7fca2a4a7740 1 Processor -- start 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5843a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1b34d0 0x562fba584a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba5855d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562fba415ab0 con 0x562fba1d41b0 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562fba483730 con 0x562fba1b34d0 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562fba57db30 con 0x562fba1ab2f0 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca21ad6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5843a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca21ad6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5843a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45646/0 (socket says 172.21.15.29:45646) 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca21ad6640 1 -- 172.21.15.29:0/3366082643 learned_addr learned my addr 172.21.15.29:0/3366082643 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca2a4a7740 10 client.0 _ll_register_callbacks cb 0x562fba4d76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca21ad6640 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba5855d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.749+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1b34d0 0x562fba584a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca21ad6640 1 -- 172.21.15.29:0/3366082643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1b34d0 msgr2=0x562fba584a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca21ad6640 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1b34d0 0x562fba584a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca21ad6640 1 -- 172.21.15.29:0/3366082643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 msgr2=0x562fba5855d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca21ad6640 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba5855d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca21ad6640 1 -- 172.21.15.29:0/3366082643 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fba23f790 con 0x562fba1ab2f0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1b34d0 0x562fba584a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca21ad6640 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5843a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fca14011fd0 tx=0x7fca14004720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a01839492c539b82 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca04ff9640 1 -- 172.21.15.29:0/3366082643 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1402d060 con 0x562fba1ab2f0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3366082643 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca14004030 con 0x562fba1ab2f0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca04ff9640 1 -- 172.21.15.29:0/3366082643 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fca14004a40 con 0x562fba1ab2f0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca04ff9640 1 -- 172.21.15.29:0/3366082643 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fca1401e040 con 0x562fba1ab2f0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca2a4a7740 10 client.27343 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.750+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3366082643 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562fba2b8620 con 0x562fba1ab2f0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca04ff9640 1 -- 172.21.15.29:0/3366082643 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca14004030 con 0x562fba1ab2f0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca04ff9640 1 -- 172.21.15.29:0/3366082643 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fca14024070 con 0x562fba1ab2f0 2024-09-21T02:56:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca2a4a7740 10 client.27343 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca2a4a7740 10 client.27343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca2a4a7740 20 client.27343 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562fba192d80 0x562fba59b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca2a4a7740 4 client.27343 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:56:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3366082643 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fba4f0640 con 0x562fba192d80 2024-09-21T02:56:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.751+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562fba192d80 0x562fba59b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.753+0000 7fca04ff9640 10 client.27343 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.753+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562fba192d80 0x562fba59b8a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2938f14c0df0518d server_cookie=e29987bd90daefd9 in_seq=0 out_seq=0 2024-09-21T02:56:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.754+0000 7fca04ff9640 1 -- 172.21.15.29:0/3366082643 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562fba4f0640 con 0x562fba192d80 2024-09-21T02:56:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.754+0000 7fca04ff9640 10 client.27343 handle_command_reply: tid=0 2024-09-21T02:56:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.754+0000 7fca2a4a7740 1 client.27343 shutdown 2024-09-21T02:56:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3366082643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 msgr2=0x562fba5843a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5843a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fca14011fd0 tx=0x7fca14004720 comp rx=0 tx=0).stop 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3366082643 shutdown_connections 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562fba192d80 0x562fba59b8a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1d41b0 0x562fba5855d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1b34d0 0x562fba584a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca04ff9640 0 client.27343 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3366082643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5843a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3366082643 >> 172.21.15.29:0/3366082643 conn(0x562fba1c8300 msgr2=0x562fba57cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3366082643 shutdown_connections 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3366082643 wait complete. 2024-09-21T02:56:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.755+0000 7fca2a4a7740 20 client.27343 trim_cache size 0 max 16384 2024-09-21T02:56:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.782+0000 7fca2a4a7740 1 Processor -- start 2024-09-21T02:56:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.782+0000 7fca2a4a7740 1 -- start start 2024-09-21T02:56:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.782+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba192d80 0x562fba57c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.782+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5841c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.782+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1b34d0 0x562fba57df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.782+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c1aa400 con 0x562fba1b34d0 2024-09-21T02:56:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.782+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c1abf40 con 0x562fba192d80 2024-09-21T02:56:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.782+0000 7fca2a4a7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c1aaad0 con 0x562fba1ab2f0 2024-09-21T02:56:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca21ad6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1b34d0 0x562fba57df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca21ad6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba192d80 0x562fba57c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca21ad6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1b34d0 0x562fba57df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51210/0 (socket says 172.21.15.29:51210) 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca21ad6640 1 -- 172.21.15.29:0/3575655151 learned_addr learned my addr 172.21.15.29:0/3575655151 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3575655151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5841c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca20ad4640 1 -- 172.21.15.29:0/3575655151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba192d80 msgr2=0x562fba57c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3575655151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba192d80 0x562fba57c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca20ad4640 1 -- 172.21.15.29:0/3575655151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1b34d0 msgr2=0x562fba57df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3575655151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1b34d0 0x562fba57df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.783+0000 7fca20ad4640 1 -- 172.21.15.29:0/3575655151 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fba584700 con 0x562fba1ab2f0 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.784+0000 7fca20ad4640 1 --2- 172.21.15.29:0/3575655151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5841c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x562fba4ee720 tx=0x7fca0c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eccc00d0d7888c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.784+0000 7fca06ffd640 1 -- 172.21.15.29:0/3575655151 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00fc80 con 0x562fba1ab2f0 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.784+0000 7fca06ffd640 1 -- 172.21.15.29:0/3575655151 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca0c0044c0 con 0x562fba1ab2f0 2024-09-21T02:56:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.784+0000 7fca06ffd640 1 -- 172.21.15.29:0/3575655151 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0047a0 con 0x562fba1ab2f0 2024-09-21T02:56:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.785+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3575655151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 msgr2=0x562fba5841c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3575655151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5841c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x562fba4ee720 tx=0x7fca0c005f90 comp rx=0 tx=0).stop 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3575655151 shutdown_connections 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3575655151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba1b34d0 0x562fba57df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3575655151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1ab2f0 0x562fba5841c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/3575655151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba192d80 0x562fba57c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3575655151 >> 172.21.15.29:0/3575655151 conn(0x562fba1c8300 msgr2=0x562fba583ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3575655151 shutdown_connections 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 -- 172.21.15.29:0/3575655151 wait complete. 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 -- start start 2024-09-21T02:56:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.786+0000 7fca2a4a7740 1 Processor -- start 2024-09-21T02:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba192d80 0x562fba5962b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca2a4a7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba5967f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba192d80 0x562fba5962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba192d80 0x562fba5962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51212/0 (socket says 172.21.15.29:51212) 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 -- 172.21.15.29:0/730144510 learned_addr learned my addr 172.21.15.29:0/730144510 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba596d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c1ab870 con 0x562fba192d80 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c1ab1a0 con 0x562fba1ab2f0 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c1a83e0 con 0x562fba1b34d0 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca20ad4640 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba5967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca2a4a7740 10 client.0 _ll_register_callbacks cb 0x7fca1810b490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba596d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 -- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 msgr2=0x562fba596d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba596d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 -- 172.21.15.29:0/730144510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 msgr2=0x562fba5967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba5967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca21ad6640 1 -- 172.21.15.29:0/730144510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fba604240 con 0x562fba192d80 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.787+0000 7fca20ad4640 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba5967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.788+0000 7fca21ad6640 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba192d80 0x562fba5962b0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fca1400f940 tx=0x7fca14005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12fdc283581e518 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.788+0000 7fca04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.788+0000 7fca04ff9640 1 -- 172.21.15.29:0/730144510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1402d040 con 0x562fba192d80 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.788+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562fba4ec0f0 con 0x562fba192d80 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.788+0000 7fca04ff9640 1 -- 172.21.15.29:0/730144510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fca14005930 con 0x562fba192d80 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.788+0000 7fca04ff9640 1 -- 172.21.15.29:0/730144510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fca14005b30 con 0x562fba192d80 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.788+0000 7fca2a4a7740 10 client.17856 fetch_fsmap learned FSMap version 37 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.788+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562fba6054b0 con 0x562fba192d80 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca04ff9640 1 -- 172.21.15.29:0/730144510 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x562fba4ec0f0 con 0x562fba192d80 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca04ff9640 1 -- 172.21.15.29:0/730144510 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fca14024070 con 0x562fba192d80 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca2a4a7740 10 client.17856 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca2a4a7740 10 client.17856 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca2a4a7740 20 client.17856 populate_metadata read hostname 'smithi029' 2024-09-21T02:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562fba1d41b0 0x562fba607e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca2a4a7740 4 client.17856 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562fba4e9520 con 0x562fba1d41b0 2024-09-21T02:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.789+0000 7fca20ad4640 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562fba1d41b0 0x562fba607e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.790+0000 7fca04ff9640 10 client.17856 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.790+0000 7fca20ad4640 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562fba1d41b0 0x562fba607e70 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2215ca2700f66ea server_cookie=92f8679898d35de1 in_seq=0 out_seq=0 2024-09-21T02:56:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:55 smithi029 ceph-mon[31006]: pgmap v923: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.9 MiB/s wr, 303 op/s 2024-09-21T02:56:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.995+0000 7fca04ff9640 1 -- 172.21.15.29:0/730144510 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562fba4e9520 con 0x562fba1d41b0 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.995+0000 7fca04ff9640 10 client.17856 handle_command_reply: tid=0 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.995+0000 7fca2a4a7740 1 client.17856 shutdown 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.995+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba192d80 msgr2=0x562fba5962b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.995+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba192d80 0x562fba5962b0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fca1400f940 tx=0x7fca14005fd0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.995+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 shutdown_connections 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.995+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fba1b34d0 0x562fba596d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.995+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x562fba1ab2f0 0x562fba5967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.996+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x562fba1d41b0 0x562fba607e70 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.996+0000 7fca2a4a7740 1 --2- 172.21.15.29:0/730144510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562fba192d80 0x562fba5962b0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.996+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 >> 172.21.15.29:0/730144510 conn(0x562fba1c8300 msgr2=0x562fba585d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.996+0000 7fca04ff9640 0 client.17856 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:56:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.996+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 shutdown_connections 2024-09-21T02:56:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.996+0000 7fca2a4a7740 1 -- 172.21.15.29:0/730144510 wait complete. 2024-09-21T02:56:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.996+0000 7fca2a4a7740 20 client.17856 trim_cache size 0 max 16384 2024-09-21T02:56:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:56:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:56:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:56:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 -- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fca080b6590 msgr2=0x7fca080b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fca080b6590 0x7fca080b8aa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fca180cac10 tx=0x7fca1810b5c0 comp rx=0 tx=0).stop 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 -- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 msgr2=0x7fca1c199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 0x7fca1c199ea0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fca18009230 tx=0x7fca18004040 comp rx=0 tx=0).stop 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 -- 172.21.15.29:0/3225409657 shutdown_connections 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fca080b6590 0x7fca080b8aa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca1c1050e0 0x7fca1c19b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca1c104740 0x7fca1c19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 --2- 172.21.15.29:0/3225409657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c1036c0 0x7fca1c199ea0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 -- 172.21.15.29:0/3225409657 >> 172.21.15.29:0/3225409657 conn(0x7fca1c0ff020 msgr2=0x7fca1c107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:56:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 -- 172.21.15.29:0/3225409657 shutdown_connections 2024-09-21T02:56:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:56:55.997+0000 7fca23539640 1 -- 172.21.15.29:0/3225409657 wait complete. 2024-09-21T02:56:56.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:55 smithi066 ceph-mon[33436]: pgmap v923: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.9 MiB/s wr, 303 op/s 2024-09-21T02:56:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:55 smithi190 ceph-mon[33871]: pgmap v923: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.9 MiB/s wr, 303 op/s 2024-09-21T02:56:56.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30322 6.13 MB/sec execute 153 sec latency 364.848 ms 2024-09-21T02:56:57.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30478 6.14 MB/sec execute 154 sec latency 148.719 ms 2024-09-21T02:56:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:57 smithi066 ceph-mon[33436]: pgmap v924: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 3.5 MiB/s wr, 265 op/s 2024-09-21T02:56:58.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:57 smithi190 ceph-mon[33871]: pgmap v924: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 3.5 MiB/s wr, 265 op/s 2024-09-21T02:56:58.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:57 smithi029 ceph-mon[31006]: pgmap v924: 129 pgs: 129 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 3.5 MiB/s wr, 265 op/s 2024-09-21T02:56:58.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30583 6.12 MB/sec execute 155 sec latency 445.342 ms 2024-09-21T02:56:59.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30674 6.09 MB/sec execute 156 sec latency 371.283 ms 2024-09-21T02:57:00.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:56:59 smithi066 ceph-mon[33436]: pgmap v925: 129 pgs: 129 active+clean; 276 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.4 MiB/s wr, 278 op/s 2024-09-21T02:57:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:56:59 smithi190 ceph-mon[33871]: pgmap v925: 129 pgs: 129 active+clean; 276 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.4 MiB/s wr, 278 op/s 2024-09-21T02:57:00.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:56:59 smithi029 ceph-mon[31006]: pgmap v925: 129 pgs: 129 active+clean; 276 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.4 MiB/s wr, 278 op/s 2024-09-21T02:57:00.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30734 6.05 MB/sec execute 157 sec latency 402.652 ms 2024-09-21T02:57:01.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30848 6.03 MB/sec execute 158 sec latency 345.201 ms 2024-09-21T02:57:02.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:01 smithi066 ceph-mon[33436]: pgmap v926: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 3.1 MiB/s wr, 299 op/s 2024-09-21T02:57:02.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:01 smithi190 ceph-mon[33871]: pgmap v926: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 3.1 MiB/s wr, 299 op/s 2024-09-21T02:57:02.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:01 smithi029 ceph-mon[31006]: pgmap v926: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 3.1 MiB/s wr, 299 op/s 2024-09-21T02:57:02.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30943 6.04 MB/sec execute 159 sec latency 445.400 ms 2024-09-21T02:57:03.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31026 6.01 MB/sec execute 160 sec latency 498.855 ms 2024-09-21T02:57:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:03 smithi066 ceph-mon[33436]: pgmap v927: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.5 MiB/s wr, 303 op/s 2024-09-21T02:57:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:03 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1263171788' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:57:04.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:03 smithi190 ceph-mon[33871]: pgmap v927: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.5 MiB/s wr, 303 op/s 2024-09-21T02:57:04.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:03 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1263171788' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:57:04.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:03 smithi029 ceph-mon[31006]: pgmap v927: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.5 MiB/s wr, 303 op/s 2024-09-21T02:57:04.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:03 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1263171788' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:57:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 -- 172.21.15.29:0/1631064738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df8105300 msgr2=0x7f9df810d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 --2- 172.21.15.29:0/1631064738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df8105300 0x7f9df810d850 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009900 tx=0x7f9de802a410 comp rx=0 tx=0).stop 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 -- 172.21.15.29:0/1631064738 shutdown_connections 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 --2- 172.21.15.29:0/1631064738 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9df810de10 0x7f9df8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 --2- 172.21.15.29:0/1631064738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df8105300 0x7f9df810d850 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 --2- 172.21.15.29:0/1631064738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9df81049b0 0x7f9df8104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 -- 172.21.15.29:0/1631064738 >> 172.21.15.29:0/1631064738 conn(0x7f9df8079730 msgr2=0x7f9df8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 -- 172.21.15.29:0/1631064738 shutdown_connections 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.502+0000 7f9e00788640 1 -- 172.21.15.29:0/1631064738 wait complete. 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9e00788640 1 Processor -- start 2024-09-21T02:57:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9e00788640 1 -- start start 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9e00788640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df81049b0 0x7f9df819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9e00788640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9df8105300 0x7f9df819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9e00788640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9df810de10 0x7f9df819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9e00788640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9df8113920 con 0x7f9df81049b0 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9e00788640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9df81137a0 con 0x7f9df8105300 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9e00788640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9df8113aa0 con 0x7f9df810de10 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9dfe524640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df81049b0 0x7f9df819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9dfe524640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df81049b0 0x7f9df819a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51228/0 (socket says 172.21.15.29:51228) 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9dfe524640 1 -- 172.21.15.29:0/1263171788 learned_addr learned my addr 172.21.15.29:0/1263171788 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.503+0000 7f9dfdd23640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9df8105300 0x7f9df819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:04.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9dfed25640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9df810de10 0x7f9df819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9dfe524640 1 -- 172.21.15.29:0/1263171788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9df810de10 msgr2=0x7f9df819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9dfe524640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9df810de10 0x7f9df819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9dfe524640 1 -- 172.21.15.29:0/1263171788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9df8105300 msgr2=0x7f9df819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9dfe524640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9df8105300 0x7f9df819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9dfe524640 1 -- 172.21.15.29:0/1263171788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df81a02a0 con 0x7f9df81049b0 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9dfe524640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df81049b0 0x7f9df819a3f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f9df400d950 tx=0x7f9df400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9def7fe640 1 -- 172.21.15.29:0/1263171788 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df4017070 con 0x7f9df81049b0 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9def7fe640 1 -- 172.21.15.29:0/1263171788 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9df4009c30 con 0x7f9df81049b0 2024-09-21T02:57:04.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df81a0590 con 0x7f9df81049b0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9def7fe640 1 -- 172.21.15.29:0/1263171788 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df400f620 con 0x7f9df81049b0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.504+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df81a0a50 con 0x7f9df81049b0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.505+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df806b560 con 0x7f9df81049b0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.507+0000 7f9def7fe640 1 -- 172.21.15.29:0/1263171788 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9df401a070 con 0x7f9df81049b0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.507+0000 7f9def7fe640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9de40b6540 0x7f9de40b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.508+0000 7f9dfdd23640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9de40b6540 0x7f9de40b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.508+0000 7f9def7fe640 1 -- 172.21.15.29:0/1263171788 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9df40e70a0 con 0x7f9df81049b0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.508+0000 7f9dfdd23640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9de40b6540 0x7f9de40b8a50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005940 tx=0x7f9de80058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.511+0000 7f9def7fe640 1 -- 172.21.15.29:0/1263171788 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9df40ad5b0 con 0x7f9df81049b0 2024-09-21T02:57:04.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.746+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9df8104dc0 con 0x7f9df81049b0 2024-09-21T02:57:04.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.747+0000 7f9def7fe640 1 -- 172.21.15.29:0/1263171788 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f9df8104dc0 con 0x7f9df81049b0 2024-09-21T02:57:04.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:57:04.461 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:57:04.461 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:57:04.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9de40b6540 msgr2=0x7f9de40b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:04.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9de40b6540 0x7f9de40b8a50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005940 tx=0x7f9de80058d0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df81049b0 msgr2=0x7f9df819a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df81049b0 0x7f9df819a3f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f9df400d950 tx=0x7f9df400de10 comp rx=0 tx=0).stop 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 shutdown_connections 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9de40b6540 0x7f9de40b8a50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9df810de10 0x7f9df819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9df8105300 0x7f9df819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 --2- 172.21.15.29:0/1263171788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df81049b0 0x7f9df819a3f0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 >> 172.21.15.29:0/1263171788 conn(0x7f9df8079730 msgr2=0x7f9df8100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.749+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 shutdown_connections 2024-09-21T02:57:04.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:02.750+0000 7f9e00788640 1 -- 172.21.15.29:0/1263171788 wait complete. 2024-09-21T02:57:04.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31115 6.00 MB/sec execute 161 sec latency 296.430 ms 2024-09-21T02:57:05.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31289 6.02 MB/sec execute 162 sec latency 127.144 ms 2024-09-21T02:57:06.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:05 smithi066 ceph-mon[33436]: pgmap v928: 129 pgs: 129 active+clean; 272 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 2.3 MiB/s wr, 297 op/s 2024-09-21T02:57:06.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:05 smithi190 ceph-mon[33871]: pgmap v928: 129 pgs: 129 active+clean; 272 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 2.3 MiB/s wr, 297 op/s 2024-09-21T02:57:06.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:05 smithi029 ceph-mon[31006]: pgmap v928: 129 pgs: 129 active+clean; 272 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 2.3 MiB/s wr, 297 op/s 2024-09-21T02:57:06.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31386 6.03 MB/sec execute 163 sec latency 229.884 ms 2024-09-21T02:57:07.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31510 6.07 MB/sec execute 164 sec latency 626.207 ms 2024-09-21T02:57:08.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:07 smithi066 ceph-mon[33436]: pgmap v929: 129 pgs: 129 active+clean; 275 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-09-21T02:57:08.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:07 smithi190 ceph-mon[33871]: pgmap v929: 129 pgs: 129 active+clean; 275 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-09-21T02:57:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:07 smithi029 ceph-mon[31006]: pgmap v929: 129 pgs: 129 active+clean; 275 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-09-21T02:57:08.725 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31675 6.10 MB/sec execute 165 sec latency 466.034 ms 2024-09-21T02:57:09.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31869 6.15 MB/sec execute 166 sec latency 97.797 ms 2024-09-21T02:57:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:09 smithi066 ceph-mon[33436]: pgmap v930: 129 pgs: 129 active+clean; 281 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 2.3 MiB/s wr, 228 op/s 2024-09-21T02:57:10.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:09 smithi190 ceph-mon[33871]: pgmap v930: 129 pgs: 129 active+clean; 281 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 2.3 MiB/s wr, 228 op/s 2024-09-21T02:57:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:09 smithi029 ceph-mon[31006]: pgmap v930: 129 pgs: 129 active+clean; 281 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 2.3 MiB/s wr, 228 op/s 2024-09-21T02:57:10.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31931 6.13 MB/sec execute 167 sec latency 395.559 ms 2024-09-21T02:57:11.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32045 6.13 MB/sec execute 168 sec latency 175.052 ms 2024-09-21T02:57:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:11 smithi066 ceph-mon[33436]: pgmap v931: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.6 MiB/s wr, 188 op/s 2024-09-21T02:57:12.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:11 smithi190 ceph-mon[33871]: pgmap v931: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.6 MiB/s wr, 188 op/s 2024-09-21T02:57:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:11 smithi029 ceph-mon[31006]: pgmap v931: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.6 MiB/s wr, 188 op/s 2024-09-21T02:57:12.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32141 6.14 MB/sec execute 169 sec latency 219.736 ms 2024-09-21T02:57:13.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32261 6.13 MB/sec execute 170 sec latency 133.105 ms 2024-09-21T02:57:14.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:13 smithi066 ceph-mon[33436]: pgmap v932: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 2.9 MiB/s wr, 166 op/s 2024-09-21T02:57:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:13 smithi190 ceph-mon[33871]: pgmap v932: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 2.9 MiB/s wr, 166 op/s 2024-09-21T02:57:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:13 smithi029 ceph-mon[31006]: pgmap v932: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 2.9 MiB/s wr, 166 op/s 2024-09-21T02:57:14.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32368 6.16 MB/sec execute 171 sec latency 499.409 ms 2024-09-21T02:57:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 -- 172.21.15.29:0/2323438895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95201093e0 msgr2=0x7f952010b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 --2- 172.21.15.29:0/2323438895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95201093e0 0x7f952010b8f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f9518009c90 tx=0x7f951802a860 comp rx=0 tx=0).stop 2024-09-21T02:57:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 -- 172.21.15.29:0/2323438895 shutdown_connections 2024-09-21T02:57:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 --2- 172.21.15.29:0/2323438895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95201093e0 0x7f952010b8f0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 --2- 172.21.15.29:0/2323438895 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9520106880 0x7f9520108ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 --2- 172.21.15.29:0/2323438895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f952006b6b0 0x7f9520106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 -- 172.21.15.29:0/2323438895 >> 172.21.15.29:0/2323438895 conn(0x7f95200fdee0 msgr2=0x7f9520100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:15.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 -- 172.21.15.29:0/2323438895 shutdown_connections 2024-09-21T02:57:15.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.247+0000 7f9527be4640 1 -- 172.21.15.29:0/2323438895 wait complete. 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.248+0000 7f9527be4640 1 Processor -- start 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9527be4640 1 -- start start 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9527be4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f952006b6b0 0x7f9520199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9527be4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9520106880 0x7f952019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9527be4640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95201093e0 0x7f952019b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9527be4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f952010f1b0 con 0x7f9520106880 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9527be4640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f952010f030 con 0x7f95201093e0 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9527be4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f952010f330 con 0x7f952006b6b0 2024-09-21T02:57:15.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f952517f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9520106880 0x7f952019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f952517f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9520106880 0x7f952019a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37078/0 (socket says 172.21.15.29:37078) 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f952517f640 1 -- 172.21.15.29:0/3366631003 learned_addr learned my addr 172.21.15.29:0/3366631003 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9525980640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f952006b6b0 0x7f9520199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.249+0000 7f9526181640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95201093e0 0x7f952019b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f952517f640 1 -- 172.21.15.29:0/3366631003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f952006b6b0 msgr2=0x7f9520199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f952517f640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f952006b6b0 0x7f9520199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f952517f640 1 -- 172.21.15.29:0/3366631003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95201093e0 msgr2=0x7f952019b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f952517f640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95201093e0 0x7f952019b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f952517f640 1 -- 172.21.15.29:0/3366631003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952019b8d0 con 0x7f9520106880 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f9525980640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f952006b6b0 0x7f9520199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f952517f640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9520106880 0x7f952019a500 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f951000e9c0 tx=0x7f951000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f9516ffd640 1 -- 172.21.15.29:0/3366631003 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951000cd80 con 0x7f9520106880 2024-09-21T02:57:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f9516ffd640 1 -- 172.21.15.29:0/3366631003 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95100054b0 con 0x7f9520106880 2024-09-21T02:57:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f9527be4640 1 -- 172.21.15.29:0/3366631003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f952019bbc0 con 0x7f9520106880 2024-09-21T02:57:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f9516ffd640 1 -- 172.21.15.29:0/3366631003 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510010430 con 0x7f9520106880 2024-09-21T02:57:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.250+0000 7f9527be4640 1 -- 172.21.15.29:0/3366631003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9520102300 con 0x7f9520106880 2024-09-21T02:57:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.254+0000 7f9516ffd640 1 -- 172.21.15.29:0/3366631003 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9510004f20 con 0x7f9520106880 2024-09-21T02:57:15.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.256+0000 7f9516ffd640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f950c0b6510 0x7f950c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.256+0000 7f9525980640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f950c0b6510 0x7f950c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.256+0000 7f9516ffd640 1 -- 172.21.15.29:0/3366631003 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f95100e7ca0 con 0x7f9520106880 2024-09-21T02:57:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.257+0000 7f9525980640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f950c0b6510 0x7f950c0b8a20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f951c0096a0 tx=0x7f951c012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.390+0000 7f952ec33740 1 Processor -- start 2024-09-21T02:57:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.390+0000 7f952ec33740 1 -- start start 2024-09-21T02:57:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952ec33740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea122e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952ec33740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea1234c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952ec33740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea124010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952ec33740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c9ea0956f0 con 0x55c9e9d5a4d0 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952ec33740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c9ea084b20 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952ec33740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c9ea088930 con 0x55c9e9d522f0 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f9526181640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea124010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f9525980640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea122e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952517f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea1234c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952517f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea1234c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47944/0 (socket says 172.21.15.29:47944) 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f9525980640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea122e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:34170/0 (socket says 172.21.15.29:34170) 2024-09-21T02:57:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952517f640 1 -- 172.21.15.29:0/2923242366 learned_addr learned my addr 172.21.15.29:0/2923242366 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952517f640 1 -- 172.21.15.29:0/2923242366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 msgr2=0x55c9ea122e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952517f640 1 --2- 172.21.15.29:0/2923242366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea122e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952517f640 1 -- 172.21.15.29:0/2923242366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 msgr2=0x55c9ea124010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952517f640 1 --2- 172.21.15.29:0/2923242366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea124010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.391+0000 7f952517f640 1 -- 172.21.15.29:0/2923242366 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9ea04a530 con 0x55c9e9d522f0 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.392+0000 7f952517f640 1 --2- 172.21.15.29:0/2923242366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea1234c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f95100aad20 tx=0x7f95100ad940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5605ec85e5c04f92 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.392+0000 7f950bfff640 1 -- 172.21.15.29:0/2923242366 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95100ad5c0 con 0x55c9e9d522f0 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.392+0000 7f950bfff640 1 -- 172.21.15.29:0/2923242366 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95100ae1b0 con 0x55c9e9d522f0 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.392+0000 7f950bfff640 1 -- 172.21.15.29:0/2923242366 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95100a9d20 con 0x55c9e9d522f0 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 -- 172.21.15.29:0/2923242366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 msgr2=0x55c9ea1234c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 --2- 172.21.15.29:0/2923242366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea1234c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f95100aad20 tx=0x7f95100ad940 comp rx=0 tx=0).stop 2024-09-21T02:57:15.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 -- 172.21.15.29:0/2923242366 shutdown_connections 2024-09-21T02:57:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 --2- 172.21.15.29:0/2923242366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea124010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 --2- 172.21.15.29:0/2923242366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea1234c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 --2- 172.21.15.29:0/2923242366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea122e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 -- 172.21.15.29:0/2923242366 >> 172.21.15.29:0/2923242366 conn(0x55c9e9d6f300 msgr2=0x55c9ea11e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 -- 172.21.15.29:0/2923242366 shutdown_connections 2024-09-21T02:57:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.393+0000 7f952ec33740 1 -- 172.21.15.29:0/2923242366 wait complete. 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.394+0000 7f952ec33740 1 -- start start 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.394+0000 7f952ec33740 1 Processor -- start 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f952ec33740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d522f0 0x55c9ea134d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f952ec33740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea135420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f952ec33740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea135f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f952ec33740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c9e9fbcab0 con 0x55c9e9d522f0 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f952ec33740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c9ea02a730 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f952ec33740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c9ea124ad0 con 0x55c9e9d5a4d0 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f952ec33740 10 client.0 _ll_register_callbacks cb 0x55c9ea07e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f9525980640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d522f0 0x55c9ea134d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f9525980640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d522f0 0x55c9ea134d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37094/0 (socket says 172.21.15.29:37094) 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f9525980640 1 -- 172.21.15.29:0/1703340999 learned_addr learned my addr 172.21.15.29:0/1703340999 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f952517f640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea135420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f9526181640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f9525980640 1 -- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d5a4d0 msgr2=0x55c9ea135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f9525980640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea135420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f9525980640 1 -- 172.21.15.29:0/1703340999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 msgr2=0x55c9ea135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.395+0000 7f9525980640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea135f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f9525980640 1 -- 172.21.15.29:0/1703340999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9e9de6790 con 0x55c9e9d522f0 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f9526181640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea135f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f9525980640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d522f0 0x55c9ea134d60 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f951c003de0 tx=0x7f951c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b3bd383dcb146c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f95097fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f952517f640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea135420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f95097fa640 1 -- 172.21.15.29:0/1703340999 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c0044b0 con 0x55c9e9d522f0 2024-09-21T02:57:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f95097fa640 1 -- 172.21.15.29:0/1703340999 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f951c01a400 con 0x55c9e9d522f0 2024-09-21T02:57:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f952ec33740 1 -- 172.21.15.29:0/1703340999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95100ae1b0 con 0x55c9e9d522f0 2024-09-21T02:57:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.396+0000 7f95097fa640 1 -- 172.21.15.29:0/1703340999 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f951c032050 con 0x55c9e9d522f0 2024-09-21T02:57:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f952ec33740 10 client.17874 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f952ec33740 1 -- 172.21.15.29:0/1703340999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c9e9e5f620 con 0x55c9e9d522f0 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f95097fa640 1 -- 172.21.15.29:0/1703340999 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95100ae1b0 con 0x55c9e9d522f0 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f95097fa640 1 -- 172.21.15.29:0/1703340999 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f951c027480 con 0x55c9e9d522f0 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f952ec33740 10 client.17874 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f952ec33740 10 client.17874 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f952ec33740 20 client.17874 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f952ec33740 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c9e9d39d80 0x55c9ea1426f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f952ec33740 4 client.17874 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.397+0000 7f952ec33740 1 -- 172.21.15.29:0/1703340999 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9ea097640 con 0x55c9e9d39d80 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.398+0000 7f952517f640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c9e9d39d80 0x55c9ea1426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.398+0000 7f95097fa640 10 client.17874 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.398+0000 7f952517f640 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c9e9d39d80 0x55c9ea1426f0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf3f5f255605166a server_cookie=79bdd68955aa255e in_seq=0 out_seq=0 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.399+0000 7f95097fa640 1 -- 172.21.15.29:0/1703340999 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c9ea097640 con 0x55c9e9d39d80 2024-09-21T02:57:15.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.399+0000 7f95097fa640 10 client.17874 handle_command_reply: tid=0 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.399+0000 7f952ec33740 1 client.17874 shutdown 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 -- 172.21.15.29:0/1703340999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d522f0 msgr2=0x55c9ea134d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d522f0 0x55c9ea134d60 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f951c003de0 tx=0x7f951c004730 comp rx=0 tx=0).stop 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 -- 172.21.15.29:0/1703340999 shutdown_connections 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea135f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea135420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c9e9d39d80 0x55c9ea1426f0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 --2- 172.21.15.29:0/1703340999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d522f0 0x55c9ea134d60 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f95097fa640 0 client.17874 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 -- 172.21.15.29:0/1703340999 >> 172.21.15.29:0/1703340999 conn(0x55c9e9d6f300 msgr2=0x55c9ea11cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 -- 172.21.15.29:0/1703340999 shutdown_connections 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 1 -- 172.21.15.29:0/1703340999 wait complete. 2024-09-21T02:57:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.400+0000 7f952ec33740 20 client.17874 trim_cache size 0 max 16384 2024-09-21T02:57:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f952ec33740 1 Processor -- start 2024-09-21T02:57:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f952ec33740 1 -- start start 2024-09-21T02:57:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f952ec33740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea11e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f952ec33740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea1195c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f952ec33740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea11d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f952ec33740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f95201aa4a0 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f952ec33740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f95201abfe0 con 0x55c9e9d5a4d0 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f952ec33740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f95201aab70 con 0x55c9e9d522f0 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.422+0000 7f9526181640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea11d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9526181640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea11d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37100/0 (socket says 172.21.15.29:37100) 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9526181640 1 -- 172.21.15.29:0/2197573578 learned_addr learned my addr 172.21.15.29:0/2197573578 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9525980640 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea11e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f952517f640 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea1195c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9526181640 1 -- 172.21.15.29:0/2197573578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 msgr2=0x55c9ea11e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9526181640 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea11e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9526181640 1 -- 172.21.15.29:0/2197573578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d5a4d0 msgr2=0x55c9ea1195c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9526181640 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea1195c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9526181640 1 -- 172.21.15.29:0/2197573578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9ea136300 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.426 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9525980640 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea11e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f9526181640 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea11d610 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f9518009c90 tx=0x7f95180093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92bc9e16ef98ad0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.423+0000 7f950b7fe640 1 -- 172.21.15.29:0/2197573578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95180314b0 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.424+0000 7f950b7fe640 1 -- 172.21.15.29:0/2197573578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9518036040 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.424+0000 7f950b7fe640 1 -- 172.21.15.29:0/2197573578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9518004290 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 -- 172.21.15.29:0/2197573578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d7b1b0 msgr2=0x55c9ea11d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea11d610 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f9518009c90 tx=0x7f95180093d0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 -- 172.21.15.29:0/2197573578 shutdown_connections 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea11d610 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea1195c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 --2- 172.21.15.29:0/2197573578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea11e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 -- 172.21.15.29:0/2197573578 >> 172.21.15.29:0/2197573578 conn(0x55c9e9d6f300 msgr2=0x55c9ea123e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 -- 172.21.15.29:0/2197573578 shutdown_connections 2024-09-21T02:57:15.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.425+0000 7f952ec33740 1 -- 172.21.15.29:0/2197573578 wait complete. 2024-09-21T02:57:15.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 1 -- start start 2024-09-21T02:57:15.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 1 Processor -- start 2024-09-21T02:57:15.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea13a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea13a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea13b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f95201ab910 con 0x55c9e9d5a4d0 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f95201ab240 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f95201a8480 con 0x55c9e9d522f0 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f952ec33740 10 client.0 _ll_register_callbacks cb 0x7f951c0041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f9526181640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea13b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.426+0000 7f9525980640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea13a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9525980640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea13a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47966/0 (socket says 172.21.15.29:47966) 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9525980640 1 -- 172.21.15.29:0/1923992378 learned_addr learned my addr 172.21.15.29:0/1923992378 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9526181640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea13b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:34202/0 (socket says 172.21.15.29:34202) 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f952517f640 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea13a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9526181640 1 -- 172.21.15.29:0/1923992378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 msgr2=0x55c9ea13a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9526181640 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea13a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9526181640 1 -- 172.21.15.29:0/1923992378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 msgr2=0x55c9ea13a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9526181640 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea13a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9526181640 1 -- 172.21.15.29:0/1923992378 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9ea13bb40 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9525980640 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea13a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f9526181640 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea13b3e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9518031040 tx=0x7f9518031760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b48ba5a2a622cedb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.427+0000 7f95097fa640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f95097fa640 1 -- 172.21.15.29:0/1923992378 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9518031c90 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f95097fa640 1 -- 172.21.15.29:0/1923992378 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9518004030 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 1 -- 172.21.15.29:0/1923992378 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9518036040 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f95097fa640 1 -- 172.21.15.29:0/1923992378 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f951805b050 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 10 client.27224 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 1 -- 172.21.15.29:0/1923992378 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c9ea13ce30 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f95097fa640 1 -- 172.21.15.29:0/1923992378 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9518036040 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f95097fa640 1 -- 172.21.15.29:0/1923992378 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f9518030760 con 0x55c9e9d7b1b0 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 10 client.27224 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 10 client.27224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 20 client.27224 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c9e9d0f1c0 0x55c9ea1af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 4 client.27224 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:57:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.428+0000 7f952ec33740 1 -- 172.21.15.29:0/1923992378 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c9ea0930f0 con 0x55c9e9d0f1c0 2024-09-21T02:57:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.429+0000 7f9525980640 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c9e9d0f1c0 0x55c9ea1af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.429+0000 7f95097fa640 10 client.27224 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.429+0000 7f9525980640 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c9e9d0f1c0 0x55c9ea1af3d0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6155d1598cf0fc75 server_cookie=3821ff621f83f6da in_seq=0 out_seq=0 2024-09-21T02:57:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f95097fa640 1 -- 172.21.15.29:0/1923992378 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c9ea0930f0 con 0x55c9e9d0f1c0 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f95097fa640 10 client.27224 handle_command_reply: tid=0 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 client.27224 shutdown 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 -- 172.21.15.29:0/1923992378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 msgr2=0x55c9ea13b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea13b3e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9518031040 tx=0x7f9518031760 comp rx=0 tx=0).stop 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 -- 172.21.15.29:0/1923992378 shutdown_connections 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c9e9d7b1b0 0x55c9ea13b3e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c9e9d0f1c0 0x55c9ea1af3d0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c9e9d5a4d0 0x55c9ea13a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 --2- 172.21.15.29:0/1923992378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9e9d522f0 0x55c9ea13a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f95097fa640 0 client.27224 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 -- 172.21.15.29:0/1923992378 >> 172.21.15.29:0/1923992378 conn(0x55c9e9d6f300 msgr2=0x55c9ea11e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 -- 172.21.15.29:0/1923992378 shutdown_connections 2024-09-21T02:57:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 1 -- 172.21.15.29:0/1923992378 wait complete. 2024-09-21T02:57:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.644+0000 7f952ec33740 20 client.27224 trim_cache size 0 max 16384 2024-09-21T02:57:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:57:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:57:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:57:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:57:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.645+0000 7f9527be4640 1 -- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f950c0b6510 msgr2=0x7f950c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.645+0000 7f9527be4640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f950c0b6510 0x7f950c0b8a20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f951c0096a0 tx=0x7f951c012040 comp rx=0 tx=0).stop 2024-09-21T02:57:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.645+0000 7f9527be4640 1 -- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9520106880 msgr2=0x7f952019a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.645+0000 7f9527be4640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9520106880 0x7f952019a500 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f951000e9c0 tx=0x7f951000ee80 comp rx=0 tx=0).stop 2024-09-21T02:57:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.646+0000 7f9527be4640 1 -- 172.21.15.29:0/3366631003 shutdown_connections 2024-09-21T02:57:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.646+0000 7f9527be4640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f950c0b6510 0x7f950c0b8a20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.646+0000 7f9527be4640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95201093e0 0x7f952019b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.646+0000 7f9527be4640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9520106880 0x7f952019a500 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.646+0000 7f9527be4640 1 --2- 172.21.15.29:0/3366631003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f952006b6b0 0x7f9520199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.646+0000 7f9527be4640 1 -- 172.21.15.29:0/3366631003 >> 172.21.15.29:0/3366631003 conn(0x7f95200fdee0 msgr2=0x7f9520104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.646+0000 7f9527be4640 1 -- 172.21.15.29:0/3366631003 shutdown_connections 2024-09-21T02:57:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:15.646+0000 7f9527be4640 1 -- 172.21.15.29:0/3366631003 wait complete. 2024-09-21T02:57:15.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32464 6.14 MB/sec execute 172 sec latency 377.759 ms 2024-09-21T02:57:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:15 smithi066 ceph-mon[33436]: pgmap v933: 129 pgs: 129 active+clean; 291 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-09-21T02:57:16.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:15 smithi190 ceph-mon[33871]: pgmap v933: 129 pgs: 129 active+clean; 291 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-09-21T02:57:16.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:15 smithi029 ceph-mon[31006]: pgmap v933: 129 pgs: 129 active+clean; 291 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-09-21T02:57:16.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32567 6.15 MB/sec execute 173 sec latency 305.063 ms 2024-09-21T02:57:17.690 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 130.2 --await-for 45.0' 2024-09-21T02:57:17.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 130.2 --await-for 45.0 2024-09-21T02:57:17.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32665 6.15 MB/sec execute 174 sec latency 253.642 ms 2024-09-21T02:57:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:17 smithi066 ceph-mon[33436]: pgmap v934: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.3 MiB/s wr, 180 op/s 2024-09-21T02:57:18.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:17 smithi190 ceph-mon[33871]: pgmap v934: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.3 MiB/s wr, 180 op/s 2024-09-21T02:57:18.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:17 smithi029 ceph-mon[31006]: pgmap v934: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 3.3 MiB/s wr, 180 op/s 2024-09-21T02:57:18.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32797 6.15 MB/sec execute 175 sec latency 107.737 ms 2024-09-21T02:57:19.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32951 6.17 MB/sec execute 176 sec latency 153.913 ms 2024-09-21T02:57:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:19 smithi066 ceph-mon[33436]: pgmap v935: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-09-21T02:57:20.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:19 smithi190 ceph-mon[33871]: pgmap v935: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-09-21T02:57:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:19 smithi029 ceph-mon[31006]: pgmap v935: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-09-21T02:57:20.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33053 6.16 MB/sec execute 177 sec latency 456.874 ms 2024-09-21T02:57:21.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33162 6.16 MB/sec execute 178 sec latency 185.185 ms 2024-09-21T02:57:21.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:21 smithi190 ceph-mon[33871]: pgmap v936: 129 pgs: 129 active+clean; 288 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 3.0 MiB/s wr, 219 op/s 2024-09-21T02:57:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:21 smithi066 ceph-mon[33436]: pgmap v936: 129 pgs: 129 active+clean; 288 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 3.0 MiB/s wr, 219 op/s 2024-09-21T02:57:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:21 smithi029 ceph-mon[31006]: pgmap v936: 129 pgs: 129 active+clean; 288 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 3.0 MiB/s wr, 219 op/s 2024-09-21T02:57:22.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33309 6.18 MB/sec execute 179 sec latency 240.394 ms 2024-09-21T02:57:23.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33475 6.18 MB/sec execute 180 sec latency 97.197 ms 2024-09-21T02:57:24.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:23 smithi066 ceph-mon[33436]: pgmap v937: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-09-21T02:57:24.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:23 smithi190 ceph-mon[33871]: pgmap v937: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-09-21T02:57:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:23 smithi029 ceph-mon[31006]: pgmap v937: 129 pgs: 129 active+clean; 295 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-09-21T02:57:24.337 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:57:24.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33545 6.16 MB/sec execute 181 sec latency 732.772 ms 2024-09-21T02:57:25.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33628 6.13 MB/sec execute 182 sec latency 211.074 ms 2024-09-21T02:57:26.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:25 smithi066 ceph-mon[33436]: pgmap v938: 129 pgs: 129 active+clean; 298 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 3.4 MiB/s wr, 253 op/s 2024-09-21T02:57:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:25 smithi190 ceph-mon[33871]: pgmap v938: 129 pgs: 129 active+clean; 298 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 3.4 MiB/s wr, 253 op/s 2024-09-21T02:57:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:25 smithi029 ceph-mon[31006]: pgmap v938: 129 pgs: 129 active+clean; 298 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 3.4 MiB/s wr, 253 op/s 2024-09-21T02:57:26.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33750 6.14 MB/sec execute 183 sec latency 266.467 ms 2024-09-21T02:57:27.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33838 6.12 MB/sec execute 184 sec latency 506.621 ms 2024-09-21T02:57:28.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:27 smithi066 ceph-mon[33436]: pgmap v939: 129 pgs: 129 active+clean; 296 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 3.2 MiB/s wr, 239 op/s 2024-09-21T02:57:28.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:27 smithi190 ceph-mon[33871]: pgmap v939: 129 pgs: 129 active+clean; 296 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 3.2 MiB/s wr, 239 op/s 2024-09-21T02:57:28.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:27 smithi029 ceph-mon[31006]: pgmap v939: 129 pgs: 129 active+clean; 296 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 3.2 MiB/s wr, 239 op/s 2024-09-21T02:57:28.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33972 6.14 MB/sec execute 185 sec latency 195.477 ms 2024-09-21T02:57:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.491+0000 7f8eae2d3640 1 -- 172.21.15.29:0/3810056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 msgr2=0x7f8ea8108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.491+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/3810056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 0x7f8ea8108740 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009900 tx=0x7f8e9802a410 comp rx=0 tx=0).stop 2024-09-21T02:57:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.491+0000 7f8eae2d3640 1 -- 172.21.15.29:0/3810056 shutdown_connections 2024-09-21T02:57:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.491+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/3810056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea810b630 0x7f8ea810db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.491+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/3810056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea8108c80 0x7f8ea810b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.491+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/3810056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 0x7f8ea8108740 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.491+0000 7f8eae2d3640 1 -- 172.21.15.29:0/3810056 >> 172.21.15.29:0/3810056 conn(0x7f8ea80fdf00 msgr2=0x7f8ea81003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.491+0000 7f8eae2d3640 1 -- 172.21.15.29:0/3810056 shutdown_connections 2024-09-21T02:57:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.492+0000 7f8eae2d3640 1 -- 172.21.15.29:0/3810056 wait complete. 2024-09-21T02:57:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.492+0000 7f8eae2d3640 1 Processor -- start 2024-09-21T02:57:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.492+0000 7f8eae2d3640 1 -- start start 2024-09-21T02:57:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8eae2d3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 0x7f8ea819c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8eae2d3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea8108c80 0x7f8ea819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8eae2d3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea810b630 0x7f8ea819d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8eae2d3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea8111430 con 0x7f8ea8108c80 2024-09-21T02:57:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8eae2d3640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81112b0 con 0x7f8ea806c240 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8eae2d3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81115b0 con 0x7f8ea810b630 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 0x7f8ea819c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8ea77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea8108c80 0x7f8ea819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 0x7f8ea819c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:54726/0 (socket says 172.21.15.29:54726) 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2271670885 learned_addr learned my addr 172.21.15.29:0/2271670885 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.493+0000 7f8eac870640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea810b630 0x7f8ea819d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2271670885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea810b630 msgr2=0x7f8ea819d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea810b630 0x7f8ea819d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea8108c80 msgr2=0x7f8ea819c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea8108c80 0x7f8ea819c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2271670885 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea819db60 con 0x7f8ea806c240 2024-09-21T02:57:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 0x7f8ea819c120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009230 tx=0x7f8e980040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea57fa640 1 -- 172.21.15.29:0/2271670885 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9803f030 con 0x7f8ea806c240 2024-09-21T02:57:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8eae2d3640 1 -- 172.21.15.29:0/2271670885 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea819ddf0 con 0x7f8ea806c240 2024-09-21T02:57:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea57fa640 1 -- 172.21.15.29:0/2271670885 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e980044c0 con 0x7f8ea806c240 2024-09-21T02:57:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8ea57fa640 1 -- 172.21.15.29:0/2271670885 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e98035070 con 0x7f8ea806c240 2024-09-21T02:57:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.494+0000 7f8eae2d3640 1 -- 172.21.15.29:0/2271670885 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea81a4540 con 0x7f8ea806c240 2024-09-21T02:57:29.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.497+0000 7f8ea57fa640 1 -- 172.21.15.29:0/2271670885 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f8e98042030 con 0x7f8ea806c240 2024-09-21T02:57:29.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.498+0000 7f8ea57fa640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e900b6510 0x7f8e900b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.498+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e900b6510 0x7f8e900b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.499+0000 7f8ea57fa640 1 -- 172.21.15.29:0/2271670885 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e981089a0 con 0x7f8ea806c240 2024-09-21T02:57:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.665+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e900b6510 0x7f8e900b8a20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c006fd0 tx=0x7f8e9c0090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eb5244740 1 Processor -- start 2024-09-21T02:57:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eb5244740 1 -- start start 2024-09-21T02:57:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b291e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b2924c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b293010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eb5244740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a86b2046f0 con 0x55a86aeea1b0 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eb5244740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a86b1f3b20 con 0x55a86aec94d0 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eb5244740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a86b1f7930 con 0x55a86aec12f0 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8ea77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b2924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b291e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b291e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34762/0 (socket says 172.21.15.29:34762) 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8ea77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b2924c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47640/0 (socket says 172.21.15.29:47640) 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2639203466 learned_addr learned my addr 172.21.15.29:0/2639203466 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.667+0000 7f8eac870640 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b293010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.668+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2639203466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 msgr2=0x55a86b2924c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.668+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b2924c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.668+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2639203466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 msgr2=0x55a86b293010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.668+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b293010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.668+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2639203466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86b1b9530 con 0x55a86aeea1b0 2024-09-21T02:57:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.668+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b2924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.668+0000 7f8eac870640 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b293010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.668+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b291e50 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f8e98005760 tx=0x7f8e980cdfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7f70ab917f5a27 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.672+0000 7f8e8e7fc640 1 -- 172.21.15.29:0/2639203466 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e98049040 con 0x55a86aeea1b0 2024-09-21T02:57:29.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.672+0000 7f8e8e7fc640 1 -- 172.21.15.29:0/2639203466 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e980d3290 con 0x55a86aeea1b0 2024-09-21T02:57:29.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.672+0000 7f8e8e7fc640 1 -- 172.21.15.29:0/2639203466 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e980d7f80 con 0x55a86aeea1b0 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.673+0000 7f8eb5244740 1 -- 172.21.15.29:0/2639203466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 msgr2=0x55a86b291e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.673+0000 7f8eb5244740 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b291e50 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f8e98005760 tx=0x7f8e980cdfe0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 -- 172.21.15.29:0/2639203466 shutdown_connections 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b293010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b2924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 --2- 172.21.15.29:0/2639203466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b291e50 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 -- 172.21.15.29:0/2639203466 >> 172.21.15.29:0/2639203466 conn(0x55a86aede300 msgr2=0x55a86b28d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 -- 172.21.15.29:0/2639203466 shutdown_connections 2024-09-21T02:57:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 -- 172.21.15.29:0/2639203466 wait complete. 2024-09-21T02:57:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 -- start start 2024-09-21T02:57:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.674+0000 7f8eb5244740 1 Processor -- start 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b29b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b29ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b29c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eb5244740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a86b12bab0 con 0x55a86aeea1b0 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eb5244740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a86b199730 con 0x55a86aec94d0 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eb5244740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a86b2938f0 con 0x55a86aec12f0 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eac870640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b29c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eb5244740 10 client.0 _ll_register_callbacks cb 0x55a86b1ed6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8ea77fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b29ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b29b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eac870640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b29c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34764/0 (socket says 172.21.15.29:34764) 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8eac870640 1 -- 172.21.15.29:0/1474685423 learned_addr learned my addr 172.21.15.29:0/1474685423 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8ea77fe640 1 -- 172.21.15.29:0/1474685423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 msgr2=0x55a86b29b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b29b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8ea77fe640 1 -- 172.21.15.29:0/1474685423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 msgr2=0x55a86b29c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b29c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.675+0000 7f8ea77fe640 1 -- 172.21.15.29:0/1474685423 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86af55790 con 0x55a86aec94d0 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b29ba10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00e5a0 tx=0x7f8e9c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ccb9f41055aaafa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8e877fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8e877fe640 1 -- 172.21.15.29:0/1474685423 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c025070 con 0x55a86aec94d0 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8e877fe640 1 -- 172.21.15.29:0/1474685423 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e9c013720 con 0x55a86aec94d0 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8eb5244740 1 -- 172.21.15.29:0/1474685423 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e980d3290 con 0x55a86aec94d0 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8e877fe640 1 -- 172.21.15.29:0/1474685423 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8e9c031050 con 0x55a86aec94d0 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8eb5244740 10 client.27236 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8eb5244740 1 -- 172.21.15.29:0/1474685423 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a86afce620 con 0x55a86aec94d0 2024-09-21T02:57:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.676+0000 7f8e877fe640 1 -- 172.21.15.29:0/1474685423 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e980d3290 con 0x55a86aec94d0 2024-09-21T02:57:29.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8e877fe640 1 -- 172.21.15.29:0/1474685423 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f8e9c021950 con 0x55a86aec94d0 2024-09-21T02:57:29.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8eb5244740 10 client.27236 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:29.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8eb5244740 10 client.27236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:57:29.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8eb5244740 20 client.27236 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:29.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8eb5244740 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a86aea8d80 0x55a86b2a8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8eb5244740 4 client.27236 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8eb5244740 1 -- 172.21.15.29:0/1474685423 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a86b206640 con 0x55a86aea8d80 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a86aea8d80 0x55a86b2a8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8e877fe640 10 client.27236 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.677+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a86aea8d80 0x55a86b2a8ad0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f441f4e45b68981 server_cookie=36f2e148ab7e35c4 in_seq=0 out_seq=0 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.679+0000 7f8e877fe640 1 -- 172.21.15.29:0/1474685423 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a86b206640 con 0x55a86aea8d80 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.679+0000 7f8e877fe640 10 client.27236 handle_command_reply: tid=0 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.679+0000 7f8eb5244740 1 client.27236 shutdown 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.679+0000 7f8eb5244740 1 -- 172.21.15.29:0/1474685423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 msgr2=0x55a86b29ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.679+0000 7f8eb5244740 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b29ba10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00e5a0 tx=0x7f8e9c004750 comp rx=0 tx=0).stop 2024-09-21T02:57:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 1 -- 172.21.15.29:0/1474685423 shutdown_connections 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a86aea8d80 0x55a86b2a8ad0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aeea1b0 0x55a86b29c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b29ba10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8e877fe640 0 client.27236 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 1 --2- 172.21.15.29:0/1474685423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec12f0 0x55a86b29b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 1 -- 172.21.15.29:0/1474685423 >> 172.21.15.29:0/1474685423 conn(0x55a86aede300 msgr2=0x55a86b28bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 1 -- 172.21.15.29:0/1474685423 shutdown_connections 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 1 -- 172.21.15.29:0/1474685423 wait complete. 2024-09-21T02:57:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.680+0000 7f8eb5244740 20 client.27236 trim_cache size 0 max 16384 2024-09-21T02:57:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eb5244740 1 Processor -- start 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eb5244740 1 -- start start 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b292c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec94d0 0x55a86b295f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aeea1b0 0x55a86b2964a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eb5244740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81ae830 con 0x55a86aec12f0 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eb5244740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81b0370 con 0x55a86aeea1b0 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eb5244740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81aef00 con 0x55a86aec94d0 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eac870640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aeea1b0 0x55a86b2964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eac870640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aeea1b0 0x55a86b2964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:54752/0 (socket says 172.21.15.29:54752) 2024-09-21T02:57:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.705+0000 7f8eac870640 1 -- 172.21.15.29:0/3015512113 learned_addr learned my addr 172.21.15.29:0/3015512113 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.706+0000 7f8eac870640 1 -- 172.21.15.29:0/3015512113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec94d0 msgr2=0x55a86b295f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:57:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.706+0000 7f8eac870640 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec94d0 0x55a86b295f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.706+0000 7f8eac870640 1 -- 172.21.15.29:0/3015512113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 msgr2=0x55a86b292c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.706+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b292c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.706+0000 7f8eac870640 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b292c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.706+0000 7f8eac870640 1 -- 172.21.15.29:0/3015512113 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86b287b40 con 0x55a86aeea1b0 2024-09-21T02:57:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.706+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b292c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:57:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.707+0000 7f8eac870640 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aeea1b0 0x55a86b2964a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0005fd0 tx=0x7f8ea0009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b9f9fa44ad5c173 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.707+0000 7f8e8dffb640 1 -- 172.21.15.29:0/3015512113 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea00120d0 con 0x55a86aeea1b0 2024-09-21T02:57:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.707+0000 7f8e8dffb640 1 -- 172.21.15.29:0/3015512113 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ea0021070 con 0x55a86aeea1b0 2024-09-21T02:57:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.707+0000 7f8e8dffb640 1 -- 172.21.15.29:0/3015512113 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0010a50 con 0x55a86aeea1b0 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.708+0000 7f8eb5244740 1 -- 172.21.15.29:0/3015512113 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aeea1b0 msgr2=0x55a86b2964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.708+0000 7f8eb5244740 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aeea1b0 0x55a86b2964a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0005fd0 tx=0x7f8ea0009500 comp rx=0 tx=0).stop 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.708+0000 7f8eb5244740 1 -- 172.21.15.29:0/3015512113 shutdown_connections 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.708+0000 7f8eb5244740 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aeea1b0 0x55a86b2964a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.708+0000 7f8eb5244740 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aec94d0 0x55a86b295f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.708+0000 7f8eb5244740 1 --2- 172.21.15.29:0/3015512113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b292c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.708+0000 7f8eb5244740 1 -- 172.21.15.29:0/3015512113 >> 172.21.15.29:0/3015512113 conn(0x55a86aede300 msgr2=0x55a86b2998e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.708+0000 7f8eb5244740 1 -- 172.21.15.29:0/3015512113 shutdown_connections 2024-09-21T02:57:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.709+0000 7f8eb5244740 1 -- 172.21.15.29:0/3015512113 wait complete. 2024-09-21T02:57:29.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.709+0000 7f8eb5244740 1 -- start start 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.709+0000 7f8eb5244740 1 Processor -- start 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b2a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b2a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8eb5244740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aeea1b0 0x55a86b2a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8eb5244740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81afca0 con 0x55a86aec12f0 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea77fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b2a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b2a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8eb5244740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81af5d0 con 0x55a86aec94d0 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea77fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b2a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:54758/0 (socket says 172.21.15.29:54758) 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea77fe640 1 -- 172.21.15.29:0/462095109 learned_addr learned my addr 172.21.15.29:0/462095109 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8eb5244740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81ac810 con 0x55a86aeea1b0 2024-09-21T02:57:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8eac870640 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aeea1b0 0x55a86b2a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8eb5244740 10 client.0 _ll_register_callbacks cb 0x7f8e9c009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea77fe640 1 -- 172.21.15.29:0/462095109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aeea1b0 msgr2=0x55a86b2a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aeea1b0 0x55a86b2a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea77fe640 1 -- 172.21.15.29:0/462095109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 msgr2=0x55a86b2a30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b2a30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.710+0000 7f8ea77fe640 1 -- 172.21.15.29:0/462095109 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86b2a2a30 con 0x55a86aec94d0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8ea77fe640 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b2a1db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c016ff0 tx=0x7f8e9c006b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84c526034372d999 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8e877fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8e877fe640 1 -- 172.21.15.29:0/462095109 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c03a060 con 0x55a86aec94d0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8e877fe640 1 -- 172.21.15.29:0/462095109 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e9c021a10 con 0x55a86aec94d0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8eb5244740 1 -- 172.21.15.29:0/462095109 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ea0021070 con 0x55a86aec94d0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8e877fe640 1 -- 172.21.15.29:0/462095109 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8e9c041050 con 0x55a86aec94d0 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8eb5244740 10 client.27242 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8eb5244740 1 -- 172.21.15.29:0/462095109 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a86b3134d0 con 0x55a86aec94d0 2024-09-21T02:57:29.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.711+0000 7f8e877fe640 1 -- 172.21.15.29:0/462095109 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e9c0044c0 con 0x55a86aec94d0 2024-09-21T02:57:29.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.712+0000 7f8e877fe640 1 -- 172.21.15.29:0/462095109 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f8e9c013720 con 0x55a86aec94d0 2024-09-21T02:57:29.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.712+0000 7f8eb5244740 10 client.27242 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.712+0000 7f8eb5244740 10 client.27242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:57:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.712+0000 7f8eb5244740 20 client.27242 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.712+0000 7f8eb5244740 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a86ae7e1c0 0x55a86b315e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.712+0000 7f8eb5244740 4 client.27242 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:57:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.712+0000 7f8eb5244740 1 -- 172.21.15.29:0/462095109 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f8ea0021070 con 0x55a86ae7e1c0 2024-09-21T02:57:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.712+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a86ae7e1c0 0x55a86b315e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.713+0000 7f8e877fe640 10 client.27242 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.713+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a86ae7e1c0 0x55a86b315e30 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0b8ed7716fb3b73 server_cookie=6ad2ad01af6f7cc0 in_seq=0 out_seq=0 2024-09-21T02:57:29.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34112 6.16 MB/sec execute 186 sec latency 123.961 ms 2024-09-21T02:57:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.869+0000 7f8e877fe640 1 -- 172.21.15.29:0/462095109 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f8ea0021070 con 0x55a86ae7e1c0 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.869+0000 7f8e877fe640 10 client.27242 handle_command_reply: tid=0 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.869+0000 7f8eb5244740 1 client.27242 shutdown 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 -- 172.21.15.29:0/462095109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 msgr2=0x55a86b2a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b2a1db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c016ff0 tx=0x7f8e9c006b50 comp rx=0 tx=0).stop 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 -- 172.21.15.29:0/462095109 shutdown_connections 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a86aeea1b0 0x55a86b2a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a86aec94d0 0x55a86b2a1db0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a86ae7e1c0 0x55a86b315e30 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 --2- 172.21.15.29:0/462095109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a86aec12f0 0x55a86b2a30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8e877fe640 0 client.27242 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 -- 172.21.15.29:0/462095109 >> 172.21.15.29:0/462095109 conn(0x55a86aede300 msgr2=0x55a86b292c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 -- 172.21.15.29:0/462095109 shutdown_connections 2024-09-21T02:57:29.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 1 -- 172.21.15.29:0/462095109 wait complete. 2024-09-21T02:57:29.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.870+0000 7f8eb5244740 20 client.27242 trim_cache size 0 max 16384 2024-09-21T02:57:29.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:57:29.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:57:29.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:57:29.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 -- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e900b6510 msgr2=0x7f8e900b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e900b6510 0x7f8e900b8a20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c006fd0 tx=0x7f8e9c0090e0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 -- 172.21.15.29:0/2271670885 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 msgr2=0x7f8ea819c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 0x7f8ea819c120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009230 tx=0x7f8e980040c0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 -- 172.21.15.29:0/2271670885 shutdown_connections 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e900b6510 0x7f8e900b8a20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea810b630 0x7f8ea819d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea8108c80 0x7f8ea819c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 --2- 172.21.15.29:0/2271670885 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea806c240 0x7f8ea819c120 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 -- 172.21.15.29:0/2271670885 >> 172.21.15.29:0/2271670885 conn(0x7f8ea80fdf00 msgr2=0x7f8ea8109a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 -- 172.21.15.29:0/2271670885 shutdown_connections 2024-09-21T02:57:29.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:29.871+0000 7f8eae2d3640 1 -- 172.21.15.29:0/2271670885 wait complete. 2024-09-21T02:57:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:29 smithi066 ceph-mon[33436]: pgmap v940: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 3.5 MiB/s wr, 254 op/s 2024-09-21T02:57:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:29 smithi190 ceph-mon[33871]: pgmap v940: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 3.5 MiB/s wr, 254 op/s 2024-09-21T02:57:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:29 smithi029 ceph-mon[31006]: pgmap v940: 129 pgs: 129 active+clean; 292 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 3.5 MiB/s wr, 254 op/s 2024-09-21T02:57:30.726 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34206 6.15 MB/sec execute 187 sec latency 279.821 ms 2024-09-21T02:57:31.727 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 6.13 MB/sec execute 188 sec latency 727.771 ms 2024-09-21T02:57:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:31 smithi029 ceph-mon[31006]: pgmap v941: 129 pgs: 129 active+clean; 291 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 -- 172.21.15.29:0/2506207562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc1094d0 msgr2=0x7f5ddc1098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 --2- 172.21.15.29:0/2506207562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc1094d0 0x7f5ddc1098e0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009900 tx=0x7f5dc402a410 comp rx=0 tx=0).stop 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 -- 172.21.15.29:0/2506207562 shutdown_connections 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 --2- 172.21.15.29:0/2506207562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ddc109e20 0x7f5ddc110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 --2- 172.21.15.29:0/2506207562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc1094d0 0x7f5ddc1098e0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 --2- 172.21.15.29:0/2506207562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ddc103b90 0x7f5ddc103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 -- 172.21.15.29:0/2506207562 >> 172.21.15.29:0/2506207562 conn(0x7f5ddc0ff5c0 msgr2=0x7f5ddc101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 -- 172.21.15.29:0/2506207562 shutdown_connections 2024-09-21T02:57:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.925+0000 7f5de0bac640 1 -- 172.21.15.29:0/2506207562 wait complete. 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.926+0000 7f5de0bac640 1 Processor -- start 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.926+0000 7f5de0bac640 1 -- start start 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5de0bac640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ddc103b90 0x7f5ddc19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5de0bac640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ddc1094d0 0x7f5ddc19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5de0bac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc109e20 0x7f5ddc19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5de0bac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc113990 con 0x7f5ddc109e20 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5de0bac640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc113810 con 0x7f5ddc1094d0 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5de0bac640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc113b10 con 0x7f5ddc103b90 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc109e20 0x7f5ddc19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc109e20 0x7f5ddc19fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34806/0 (socket says 172.21.15.29:34806) 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 -- 172.21.15.29:0/3901087650 learned_addr learned my addr 172.21.15.29:0/3901087650 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5dda59c640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ddc103b90 0x7f5ddc19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ddc1094d0 0x7f5ddc19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 -- 172.21.15.29:0/3901087650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ddc103b90 msgr2=0x7f5ddc19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ddc103b90 0x7f5ddc19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 -- 172.21.15.29:0/3901087650 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ddc1094d0 msgr2=0x7f5ddc19a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ddc1094d0 0x7f5ddc19a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 -- 172.21.15.29:0/3901087650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc1a0350 con 0x7f5ddc109e20 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ddc1094d0 0x7f5ddc19a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.927+0000 7f5ddad9d640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc109e20 0x7f5ddc19fc20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400eb10 tx=0x7f5dd400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.928+0000 7f5dd37fe640 1 -- 172.21.15.29:0/3901087650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4004b20 con 0x7f5ddc109e20 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.928+0000 7f5de0bac640 1 -- 172.21.15.29:0/3901087650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ddc1a0640 con 0x7f5ddc109e20 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.928+0000 7f5dd37fe640 1 -- 172.21.15.29:0/3901087650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dd4004cc0 con 0x7f5ddc109e20 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.928+0000 7f5dd37fe640 1 -- 172.21.15.29:0/3901087650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4010690 con 0x7f5ddc109e20 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.928+0000 7f5de0bac640 1 -- 172.21.15.29:0/3901087650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ddc1a6da0 con 0x7f5ddc109e20 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.930+0000 7f5dd37fe640 1 -- 172.21.15.29:0/3901087650 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f5dd4010830 con 0x7f5ddc109e20 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.930+0000 7f5de7120740 1 Processor -- start 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5de7120740 1 -- start start 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5de7120740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb39600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5de7120740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb39b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5de7120740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb3a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5de7120740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a03baafb90 con 0x55a03b792840 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5de7120740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a03ba9efc0 con 0x55a03b771b60 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5de7120740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a03baa2dd0 con 0x55a03b769980 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb39600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb39600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34808/0 (socket says 172.21.15.29:34808) 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dd3fff640 1 -- 172.21.15.29:0/915175030 learned_addr learned my addr 172.21.15.29:0/915175030 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dda59c640 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb39b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb3a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dd37fe640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5dc80b6510 0x7f5dc80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dda59c640 1 -- 172.21.15.29:0/915175030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 msgr2=0x55a03bb3a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dda59c640 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb3a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dda59c640 1 -- 172.21.15.29:0/915175030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 msgr2=0x55a03bb39600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dda59c640 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb39600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dda59c640 1 -- 172.21.15.29:0/915175030 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a03ba26060 con 0x55a03b769980 2024-09-21T02:57:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.931+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb3a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.932+0000 7f5ddad9d640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5dc80b6510 0x7f5dc80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.932+0000 7f5dda59c640 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb39b40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc004630 tx=0x7f5dcc00bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efc7051c2728962e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.932+0000 7f5dd37fe640 1 -- 172.21.15.29:0/3901087650 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5dd40e7da0 con 0x7f5ddc109e20 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.932+0000 7f5dd0ff9640 1 -- 172.21.15.29:0/915175030 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc003bd0 con 0x55a03b769980 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.932+0000 7f5dd0ff9640 1 -- 172.21.15.29:0/915175030 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dcc015070 con 0x55a03b769980 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.932+0000 7f5dd0ff9640 1 -- 172.21.15.29:0/915175030 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc004c00 con 0x55a03b769980 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 -- 172.21.15.29:0/915175030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 msgr2=0x55a03bb39b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5ddad9d640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5dc80b6510 0x7f5dc80b8a20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009ae0 tx=0x7f5dd400cee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb39b40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc004630 tx=0x7f5dcc00bb20 comp rx=0 tx=0).stop 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 -- 172.21.15.29:0/915175030 shutdown_connections 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb3a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb39b40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 --2- 172.21.15.29:0/915175030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb39600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 -- 172.21.15.29:0/915175030 >> 172.21.15.29:0/915175030 conn(0x55a03b786990 msgr2=0x55a03bb374f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 -- 172.21.15.29:0/915175030 shutdown_connections 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 -- 172.21.15.29:0/915175030 wait complete. 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.933+0000 7f5de7120740 1 -- start start 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5de7120740 1 Processor -- start 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5de7120740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b769980 0x55a03bb45240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5de7120740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb45800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5de7120740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b792840 0x55a03bb510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5de7120740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a03b9007a0 con 0x55a03b769980 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5de7120740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a03ba440a0 con 0x55a03b771b60 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5de7120740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a03bb47de0 con 0x55a03b792840 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5de7120740 10 client.0 _ll_register_callbacks cb 0x55a03ba213c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b769980 0x55a03bb45240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b769980 0x55a03bb45240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34810/0 (socket says 172.21.15.29:34810) 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 -- 172.21.15.29:0/3340894010 learned_addr learned my addr 172.21.15.29:0/3340894010 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b792840 0x55a03bb510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dda59c640 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb45800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 -- 172.21.15.29:0/3340894010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b792840 msgr2=0x55a03bb510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b792840 0x55a03bb510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 -- 172.21.15.29:0/3340894010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 msgr2=0x55a03bb45800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb45800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dd3fff640 1 -- 172.21.15.29:0/3340894010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a03ba32880 con 0x55a03b769980 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.934+0000 7f5dda59c640 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb45800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b769980 0x55a03bb45240 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f5dc402a930 tx=0x7f5dc40056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e551989e0ab2b508 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5dc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/3340894010 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4043030 con 0x55a03b769980 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/3340894010 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5dc40037f0 con 0x55a03b769980 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5de7120740 1 -- 172.21.15.29:0/3340894010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dcc015070 con 0x55a03b769980 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/3340894010 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5dc4053050 con 0x55a03b769980 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5de7120740 10 client.17907 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5de7120740 1 -- 172.21.15.29:0/3340894010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a03b80cb90 con 0x55a03b769980 2024-09-21T02:57:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/3340894010 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dcc015070 con 0x55a03b769980 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.935+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/3340894010 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f5dc403d8a0 con 0x55a03b769980 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5de7120740 10 client.17907 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5de7120740 10 client.17907 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5de7120740 20 client.17907 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5de7120740 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a03b7511c0 0x55a03bb57130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5de7120740 4 client.17907 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5de7120740 1 -- 172.21.15.29:0/3340894010 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a03bab1ae0 con 0x55a03b7511c0 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5dda59c640 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a03b7511c0 0x55a03bb57130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5dc1ffb640 10 client.17907 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.936+0000 7f5dda59c640 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a03b7511c0 0x55a03bb57130 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6745b1900403b5cd server_cookie=7f5721bae5743beb in_seq=0 out_seq=0 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/3340894010 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a03bab1ae0 con 0x55a03b7511c0 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5dc1ffb640 10 client.17907 handle_command_reply: tid=0 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 client.17907 shutdown 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 -- 172.21.15.29:0/3340894010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b769980 msgr2=0x55a03bb45240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b769980 0x55a03bb45240 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f5dc402a930 tx=0x7f5dc40056d0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 -- 172.21.15.29:0/3340894010 shutdown_connections 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b792840 0x55a03bb510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb45800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a03b7511c0 0x55a03bb57130 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5dc1ffb640 0 client.17907 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 --2- 172.21.15.29:0/3340894010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b769980 0x55a03bb45240 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.937+0000 7f5de7120740 1 -- 172.21.15.29:0/3340894010 >> 172.21.15.29:0/3340894010 conn(0x55a03b786990 msgr2=0x55a03bb3c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.938+0000 7f5de7120740 1 -- 172.21.15.29:0/3340894010 shutdown_connections 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.938+0000 7f5de7120740 1 -- 172.21.15.29:0/3340894010 wait complete. 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.938+0000 7f5de7120740 20 client.17907 trim_cache size 0 max 16384 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5de7120740 1 Processor -- start 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5de7120740 1 -- start start 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5de7120740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb50fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5de7120740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb51510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5de7120740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb42230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5de7120740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc1b0970 con 0x55a03b792840 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5de7120740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc1b24b0 con 0x55a03b771b60 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5de7120740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc1b1040 con 0x55a03b769980 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb42230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5dda59c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb51510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb42230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34820/0 (socket says 172.21.15.29:34820) 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5dd3fff640 1 -- 172.21.15.29:0/3866896460 learned_addr learned my addr 172.21.15.29:0/3866896460 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.959+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/3866896460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb50fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dda59c640 1 -- 172.21.15.29:0/3866896460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 msgr2=0x55a03bb50fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dda59c640 1 --2- 172.21.15.29:0/3866896460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb50fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dda59c640 1 -- 172.21.15.29:0/3866896460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 msgr2=0x55a03bb42230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dda59c640 1 --2- 172.21.15.29:0/3866896460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb42230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dda59c640 1 -- 172.21.15.29:0/3866896460 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a03bb45c60 con 0x55a03b771b60 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dda59c640 1 --2- 172.21.15.29:0/3866896460 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb51510 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0048a0 tx=0x7f5dcc013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8db542103735a586 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dc3fff640 1 -- 172.21.15.29:0/3866896460 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc0046e0 con 0x55a03b771b60 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dc3fff640 1 -- 172.21.15.29:0/3866896460 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dcc015040 con 0x55a03b771b60 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.960+0000 7f5dc3fff640 1 -- 172.21.15.29:0/3866896460 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc028c60 con 0x55a03b771b60 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 -- 172.21.15.29:0/3866896460 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 msgr2=0x55a03bb51510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 --2- 172.21.15.29:0/3866896460 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb51510 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0048a0 tx=0x7f5dcc013040 comp rx=0 tx=0).stop 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 -- 172.21.15.29:0/3866896460 shutdown_connections 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 --2- 172.21.15.29:0/3866896460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bb42230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 --2- 172.21.15.29:0/3866896460 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb51510 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 --2- 172.21.15.29:0/3866896460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb50fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 -- 172.21.15.29:0/3866896460 >> 172.21.15.29:0/3866896460 conn(0x55a03b786990 msgr2=0x55a03bb46560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 -- 172.21.15.29:0/3866896460 shutdown_connections 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.961+0000 7f5de7120740 1 -- 172.21.15.29:0/3866896460 wait complete. 2024-09-21T02:57:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 1 -- start start 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 1 Processor -- start 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb4a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb4ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bbc1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc1b1de0 con 0x55a03b792840 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc1b1710 con 0x55a03b771b60 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc1ae950 con 0x55a03b769980 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5de7120740 10 client.0 _ll_register_callbacks cb 0x7f5dc4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bbc1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb4a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5dd3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb4a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47706/0 (socket says 172.21.15.29:47706) 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5dd3fff640 1 -- 172.21.15.29:0/614862046 learned_addr learned my addr 172.21.15.29:0/614862046 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5dda59c640 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb4ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.962+0000 7f5dd3fff640 1 -- 172.21.15.29:0/614862046 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 msgr2=0x55a03bb4ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb4ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dd3fff640 1 -- 172.21.15.29:0/614862046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 msgr2=0x55a03bbc1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bbc1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dd3fff640 1 -- 172.21.15.29:0/614862046 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a03bbc2040 con 0x55a03b769980 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dd3fff640 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb4a720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5dc403dde0 tx=0x7f5dc4032eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce557b4f62fc845 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/614862046 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc403de60 con 0x55a03b769980 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5de7120740 1 -- 172.21.15.29:0/614862046 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dcc015040 con 0x55a03b769980 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/614862046 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f5dc4031a20 con 0x55a03b769980 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/614862046 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5dc4043040 con 0x55a03b769980 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5de7120740 10 client.27403 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5de7120740 1 -- 172.21.15.29:0/614862046 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a03bbc3270 con 0x55a03b769980 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.963+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/614862046 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dcc015040 con 0x55a03b769980 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/614862046 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f5dc404dbb0 con 0x55a03b769980 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5de7120740 10 client.27403 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5de7120740 10 client.27403 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5de7120740 20 client.27403 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5de7120740 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a03b726570 0x55a03bbc59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5de7120740 4 client.27403 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.2, "await_for": 45.0}] 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5de7120740 1 -- 172.21.15.29:0/614862046 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.2, "await_for": 45.0}) -- 0x55a03baad590 con 0x55a03b726570 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5dda59c640 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a03b726570 0x55a03bbc59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5dda59c640 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a03b726570 0x55a03bbc59b0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a78254091caf9e9d server_cookie=65ecb1f8957f4c20 in_seq=0 out_seq=0 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:30.964+0000 7f5dc1ffb640 10 client.27403 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5dc1ffb640 1 -- 172.21.15.29:0/614862046 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a03baad590 con 0x55a03b726570 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5dc1ffb640 10 client.27403 handle_command_reply: tid=0 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 client.27403 shutdown 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 -- 172.21.15.29:0/614862046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 msgr2=0x55a03bb4a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb4a720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5dc403dde0 tx=0x7f5dc4032eb0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 -- 172.21.15.29:0/614862046 shutdown_connections 2024-09-21T02:57:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a03b726570 0x55a03bbc59b0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a03b792840 0x55a03bbc1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5dc1ffb640 0 client.27403 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a03b771b60 0x55a03bb4ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 --2- 172.21.15.29:0/614862046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a03b769980 0x55a03bb4a720 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 -- 172.21.15.29:0/614862046 >> 172.21.15.29:0/614862046 conn(0x55a03b786990 msgr2=0x55a03bb35a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 -- 172.21.15.29:0/614862046 shutdown_connections 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 70, 2024-09-21T02:57:31.960 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "28a84c4a": { 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 70, 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 130.2, 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:57:31.961 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 1 -- 172.21.15.29:0/614862046 wait complete. 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.523+0000 7f5de7120740 20 client.27403 trim_cache size 0 max 16384 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 -- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5dc80b6510 msgr2=0x7f5dc80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5dc80b6510 0x7f5dc80b8a20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009ae0 tx=0x7f5dd400cee0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 -- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc109e20 msgr2=0x7f5ddc19fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc109e20 0x7f5ddc19fc20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400eb10 tx=0x7f5dd400c2d0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 -- 172.21.15.29:0/3901087650 shutdown_connections 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5dc80b6510 0x7f5dc80b8a20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ddc109e20 0x7f5ddc19fc20 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ddc1094d0 0x7f5ddc19a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 --2- 172.21.15.29:0/3901087650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ddc103b90 0x7f5ddc19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 -- 172.21.15.29:0/3901087650 >> 172.21.15.29:0/3901087650 conn(0x7f5ddc0ff5c0 msgr2=0x7f5ddc1019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 -- 172.21.15.29:0/3901087650 shutdown_connections 2024-09-21T02:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:31.524+0000 7f5de0bac640 1 -- 172.21.15.29:0/3901087650 wait complete. 2024-09-21T02:57:32.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:31 smithi066 ceph-mon[33436]: pgmap v941: 129 pgs: 129 active+clean; 291 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-09-21T02:57:32.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:31 smithi190 ceph-mon[33871]: pgmap v941: 129 pgs: 129 active+clean; 291 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-09-21T02:57:32.727 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 6.10 MB/sec execute 189 sec latency 1727.881 ms 2024-09-21T02:57:33.727 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 6.07 MB/sec execute 190 sec latency 2728.002 ms 2024-09-21T02:57:34.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:33 smithi066 ceph-mon[33436]: pgmap v942: 129 pgs: 129 active+clean; 280 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.4 MiB/s wr, 252 op/s 2024-09-21T02:57:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:33 smithi190 ceph-mon[33871]: pgmap v942: 129 pgs: 129 active+clean; 280 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.4 MiB/s wr, 252 op/s 2024-09-21T02:57:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:33 smithi029 ceph-mon[31006]: pgmap v942: 129 pgs: 129 active+clean; 280 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.4 MiB/s wr, 252 op/s 2024-09-21T02:57:34.727 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 6.03 MB/sec execute 191 sec latency 3728.107 ms 2024-09-21T02:57:35.727 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 6.00 MB/sec execute 192 sec latency 4728.206 ms 2024-09-21T02:57:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:35 smithi066 ceph-mon[33436]: pgmap v943: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-09-21T02:57:36.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:35 smithi190 ceph-mon[33871]: pgmap v943: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-09-21T02:57:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:35 smithi029 ceph-mon[31006]: pgmap v943: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-09-21T02:57:36.727 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 5.97 MB/sec execute 193 sec latency 5728.309 ms 2024-09-21T02:57:37.727 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 5.94 MB/sec execute 194 sec latency 6728.397 ms 2024-09-21T02:57:37.799 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '28a84c4a', quiesce took 43.7 seconds. Will release after: -33.5 2024-09-21T02:57:37.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:57:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:37 smithi066 ceph-mon[33436]: pgmap v944: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 2.1 MiB/s wr, 255 op/s 2024-09-21T02:57:38.110 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:57:38.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:37 smithi190 ceph-mon[33871]: pgmap v944: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 2.1 MiB/s wr, 255 op/s 2024-09-21T02:57:38.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:37 smithi029 ceph-mon[31006]: pgmap v944: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 2.1 MiB/s wr, 255 op/s 2024-09-21T02:57:38.727 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 5.91 MB/sec execute 195 sec latency 7728.451 ms 2024-09-21T02:57:39.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 5.88 MB/sec execute 196 sec latency 8728.808 ms 2024-09-21T02:57:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 -- 172.21.15.29:0/3259224488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8108d20 msgr2=0x7f40f810f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 --2- 172.21.15.29:0/3259224488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8108d20 0x7f40f810f550 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009ca0 tx=0x7f40f002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 -- 172.21.15.29:0/3259224488 shutdown_connections 2024-09-21T02:57:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 --2- 172.21.15.29:0/3259224488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8108d20 0x7f40f810f550 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 --2- 172.21.15.29:0/3259224488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f81083d0 0x7f40f81087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 --2- 172.21.15.29:0/3259224488 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f40f8102ae0 0x7f40f8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 -- 172.21.15.29:0/3259224488 >> 172.21.15.29:0/3259224488 conn(0x7f40f8079790 msgr2=0x7f40f8079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 -- 172.21.15.29:0/3259224488 shutdown_connections 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.831+0000 7f40fe61d640 1 -- 172.21.15.29:0/3259224488 wait complete. 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.832+0000 7f40fe61d640 1 Processor -- start 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.832+0000 7f40fe61d640 1 -- start start 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40fe61d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8102ae0 0x7f40f819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40fe61d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f40f81083d0 0x7f40f819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40fe61d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f8108d20 0x7f40f819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40fe61d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f40f81128e0 con 0x7f40f8102ae0 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40fe61d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f40f8112760 con 0x7f40f81083d0 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40fe61d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40f8112a60 con 0x7f40f8108d20 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8102ae0 0x7f40f819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8102ae0 0x7f40f819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57652/0 (socket says 172.21.15.29:57652) 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 -- 172.21.15.29:0/2205685560 learned_addr learned my addr 172.21.15.29:0/2205685560 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40fcbba640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f8108d20 0x7f40f819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f77fe640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f40f81083d0 0x7f40f819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 -- 172.21.15.29:0/2205685560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f8108d20 msgr2=0x7f40f819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f8108d20 0x7f40f819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 -- 172.21.15.29:0/2205685560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f40f81083d0 msgr2=0x7f40f819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f40f81083d0 0x7f40f819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 -- 172.21.15.29:0/2205685560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f81a02d0 con 0x7f40f8102ae0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f77fe640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f40f81083d0 0x7f40f819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f7fff640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8102ae0 0x7f40f819a360 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f40e800d950 tx=0x7f40e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.833+0000 7f40f57fa640 1 -- 172.21.15.29:0/2205685560 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8017030 con 0x7f40f8102ae0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.834+0000 7f40f57fa640 1 -- 172.21.15.29:0/2205685560 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f40e8004020 con 0x7f40f8102ae0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.834+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f81a05c0 con 0x7f40f8102ae0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.834+0000 7f40f57fa640 1 -- 172.21.15.29:0/2205685560 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e80055d0 con 0x7f40f8102ae0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.834+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f81a6d20 con 0x7f40f8102ae0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.835+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f8102ef0 con 0x7f40f8102ae0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.836+0000 7f40f57fa640 1 -- 172.21.15.29:0/2205685560 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f40e8009a30 con 0x7f40f8102ae0 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.838+0000 7f40f57fa640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f40e00b6510 0x7f40e00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.838+0000 7f40f77fe640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f40e00b6510 0x7f40e00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.838+0000 7f40f57fa640 1 -- 172.21.15.29:0/2205685560 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f40e80e7730 con 0x7f40f8102ae0 2024-09-21T02:57:39.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:39 smithi029 ceph-mon[31006]: pgmap v945: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.9 MiB/s wr, 244 op/s 2024-09-21T02:57:39.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:39 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2205685560' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:57:39.820 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:57:39.820 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.838+0000 7f40f77fe640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f40e00b6510 0x7f40e00b8a20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f40ec005fd0 tx=0x7f40ec0094d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:38.841+0000 7f40f57fa640 1 -- 172.21.15.29:0/2205685560 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f40e80adc40 con 0x7f40f8102ae0 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.067+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40f819b490 con 0x7f40f8102ae0 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.068+0000 7f40f57fa640 1 -- 172.21.15.29:0/2205685560 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f40e80b2020 con 0x7f40f8102ae0 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f40e00b6510 msgr2=0x7f40e00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f40e00b6510 0x7f40e00b8a20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f40ec005fd0 tx=0x7f40ec0094d0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8102ae0 msgr2=0x7f40f819a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8102ae0 0x7f40f819a360 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f40e800d950 tx=0x7f40e800de10 comp rx=0 tx=0).stop 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 shutdown_connections 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f40e00b6510 0x7f40e00b8a20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f8108d20 0x7f40f819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f40f81083d0 0x7f40f819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 --2- 172.21.15.29:0/2205685560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40f8102ae0 0x7f40f819a360 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.070+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 >> 172.21.15.29:0/2205685560 conn(0x7f40f8079790 msgr2=0x7f40f8101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.071+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 shutdown_connections 2024-09-21T02:57:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:39.071+0000 7f40fe61d640 1 -- 172.21.15.29:0/2205685560 wait complete. 2024-09-21T02:57:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:39 smithi066 ceph-mon[33436]: pgmap v945: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.9 MiB/s wr, 244 op/s 2024-09-21T02:57:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:39 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2205685560' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:57:40.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:39 smithi190 ceph-mon[33871]: pgmap v945: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.9 MiB/s wr, 244 op/s 2024-09-21T02:57:40.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:39 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2205685560' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:57:40.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 5.85 MB/sec execute 197 sec latency 9728.932 ms 2024-09-21T02:57:40.936 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 28a84c4a --release --await' 2024-09-21T02:57:40.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 28a84c4a --release --await 2024-09-21T02:57:41.205 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:57:41.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 5.82 MB/sec execute 198 sec latency 10728.992 ms 2024-09-21T02:57:41.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1294313837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 msgr2=0x7fd0f4103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1294313837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 0x7fd0f4103fa0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009ca0 tx=0x7fd0e402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1294313837 shutdown_connections 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1294313837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0f4109e20 0x7fd0f4110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1294313837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f41094d0 0x7fd0f41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1294313837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 0x7fd0f4103fa0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1294313837 >> 172.21.15.29:0/1294313837 conn(0x7fd0f40ff600 msgr2=0x7fd0f4101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1294313837 shutdown_connections 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.775+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1294313837 wait complete. 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.776+0000 7fd0f8ff7640 1 Processor -- start 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.776+0000 7fd0f8ff7640 1 -- start start 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f8ff7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 0x7fd0f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f8ff7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f41094d0 0x7fd0f4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f8ff7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0f4109e20 0x7fd0f4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f8ff7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f41139e0 con 0x7fd0f4103b90 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f8ff7640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f4113860 con 0x7fd0f4109e20 2024-09-21T02:57:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f8ff7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f4113b60 con 0x7fd0f41094d0 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 0x7fd0f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 0x7fd0f4075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57670/0 (socket says 172.21.15.29:57670) 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f259c640 1 -- 172.21.15.29:0/1156715146 learned_addr learned my addr 172.21.15.29:0/1156715146 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0f4109e20 0x7fd0f4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f41094d0 0x7fd0f4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f259c640 1 -- 172.21.15.29:0/1156715146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f41094d0 msgr2=0x7fd0f4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f259c640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f41094d0 0x7fd0f4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f259c640 1 -- 172.21.15.29:0/1156715146 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0f4109e20 msgr2=0x7fd0f4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f259c640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0f4109e20 0x7fd0f4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f259c640 1 -- 172.21.15.29:0/1156715146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f41a86b0 con 0x7fd0f4103b90 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f41094d0 0x7fd0f4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.777+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0f4109e20 0x7fd0f4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.778+0000 7fd0f259c640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 0x7fd0f4075710 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009dc0 tx=0x7fd0e4003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.778+0000 7fd0eb7fe640 1 -- 172.21.15.29:0/1156715146 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4032dd0 con 0x7fd0f4103b90 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.778+0000 7fd0eb7fe640 1 -- 172.21.15.29:0/1156715146 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0e40044d0 con 0x7fd0f4103b90 2024-09-21T02:57:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.778+0000 7fd0eb7fe640 1 -- 172.21.15.29:0/1156715146 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4030c50 con 0x7fd0f4103b90 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.778+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1156715146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f41a8940 con 0x7fd0f4103b90 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.778+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1156715146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f41a8e30 con 0x7fd0f4103b90 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0ff56b740 1 Processor -- start 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0ff56b740 1 -- start start 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a640253510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 0x55a640253a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a640253f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a64013edc0 con 0x55a63fead520 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a6401cb1f0 con 0x55a63fe8c840 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a6401ba620 con 0x55a63fe84660 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a640253510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a640253510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57672/0 (socket says 172.21.15.29:57672) 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.781+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/2787881202 learned_addr learned my addr 172.21.15.29:0/2787881202 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a640253f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/2787881202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 msgr2=0x55a640253a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 0x55a640253a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/2787881202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 msgr2=0x55a640253f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a640253f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/2787881202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a640182fd0 con 0x55a63fead520 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a640253f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0eb7fe640 1 -- 172.21.15.29:0/1156715146 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd0e4004680 con 0x7fd0f4103b90 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a640253510 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec010400 tx=0x7fd0ec0108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5214032ff76bd6aa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0e8ff9640 1 -- 172.21.15.29:0/2787881202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec004230 con 0x55a63fead520 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0e8ff9640 1 -- 172.21.15.29:0/2787881202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0ec0043d0 con 0x55a63fead520 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.782+0000 7fd0e8ff9640 1 -- 172.21.15.29:0/2787881202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec003960 con 0x55a63fead520 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.783+0000 7fd0eb7fe640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0e00b6510 0x7fd0e00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.783+0000 7fd0f259c640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0e00b6510 0x7fd0e00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.783+0000 7fd0eb7fe640 1 -- 172.21.15.29:0/1156715146 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd0e41087c0 con 0x7fd0f4103b90 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.783+0000 7fd0ff56b740 1 -- 172.21.15.29:0/2787881202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 msgr2=0x55a640253510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.783+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a640253510 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec010400 tx=0x7fd0ec0108c0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 -- 172.21.15.29:0/2787881202 shutdown_connections 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a640253f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 0x55a640253a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/2787881202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a640253510 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 -- 172.21.15.29:0/2787881202 >> 172.21.15.29:0/2787881202 conn(0x55a63fea1910 msgr2=0x55a640251400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0f259c640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0e00b6510 0x7fd0e00b8a20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4031040 tx=0x7fd0e40c2e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 -- 172.21.15.29:0/2787881202 shutdown_connections 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 -- 172.21.15.29:0/2787881202 wait complete. 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 -- start start 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.784+0000 7fd0ff56b740 1 Processor -- start 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 0x55a64025ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a64025f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a64026ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a640155300 con 0x55a63fead520 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a6401be430 con 0x55a63fe8c840 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a64001bbb0 con 0x55a63fe84660 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0ff56b740 10 client.0 _ll_register_callbacks cb 0x55a64016f120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a64026ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 0x55a64025ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 0x55a64025ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54116/0 (socket says 172.21.15.29:54116) 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/3413316290 learned_addr learned my addr 172.21.15.29:0/3413316290 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a64025f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/3413316290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 msgr2=0x55a64025ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 0x55a64025ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/3413316290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 msgr2=0x55a64025f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a64025f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.785+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/3413316290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a640144780 con 0x55a63fead520 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a64026ae90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec000c00 tx=0x7fd0ec011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340a6686dd7970b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/3413316290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec024040 con 0x55a63fead520 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/3413316290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd0ec02b630 con 0x55a63fead520 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0ff56b740 1 -- 172.21.15.29:0/3413316290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0ec0043d0 con 0x55a63fead520 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/3413316290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd0ec033050 con 0x55a63fead520 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0ff56b740 10 client.17931 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0ff56b740 1 -- 172.21.15.29:0/3413316290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a63ff18b00 con 0x55a63fead520 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/3413316290 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0ec0043d0 con 0x55a63fead520 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.786+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/3413316290 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fd0ec02d900 con 0x55a63fead520 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0ff56b740 10 client.17931 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0ff56b740 10 client.17931 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0ff56b740 20 client.17931 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a63fe6bea0 0x55a640270d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0ff56b740 4 client.17931 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:57:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0ff56b740 1 -- 172.21.15.29:0/3413316290 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6401cd140 con 0x55a63fe6bea0 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a63fe6bea0 0x55a640270d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0d9ffb640 10 client.17931 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.787+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a63fe6bea0 0x55a640270d90 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a31ee392f564b601 server_cookie=54a8494696259f50 in_seq=0 out_seq=0 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/3413316290 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a6401cd140 con 0x55a63fe6bea0 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0d9ffb640 10 client.17931 handle_command_reply: tid=0 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 client.17931 shutdown 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 -- 172.21.15.29:0/3413316290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 msgr2=0x55a64026ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a64026ae90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec000c00 tx=0x7fd0ec011f60 comp rx=0 tx=0).stop 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 -- 172.21.15.29:0/3413316290 shutdown_connections 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a63fe6bea0 0x55a640270d90 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fead520 0x55a64026ae90 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0d9ffb640 0 client.17931 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe8c840 0x55a64025f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/3413316290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe84660 0x55a64025ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 -- 172.21.15.29:0/3413316290 >> 172.21.15.29:0/3413316290 conn(0x55a63fea1910 msgr2=0x55a6402566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 -- 172.21.15.29:0/3413316290 shutdown_connections 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 1 -- 172.21.15.29:0/3413316290 wait complete. 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.788+0000 7fd0ff56b740 20 client.17931 trim_cache size 0 max 16384 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0ff56b740 1 Processor -- start 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0ff56b740 1 -- start start 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe6bea0 0x55a640254bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a64026ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe8c840 0x55a640260ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f41b2c10 con 0x55a63fe8c840 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f41b4750 con 0x55a63fe84660 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f41b32e0 con 0x55a63fe6bea0 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe8c840 0x55a640260ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe8c840 0x55a640260ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57680/0 (socket says 172.21.15.29:57680) 2024-09-21T02:57:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/78563936 learned_addr learned my addr 172.21.15.29:0/78563936 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.806+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe6bea0 0x55a640254bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a64026ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/78563936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe6bea0 msgr2=0x55a640254bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe6bea0 0x55a640254bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/78563936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 msgr2=0x55a64026ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a64026ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/78563936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a63ff8f960 con 0x55a63fe8c840 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe8c840 0x55a640260ad0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec02de70 tx=0x7fd0ec005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb2cbcc1dfc6602 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0dbfff640 1 -- 172.21.15.29:0/78563936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec01bb20 con 0x55a63fe8c840 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0dbfff640 1 -- 172.21.15.29:0/78563936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0ec01bcc0 con 0x55a63fe8c840 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.807+0000 7fd0dbfff640 1 -- 172.21.15.29:0/78563936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec0035f0 con 0x55a63fe8c840 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 -- 172.21.15.29:0/78563936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe8c840 msgr2=0x55a640260ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe8c840 0x55a640260ad0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec02de70 tx=0x7fd0ec005cc0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 -- 172.21.15.29:0/78563936 shutdown_connections 2024-09-21T02:57:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe8c840 0x55a640260ad0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a64026ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/78563936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe6bea0 0x55a640254bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 -- 172.21.15.29:0/78563936 >> 172.21.15.29:0/78563936 conn(0x55a63fea1910 msgr2=0x55a64025d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 -- 172.21.15.29:0/78563936 shutdown_connections 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.808+0000 7fd0ff56b740 1 -- 172.21.15.29:0/78563936 wait complete. 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 1 -- start start 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 1 Processor -- start 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe6bea0 0x55a6402635b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a640263af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe8c840 0x55a6402da5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f41b4080 con 0x55a63fe6bea0 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f41b39b0 con 0x55a63fe84660 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f41b0bf0 con 0x55a63fe8c840 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0ff56b740 10 client.0 _ll_register_callbacks cb 0x7fd0dc009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe6bea0 0x55a6402635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0f2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe6bea0 0x55a6402635b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57688/0 (socket says 172.21.15.29:57688) 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0f2d9d640 1 -- 172.21.15.29:0/1423737266 learned_addr learned my addr 172.21.15.29:0/1423737266 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.809+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a640263af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe8c840 0x55a6402da5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0f1d9b640 1 -- 172.21.15.29:0/1423737266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe8c840 msgr2=0x55a6402da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe8c840 0x55a6402da5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0f1d9b640 1 -- 172.21.15.29:0/1423737266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe6bea0 msgr2=0x55a6402635b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe6bea0 0x55a6402635b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0f1d9b640 1 -- 172.21.15.29:0/1423737266 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6402dab00 con 0x55a63fe84660 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe6bea0 0x55a6402635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0f1d9b640 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a640263af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55a640271330 tx=0x7fd0dc0044d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce01f3478c114cf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/1423737266 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc012d60 con 0x55a63fe84660 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0ff56b740 1 -- 172.21.15.29:0/1423737266 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0ec01bcc0 con 0x55a63fe84660 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/1423737266 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd0dc005020 con 0x55a63fe84660 2024-09-21T02:57:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/1423737266 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd0dc031050 con 0x55a63fe84660 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0ff56b740 10 client.27281 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.810+0000 7fd0ff56b740 1 -- 172.21.15.29:0/1423737266 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a6402dbe20 con 0x55a63fe84660 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/1423737266 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0ec01bcc0 con 0x55a63fe84660 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/1423737266 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fd0dc023070 con 0x55a63fe84660 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0ff56b740 10 client.27281 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0ff56b740 10 client.27281 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0ff56b740 20 client.27281 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a63fead520 0x55a6402de550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0ff56b740 4 client.27281 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "28a84c4a", "release": true, "await": true}] 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0ff56b740 1 -- 172.21.15.29:0/1423737266 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "28a84c4a", "release": true, "await": true}) -- 0x55a6401c8bf0 con 0x55a63fead520 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.811+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a63fead520 0x55a6402de550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.812+0000 7fd0f2d9d640 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a63fead520 0x55a6402de550 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a527d10b1d7e482 server_cookie=e6ae064ebb8bbcf2 in_seq=0 out_seq=0 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.812+0000 7fd0d9ffb640 10 client.27281 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0d9ffb640 1 -- 172.21.15.29:0/1423737266 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a6401c8bf0 con 0x55a63fead520 2024-09-21T02:57:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0d9ffb640 10 client.27281 handle_command_reply: tid=0 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 client.27281 shutdown 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 -- 172.21.15.29:0/1423737266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 msgr2=0x55a640263af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a640263af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55a640271330 tx=0x7fd0dc0044d0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 -- 172.21.15.29:0/1423737266 shutdown_connections 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a63fe8c840 0x55a6402da5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a63fe84660 0x55a640263af0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a63fead520 0x55a6402de550 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 --2- 172.21.15.29:0/1423737266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a63fe6bea0 0x55a6402635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 -- 172.21.15.29:0/1423737266 >> 172.21.15.29:0/1423737266 conn(0x55a63fea1910 msgr2=0x55a64025ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0d9ffb640 0 client.27281 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 -- 172.21.15.29:0/1423737266 shutdown_connections 2024-09-21T02:57:41.896 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 72, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "28a84c4a": { 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 72, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 130.2, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:57:41.897 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.874+0000 7fd0ff56b740 1 -- 172.21.15.29:0/1423737266 wait complete. 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.875+0000 7fd0ff56b740 20 client.27281 trim_cache size 0 max 16384 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0e00b6510 msgr2=0x7fd0e00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0e00b6510 0x7fd0e00b8a20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4031040 tx=0x7fd0e40c2e80 comp rx=0 tx=0).stop 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 msgr2=0x7fd0f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 0x7fd0f4075710 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009dc0 tx=0x7fd0e4003f80 comp rx=0 tx=0).stop 2024-09-21T02:57:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1156715146 shutdown_connections 2024-09-21T02:57:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd0e00b6510 0x7fd0e00b8a20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0f4109e20 0x7fd0f4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f41094d0 0x7fd0f4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 --2- 172.21.15.29:0/1156715146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f4103b90 0x7fd0f4075710 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1156715146 >> 172.21.15.29:0/1156715146 conn(0x7fd0f40ff600 msgr2=0x7fd0f4101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1156715146 shutdown_connections 2024-09-21T02:57:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:57:41.876+0000 7fd0f8ff7640 1 -- 172.21.15.29:0/1156715146 wait complete. 2024-09-21T02:57:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:41 smithi066 ceph-mon[33436]: pgmap v946: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.3 MiB/s wr, 190 op/s 2024-09-21T02:57:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:41 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:57:42.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:41 smithi190 ceph-mon[33871]: pgmap v946: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.3 MiB/s wr, 190 op/s 2024-09-21T02:57:42.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:41 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:57:42.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:41 smithi029 ceph-mon[31006]: pgmap v946: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.3 MiB/s wr, 190 op/s 2024-09-21T02:57:42.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:41 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:57:42.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34387 5.83 MB/sec execute 199 sec latency 10892.217 ms 2024-09-21T02:57:42.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:57:42.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:57:42.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:57:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:42.999+0000 7ff7907b0640 1 -- 172.21.15.29:0/3074635871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7881068e0 msgr2=0x7ff788108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:42.999+0000 7ff7907b0640 1 --2- 172.21.15.29:0/3074635871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7881068e0 0x7ff788108d00 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7ff778009900 tx=0x7ff77802a410 comp rx=0 tx=0).stop 2024-09-21T02:57:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 -- 172.21.15.29:0/3074635871 shutdown_connections 2024-09-21T02:57:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 --2- 172.21.15.29:0/3074635871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7881093f0 0x7ff78810b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 --2- 172.21.15.29:0/3074635871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7881068e0 0x7ff788108d00 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 --2- 172.21.15.29:0/3074635871 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff78806b650 0x7ff7881063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 -- 172.21.15.29:0/3074635871 >> 172.21.15.29:0/3074635871 conn(0x7ff7880fdee0 msgr2=0x7ff788100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 -- 172.21.15.29:0/3074635871 shutdown_connections 2024-09-21T02:57:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 -- 172.21.15.29:0/3074635871 wait complete. 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 Processor -- start 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.000+0000 7ff7907b0640 1 -- start start 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff7907b0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff78806b650 0x7ff788199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff7907b0640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7881068e0 0x7ff78819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff7907b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7881093f0 0x7ff78819b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff7907b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff78810f190 con 0x7ff7881093f0 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff7907b0640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff78810f010 con 0x7ff7881068e0 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff7907b0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff78810f310 con 0x7ff78806b650 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff78806b650 0x7ff788199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff78806b650 0x7ff788199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54146/0 (socket says 172.21.15.29:54146) 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78ed4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7881093f0 0x7ff78819b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78e54c640 1 -- 172.21.15.29:0/4116598021 learned_addr learned my addr 172.21.15.29:0/4116598021 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7881068e0 0x7ff78819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78e54c640 1 -- 172.21.15.29:0/4116598021 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7881068e0 msgr2=0x7ff78819a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78e54c640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7881068e0 0x7ff78819a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78e54c640 1 -- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7881093f0 msgr2=0x7ff78819b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78e54c640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7881093f0 0x7ff78819b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.001+0000 7ff78e54c640 1 -- 172.21.15.29:0/4116598021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78819b8b0 con 0x7ff78806b650 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.002+0000 7ff78e54c640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff78806b650 0x7ff788199e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff78400ea60 tx=0x7ff78400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.002+0000 7ff77f7fe640 1 -- 172.21.15.29:0/4116598021 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78400cde0 con 0x7ff78806b650 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.002+0000 7ff77f7fe640 1 -- 172.21.15.29:0/4116598021 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7840035f0 con 0x7ff78806b650 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.002+0000 7ff7907b0640 1 -- 172.21.15.29:0/4116598021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78819bba0 con 0x7ff78806b650 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.002+0000 7ff77f7fe640 1 -- 172.21.15.29:0/4116598021 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784010690 con 0x7ff78806b650 2024-09-21T02:57:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.002+0000 7ff7907b0640 1 -- 172.21.15.29:0/4116598021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7881a2290 con 0x7ff78806b650 2024-09-21T02:57:43.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.005+0000 7ff77f7fe640 1 -- 172.21.15.29:0/4116598021 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff784004000 con 0x7ff78806b650 2024-09-21T02:57:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff7977f4740 1 Processor -- start 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff7977f4740 1 -- start start 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff77f7fe640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7740b6510 0x7ff7740b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f8fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bbf2f0 0x555e28f904c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7740b6510 0x7ff7740b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f91010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff7977f4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555e28f026f0 con 0x555e28bc74d0 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff7977f4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555e28ef1b20 con 0x555e28bbf2f0 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff77f7fe640 1 -- 172.21.15.29:0/4116598021 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff7840e79d0 con 0x7ff78806b650 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.006+0000 7ff7977f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555e28ef5930 con 0x555e28be81b0 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f8fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78ed4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f91010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f8fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54148/0 (socket says 172.21.15.29:54148) 2024-09-21T02:57:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 -- 172.21.15.29:0/328932029 learned_addr learned my addr 172.21.15.29:0/328932029 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bbf2f0 0x555e28f904c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 -- 172.21.15.29:0/328932029 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bbf2f0 msgr2=0x555e28f904c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bbf2f0 0x555e28f904c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 -- 172.21.15.29:0/328932029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 msgr2=0x555e28f91010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f91010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 -- 172.21.15.29:0/328932029 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e28eb7530 con 0x555e28be81b0 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bbf2f0 0x555e28f904c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7740b6510 0x7ff7740b8a20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff780013c10 tx=0x7ff780009db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff78e54c640 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f8fe50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff784000c00 tx=0x7ff7840ad130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f51afc63a907931 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.007+0000 7ff77cff9640 1 -- 172.21.15.29:0/328932029 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7840b2020 con 0x555e28be81b0 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.008+0000 7ff77cff9640 1 -- 172.21.15.29:0/328932029 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7840ad2f0 con 0x555e28be81b0 2024-09-21T02:57:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.008+0000 7ff77cff9640 1 -- 172.21.15.29:0/328932029 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7840a9a50 con 0x555e28be81b0 2024-09-21T02:57:43.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:57:43.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:57:43.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:57:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 -- 172.21.15.29:0/328932029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 msgr2=0x555e28f8fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f8fe50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff784000c00 tx=0x7ff7840ad130 comp rx=0 tx=0).stop 2024-09-21T02:57:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 -- 172.21.15.29:0/328932029 shutdown_connections 2024-09-21T02:57:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f91010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bbf2f0 0x555e28f904c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 --2- 172.21.15.29:0/328932029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f8fe50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 -- 172.21.15.29:0/328932029 >> 172.21.15.29:0/328932029 conn(0x555e28bdc300 msgr2=0x555e28f8b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 -- 172.21.15.29:0/328932029 shutdown_connections 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.009+0000 7ff7977f4740 1 -- 172.21.15.29:0/328932029 wait complete. 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.010+0000 7ff7977f4740 1 -- start start 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.010+0000 7ff7977f4740 1 Processor -- start 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.010+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28bbf2f0 0x555e28f995f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.010+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f99cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.010+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28be81b0 0x555e28f9a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.010+0000 7ff7977f4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555e28e29ab0 con 0x555e28bc74d0 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.010+0000 7ff7977f4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x555e28e97730 con 0x555e28be81b0 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff7977f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555e28f91b30 con 0x555e28bbf2f0 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28bbf2f0 0x555e28f995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28bbf2f0 0x555e28f995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54158/0 (socket says 172.21.15.29:54158) 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 -- 172.21.15.29:0/2521868446 learned_addr learned my addr 172.21.15.29:0/2521868446 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f99cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff7977f4740 10 client.0 _ll_register_callbacks cb 0x555e28eeb6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28be81b0 0x555e28f9a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 -- 172.21.15.29:0/2521868446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28be81b0 msgr2=0x555e28f9a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28be81b0 0x555e28f9a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 -- 172.21.15.29:0/2521868446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 msgr2=0x555e28f99cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f99cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 -- 172.21.15.29:0/2521868446 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e28c53790 con 0x555e28bbf2f0 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff7977f4740 1 -- 172.21.15.29:0/2521868446 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7840ad2f0 con 0x555e28bbf2f0 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28be81b0 0x555e28f9a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f99cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff78e54c640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28bbf2f0 0x555e28f995f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff78400ea30 tx=0x7ff784003cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=978fc2165fe20503 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.011+0000 7ff771ffb640 10 client.27433 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:57:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff771ffb640 1 -- 172.21.15.29:0/2521868446 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7840aed10 con 0x555e28bbf2f0 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff771ffb640 1 -- 172.21.15.29:0/2521868446 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff7840a9130 con 0x555e28bbf2f0 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff771ffb640 1 -- 172.21.15.29:0/2521868446 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff7840a9330 con 0x555e28bbf2f0 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff7977f4740 10 client.27433 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff7977f4740 1 -- 172.21.15.29:0/2521868446 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=37}) -- 0x555e28ccc620 con 0x555e28bbf2f0 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff771ffb640 1 -- 172.21.15.29:0/2521868446 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff7840ac4b0 con 0x555e28bbf2f0 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff7977f4740 10 client.27433 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff7977f4740 10 client.27433 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff7977f4740 20 client.27433 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555e28ba6d80 0x555e28fa6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff7977f4740 4 client.27433 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff7977f4740 1 -- 172.21.15.29:0/2521868446 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e28f04640 con 0x555e28ba6d80 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.012+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555e28ba6d80 0x555e28fa6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.013+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555e28ba6d80 0x555e28fa6800 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0ec0417ca53e78c server_cookie=cbdf4f54b4288d89 in_seq=0 out_seq=0 2024-09-21T02:57:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.013+0000 7ff771ffb640 10 client.27433 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff771ffb640 1 -- 172.21.15.29:0/2521868446 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555e28f04640 con 0x555e28ba6d80 2024-09-21T02:57:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff771ffb640 10 client.27433 handle_command_reply: tid=0 2024-09-21T02:57:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 client.27433 shutdown 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 -- 172.21.15.29:0/2521868446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28bbf2f0 msgr2=0x555e28f995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28bbf2f0 0x555e28f995f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff78400ea30 tx=0x7ff784003cf0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 -- 172.21.15.29:0/2521868446 shutdown_connections 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28be81b0 0x555e28f9a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555e28ba6d80 0x555e28fa6800 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bc74d0 0x555e28f99cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff771ffb640 0 client.27433 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2521868446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28bbf2f0 0x555e28f995f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 -- 172.21.15.29:0/2521868446 >> 172.21.15.29:0/2521868446 conn(0x555e28bdc300 msgr2=0x555e28f89ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 -- 172.21.15.29:0/2521868446 shutdown_connections 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 1 -- 172.21.15.29:0/2521868446 wait complete. 2024-09-21T02:57:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.014+0000 7ff7977f4740 20 client.27433 trim_cache size 0 max 16384 2024-09-21T02:57:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.040+0000 7ff7977f4740 1 Processor -- start 2024-09-21T02:57:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.040+0000 7ff7977f4740 1 -- start start 2024-09-21T02:57:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28f954a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e28f959e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f95f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff7977f4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff7881ac5a0 con 0x555e28bbf2f0 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff7977f4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff7881ae0e0 con 0x555e28bc74d0 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff7977f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff7881acc70 con 0x555e28be81b0 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28f954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28f954a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57756/0 (socket says 172.21.15.29:57756) 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78e54c640 1 -- 172.21.15.29:0/776657249 learned_addr learned my addr 172.21.15.29:0/776657249 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f95f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e28f959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78ed4d640 1 -- 172.21.15.29:0/776657249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 msgr2=0x555e28f959e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e28f959e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78ed4d640 1 -- 172.21.15.29:0/776657249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 msgr2=0x555e28f954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28f954a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78ed4d640 1 -- 172.21.15.29:0/776657249 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e28f91690 con 0x555e28be81b0 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78e54c640 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28f954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.041+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e28f959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.042+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f95f20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff78000fc20 tx=0x7ff7800094e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb7f4c12d6bd8a9a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.042+0000 7ff773fff640 1 -- 172.21.15.29:0/776657249 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780026070 con 0x555e28be81b0 2024-09-21T02:57:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.042+0000 7ff773fff640 1 -- 172.21.15.29:0/776657249 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff780013170 con 0x555e28be81b0 2024-09-21T02:57:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.042+0000 7ff773fff640 1 -- 172.21.15.29:0/776657249 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7800107b0 con 0x555e28be81b0 2024-09-21T02:57:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 -- 172.21.15.29:0/776657249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 msgr2=0x555e28f95f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f95f20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff78000fc20 tx=0x7ff7800094e0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 -- 172.21.15.29:0/776657249 shutdown_connections 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e28f95f20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e28f959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 --2- 172.21.15.29:0/776657249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28f954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 -- 172.21.15.29:0/776657249 >> 172.21.15.29:0/776657249 conn(0x555e28bdc300 msgr2=0x555e28f8ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 -- 172.21.15.29:0/776657249 shutdown_connections 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.043+0000 7ff7977f4740 1 -- 172.21.15.29:0/776657249 wait complete. 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 1 -- start start 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 1 Processor -- start 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28fa14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e2900cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e2900f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff7881ada10 con 0x555e28bbf2f0 2024-09-21T02:57:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff7881ad340 con 0x555e28bc74d0 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff7881aa580 con 0x555e28be81b0 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff7977f4740 10 client.0 _ll_register_callbacks cb 0x7ff780013760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28fa14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff78e54c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28fa14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57772/0 (socket says 172.21.15.29:57772) 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff78e54c640 1 -- 172.21.15.29:0/2373883441 learned_addr learned my addr 172.21.15.29:0/2373883441 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e2900f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.044+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e2900cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff78e54c640 1 -- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 msgr2=0x555e2900f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff78e54c640 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e2900f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff78e54c640 1 -- 172.21.15.29:0/2373883441 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 msgr2=0x555e2900cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff78e54c640 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e2900cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff78e54c640 1 -- 172.21.15.29:0/2373883441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e2900fe70 con 0x555e28bbf2f0 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff78ed4d640 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e2900f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff78e54c640 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28fa14a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff7840e51a0 tx=0x7ff784003a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af6791eab3ba656 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff771ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff771ffb640 1 -- 172.21.15.29:0/2373883441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784071070 con 0x555e28bbf2f0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff771ffb640 1 -- 172.21.15.29:0/2373883441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff7840b3450 con 0x555e28bbf2f0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff7977f4740 1 -- 172.21.15.29:0/2373883441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff780013170 con 0x555e28bbf2f0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff771ffb640 1 -- 172.21.15.29:0/2373883441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff7840adee0 con 0x555e28bbf2f0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff7977f4740 10 client.17955 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.045+0000 7ff7977f4740 1 -- 172.21.15.29:0/2373883441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555e2900fe70 con 0x555e28bbf2f0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff771ffb640 1 -- 172.21.15.29:0/2373883441 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff780013170 con 0x555e28bbf2f0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff771ffb640 1 -- 172.21.15.29:0/2373883441 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff7840b3650 con 0x555e28bbf2f0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff7977f4740 10 client.17955 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff7977f4740 10 client.17955 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff7977f4740 20 client.17955 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555e28b7c1c0 0x555e29013750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff7977f4740 4 client.17955 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff7977f4740 1 -- 172.21.15.29:0/2373883441 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555e28f000f0 con 0x555e28b7c1c0 2024-09-21T02:57:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.046+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555e28b7c1c0 0x555e29013750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.047+0000 7ff78dd4b640 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555e28b7c1c0 0x555e29013750 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a26864ca5dd4a45 server_cookie=7ca0ecef5fc4d3d0 in_seq=0 out_seq=0 2024-09-21T02:57:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.047+0000 7ff771ffb640 10 client.17955 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:43.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:57:43.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:57:43.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:57:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff771ffb640 1 -- 172.21.15.29:0/2373883441 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555e28f000f0 con 0x555e28b7c1c0 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff771ffb640 10 client.17955 handle_command_reply: tid=0 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 client.17955 shutdown 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 -- 172.21.15.29:0/2373883441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 msgr2=0x555e28fa14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28fa14a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff7840e51a0 tx=0x7ff784003a30 comp rx=0 tx=0).stop 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 -- 172.21.15.29:0/2373883441 shutdown_connections 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e28be81b0 0x555e2900f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x555e28bc74d0 0x555e2900cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x555e28b7c1c0 0x555e29013750 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 --2- 172.21.15.29:0/2373883441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555e28bbf2f0 0x555e28fa14a0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 -- 172.21.15.29:0/2373883441 >> 172.21.15.29:0/2373883441 conn(0x555e28bdc300 msgr2=0x555e28f8ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff771ffb640 0 client.17955 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 -- 172.21.15.29:0/2373883441 shutdown_connections 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 1 -- 172.21.15.29:0/2373883441 wait complete. 2024-09-21T02:57:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.157+0000 7ff7977f4740 20 client.17955 trim_cache size 0 max 16384 2024-09-21T02:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:57:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.159+0000 7ff7907b0640 1 -- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7740b6510 msgr2=0x7ff7740b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.159+0000 7ff7907b0640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7740b6510 0x7ff7740b8a20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff780013c10 tx=0x7ff780009db0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.159+0000 7ff7907b0640 1 -- 172.21.15.29:0/4116598021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff78806b650 msgr2=0x7ff788199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.159+0000 7ff7907b0640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff78806b650 0x7ff788199e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff78400ea60 tx=0x7ff78400ef20 comp rx=0 tx=0).stop 2024-09-21T02:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.159+0000 7ff7907b0640 1 -- 172.21.15.29:0/4116598021 shutdown_connections 2024-09-21T02:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.159+0000 7ff7907b0640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff7740b6510 0x7ff7740b8a20 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.159+0000 7ff7907b0640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7881093f0 0x7ff78819b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.159+0000 7ff7907b0640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7881068e0 0x7ff78819a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.160+0000 7ff7907b0640 1 --2- 172.21.15.29:0/4116598021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff78806b650 0x7ff788199e70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.160+0000 7ff7907b0640 1 -- 172.21.15.29:0/4116598021 >> 172.21.15.29:0/4116598021 conn(0x7ff7880fdee0 msgr2=0x7ff788104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.160+0000 7ff7907b0640 1 -- 172.21.15.29:0/4116598021 shutdown_connections 2024-09-21T02:57:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:43.160+0000 7ff7907b0640 1 -- 172.21.15.29:0/4116598021 wait complete. 2024-09-21T02:57:43.238 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '28a84c4a', total seconds elapsed: 49.2 2024-09-21T02:57:43.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34531 5.81 MB/sec execute 200 sec latency 132.774 ms 2024-09-21T02:57:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:43 smithi066 ceph-mon[33436]: pgmap v947: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 860 KiB/s wr, 150 op/s 2024-09-21T02:57:44.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:43 smithi190 ceph-mon[33871]: pgmap v947: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 860 KiB/s wr, 150 op/s 2024-09-21T02:57:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:43 smithi029 ceph-mon[31006]: pgmap v947: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 860 KiB/s wr, 150 op/s 2024-09-21T02:57:44.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34692 5.80 MB/sec execute 201 sec latency 72.963 ms 2024-09-21T02:57:45.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34845 5.80 MB/sec execute 202 sec latency 84.977 ms 2024-09-21T02:57:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:45 smithi066 ceph-mon[33436]: pgmap v948: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 599 KiB/s wr, 129 op/s 2024-09-21T02:57:46.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:45 smithi190 ceph-mon[33871]: pgmap v948: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 599 KiB/s wr, 129 op/s 2024-09-21T02:57:46.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:45 smithi029 ceph-mon[31006]: pgmap v948: 129 pgs: 129 active+clean; 276 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 599 KiB/s wr, 129 op/s 2024-09-21T02:57:46.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 35004 5.80 MB/sec execute 203 sec latency 77.467 ms 2024-09-21T02:57:47.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 35134 5.79 MB/sec execute 204 sec latency 392.673 ms 2024-09-21T02:57:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:47 smithi066 ceph-mon[33436]: pgmap v949: 129 pgs: 129 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 464 KiB/s wr, 97 op/s 2024-09-21T02:57:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:47 smithi190 ceph-mon[33871]: pgmap v949: 129 pgs: 129 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 464 KiB/s wr, 97 op/s 2024-09-21T02:57:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:47 smithi029 ceph-mon[31006]: pgmap v949: 129 pgs: 129 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 464 KiB/s wr, 97 op/s 2024-09-21T02:57:48.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 35271 5.80 MB/sec execute 205 sec latency 494.063 ms 2024-09-21T02:57:49.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 35445 5.83 MB/sec execute 206 sec latency 116.693 ms 2024-09-21T02:57:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:49 smithi066 ceph-mon[33436]: pgmap v950: 129 pgs: 129 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 708 KiB/s wr, 110 op/s 2024-09-21T02:57:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:49 smithi190 ceph-mon[33871]: pgmap v950: 129 pgs: 129 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 708 KiB/s wr, 110 op/s 2024-09-21T02:57:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:49 smithi029 ceph-mon[31006]: pgmap v950: 129 pgs: 129 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 708 KiB/s wr, 110 op/s 2024-09-21T02:57:50.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 35623 5.87 MB/sec execute 207 sec latency 193.381 ms 2024-09-21T02:57:51.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 35839 5.92 MB/sec execute 208 sec latency 98.817 ms 2024-09-21T02:57:52.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:51 smithi066 ceph-mon[33436]: pgmap v951: 129 pgs: 129 active+clean; 282 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-09-21T02:57:52.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:51 smithi190 ceph-mon[33871]: pgmap v951: 129 pgs: 129 active+clean; 282 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-09-21T02:57:52.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:51 smithi029 ceph-mon[31006]: pgmap v951: 129 pgs: 129 active+clean; 282 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-09-21T02:57:52.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36011 5.96 MB/sec execute 209 sec latency 102.442 ms 2024-09-21T02:57:53.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36137 5.97 MB/sec execute 210 sec latency 104.827 ms 2024-09-21T02:57:54.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:53 smithi066 ceph-mon[33436]: pgmap v952: 129 pgs: 129 active+clean; 287 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-09-21T02:57:54.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:53 smithi190 ceph-mon[33871]: pgmap v952: 129 pgs: 129 active+clean; 287 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-09-21T02:57:54.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:53 smithi029 ceph-mon[31006]: pgmap v952: 129 pgs: 129 active+clean; 287 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-09-21T02:57:54.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36273 5.98 MB/sec execute 211 sec latency 90.833 ms 2024-09-21T02:57:55.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36414 6.01 MB/sec execute 212 sec latency 384.979 ms 2024-09-21T02:57:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:55 smithi066 ceph-mon[33436]: pgmap v953: 129 pgs: 129 active+clean; 293 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-09-21T02:57:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:55 smithi190 ceph-mon[33871]: pgmap v953: 129 pgs: 129 active+clean; 293 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-09-21T02:57:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:55 smithi029 ceph-mon[31006]: pgmap v953: 129 pgs: 129 active+clean; 293 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-09-21T02:57:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 -- 172.21.15.29:0/2096816563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c1050e0 msgr2=0x7fb67c10ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2096816563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c1050e0 0x7fb67c10ba80 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb674009ca0 tx=0x7fb67402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 -- 172.21.15.29:0/2096816563 shutdown_connections 2024-09-21T02:57:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2096816563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c1050e0 0x7fb67c10ba80 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2096816563 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb67c104740 0x7fb67c104b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2096816563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb67c1036c0 0x7fb67c103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 -- 172.21.15.29:0/2096816563 >> 172.21.15.29:0/2096816563 conn(0x7fb67c0fefe0 msgr2=0x7fb67c1014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 -- 172.21.15.29:0/2096816563 shutdown_connections 2024-09-21T02:57:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.288+0000 7fb682e0e640 1 -- 172.21.15.29:0/2096816563 wait complete. 2024-09-21T02:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb682e0e640 1 Processor -- start 2024-09-21T02:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb682e0e640 1 -- start start 2024-09-21T02:57:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb682e0e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb67c1036c0 0x7fb67c076e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb682e0e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb67c104740 0x7fb67c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb682e0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c1050e0 0x7fb67c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb682e0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10f170 con 0x7fb67c1050e0 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb682e0e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10eff0 con 0x7fb67c1036c0 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb682e0e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10f2f0 con 0x7fb67c104740 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb680baa640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb67c1036c0 0x7fb67c076e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb67c104740 0x7fb67c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb680baa640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb67c1036c0 0x7fb67c076e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:60574/0 (socket says 172.21.15.29:60574) 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb680baa640 1 -- 172.21.15.29:0/2391102721 learned_addr learned my addr 172.21.15.29:0/2391102721 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.289+0000 7fb6813ab640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c1050e0 0x7fb67c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb67bfff640 1 -- 172.21.15.29:0/2391102721 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb67c1036c0 msgr2=0x7fb67c076e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb67c1036c0 0x7fb67c076e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb67bfff640 1 -- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c1050e0 msgr2=0x7fb67c075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c1050e0 0x7fb67c075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb67bfff640 1 -- 172.21.15.29:0/2391102721 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c076380 con 0x7fb67c104740 2024-09-21T02:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb67c104740 0x7fb67c075710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb66800e9f0 tx=0x7fb66800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb679ffb640 1 -- 172.21.15.29:0/2391102721 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66800cd80 con 0x7fb67c104740 2024-09-21T02:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb682e0e640 1 -- 172.21.15.29:0/2391102721 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c1a3ea0 con 0x7fb67c104740 2024-09-21T02:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb679ffb640 1 -- 172.21.15.29:0/2391102721 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6680035f0 con 0x7fb67c104740 2024-09-21T02:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb679ffb640 1 -- 172.21.15.29:0/2391102721 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668010670 con 0x7fb67c104740 2024-09-21T02:57:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.290+0000 7fb682e0e640 1 -- 172.21.15.29:0/2391102721 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c1a42b0 con 0x7fb67c104740 2024-09-21T02:57:56.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.294+0000 7fb689e58740 1 Processor -- start 2024-09-21T02:57:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.294+0000 7fb689e58740 1 -- start start 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb689e58740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb689e58740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb689e58740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb689e58740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56138e23d6f0 con 0x56138df024d0 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb689e58740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56138e22cb20 con 0x56138df231b0 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb689e58740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56138e230930 con 0x56138defa2f0 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb6813ab640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb679ffb640 1 -- 172.21.15.29:0/2391102721 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb668004000 con 0x7fb67c104740 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb6813ab640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:60582/0 (socket says 172.21.15.29:60582) 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb680baa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb6813ab640 1 -- 172.21.15.29:0/4109416490 learned_addr learned my addr 172.21.15.29:0/4109416490 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb67bfff640 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb6813ab640 1 -- 172.21.15.29:0/4109416490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 msgr2=0x56138e2cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb6813ab640 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.295+0000 7fb6813ab640 1 -- 172.21.15.29:0/4109416490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 msgr2=0x56138e2cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb6813ab640 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb6813ab640 1 -- 172.21.15.29:0/4109416490 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56138e1f2530 con 0x56138df231b0 2024-09-21T02:57:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb67bfff640 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:57:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb6813ab640 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2cae50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb674004ac0 tx=0x7fb674009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=519f365f10d9aec8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb679ffb640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb66c0b6590 0x7fb66c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb666ffd640 1 -- 172.21.15.29:0/4109416490 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674031a80 con 0x56138df231b0 2024-09-21T02:57:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb666ffd640 1 -- 172.21.15.29:0/4109416490 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb674036070 con 0x56138df231b0 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb666ffd640 1 -- 172.21.15.29:0/4109416490 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6740043a0 con 0x56138df231b0 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.296+0000 7fb680baa640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb66c0b6590 0x7fb66c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.297+0000 7fb679ffb640 1 -- 172.21.15.29:0/2391102721 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb6680e7900 con 0x7fb67c104740 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb680baa640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb66c0b6590 0x7fb66c0b8aa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb670009640 tx=0x7fb670005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 -- 172.21.15.29:0/4109416490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 msgr2=0x56138e2cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2cae50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb674004ac0 tx=0x7fb674009800 comp rx=0 tx=0).stop 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 -- 172.21.15.29:0/4109416490 shutdown_connections 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 --2- 172.21.15.29:0/4109416490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2cae50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 -- 172.21.15.29:0/4109416490 >> 172.21.15.29:0/4109416490 conn(0x56138df17300 msgr2=0x56138e2c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 -- 172.21.15.29:0/4109416490 shutdown_connections 2024-09-21T02:57:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.298+0000 7fb689e58740 1 -- 172.21.15.29:0/4109416490 wait complete. 2024-09-21T02:57:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.299+0000 7fb689e58740 1 -- start start 2024-09-21T02:57:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.299+0000 7fb689e58740 1 Processor -- start 2024-09-21T02:57:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.300+0000 7fb689e58740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138defa2f0 0x56138e2d44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.300+0000 7fb689e58740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138df024d0 0x56138e2d4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb689e58740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2d5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb689e58740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56138e164ab0 con 0x56138defa2f0 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb689e58740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56138e1d2730 con 0x56138df231b0 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb689e58740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56138e2cc950 con 0x56138df024d0 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb689e58740 10 client.0 _ll_register_callbacks cb 0x56138e2266b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138df024d0 0x56138e2d4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb680baa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138defa2f0 0x56138e2d44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb680baa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138defa2f0 0x56138e2d44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50292/0 (socket says 172.21.15.29:50292) 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138df024d0 0x56138e2d4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55164/0 (socket says 172.21.15.29:55164) 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb680baa640 1 -- 172.21.15.29:0/2013504843 learned_addr learned my addr 172.21.15.29:0/2013504843 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb67bfff640 1 -- 172.21.15.29:0/2013504843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 msgr2=0x56138e2d5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2d5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb67bfff640 1 -- 172.21.15.29:0/2013504843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138defa2f0 msgr2=0x56138e2d44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138defa2f0 0x56138e2d44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb67bfff640 1 -- 172.21.15.29:0/2013504843 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56138df8e790 con 0x56138df024d0 2024-09-21T02:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138df024d0 0x56138e2d4b90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb6680e50d0 tx=0x7fb6680ad5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4e5127ec2b1c094 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.301+0000 7fb65bfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.302+0000 7fb65bfff640 1 -- 172.21.15.29:0/2013504843 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680042f0 con 0x56138df024d0 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.302+0000 7fb65bfff640 1 -- 172.21.15.29:0/2013504843 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb6680a9980 con 0x56138df024d0 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.302+0000 7fb689e58740 1 -- 172.21.15.29:0/2013504843 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb674036070 con 0x56138df024d0 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.302+0000 7fb65bfff640 1 -- 172.21.15.29:0/2013504843 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb6680ad220 con 0x56138df024d0 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.302+0000 7fb689e58740 10 client.27451 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.302+0000 7fb689e58740 1 -- 172.21.15.29:0/2013504843 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56138e007620 con 0x56138df024d0 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.302+0000 7fb65bfff640 1 -- 172.21.15.29:0/2013504843 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb674036070 con 0x56138df024d0 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.303+0000 7fb65bfff640 1 -- 172.21.15.29:0/2013504843 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fb6680ac3e0 con 0x56138df024d0 2024-09-21T02:57:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.303+0000 7fb689e58740 10 client.27451 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.303+0000 7fb689e58740 10 client.27451 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.303+0000 7fb689e58740 20 client.27451 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.303+0000 7fb689e58740 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56138dee1d80 0x56138e2e1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.303+0000 7fb689e58740 4 client.27451 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.303+0000 7fb689e58740 1 -- 172.21.15.29:0/2013504843 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56138e23f640 con 0x56138dee1d80 2024-09-21T02:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.303+0000 7fb680baa640 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56138dee1d80 0x56138e2e1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.304+0000 7fb65bfff640 10 client.27451 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.304+0000 7fb680baa640 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56138dee1d80 0x56138e2e1c30 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0df15cf28a7ed0a server_cookie=2c4f9361876f01a in_seq=0 out_seq=0 2024-09-21T02:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.305+0000 7fb65bfff640 1 -- 172.21.15.29:0/2013504843 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56138e23f640 con 0x56138dee1d80 2024-09-21T02:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.305+0000 7fb65bfff640 10 client.27451 handle_command_reply: tid=0 2024-09-21T02:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.305+0000 7fb689e58740 1 client.27451 shutdown 2024-09-21T02:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.305+0000 7fb689e58740 1 -- 172.21.15.29:0/2013504843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138df024d0 msgr2=0x56138e2d4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.305+0000 7fb689e58740 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138df024d0 0x56138e2d4b90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb6680e50d0 tx=0x7fb6680ad5a0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 1 -- 172.21.15.29:0/2013504843 shutdown_connections 2024-09-21T02:57:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2d5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138df024d0 0x56138e2d4b90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56138dee1d80 0x56138e2e1c30 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 1 --2- 172.21.15.29:0/2013504843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138defa2f0 0x56138e2d44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 1 -- 172.21.15.29:0/2013504843 >> 172.21.15.29:0/2013504843 conn(0x56138df17300 msgr2=0x56138e2c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb65bfff640 0 client.27451 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 1 -- 172.21.15.29:0/2013504843 shutdown_connections 2024-09-21T02:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 1 -- 172.21.15.29:0/2013504843 wait complete. 2024-09-21T02:57:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.306+0000 7fb689e58740 20 client.27451 trim_cache size 0 max 16384 2024-09-21T02:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.329+0000 7fb689e58740 1 Processor -- start 2024-09-21T02:57:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.329+0000 7fb689e58740 1 -- start start 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb689e58740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2d5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb689e58740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb689e58740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2d36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb689e58740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1ae660 con 0x56138df024d0 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb689e58740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1b01a0 con 0x56138df231b0 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50296/0 (socket says 172.21.15.29:50296) 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb67bfff640 1 -- 172.21.15.29:0/124125167 learned_addr learned my addr 172.21.15.29:0/124125167 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb680baa640 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2d5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb689e58740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1aed30 con 0x56138defa2f0 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb6813ab640 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2d36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb67bfff640 1 -- 172.21.15.29:0/124125167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 msgr2=0x56138e2d5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb67bfff640 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2d5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb67bfff640 1 -- 172.21.15.29:0/124125167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 msgr2=0x56138e2d36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb67bfff640 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2d36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.330+0000 7fb67bfff640 1 -- 172.21.15.29:0/124125167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56138e2c6890 con 0x56138df024d0 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.331+0000 7fb67bfff640 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc570 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb6680a9ce0 tx=0x7fb6680a95f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27061c596881e68 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.331+0000 7fb6667fc640 1 -- 172.21.15.29:0/124125167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680042f0 con 0x56138df024d0 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.331+0000 7fb6667fc640 1 -- 172.21.15.29:0/124125167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6680ade10 con 0x56138df024d0 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.331+0000 7fb6667fc640 1 -- 172.21.15.29:0/124125167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680b3380 con 0x56138df024d0 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 -- 172.21.15.29:0/124125167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 msgr2=0x56138e2dc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc570 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb6680a9ce0 tx=0x7fb6680a95f0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 -- 172.21.15.29:0/124125167 shutdown_connections 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2d36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc570 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 --2- 172.21.15.29:0/124125167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2d5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 -- 172.21.15.29:0/124125167 >> 172.21.15.29:0/124125167 conn(0x56138df17300 msgr2=0x56138e2c15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 -- 172.21.15.29:0/124125167 shutdown_connections 2024-09-21T02:57:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 -- 172.21.15.29:0/124125167 wait complete. 2024-09-21T02:57:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 -- start start 2024-09-21T02:57:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.332+0000 7fb689e58740 1 Processor -- start 2024-09-21T02:57:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb689e58740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2dbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb689e58740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb689e58740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2dac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb689e58740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1afad0 con 0x56138df024d0 2024-09-21T02:57:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb689e58740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1af400 con 0x56138df231b0 2024-09-21T02:57:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb689e58740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1ac640 con 0x56138defa2f0 2024-09-21T02:57:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb689e58740 10 client.0 _ll_register_callbacks cb 0x7fb6680056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb680baa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2dbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50310/0 (socket says 172.21.15.29:50310) 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb67bfff640 1 -- 172.21.15.29:0/2202288932 learned_addr learned my addr 172.21.15.29:0/2202288932 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.333+0000 7fb6813ab640 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb680baa640 1 -- 172.21.15.29:0/2202288932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 msgr2=0x56138e2dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb680baa640 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2dac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb680baa640 1 -- 172.21.15.29:0/2202288932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 msgr2=0x56138e2dc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb680baa640 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb680baa640 1 -- 172.21.15.29:0/2202288932 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56138e2db3e0 con 0x56138defa2f0 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb680baa640 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2dbe30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x56138e2e2350 tx=0x7fb6700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a3e09c194ed9579 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb65bfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:57:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb65bfff640 1 -- 172.21.15.29:0/2202288932 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67001dd90 con 0x56138defa2f0 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb65bfff640 1 -- 172.21.15.29:0/2202288932 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb670018030 con 0x56138defa2f0 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb689e58740 1 -- 172.21.15.29:0/2202288932 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6680ade10 con 0x56138defa2f0 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.334+0000 7fb65bfff640 1 -- 172.21.15.29:0/2202288932 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb67003d050 con 0x56138defa2f0 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb689e58740 10 client.27457 fetch_fsmap learned FSMap version 37 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb689e58740 1 -- 172.21.15.29:0/2202288932 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56138e2da2f0 con 0x56138defa2f0 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb65bfff640 1 -- 172.21.15.29:0/2202288932 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6680ade10 con 0x56138defa2f0 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb65bfff640 1 -- 172.21.15.29:0/2202288932 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fb670004b10 con 0x56138defa2f0 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb689e58740 10 client.27457 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb689e58740 10 client.27457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb689e58740 20 client.27457 populate_metadata read hostname 'smithi029' 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb689e58740 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56138deb71c0 0x56138e34eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb689e58740 4 client.27457 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb689e58740 1 -- 172.21.15.29:0/2202288932 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56138e23b0f0 con 0x56138deb71c0 2024-09-21T02:57:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.335+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56138deb71c0 0x56138e34eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:57:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.336+0000 7fb65bfff640 10 client.27457 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.336+0000 7fb67bfff640 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56138deb71c0 0x56138e34eca0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4eead5a0fb347b16 server_cookie=c2eb5524c49ba27a in_seq=0 out_seq=0 2024-09-21T02:57:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.521+0000 7fb65bfff640 1 -- 172.21.15.29:0/2202288932 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56138e23b0f0 con 0x56138deb71c0 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.521+0000 7fb65bfff640 10 client.27457 handle_command_reply: tid=0 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.521+0000 7fb689e58740 1 client.27457 shutdown 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 -- 172.21.15.29:0/2202288932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 msgr2=0x56138e2dbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2dbe30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x56138e2e2350 tx=0x7fb6700046c0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 -- 172.21.15.29:0/2202288932 shutdown_connections 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56138df231b0 0x56138e2dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56138deb71c0 0x56138e34eca0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56138df024d0 0x56138e2dc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 --2- 172.21.15.29:0/2202288932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56138defa2f0 0x56138e2dbe30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb65bfff640 0 client.27457 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 -- 172.21.15.29:0/2202288932 >> 172.21.15.29:0/2202288932 conn(0x56138df17300 msgr2=0x56138e2c15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 -- 172.21.15.29:0/2202288932 shutdown_connections 2024-09-21T02:57:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 1 -- 172.21.15.29:0/2202288932 wait complete. 2024-09-21T02:57:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.522+0000 7fb689e58740 20 client.27457 trim_cache size 0 max 16384 2024-09-21T02:57:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:57:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:57:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:57:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 -- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb66c0b6590 msgr2=0x7fb66c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb66c0b6590 0x7fb66c0b8aa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb670009640 tx=0x7fb670005f90 comp rx=0 tx=0).stop 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 -- 172.21.15.29:0/2391102721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb67c104740 msgr2=0x7fb67c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb67c104740 0x7fb67c075710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb66800e9f0 tx=0x7fb66800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 -- 172.21.15.29:0/2391102721 shutdown_connections 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb66c0b6590 0x7fb66c0b8aa0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c1050e0 0x7fb67c075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb67c104740 0x7fb67c075710 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 --2- 172.21.15.29:0/2391102721 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb67c1036c0 0x7fb67c076e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 -- 172.21.15.29:0/2391102721 >> 172.21.15.29:0/2391102721 conn(0x7fb67c0fefe0 msgr2=0x7fb67c107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:57:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 -- 172.21.15.29:0/2391102721 shutdown_connections 2024-09-21T02:57:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:57:56.523+0000 7fb682e0e640 1 -- 172.21.15.29:0/2391102721 wait complete. 2024-09-21T02:57:56.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36533 6.01 MB/sec execute 213 sec latency 198.815 ms 2024-09-21T02:57:57.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36659 6.00 MB/sec execute 214 sec latency 100.095 ms 2024-09-21T02:57:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:57 smithi066 ceph-mon[33436]: pgmap v954: 129 pgs: 129 active+clean; 298 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-09-21T02:57:58.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:57 smithi190 ceph-mon[33871]: pgmap v954: 129 pgs: 129 active+clean; 298 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-09-21T02:57:58.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:57 smithi029 ceph-mon[31006]: pgmap v954: 129 pgs: 129 active+clean; 298 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-09-21T02:57:58.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36801 6.02 MB/sec execute 215 sec latency 88.686 ms 2024-09-21T02:57:59.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36935 6.04 MB/sec execute 216 sec latency 95.178 ms 2024-09-21T02:58:00.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:57:59 smithi066 ceph-mon[33436]: pgmap v955: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 4.0 MiB/s wr, 230 op/s 2024-09-21T02:58:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:57:59 smithi190 ceph-mon[33871]: pgmap v955: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 4.0 MiB/s wr, 230 op/s 2024-09-21T02:58:00.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:57:59 smithi029 ceph-mon[31006]: pgmap v955: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 4.0 MiB/s wr, 230 op/s 2024-09-21T02:58:00.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37064 6.03 MB/sec execute 217 sec latency 89.854 ms 2024-09-21T02:58:01.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37182 6.01 MB/sec execute 218 sec latency 206.342 ms 2024-09-21T02:58:02.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:01 smithi066 ceph-mon[33436]: pgmap v956: 129 pgs: 129 active+clean; 299 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 4.4 MiB/s wr, 262 op/s 2024-09-21T02:58:02.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:01 smithi190 ceph-mon[33871]: pgmap v956: 129 pgs: 129 active+clean; 299 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 4.4 MiB/s wr, 262 op/s 2024-09-21T02:58:02.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:01 smithi029 ceph-mon[31006]: pgmap v956: 129 pgs: 129 active+clean; 299 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 4.4 MiB/s wr, 262 op/s 2024-09-21T02:58:02.182 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.3) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:58:02.182 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.15 2024-09-21T02:58:02.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:58:02.601 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:58:02.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37327 6.01 MB/sec execute 219 sec latency 106.063 ms 2024-09-21T02:58:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.317+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3953862907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30bc1094e0 msgr2=0x7f30bc1098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.317+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3953862907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30bc1094e0 0x7f30bc1098f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009900 tx=0x7f30ac02a410 comp rx=0 tx=0).stop 2024-09-21T02:58:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.317+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3953862907 shutdown_connections 2024-09-21T02:58:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.317+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3953862907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30bc109e30 0x7f30bc110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.317+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3953862907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30bc1094e0 0x7f30bc1098f0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.317+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3953862907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30bc103ba0 0x7f30bc103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.317+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3953862907 >> 172.21.15.29:0/3953862907 conn(0x7f30bc0ff630 msgr2=0x7f30bc101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.317+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3953862907 shutdown_connections 2024-09-21T02:58:03.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.318+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3953862907 wait complete. 2024-09-21T02:58:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.319+0000 7f30c3d7a640 1 Processor -- start 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.319+0000 7f30c3d7a640 1 -- start start 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.319+0000 7f30c3d7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30bc103ba0 0x7f30bc077120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.319+0000 7f30c3d7a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30bc1094e0 0x7f30bc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.319+0000 7f30c3d7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30bc109e30 0x7f30bc075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.319+0000 7f30c1b16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30bc103ba0 0x7f30bc077120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1b16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30bc103ba0 0x7f30bc077120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55186/0 (socket says 172.21.15.29:55186) 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1b16640 1 -- 172.21.15.29:0/3155702408 learned_addr learned my addr 172.21.15.29:0/3155702408 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc1139a0 con 0x7f30bc109e30 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc113820 con 0x7f30bc1094e0 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc113b20 con 0x7f30bc103ba0 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1315640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30bc1094e0 0x7f30bc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c2317640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30bc109e30 0x7f30bc075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1b16640 1 -- 172.21.15.29:0/3155702408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30bc1094e0 msgr2=0x7f30bc075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1b16640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30bc1094e0 0x7f30bc075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1b16640 1 -- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30bc109e30 msgr2=0x7f30bc075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1b16640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30bc109e30 0x7f30bc075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1b16640 1 -- 172.21.15.29:0/3155702408 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30bc076470 con 0x7f30bc103ba0 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.320+0000 7f30c1b16640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30bc103ba0 0x7f30bc077120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f30b800ea60 tx=0x7f30b800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.321+0000 7f30b2ffd640 1 -- 172.21.15.29:0/3155702408 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b800cde0 con 0x7f30bc103ba0 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.321+0000 7f30b2ffd640 1 -- 172.21.15.29:0/3155702408 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30b80035f0 con 0x7f30bc103ba0 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.321+0000 7f30b2ffd640 1 -- 172.21.15.29:0/3155702408 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b8010690 con 0x7f30bc103ba0 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.321+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30bc1acdb0 con 0x7f30bc103ba0 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.321+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30bc1ad1b0 con 0x7f30bc103ba0 2024-09-21T02:58:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.322+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30bc103fb0 con 0x7f30bc103ba0 2024-09-21T02:58:03.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.324+0000 7f30b2ffd640 1 -- 172.21.15.29:0/3155702408 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f30b8004000 con 0x7f30bc103ba0 2024-09-21T02:58:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.325+0000 7f30b2ffd640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30a80b6590 0x7f30a80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.325+0000 7f30c1315640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30a80b6590 0x7f30a80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.325+0000 7f30b2ffd640 1 -- 172.21.15.29:0/3155702408 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f30b80e7c20 con 0x7f30bc103ba0 2024-09-21T02:58:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.326+0000 7f30c1315640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30a80b6590 0x7f30a80b8aa0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f30ac005940 tx=0x7f30ac0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.328+0000 7f30b2ffd640 1 -- 172.21.15.29:0/3155702408 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f30b80ae130 con 0x7f30bc103ba0 2024-09-21T02:58:03.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.707+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30bc02e6e0 con 0x7f30bc103ba0 2024-09-21T02:58:03.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.708+0000 7f30b2ffd640 1 -- 172.21.15.29:0/3155702408 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f30b80b2510 con 0x7f30bc103ba0 2024-09-21T02:58:03.709 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:58:03.710 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:58:03.710 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:58:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30a80b6590 msgr2=0x7f30a80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30a80b6590 0x7f30a80b8aa0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f30ac005940 tx=0x7f30ac0058d0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30bc103ba0 msgr2=0x7f30bc077120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30bc103ba0 0x7f30bc077120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f30b800ea60 tx=0x7f30b800ef20 comp rx=0 tx=0).stop 2024-09-21T02:58:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 shutdown_connections 2024-09-21T02:58:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f30a80b6590 0x7f30a80b8aa0 secure :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f30ac005940 tx=0x7f30ac0058d0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30bc109e30 0x7f30bc075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30bc1094e0 0x7f30bc075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 --2- 172.21.15.29:0/3155702408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30bc103ba0 0x7f30bc077120 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.712+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 >> 172.21.15.29:0/3155702408 conn(0x7f30bc0ff630 msgr2=0x7f30bc101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.713+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 shutdown_connections 2024-09-21T02:58:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:03.713+0000 7f30c3d7a640 1 -- 172.21.15.29:0/3155702408 wait complete. 2024-09-21T02:58:03.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37466 6.01 MB/sec execute 220 sec latency 139.229 ms 2024-09-21T02:58:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:03 smithi066 ceph-mon[33436]: pgmap v957: 129 pgs: 129 active+clean; 299 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 4.5 MiB/s wr, 252 op/s 2024-09-21T02:58:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:03 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3155702408' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:04.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:03 smithi190 ceph-mon[33871]: pgmap v957: 129 pgs: 129 active+clean; 299 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 4.5 MiB/s wr, 252 op/s 2024-09-21T02:58:04.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:03 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3155702408' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:04.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:03 smithi029 ceph-mon[31006]: pgmap v957: 129 pgs: 129 active+clean; 299 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 4.5 MiB/s wr, 252 op/s 2024-09-21T02:58:04.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:03 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3155702408' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:04.633 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 142.15 --await-for 45.0' 2024-09-21T02:58:04.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 142.15 --await-for 45.0 2024-09-21T02:58:04.729 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37592 6.00 MB/sec execute 221 sec latency 173.333 ms 2024-09-21T02:58:05.022 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:58:05.729 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37718 6.01 MB/sec execute 222 sec latency 170.278 ms 2024-09-21T02:58:06.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:05 smithi066 ceph-mon[33436]: pgmap v958: 129 pgs: 129 active+clean; 300 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-09-21T02:58:06.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:05 smithi190 ceph-mon[33871]: pgmap v958: 129 pgs: 129 active+clean; 300 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-09-21T02:58:06.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:05 smithi029 ceph-mon[31006]: pgmap v958: 129 pgs: 129 active+clean; 300 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-09-21T02:58:06.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.190+0000 7fd90c6b3640 1 -- 172.21.15.29:0/321007784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904103b90 msgr2=0x7fd904103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.190+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/321007784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904103b90 0x7fd904103fa0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fd900009900 tx=0x7fd90002a410 comp rx=0 tx=0).stop 2024-09-21T02:58:06.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.190+0000 7fd90c6b3640 1 -- 172.21.15.29:0/321007784 shutdown_connections 2024-09-21T02:58:06.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.190+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/321007784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd904109e20 0x7fd904110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.190+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/321007784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9041094d0 0x7fd9041098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.190+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/321007784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904103b90 0x7fd904103fa0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.190+0000 7fd90c6b3640 1 -- 172.21.15.29:0/321007784 >> 172.21.15.29:0/321007784 conn(0x7fd9040ff5e0 msgr2=0x7fd904101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.191+0000 7fd90c6b3640 1 -- 172.21.15.29:0/321007784 shutdown_connections 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.191+0000 7fd90c6b3640 1 -- 172.21.15.29:0/321007784 wait complete. 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.191+0000 7fd90c6b3640 1 Processor -- start 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90c6b3640 1 -- start start 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90c6b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd904103b90 0x7fd90419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90c6b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9041094d0 0x7fd90419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90c6b3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd904109e20 0x7fd9041a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90c6b3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd904113990 con 0x7fd9041094d0 2024-09-21T02:58:06.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd909c4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9041094d0 0x7fd90419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd909c4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9041094d0 0x7fd90419cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42514/0 (socket says 172.21.15.29:42514) 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd909c4e640 1 -- 172.21.15.29:0/2745358174 learned_addr learned my addr 172.21.15.29:0/2745358174 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90ac50640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd904109e20 0x7fd9041a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90a44f640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd904103b90 0x7fd90419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd904113810 con 0x7fd904109e20 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.192+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd904113b10 con 0x7fd904103b90 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.193+0000 7fd90a44f640 1 -- 172.21.15.29:0/2745358174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd904109e20 msgr2=0x7fd9041a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.193+0000 7fd90a44f640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd904109e20 0x7fd9041a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.193+0000 7fd90a44f640 1 -- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9041094d0 msgr2=0x7fd90419cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.193+0000 7fd90a44f640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9041094d0 0x7fd90419cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.193+0000 7fd90a44f640 1 -- 172.21.15.29:0/2745358174 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9041a24d0 con 0x7fd904103b90 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.193+0000 7fd90ac50640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd904109e20 0x7fd9041a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.193+0000 7fd90a44f640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd904103b90 0x7fd90419c560 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd900005940 tx=0x7fd900004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.193+0000 7fd8fb7fe640 1 -- 172.21.15.29:0/2745358174 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90003f070 con 0x7fd904103b90 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.194+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9041a2760 con 0x7fd904103b90 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.194+0000 7fd8fb7fe640 1 -- 172.21.15.29:0/2745358174 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd900003a80 con 0x7fd904103b90 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.194+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9041a8f20 con 0x7fd904103b90 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.194+0000 7fd8fb7fe640 1 -- 172.21.15.29:0/2745358174 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd900035070 con 0x7fd904103b90 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.197+0000 7fd912c27740 1 Processor -- start 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.197+0000 7fd912c27740 1 -- start start 2024-09-21T02:58:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd8fb7fe640 1 -- 172.21.15.29:0/2745358174 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd900043400 con 0x7fd904103b90 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd912c27740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a801e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd912c27740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a8023b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd912c27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8028f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd912c27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cb4a778bb0 con 0x55cb4a43ab70 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd912c27740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55cb4a767fe0 con 0x55cb4a432990 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd912c27740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cb4a76bdf0 con 0x55cb4a45b850 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd90ac50640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a801e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd90ac50640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a801e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:57436/0 (socket says 172.21.15.29:57436) 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd90ac50640 1 -- 172.21.15.29:0/3856299598 learned_addr learned my addr 172.21.15.29:0/3856299598 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd90ac50640 1 -- 172.21.15.29:0/3856299598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 msgr2=0x55cb4a8023b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:58:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd90ac50640 1 --2- 172.21.15.29:0/3856299598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a8023b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd90ac50640 1 -- 172.21.15.29:0/3856299598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 msgr2=0x55cb4a8028f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd90ac50640 1 --2- 172.21.15.29:0/3856299598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8028f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.198+0000 7fd90ac50640 1 -- 172.21.15.29:0/3856299598 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb4a4d5ba0 con 0x55cb4a45b850 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.199+0000 7fd90ac50640 1 --2- 172.21.15.29:0/3856299598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a801e70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc010750 tx=0x7fd8fc010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3b9e76e952e3d94 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.199+0000 7fd8f8ff9640 1 -- 172.21.15.29:0/3856299598 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc00ead0 con 0x55cb4a45b850 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.199+0000 7fd8f8ff9640 1 -- 172.21.15.29:0/3856299598 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8fc010e60 con 0x55cb4a45b850 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.199+0000 7fd8fb7fe640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd8f00b6590 0x7fd8f00b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.199+0000 7fd8f8ff9640 1 -- 172.21.15.29:0/3856299598 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc0113e0 con 0x55cb4a45b850 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.200+0000 7fd8fb7fe640 1 -- 172.21.15.29:0/2745358174 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd9001074f0 con 0x7fd904103b90 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.200+0000 7fd90a44f640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd8f00b6590 0x7fd8f00b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.200+0000 7fd912c27740 1 -- 172.21.15.29:0/3856299598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 msgr2=0x55cb4a801e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.200+0000 7fd912c27740 1 --2- 172.21.15.29:0/3856299598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a801e70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc010750 tx=0x7fd8fc010c10 comp rx=0 tx=0).stop 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.201+0000 7fd912c27740 1 -- 172.21.15.29:0/3856299598 shutdown_connections 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.201+0000 7fd912c27740 1 --2- 172.21.15.29:0/3856299598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.201+0000 7fd912c27740 1 --2- 172.21.15.29:0/3856299598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a8023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.201+0000 7fd912c27740 1 --2- 172.21.15.29:0/3856299598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a801e70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.201+0000 7fd912c27740 1 -- 172.21.15.29:0/3856299598 >> 172.21.15.29:0/3856299598 conn(0x55cb4a44fc40 msgr2=0x55cb4a7ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.201+0000 7fd90a44f640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd8f00b6590 0x7fd8f00b8aa0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd900104c70 tx=0x7fd900009560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.201+0000 7fd912c27740 1 -- 172.21.15.29:0/3856299598 shutdown_connections 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.201+0000 7fd912c27740 1 -- 172.21.15.29:0/3856299598 wait complete. 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.202+0000 7fd912c27740 1 -- start start 2024-09-21T02:58:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.202+0000 7fd912c27740 1 Processor -- start 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.202+0000 7fd912c27740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a80dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.202+0000 7fd912c27740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a43ab70 0x55cb4a80e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.202+0000 7fd912c27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a45b850 0x55cb4a8199d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.202+0000 7fd912c27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cb4a5c97b0 con 0x55cb4a45b850 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.202+0000 7fd912c27740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55cb4a702e90 con 0x55cb4a432990 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.202+0000 7fd912c27740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cb4a810650 con 0x55cb4a43ab70 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd912c27740 10 client.0 _ll_register_callbacks cb 0x55cb4a765710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd909c4e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a43ab70 0x55cb4a80e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd909c4e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a43ab70 0x55cb4a80e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:57450/0 (socket says 172.21.15.29:57450) 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd909c4e640 1 -- 172.21.15.29:0/706629027 learned_addr learned my addr 172.21.15.29:0/706629027 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd90ac50640 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a45b850 0x55cb4a8199d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd909c4e640 1 -- 172.21.15.29:0/706629027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 msgr2=0x55cb4a80dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd909c4e640 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a80dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd909c4e640 1 -- 172.21.15.29:0/706629027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a45b850 msgr2=0x55cb4a8199d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd909c4e640 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a45b850 0x55cb4a8199d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd909c4e640 1 -- 172.21.15.29:0/706629027 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb4a819fe0 con 0x55cb4a43ab70 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.203+0000 7fd90ac50640 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a45b850 0x55cb4a8199d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd909c4e640 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a43ab70 0x55cb4a80e050 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd8f40116f0 tx=0x7fd8f4011bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97bc87c5d5fbf86c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd8edffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:58:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd8edffb640 1 -- 172.21.15.29:0/706629027 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f401e060 con 0x55cb4a43ab70 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd8edffb640 1 -- 172.21.15.29:0/706629027 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd8f400fdf0 con 0x55cb4a43ab70 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd912c27740 1 -- 172.21.15.29:0/706629027 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8fc010e60 con 0x55cb4a43ab70 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd8edffb640 1 -- 172.21.15.29:0/706629027 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd8f4025050 con 0x55cb4a43ab70 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd912c27740 10 client.27475 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd912c27740 1 -- 172.21.15.29:0/706629027 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cb4a81c9a0 con 0x55cb4a43ab70 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.204+0000 7fd8edffb640 1 -- 172.21.15.29:0/706629027 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8fc010e60 con 0x55cb4a43ab70 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.205+0000 7fd8edffb640 1 -- 172.21.15.29:0/706629027 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fd8f4011ce0 con 0x55cb4a43ab70 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.205+0000 7fd912c27740 10 client.27475 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.205+0000 7fd912c27740 10 client.27475 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.205+0000 7fd912c27740 20 client.27475 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.205+0000 7fd912c27740 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cb4a41a1d0 0x55cb4a820440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.205+0000 7fd912c27740 4 client.27475 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.205+0000 7fd912c27740 1 -- 172.21.15.29:0/706629027 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb4a77ab00 con 0x55cb4a41a1d0 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.205+0000 7fd90ac50640 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cb4a41a1d0 0x55cb4a820440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.206+0000 7fd8edffb640 10 client.27475 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.206+0000 7fd90ac50640 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cb4a41a1d0 0x55cb4a820440 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec2c4bccef9b94a6 server_cookie=daa5c5b188186a93 in_seq=0 out_seq=0 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.206+0000 7fd8edffb640 1 -- 172.21.15.29:0/706629027 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cb4a77ab00 con 0x55cb4a41a1d0 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.206+0000 7fd8edffb640 10 client.27475 handle_command_reply: tid=0 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.206+0000 7fd912c27740 1 client.27475 shutdown 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 -- 172.21.15.29:0/706629027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a43ab70 msgr2=0x55cb4a80e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a43ab70 0x55cb4a80e050 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd8f40116f0 tx=0x7fd8f4011bb0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 -- 172.21.15.29:0/706629027 shutdown_connections 2024-09-21T02:58:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cb4a41a1d0 0x55cb4a820440 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a45b850 0x55cb4a8199d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a43ab70 0x55cb4a80e050 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 --2- 172.21.15.29:0/706629027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a80dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd8edffb640 0 client.27475 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 -- 172.21.15.29:0/706629027 >> 172.21.15.29:0/706629027 conn(0x55cb4a44fc40 msgr2=0x55cb4a805000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 -- 172.21.15.29:0/706629027 shutdown_connections 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 1 -- 172.21.15.29:0/706629027 wait complete. 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.207+0000 7fd912c27740 20 client.27475 trim_cache size 0 max 16384 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.232+0000 7fd912c27740 1 Processor -- start 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.232+0000 7fd912c27740 1 -- start start 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd912c27740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a8070d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd912c27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a807610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd912c27740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a804f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd912c27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e800fb40 con 0x55cb4a43ab70 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd912c27740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e8011680 con 0x55cb4a432990 2024-09-21T02:58:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd912c27740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e8010210 con 0x55cb4a45b850 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a807610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd90ac50640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a804f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a807610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42532/0 (socket says 172.21.15.29:42532) 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 -- 172.21.15.29:0/86378934 learned_addr learned my addr 172.21.15.29:0/86378934 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd90ac50640 1 --2- 172.21.15.29:0/86378934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a8070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 -- 172.21.15.29:0/86378934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 msgr2=0x55cb4a804f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 --2- 172.21.15.29:0/86378934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a804f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 -- 172.21.15.29:0/86378934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 msgr2=0x55cb4a8070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 --2- 172.21.15.29:0/86378934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a8070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 -- 172.21.15.29:0/86378934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb4a804160 con 0x55cb4a43ab70 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.233+0000 7fd909c4e640 1 --2- 172.21.15.29:0/86378934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a807610 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4010010 tx=0x7fd8f400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d167dcaaadfffa4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.234+0000 7fd8effff640 1 -- 172.21.15.29:0/86378934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400fdf0 con 0x55cb4a43ab70 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.234+0000 7fd8effff640 1 -- 172.21.15.29:0/86378934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8f402e070 con 0x55cb4a43ab70 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.234+0000 7fd8effff640 1 -- 172.21.15.29:0/86378934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40298c0 con 0x55cb4a43ab70 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 -- 172.21.15.29:0/86378934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 msgr2=0x55cb4a807610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 --2- 172.21.15.29:0/86378934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a807610 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4010010 tx=0x7fd8f400f790 comp rx=0 tx=0).stop 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 -- 172.21.15.29:0/86378934 shutdown_connections 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 --2- 172.21.15.29:0/86378934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a45b850 0x55cb4a804f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 --2- 172.21.15.29:0/86378934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a807610 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 --2- 172.21.15.29:0/86378934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a432990 0x55cb4a8070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 -- 172.21.15.29:0/86378934 >> 172.21.15.29:0/86378934 conn(0x55cb4a44fc40 msgr2=0x55cb4a808a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 -- 172.21.15.29:0/86378934 shutdown_connections 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 -- 172.21.15.29:0/86378934 wait complete. 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 -- start start 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.235+0000 7fd912c27740 1 Processor -- start 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd912c27740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a432990 0x55cb4a811b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd912c27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8120b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd912c27740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a45b850 0x55cb4a889e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd912c27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e8010fb0 con 0x55cb4a43ab70 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd912c27740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e80108e0 con 0x55cb4a45b850 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd912c27740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e800db20 con 0x55cb4a432990 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd912c27740 10 client.0 _ll_register_callbacks cb 0x7fd8f4009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8120b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42540/0 (socket says 172.21.15.29:42540) 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 -- 172.21.15.29:0/1551130835 learned_addr learned my addr 172.21.15.29:0/1551130835 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd90ac50640 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a432990 0x55cb4a811b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd90ac50640 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a45b850 0x55cb4a889e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 -- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a432990 msgr2=0x55cb4a811b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a432990 0x55cb4a811b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 -- 172.21.15.29:0/1551130835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a45b850 msgr2=0x55cb4a889e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a45b850 0x55cb4a889e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 -- 172.21.15.29:0/1551130835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb4a8128a0 con 0x55cb4a43ab70 2024-09-21T02:58:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd90ac50640 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a432990 0x55cb4a811b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd909c4e640 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8120b0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400e620 tx=0x7fd8f400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb167ee7177789a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd8edffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd8edffb640 1 -- 172.21.15.29:0/1551130835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400e920 con 0x55cb4a43ab70 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.236+0000 7fd912c27740 1 -- 172.21.15.29:0/1551130835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8f402e070 con 0x55cb4a43ab70 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd8edffb640 1 -- 172.21.15.29:0/1551130835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd8f4005110 con 0x55cb4a43ab70 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd8edffb640 1 -- 172.21.15.29:0/1551130835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd8f4005370 con 0x55cb4a43ab70 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd912c27740 10 client.17991 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd912c27740 1 -- 172.21.15.29:0/1551130835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cb4a88b310 con 0x55cb4a43ab70 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd8edffb640 1 -- 172.21.15.29:0/1551130835 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8f402e070 con 0x55cb4a43ab70 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd8edffb640 1 -- 172.21.15.29:0/1551130835 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fd8f403b030 con 0x55cb4a43ab70 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd912c27740 10 client.17991 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd912c27740 10 client.17991 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:58:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd912c27740 20 client.17991 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd912c27740 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cb4a3ef580 0x55cb4a88da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd912c27740 4 client.17991 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.15, "await_for": 45.0}] 2024-09-21T02:58:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd912c27740 1 -- 172.21.15.29:0/1551130835 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.15, "await_for": 45.0}) -- 0x55cb4a7765b0 con 0x55cb4a3ef580 2024-09-21T02:58:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.237+0000 7fd90ac50640 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cb4a3ef580 0x55cb4a88da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.238+0000 7fd8edffb640 10 client.17991 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.238+0000 7fd90ac50640 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cb4a3ef580 0x55cb4a88da70 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41c173d38031ffaf server_cookie=e206ca37cf749486 in_seq=0 out_seq=0 2024-09-21T02:58:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.719+0000 7fd8edffb640 1 -- 172.21.15.29:0/1551130835 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cb4a7765b0 con 0x55cb4a3ef580 2024-09-21T02:58:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.719+0000 7fd8edffb640 10 client.17991 handle_command_reply: tid=0 2024-09-21T02:58:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.719+0000 7fd912c27740 1 client.17991 shutdown 2024-09-21T02:58:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.719+0000 7fd912c27740 1 -- 172.21.15.29:0/1551130835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 msgr2=0x55cb4a8120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.719+0000 7fd912c27740 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8120b0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400e620 tx=0x7fd8f400ef30 comp rx=0 tx=0).stop 2024-09-21T02:58:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 1 -- 172.21.15.29:0/1551130835 shutdown_connections 2024-09-21T02:58:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cb4a45b850 0x55cb4a889e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cb4a3ef580 0x55cb4a88da70 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb4a43ab70 0x55cb4a8120b0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 1 --2- 172.21.15.29:0/1551130835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb4a432990 0x55cb4a811b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd8edffb640 0 client.17991 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 1 -- 172.21.15.29:0/1551130835 >> 172.21.15.29:0/1551130835 conn(0x55cb4a44fc40 msgr2=0x55cb4a808a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 1 -- 172.21.15.29:0/1551130835 shutdown_connections 2024-09-21T02:58:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 1 -- 172.21.15.29:0/1551130835 wait complete. 2024-09-21T02:58:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.720+0000 7fd912c27740 20 client.17991 trim_cache size 0 max 16384 2024-09-21T02:58:06.723 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 74, 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "7776ac5f": { 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 74, 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:58:06.724 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.2, 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:06.725 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:06.726 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:06.726 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:06.726 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:58:06.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd8f00b6590 msgr2=0x7fd8f00b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd8f00b6590 0x7fd8f00b8aa0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd900104c70 tx=0x7fd900009560 comp rx=0 tx=0).stop 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd904103b90 msgr2=0x7fd90419c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd904103b90 0x7fd90419c560 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd900005940 tx=0x7fd900004040 comp rx=0 tx=0).stop 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 shutdown_connections 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd8f00b6590 0x7fd8f00b8aa0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd904109e20 0x7fd9041a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9041094d0 0x7fd90419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 --2- 172.21.15.29:0/2745358174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd904103b90 0x7fd90419c560 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 >> 172.21.15.29:0/2745358174 conn(0x7fd9040ff5e0 msgr2=0x7fd90410e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 shutdown_connections 2024-09-21T02:58:06.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:06.721+0000 7fd90c6b3640 1 -- 172.21.15.29:0/2745358174 wait complete. 2024-09-21T02:58:06.728 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37797 6.01 MB/sec execute 223 sec latency 523.066 ms 2024-09-21T02:58:07.442 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7776ac5f', quiesce took 5.3 seconds. Will release after: 16.849999999999998 2024-09-21T02:58:07.729 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.98 MB/sec execute 224 sec latency 1497.560 ms 2024-09-21T02:58:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:07 smithi066 ceph-mon[33436]: pgmap v959: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 3.6 MiB/s wr, 276 op/s 2024-09-21T02:58:08.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:07 smithi190 ceph-mon[33871]: pgmap v959: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 3.6 MiB/s wr, 276 op/s 2024-09-21T02:58:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:07 smithi029 ceph-mon[31006]: pgmap v959: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 3.6 MiB/s wr, 276 op/s 2024-09-21T02:58:08.729 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.96 MB/sec execute 225 sec latency 2497.663 ms 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 -- 172.21.15.29:0/1514958824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103fe0 msgr2=0x7f98ac10a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 --2- 172.21.15.29:0/1514958824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103fe0 0x7f98ac10a980 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009ca0 tx=0x7f98a402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 -- 172.21.15.29:0/1514958824 shutdown_connections 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 --2- 172.21.15.29:0/1514958824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103fe0 0x7f98ac10a980 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 --2- 172.21.15.29:0/1514958824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98ac103640 0x7f98ac103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 --2- 172.21.15.29:0/1514958824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98ac1025c0 0x7f98ac102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 -- 172.21.15.29:0/1514958824 >> 172.21.15.29:0/1514958824 conn(0x7f98ac0fdf00 msgr2=0x7f98ac1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 -- 172.21.15.29:0/1514958824 shutdown_connections 2024-09-21T02:58:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.638+0000 7f98b3132640 1 -- 172.21.15.29:0/1514958824 wait complete. 2024-09-21T02:58:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98b3132640 1 Processor -- start 2024-09-21T02:58:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98b3132640 1 -- start start 2024-09-21T02:58:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98b3132640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98ac1025c0 0x7f98ac106190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98b3132640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103640 0x7f98ac104a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98b3132640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98ac103fe0 0x7f98ac104f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98b3132640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac10e060 con 0x7f98ac103640 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98b3132640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac10dee0 con 0x7f98ac103fe0 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98b3132640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac10e1e0 con 0x7f98ac1025c0 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103640 0x7f98ac104a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103640 0x7f98ac104a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42562/0 (socket says 172.21.15.29:42562) 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.639+0000 7f98abfff640 1 -- 172.21.15.29:0/3837760954 learned_addr learned my addr 172.21.15.29:0/3837760954 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98b0ece640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98ac1025c0 0x7f98ac106190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98b16cf640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98ac103fe0 0x7f98ac104f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98abfff640 1 -- 172.21.15.29:0/3837760954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98ac1025c0 msgr2=0x7f98ac106190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98abfff640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98ac1025c0 0x7f98ac106190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98abfff640 1 -- 172.21.15.29:0/3837760954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98ac103fe0 msgr2=0x7f98ac104f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98abfff640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98ac103fe0 0x7f98ac104f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98abfff640 1 -- 172.21.15.29:0/3837760954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ac1056c0 con 0x7f98ac103640 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98b16cf640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98ac103fe0 0x7f98ac104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98abfff640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103640 0x7f98ac104a50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f989400d950 tx=0x7f989400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98a9ffb640 1 -- 172.21.15.29:0/3837760954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9894017030 con 0x7f98ac103640 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98a9ffb640 1 -- 172.21.15.29:0/3837760954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9894004020 con 0x7f98ac103640 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98a9ffb640 1 -- 172.21.15.29:0/3837760954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98940055d0 con 0x7f98ac103640 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98b3132640 1 -- 172.21.15.29:0/3837760954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98ac1a3ec0 con 0x7f98ac103640 2024-09-21T02:58:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.640+0000 7f98b3132640 1 -- 172.21.15.29:0/3837760954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98ac1a42d0 con 0x7f98ac103640 2024-09-21T02:58:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98ba0a0740 1 Processor -- start 2024-09-21T02:58:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98a9ffb640 1 -- 172.21.15.29:0/3837760954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9894009a30 con 0x7f98ac103640 2024-09-21T02:58:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98ba0a0740 1 -- start start 2024-09-21T02:58:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141901b0 0x55d914537e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d9145384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d914539010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d9144aa6f0 con 0x55d9141672f0 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d914499b20 con 0x55d9141901b0 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d91449d930 con 0x55d91416f4d0 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.643+0000 7f98b0ece640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d9145384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b0ece640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d9145384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42570/0 (socket says 172.21.15.29:42570) 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b0ece640 1 -- 172.21.15.29:0/1203112584 learned_addr learned my addr 172.21.15.29:0/1203112584 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b16cf640 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141901b0 0x55d914537e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98abfff640 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d914539010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b0ece640 1 -- 172.21.15.29:0/1203112584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 msgr2=0x55d914539010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b0ece640 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d914539010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b0ece640 1 -- 172.21.15.29:0/1203112584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141901b0 msgr2=0x55d914537e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b0ece640 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141901b0 0x55d914537e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b0ece640 1 -- 172.21.15.29:0/1203112584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d91445f530 con 0x55d9141672f0 2024-09-21T02:58:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98abfff640 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d914539010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:58:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98a9ffb640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f98980b6590 0x7f98980b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b0ece640 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d9145384c0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f989c009c80 tx=0x7f989c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c682c652cd4ce775 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98b16cf640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f98980b6590 0x7f98980b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98a2ffd640 1 -- 172.21.15.29:0/1203112584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989c012450 con 0x55d9141672f0 2024-09-21T02:58:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98a9ffb640 1 -- 172.21.15.29:0/3837760954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f98940e7120 con 0x7f98ac103640 2024-09-21T02:58:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98a2ffd640 1 -- 172.21.15.29:0/1203112584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f989c012a80 con 0x55d9141672f0 2024-09-21T02:58:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.644+0000 7f98a2ffd640 1 -- 172.21.15.29:0/1203112584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989c00fac0 con 0x55d9141672f0 2024-09-21T02:58:09.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.645+0000 7f98ba0a0740 1 -- 172.21.15.29:0/1203112584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 msgr2=0x55d9145384c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.645+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d9145384c0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f989c009c80 tx=0x7f989c0117b0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.645+0000 7f98b16cf640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f98980b6590 0x7f98980b8aa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009a60 tx=0x7f98a4004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:09.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.645+0000 7f98ba0a0740 1 -- 172.21.15.29:0/1203112584 shutdown_connections 2024-09-21T02:58:09.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.645+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d914539010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.645+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d9145384c0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.645+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/1203112584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141901b0 0x55d914537e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.645+0000 7f98ba0a0740 1 -- 172.21.15.29:0/1203112584 >> 172.21.15.29:0/1203112584 conn(0x55d914184300 msgr2=0x55d9145338e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 -- 172.21.15.29:0/1203112584 shutdown_connections 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 -- 172.21.15.29:0/1203112584 wait complete. 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 -- start start 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 Processor -- start 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141672f0 0x55d9145415a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 0x55d914541c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d9145427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d9143d1ab0 con 0x55d9141901b0 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d91443f730 con 0x55d91416f4d0 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.646+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d914539b30 con 0x55d9141672f0 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98ba0a0740 10 client.0 _ll_register_callbacks cb 0x55d9144936b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d9145427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d9145427d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42576/0 (socket says 172.21.15.29:42576) 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 -- 172.21.15.29:0/245169099 learned_addr learned my addr 172.21.15.29:0/245169099 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98abfff640 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 0x55d914541c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b0ece640 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141672f0 0x55d9145415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 -- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141672f0 msgr2=0x55d9145415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141672f0 0x55d9145415a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 -- 172.21.15.29:0/245169099 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 msgr2=0x55d914541c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 0x55d914541c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 -- 172.21.15.29:0/245169099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9141fb790 con 0x55d9141901b0 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b0ece640 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141672f0 0x55d9145415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98b16cf640 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d9145427d0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f98a4035000 tx=0x7f98a402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af362b2b5de392b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f988bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f988bfff640 1 -- 172.21.15.29:0/245169099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a40045c0 con 0x55d9141901b0 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f98ba0a0740 1 -- 172.21.15.29:0/245169099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f989c012a80 con 0x55d9141901b0 2024-09-21T02:58:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.647+0000 7f988bfff640 1 -- 172.21.15.29:0/245169099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f98a4004760 con 0x55d9141901b0 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f988bfff640 1 -- 172.21.15.29:0/245169099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f98a40305c0 con 0x55d9141901b0 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f98ba0a0740 10 client.18003 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f98ba0a0740 1 -- 172.21.15.29:0/245169099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d914274620 con 0x55d9141901b0 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f988bfff640 1 -- 172.21.15.29:0/245169099 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f989c012a80 con 0x55d9141901b0 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f988bfff640 1 -- 172.21.15.29:0/245169099 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f98a4050070 con 0x55d9141901b0 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f98ba0a0740 10 client.18003 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f98ba0a0740 10 client.18003 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f98ba0a0740 20 client.18003 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d91414ed80 0x55d91454eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f98ba0a0740 4 client.18003 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.648+0000 7f98ba0a0740 1 -- 172.21.15.29:0/245169099 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9144ac640 con 0x55d91414ed80 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.649+0000 7f98b0ece640 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d91414ed80 0x55d91454eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.649+0000 7f98b0ece640 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d91414ed80 0x55d91454eda0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90a859f176b2f43c server_cookie=4406d84f2aac4036 in_seq=0 out_seq=0 2024-09-21T02:58:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.649+0000 7f988bfff640 10 client.18003 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f988bfff640 1 -- 172.21.15.29:0/245169099 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d9144ac640 con 0x55d91414ed80 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f988bfff640 10 client.18003 handle_command_reply: tid=0 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 client.18003 shutdown 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 -- 172.21.15.29:0/245169099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 msgr2=0x55d9145427d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d9145427d0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f98a4035000 tx=0x7f98a402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 -- 172.21.15.29:0/245169099 shutdown_connections 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d91414ed80 0x55d91454eda0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d9145427d0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 0x55d914541c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f988bfff640 0 client.18003 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/245169099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141672f0 0x55d9145415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 -- 172.21.15.29:0/245169099 >> 172.21.15.29:0/245169099 conn(0x55d914184300 msgr2=0x55d914531ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 -- 172.21.15.29:0/245169099 shutdown_connections 2024-09-21T02:58:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 1 -- 172.21.15.29:0/245169099 wait complete. 2024-09-21T02:58:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.650+0000 7f98ba0a0740 20 client.18003 trim_cache size 0 max 16384 2024-09-21T02:58:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98ba0a0740 1 Processor -- start 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98ba0a0740 1 -- start start 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141672f0 0x55d91453fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d91453fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d914537df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1ae7c0 con 0x55d9141901b0 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1b0300 con 0x55d9141672f0 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1aee90 con 0x55d91416f4d0 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98b16cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d914537df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98b16cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d914537df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42592/0 (socket says 172.21.15.29:42592) 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.665+0000 7f98b16cf640 1 -- 172.21.15.29:0/567224287 learned_addr learned my addr 172.21.15.29:0/567224287 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98b0ece640 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141672f0 0x55d91453fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98abfff640 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d91453fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98b16cf640 1 -- 172.21.15.29:0/567224287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 msgr2=0x55d91453fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98b16cf640 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d91453fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98b16cf640 1 -- 172.21.15.29:0/567224287 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141672f0 msgr2=0x55d91453fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98b16cf640 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141672f0 0x55d91453fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98b16cf640 1 -- 172.21.15.29:0/567224287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d91453f730 con 0x55d9141901b0 2024-09-21T02:58:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98b16cf640 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d914537df0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f98a40524e0 tx=0x7f98a4031800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e4c8ad50d43acb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98a27fc640 1 -- 172.21.15.29:0/567224287 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a40044d0 con 0x55d9141901b0 2024-09-21T02:58:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98a27fc640 1 -- 172.21.15.29:0/567224287 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98a4036050 con 0x55d9141901b0 2024-09-21T02:58:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.666+0000 7f98a27fc640 1 -- 172.21.15.29:0/567224287 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a40309c0 con 0x55d9141901b0 2024-09-21T02:58:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 -- 172.21.15.29:0/567224287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 msgr2=0x55d914537df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d914537df0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f98a40524e0 tx=0x7f98a4031800 comp rx=0 tx=0).stop 2024-09-21T02:58:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 -- 172.21.15.29:0/567224287 shutdown_connections 2024-09-21T02:58:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141901b0 0x55d914537df0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d91416f4d0 0x55d91453fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/567224287 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d9141672f0 0x55d91453fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 -- 172.21.15.29:0/567224287 >> 172.21.15.29:0/567224287 conn(0x55d914184300 msgr2=0x55d91452e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 -- 172.21.15.29:0/567224287 shutdown_connections 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 -- 172.21.15.29:0/567224287 wait complete. 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 -- start start 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.667+0000 7f98ba0a0740 1 Processor -- start 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d914549190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 0x55d914547e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98ba0a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141901b0 0x55d914548390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1afc30 con 0x55d9141672f0 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1af560 con 0x55d91416f4d0 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98ba0a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1ac7a0 con 0x55d9141901b0 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98ba0a0740 10 client.0 _ll_register_callbacks cb 0x7f98a4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b0ece640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d914549190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b0ece640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d914549190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42604/0 (socket says 172.21.15.29:42604) 2024-09-21T02:58:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b0ece640 1 -- 172.21.15.29:0/2866966054 learned_addr learned my addr 172.21.15.29:0/2866966054 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98abfff640 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 0x55d914547e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b16cf640 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141901b0 0x55d914548390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b0ece640 1 -- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141901b0 msgr2=0x55d914548390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b0ece640 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141901b0 0x55d914548390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b0ece640 1 -- 172.21.15.29:0/2866966054 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 msgr2=0x55d914547e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b0ece640 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 0x55d914547e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b0ece640 1 -- 172.21.15.29:0/2866966054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d914548900 con 0x55d9141672f0 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.668+0000 7f98b16cf640 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141901b0 0x55d914548390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98b0ece640 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d914549190 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f989c000c00 tx=0x7f989c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9866778ffccfa77e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f988bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f988bfff640 1 -- 172.21.15.29:0/2866966054 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989c030070 con 0x55d9141672f0 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f988bfff640 1 -- 172.21.15.29:0/2866966054 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f989c005250 con 0x55d9141672f0 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 1 -- 172.21.15.29:0/2866966054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98a4036050 con 0x55d9141672f0 2024-09-21T02:58:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f988bfff640 1 -- 172.21.15.29:0/2866966054 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f989c03b050 con 0x55d9141672f0 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 10 client.18009 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 1 -- 172.21.15.29:0/2866966054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d914547760 con 0x55d9141672f0 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f988bfff640 1 -- 172.21.15.29:0/2866966054 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98a4036050 con 0x55d9141672f0 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f988bfff640 1 -- 172.21.15.29:0/2866966054 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f989c0035f0 con 0x55d9141672f0 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 10 client.18009 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 10 client.18009 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 20 client.18009 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d9141241c0 0x55d9145bbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 4 client.18009 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.669+0000 7f98ba0a0740 1 -- 172.21.15.29:0/2866966054 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9144a80f0 con 0x55d9141241c0 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.670+0000 7f98abfff640 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d9141241c0 0x55d9145bbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.670+0000 7f988bfff640 10 client.18009 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.670+0000 7f98abfff640 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d9141241c0 0x55d9145bbbb0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88713fe86472ca35 server_cookie=3871888a69b114bb in_seq=0 out_seq=0 2024-09-21T02:58:09.729 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.93 MB/sec execute 226 sec latency 3498.442 ms 2024-09-21T02:58:09.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.791+0000 7f988bfff640 1 -- 172.21.15.29:0/2866966054 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d9144a80f0 con 0x55d9141241c0 2024-09-21T02:58:09.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.791+0000 7f988bfff640 10 client.18009 handle_command_reply: tid=0 2024-09-21T02:58:09.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.791+0000 7f98ba0a0740 1 client.18009 shutdown 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.791+0000 7f98ba0a0740 1 -- 172.21.15.29:0/2866966054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 msgr2=0x55d914549190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.791+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d914549190 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f989c000c00 tx=0x7f989c009b80 comp rx=0 tx=0).stop 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.791+0000 7f98ba0a0740 1 -- 172.21.15.29:0/2866966054 shutdown_connections 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.791+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9141901b0 0x55d914548390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.792+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d91416f4d0 0x55d914547e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.792+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d9141241c0 0x55d9145bbbb0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.792+0000 7f98ba0a0740 1 --2- 172.21.15.29:0/2866966054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9141672f0 0x55d914549190 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.792+0000 7f988bfff640 0 client.18009 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.792+0000 7f98ba0a0740 1 -- 172.21.15.29:0/2866966054 >> 172.21.15.29:0/2866966054 conn(0x55d914184300 msgr2=0x55d91453f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.792+0000 7f98ba0a0740 1 -- 172.21.15.29:0/2866966054 shutdown_connections 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.792+0000 7f98ba0a0740 1 -- 172.21.15.29:0/2866966054 wait complete. 2024-09-21T02:58:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.792+0000 7f98ba0a0740 20 client.18009 trim_cache size 0 max 16384 2024-09-21T02:58:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:58:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:58:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:58:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:58:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 -- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f98980b6590 msgr2=0x7f98980b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f98980b6590 0x7f98980b8aa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009a60 tx=0x7f98a4004a40 comp rx=0 tx=0).stop 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 -- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103640 msgr2=0x7f98ac104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103640 0x7f98ac104a50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f989400d950 tx=0x7f989400de10 comp rx=0 tx=0).stop 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 -- 172.21.15.29:0/3837760954 shutdown_connections 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f98980b6590 0x7f98980b8aa0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98ac103fe0 0x7f98ac104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac103640 0x7f98ac104a50 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 --2- 172.21.15.29:0/3837760954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98ac1025c0 0x7f98ac106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 -- 172.21.15.29:0/3837760954 >> 172.21.15.29:0/3837760954 conn(0x7f98ac0fdf00 msgr2=0x7f98ac0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 -- 172.21.15.29:0/3837760954 shutdown_connections 2024-09-21T02:58:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:09.793+0000 7f98b3132640 1 -- 172.21.15.29:0/3837760954 wait complete. 2024-09-21T02:58:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:09 smithi066 ceph-mon[33436]: pgmap v960: 129 pgs: 129 active+clean; 297 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 3.6 MiB/s wr, 283 op/s 2024-09-21T02:58:10.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:09 smithi190 ceph-mon[33871]: pgmap v960: 129 pgs: 129 active+clean; 297 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 3.6 MiB/s wr, 283 op/s 2024-09-21T02:58:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:09 smithi029 ceph-mon[31006]: pgmap v960: 129 pgs: 129 active+clean; 297 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 3.6 MiB/s wr, 283 op/s 2024-09-21T02:58:10.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.90 MB/sec execute 227 sec latency 4498.543 ms 2024-09-21T02:58:11.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.88 MB/sec execute 228 sec latency 5498.630 ms 2024-09-21T02:58:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:11 smithi066 ceph-mon[33436]: pgmap v961: 129 pgs: 129 active+clean; 291 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 3.2 MiB/s wr, 267 op/s 2024-09-21T02:58:12.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:11 smithi190 ceph-mon[33871]: pgmap v961: 129 pgs: 129 active+clean; 291 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 3.2 MiB/s wr, 267 op/s 2024-09-21T02:58:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:11 smithi029 ceph-mon[31006]: pgmap v961: 129 pgs: 129 active+clean; 291 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 3.2 MiB/s wr, 267 op/s 2024-09-21T02:58:12.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.85 MB/sec execute 229 sec latency 6498.704 ms 2024-09-21T02:58:13.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.83 MB/sec execute 230 sec latency 7498.757 ms 2024-09-21T02:58:14.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:13 smithi066 ceph-mon[33436]: pgmap v962: 129 pgs: 129 active+clean; 288 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-09-21T02:58:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:13 smithi190 ceph-mon[33871]: pgmap v962: 129 pgs: 129 active+clean; 288 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-09-21T02:58:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:13 smithi029 ceph-mon[31006]: pgmap v962: 129 pgs: 129 active+clean; 288 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-09-21T02:58:14.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.80 MB/sec execute 231 sec latency 8498.821 ms 2024-09-21T02:58:15.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.78 MB/sec execute 232 sec latency 9498.860 ms 2024-09-21T02:58:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:15 smithi029 ceph-mon[31006]: pgmap v963: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-09-21T02:58:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:15 smithi066 ceph-mon[33436]: pgmap v963: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-09-21T02:58:16.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:15 smithi190 ceph-mon[33871]: pgmap v963: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-09-21T02:58:16.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.75 MB/sec execute 233 sec latency 10498.898 ms 2024-09-21T02:58:17.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.73 MB/sec execute 234 sec latency 11498.997 ms 2024-09-21T02:58:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:17 smithi066 ceph-mon[33436]: pgmap v964: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-09-21T02:58:18.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:17 smithi190 ceph-mon[33871]: pgmap v964: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-09-21T02:58:18.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:17 smithi029 ceph-mon[31006]: pgmap v964: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-09-21T02:58:18.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.70 MB/sec execute 235 sec latency 12499.111 ms 2024-09-21T02:58:19.730 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.68 MB/sec execute 236 sec latency 13499.212 ms 2024-09-21T02:58:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:19 smithi066 ceph-mon[33436]: pgmap v965: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-09-21T02:58:20.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:19 smithi190 ceph-mon[33871]: pgmap v965: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-09-21T02:58:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:19 smithi029 ceph-mon[31006]: pgmap v965: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-09-21T02:58:20.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.65 MB/sec execute 237 sec latency 14499.340 ms 2024-09-21T02:58:21.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.63 MB/sec execute 238 sec latency 15499.445 ms 2024-09-21T02:58:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:21 smithi066 ceph-mon[33436]: pgmap v966: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 435 KiB/s wr, 92 op/s 2024-09-21T02:58:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:21 smithi190 ceph-mon[33871]: pgmap v966: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 435 KiB/s wr, 92 op/s 2024-09-21T02:58:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:21 smithi029 ceph-mon[31006]: pgmap v966: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 435 KiB/s wr, 92 op/s 2024-09-21T02:58:22.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.61 MB/sec execute 239 sec latency 16499.569 ms 2024-09-21T02:58:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 -- 172.21.15.29:0/2219091404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 msgr2=0x7fb044102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 --2- 172.21.15.29:0/2219091404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 0x7fb044102a40 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fb034009ca0 tx=0x7fb03402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 -- 172.21.15.29:0/2219091404 shutdown_connections 2024-09-21T02:58:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 --2- 172.21.15.29:0/2219091404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb044103ff0 0x7fb04410a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 --2- 172.21.15.29:0/2219091404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb044103650 0x7fb044103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 --2- 172.21.15.29:0/2219091404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 0x7fb044102a40 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 -- 172.21.15.29:0/2219091404 >> 172.21.15.29:0/2219091404 conn(0x7fb0440fdf30 msgr2=0x7fb0441003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 -- 172.21.15.29:0/2219091404 shutdown_connections 2024-09-21T02:58:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.908+0000 7fb0490ec640 1 -- 172.21.15.29:0/2219091404 wait complete. 2024-09-21T02:58:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb0490ec640 1 Processor -- start 2024-09-21T02:58:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb0490ec640 1 -- start start 2024-09-21T02:58:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb0490ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 0x7fb044198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb0490ec640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb044103650 0x7fb044199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb0490ec640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb044103ff0 0x7fb04419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb0490ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb04410e080 con 0x7fb0441025d0 2024-09-21T02:58:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb0490ec640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb04410df00 con 0x7fb044103ff0 2024-09-21T02:58:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb0490ec640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb04410e200 con 0x7fb044103650 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.909+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 0x7fb044198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 0x7fb044198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33924/0 (socket says 172.21.15.29:33924) 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb042d9d640 1 -- 172.21.15.29:0/1455067284 learned_addr learned my addr 172.21.15.29:0/1455067284 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb04359e640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb044103ff0 0x7fb04419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb04259c640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb044103650 0x7fb044199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb042d9d640 1 -- 172.21.15.29:0/1455067284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb044103650 msgr2=0x7fb044199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb042d9d640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb044103650 0x7fb044199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb042d9d640 1 -- 172.21.15.29:0/1455067284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb044103ff0 msgr2=0x7fb04419a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb042d9d640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb044103ff0 0x7fb04419a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb042d9d640 1 -- 172.21.15.29:0/1455067284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04419a840 con 0x7fb0441025d0 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb042d9d640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 0x7fb044198e00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fb034006010 tx=0x7fb034003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb03bfff640 1 -- 172.21.15.29:0/1455067284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034004450 con 0x7fb0441025d0 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb03bfff640 1 -- 172.21.15.29:0/1455067284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb034040070 con 0x7fb0441025d0 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb0490ec640 1 -- 172.21.15.29:0/1455067284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04419aad0 con 0x7fb0441025d0 2024-09-21T02:58:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb03bfff640 1 -- 172.21.15.29:0/1455067284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034045470 con 0x7fb0441025d0 2024-09-21T02:58:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.910+0000 7fb0490ec640 1 -- 172.21.15.29:0/1455067284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0441a1220 con 0x7fb0441025d0 2024-09-21T02:58:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.914+0000 7fb050059740 1 Processor -- start 2024-09-21T02:58:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.914+0000 7fb050059740 1 -- start start 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.914+0000 7fb050059740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5ca6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.914+0000 7fb050059740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 0x5561b5ca74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.914+0000 7fb050059740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5ca8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.914+0000 7fb050059740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5561b5c196f0 con 0x5561b58de4d0 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.914+0000 7fb050059740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5561b5c08b20 con 0x5561b58d62f0 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.914+0000 7fb050059740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5561b5c0c930 con 0x5561b58ff1b0 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5ca8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5ca8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33932/0 (socket says 172.21.15.29:33932) 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 -- 172.21.15.29:0/2572300808 learned_addr learned my addr 172.21.15.29:0/2572300808 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04359e640 1 --2- 172.21.15.29:0/2572300808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5ca6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb03bfff640 1 -- 172.21.15.29:0/1455067284 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb0340045f0 con 0x7fb0441025d0 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 -- 172.21.15.29:0/2572300808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 msgr2=0x5561b5ca6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 --2- 172.21.15.29:0/2572300808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5ca6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 -- 172.21.15.29:0/2572300808 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 msgr2=0x5561b5ca74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 --2- 172.21.15.29:0/2572300808 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 0x5561b5ca74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 -- 172.21.15.29:0/2572300808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561b5bce530 con 0x5561b58de4d0 2024-09-21T02:58:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb04259c640 1 --2- 172.21.15.29:0/2572300808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5ca8010 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00b760 tx=0x7fb02c00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fb0f631572c2cc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb0397fa640 1 -- 172.21.15.29:0/2572300808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c003dd0 con 0x5561b58de4d0 2024-09-21T02:58:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.915+0000 7fb0397fa640 1 -- 172.21.15.29:0/2572300808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb02c015070 con 0x5561b58de4d0 2024-09-21T02:58:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.916+0000 7fb0397fa640 1 -- 172.21.15.29:0/2572300808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c004c80 con 0x5561b58de4d0 2024-09-21T02:58:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.916+0000 7fb03bfff640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0300b6510 0x7fb0300b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.916+0000 7fb04359e640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0300b6510 0x7fb0300b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.916+0000 7fb03bfff640 1 -- 172.21.15.29:0/1455067284 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb034108820 con 0x7fb0441025d0 2024-09-21T02:58:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.916+0000 7fb050059740 1 -- 172.21.15.29:0/2572300808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 msgr2=0x5561b5ca8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.916+0000 7fb050059740 1 --2- 172.21.15.29:0/2572300808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5ca8010 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00b760 tx=0x7fb02c00bc20 comp rx=0 tx=0).stop 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 -- 172.21.15.29:0/2572300808 shutdown_connections 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 --2- 172.21.15.29:0/2572300808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5ca8010 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 --2- 172.21.15.29:0/2572300808 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 0x5561b5ca74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 --2- 172.21.15.29:0/2572300808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5ca6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 -- 172.21.15.29:0/2572300808 >> 172.21.15.29:0/2572300808 conn(0x5561b58f3300 msgr2=0x5561b5ca28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 -- 172.21.15.29:0/2572300808 shutdown_connections 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb04359e640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0300b6510 0x7fb0300b8a20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb03c005fd0 tx=0x7fb03c005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 -- 172.21.15.29:0/2572300808 wait complete. 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 -- start start 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.917+0000 7fb050059740 1 Processor -- start 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb050059740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5cb0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb050059740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58de4d0 0x5561b5cb0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb050059740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5cb1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb050059740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5561b5b40ab0 con 0x5561b58d62f0 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb050059740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5561b5bae730 con 0x5561b58de4d0 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb050059740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5561b5ca8b30 con 0x5561b58ff1b0 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb050059740 10 client.0 _ll_register_callbacks cb 0x5561b5c026b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5cb0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5cb0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33942/0 (socket says 172.21.15.29:33942) 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 -- 172.21.15.29:0/2167053277 learned_addr learned my addr 172.21.15.29:0/2167053277 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb04359e640 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5cb1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb04259c640 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58de4d0 0x5561b5cb0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 -- 172.21.15.29:0/2167053277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 msgr2=0x5561b5cb1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5cb1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 -- 172.21.15.29:0/2167053277 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58de4d0 msgr2=0x5561b5cb0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58de4d0 0x5561b5cb0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 -- 172.21.15.29:0/2167053277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561b596a790 con 0x5561b58d62f0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb042d9d640 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5cb0660 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fb0340cac00 tx=0x7fb0340ca590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ed135bf5726f44 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.918+0000 7fb02a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb02a7fc640 1 -- 172.21.15.29:0/2167053277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0340ce140 con 0x5561b58d62f0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb02a7fc640 1 -- 172.21.15.29:0/2167053277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb0340cd890 con 0x5561b58d62f0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 1 -- 172.21.15.29:0/2167053277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb02c015070 con 0x5561b58d62f0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb02a7fc640 1 -- 172.21.15.29:0/2167053277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb0340ced30 con 0x5561b58d62f0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 10 client.18021 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 1 -- 172.21.15.29:0/2167053277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5561b59e3620 con 0x5561b58d62f0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb02a7fc640 1 -- 172.21.15.29:0/2167053277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb02c015070 con 0x5561b58d62f0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb02a7fc640 1 -- 172.21.15.29:0/2167053277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fb03404d030 con 0x5561b58d62f0 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 10 client.18021 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 10 client.18021 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:58:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 20 client.18021 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5561b58bdd80 0x5561b5cbdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 4 client.18021 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:58:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.919+0000 7fb050059740 1 -- 172.21.15.29:0/2167053277 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561b5c1b640 con 0x5561b58bdd80 2024-09-21T02:58:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.920+0000 7fb04259c640 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5561b58bdd80 0x5561b5cbdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.920+0000 7fb04259c640 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5561b58bdd80 0x5561b5cbdd10 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f011b974770a0190 server_cookie=6108cc400d117647 in_seq=0 out_seq=0 2024-09-21T02:58:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.920+0000 7fb02a7fc640 10 client.18021 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb02a7fc640 1 -- 172.21.15.29:0/2167053277 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5561b5c1b640 con 0x5561b58bdd80 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb02a7fc640 10 client.18021 handle_command_reply: tid=0 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 client.18021 shutdown 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 -- 172.21.15.29:0/2167053277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 msgr2=0x5561b5cb0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5cb0660 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fb0340cac00 tx=0x7fb0340ca590 comp rx=0 tx=0).stop 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 -- 172.21.15.29:0/2167053277 shutdown_connections 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5cb1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58de4d0 0x5561b5cb0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5561b58bdd80 0x5561b5cbdd10 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 --2- 172.21.15.29:0/2167053277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5cb0660 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb02a7fc640 0 client.18021 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 -- 172.21.15.29:0/2167053277 >> 172.21.15.29:0/2167053277 conn(0x5561b58f3300 msgr2=0x5561b5ca0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 -- 172.21.15.29:0/2167053277 shutdown_connections 2024-09-21T02:58:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 1 -- 172.21.15.29:0/2167053277 wait complete. 2024-09-21T02:58:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.921+0000 7fb050059740 20 client.18021 trim_cache size 0 max 16384 2024-09-21T02:58:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.936+0000 7fb050059740 1 Processor -- start 2024-09-21T02:58:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.936+0000 7fb050059740 1 -- start start 2024-09-21T02:58:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb050059740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5caf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb050059740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58de4d0 0x5561b5c9d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb050059740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58ff1b0 0x5561b5ca1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb050059740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441ab380 con 0x5561b58d62f0 2024-09-21T02:58:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb050059740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441acec0 con 0x5561b58ff1b0 2024-09-21T02:58:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb050059740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441aba50 con 0x5561b58de4d0 2024-09-21T02:58:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5caf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5caf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33944/0 (socket says 172.21.15.29:33944) 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 -- 172.21.15.29:0/2774871711 learned_addr learned my addr 172.21.15.29:0/2774871711 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb04259c640 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58de4d0 0x5561b5c9d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb04359e640 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58ff1b0 0x5561b5ca1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 -- 172.21.15.29:0/2774871711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58de4d0 msgr2=0x5561b5c9d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58de4d0 0x5561b5c9d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 -- 172.21.15.29:0/2774871711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58ff1b0 msgr2=0x5561b5ca1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58ff1b0 0x5561b5ca1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 -- 172.21.15.29:0/2774871711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561b5ca8a10 con 0x5561b58d62f0 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb042d9d640 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5caf860 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fb0340cb8a0 tx=0x7fb0340ca600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b438bd0923e63b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb038ff9640 1 -- 172.21.15.29:0/2774871711 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0340ce140 con 0x5561b58d62f0 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb038ff9640 1 -- 172.21.15.29:0/2774871711 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb034036040 con 0x5561b58d62f0 2024-09-21T02:58:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.937+0000 7fb038ff9640 1 -- 172.21.15.29:0/2774871711 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0340cdaf0 con 0x5561b58d62f0 2024-09-21T02:58:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 -- 172.21.15.29:0/2774871711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 msgr2=0x5561b5caf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5caf860 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fb0340cb8a0 tx=0x7fb0340ca600 comp rx=0 tx=0).stop 2024-09-21T02:58:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 -- 172.21.15.29:0/2774871711 shutdown_connections 2024-09-21T02:58:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58ff1b0 0x5561b5ca1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58de4d0 0x5561b5c9d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 --2- 172.21.15.29:0/2774871711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58d62f0 0x5561b5caf860 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 -- 172.21.15.29:0/2774871711 >> 172.21.15.29:0/2774871711 conn(0x5561b58f3300 msgr2=0x5561b5ca2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 -- 172.21.15.29:0/2774871711 shutdown_connections 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.938+0000 7fb050059740 1 -- 172.21.15.29:0/2774871711 wait complete. 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 1 -- start start 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 1 Processor -- start 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 0x5561b5cb8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5cb8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5cb6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441ac7f0 con 0x5561b58de4d0 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441ac120 con 0x5561b58d62f0 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441a9360 con 0x5561b58ff1b0 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb050059740 10 client.0 _ll_register_callbacks cb 0x7fb034031880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb04259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5cb8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb04259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5cb8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33954/0 (socket says 172.21.15.29:33954) 2024-09-21T02:58:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb04259c640 1 -- 172.21.15.29:0/3712628749 learned_addr learned my addr 172.21.15.29:0/3712628749 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb042d9d640 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 0x5561b5cb8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.939+0000 7fb04359e640 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5cb6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb04259c640 1 -- 172.21.15.29:0/3712628749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 msgr2=0x5561b5cb6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb04259c640 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5cb6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb04259c640 1 -- 172.21.15.29:0/3712628749 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 msgr2=0x5561b5cb8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb04259c640 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 0x5561b5cb8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb04259c640 1 -- 172.21.15.29:0/3712628749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561b5cb75f0 con 0x5561b58de4d0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb04259c640 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5cb8720 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fb02c006010 tx=0x7fb02c005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da36cc179e4ab743 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb02a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb02a7fc640 1 -- 172.21.15.29:0/3712628749 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00be70 con 0x5561b58de4d0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb02a7fc640 1 -- 172.21.15.29:0/3712628749 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb02c027410 con 0x5561b58de4d0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb050059740 1 -- 172.21.15.29:0/3712628749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb034036040 con 0x5561b58de4d0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb02a7fc640 1 -- 172.21.15.29:0/3712628749 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb02c039050 con 0x5561b58de4d0 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb050059740 10 client.18027 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb050059740 1 -- 172.21.15.29:0/3712628749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5561b5cb64d0 con 0x5561b58de4d0 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.940+0000 7fb02a7fc640 1 -- 172.21.15.29:0/3712628749 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb034036040 con 0x5561b58de4d0 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb02a7fc640 1 -- 172.21.15.29:0/3712628749 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fb02c011640 con 0x5561b58de4d0 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb050059740 10 client.18027 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb050059740 10 client.18027 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb050059740 20 client.18027 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb050059740 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5561b58931c0 0x5561b5d2aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb050059740 4 client.18027 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb050059740 1 -- 172.21.15.29:0/3712628749 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5561b5c170f0 con 0x5561b58931c0 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb042d9d640 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5561b58931c0 0x5561b5d2aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb02a7fc640 10 client.18027 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.941+0000 7fb042d9d640 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5561b58931c0 0x5561b5d2aab0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8da7e5e8dd310d33 server_cookie=268d292c60e0158f in_seq=0 out_seq=0 2024-09-21T02:58:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb02a7fc640 1 -- 172.21.15.29:0/3712628749 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5561b5c170f0 con 0x5561b58931c0 2024-09-21T02:58:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb02a7fc640 10 client.18027 handle_command_reply: tid=0 2024-09-21T02:58:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 client.18027 shutdown 2024-09-21T02:58:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 -- 172.21.15.29:0/3712628749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 msgr2=0x5561b5cb8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5cb8720 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fb02c006010 tx=0x7fb02c005ce0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 -- 172.21.15.29:0/3712628749 shutdown_connections 2024-09-21T02:58:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561b58ff1b0 0x5561b5cb6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5561b58931c0 0x5561b5d2aab0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561b58de4d0 0x5561b5cb8720 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 --2- 172.21.15.29:0/3712628749 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561b58d62f0 0x5561b5cb8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb02a7fc640 0 client.18027 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.974+0000 7fb050059740 1 -- 172.21.15.29:0/3712628749 >> 172.21.15.29:0/3712628749 conn(0x5561b58f3300 msgr2=0x5561b5ca2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb050059740 1 -- 172.21.15.29:0/3712628749 shutdown_connections 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb050059740 1 -- 172.21.15.29:0/3712628749 wait complete. 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb050059740 20 client.18027 trim_cache size 0 max 16384 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:58:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:58:22.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 -- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0300b6510 msgr2=0x7fb0300b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0300b6510 0x7fb0300b8a20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb03c005fd0 tx=0x7fb03c005cc0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 -- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 msgr2=0x7fb044198e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:22.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 0x7fb044198e00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fb034006010 tx=0x7fb034003c20 comp rx=0 tx=0).stop 2024-09-21T02:58:22.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 -- 172.21.15.29:0/1455067284 shutdown_connections 2024-09-21T02:58:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb0300b6510 0x7fb0300b8a20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb044103ff0 0x7fb04419a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb044103650 0x7fb044199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 --2- 172.21.15.29:0/1455067284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0441025d0 0x7fb044198e00 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.975+0000 7fb0490ec640 1 -- 172.21.15.29:0/1455067284 >> 172.21.15.29:0/1455067284 conn(0x7fb0440fdf30 msgr2=0x7fb0441069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.976+0000 7fb0490ec640 1 -- 172.21.15.29:0/1455067284 shutdown_connections 2024-09-21T02:58:22.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:22.976+0000 7fb0490ec640 1 -- 172.21.15.29:0/1455067284 wait complete. 2024-09-21T02:58:23.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.58 MB/sec execute 240 sec latency 17499.696 ms 2024-09-21T02:58:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:23 smithi066 ceph-mon[33436]: pgmap v967: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 198 KiB/s wr, 55 op/s 2024-09-21T02:58:24.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:23 smithi190 ceph-mon[33871]: pgmap v967: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 198 KiB/s wr, 55 op/s 2024-09-21T02:58:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:23 smithi029 ceph-mon[31006]: pgmap v967: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 198 KiB/s wr, 55 op/s 2024-09-21T02:58:24.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:58:24.569 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:58:24.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.56 MB/sec execute 241 sec latency 18499.800 ms 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.170+0000 7f54cdb86640 1 -- 172.21.15.29:0/395815847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8108d20 msgr2=0x7f54c810f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.170+0000 7f54cdb86640 1 --2- 172.21.15.29:0/395815847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8108d20 0x7f54c810f550 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f54c0009ca0 tx=0x7f54c002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 -- 172.21.15.29:0/395815847 shutdown_connections 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 --2- 172.21.15.29:0/395815847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c8108d20 0x7f54c810f550 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 --2- 172.21.15.29:0/395815847 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54c81083d0 0x7f54c81087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 --2- 172.21.15.29:0/395815847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54c8102ae0 0x7f54c8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 -- 172.21.15.29:0/395815847 >> 172.21.15.29:0/395815847 conn(0x7f54c8079760 msgr2=0x7f54c8079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 -- 172.21.15.29:0/395815847 shutdown_connections 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 -- 172.21.15.29:0/395815847 wait complete. 2024-09-21T02:58:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 Processor -- start 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.171+0000 7f54cdb86640 1 -- start start 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54cdb86640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54c8102ae0 0x7f54c819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54cdb86640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c81083d0 0x7f54c819b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54cdb86640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54c8108d20 0x7f54c819bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54cdb86640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f54c8112890 con 0x7f54c81083d0 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54cdb86640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f54c8112710 con 0x7f54c8102ae0 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54cdb86640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f54c8112a10 con 0x7f54c8108d20 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c81083d0 0x7f54c819b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c81083d0 0x7f54c819b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48948/0 (socket says 172.21.15.29:48948) 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 -- 172.21.15.29:0/727222345 learned_addr learned my addr 172.21.15.29:0/727222345 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c77fe640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54c8102ae0 0x7f54c819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c7fff640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54c8108d20 0x7f54c819bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 -- 172.21.15.29:0/727222345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54c8108d20 msgr2=0x7f54c819bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54c8108d20 0x7f54c819bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 -- 172.21.15.29:0/727222345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54c8102ae0 msgr2=0x7f54c819b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54c8102ae0 0x7f54c819b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 -- 172.21.15.29:0/727222345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c81a6470 con 0x7f54c81083d0 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c7fff640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54c8108d20 0x7f54c819bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.172+0000 7f54c6ffd640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c81083d0 0x7f54c819b720 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00d770 tx=0x7f54bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.173+0000 7f54c4ff9640 1 -- 172.21.15.29:0/727222345 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc017070 con 0x7f54c81083d0 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.173+0000 7f54c4ff9640 1 -- 172.21.15.29:0/727222345 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54bc009c30 con 0x7f54c81083d0 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.173+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c81a6760 con 0x7f54c81083d0 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.173+0000 7f54c4ff9640 1 -- 172.21.15.29:0/727222345 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc00f620 con 0x7f54c81083d0 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.173+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c81a6ca0 con 0x7f54c81083d0 2024-09-21T02:58:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.174+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54c8103d10 con 0x7f54c81083d0 2024-09-21T02:58:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.175+0000 7f54c4ff9640 1 -- 172.21.15.29:0/727222345 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f54bc01a070 con 0x7f54c81083d0 2024-09-21T02:58:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.176+0000 7f54c4ff9640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f54b00b6540 0x7f54b00b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.176+0000 7f54c77fe640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f54b00b6540 0x7f54b00b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.176+0000 7f54c4ff9640 1 -- 172.21.15.29:0/727222345 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f54bc0e70b0 con 0x7f54c81083d0 2024-09-21T02:58:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.177+0000 7f54c77fe640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f54b00b6540 0x7f54b00b8a50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f54c806c6a0 tx=0x7f54b800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.179+0000 7f54c4ff9640 1 -- 172.21.15.29:0/727222345 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f54bc0ad5c0 con 0x7f54c81083d0 2024-09-21T02:58:25.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.406+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54c806bf50 con 0x7f54c81083d0 2024-09-21T02:58:25.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.406+0000 7f54c4ff9640 1 -- 172.21.15.29:0/727222345 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f54bc0b19a0 con 0x7f54c81083d0 2024-09-21T02:58:25.408 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:58:25.408 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:58:25.408 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:58:25.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.408+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f54b00b6540 msgr2=0x7f54b00b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:25.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.408+0000 7f54cdb86640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f54b00b6540 0x7f54b00b8a50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f54c806c6a0 tx=0x7f54b800b390 comp rx=0 tx=0).stop 2024-09-21T02:58:25.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.408+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c81083d0 msgr2=0x7f54c819b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:25.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.408+0000 7f54cdb86640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c81083d0 0x7f54c819b720 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00d770 tx=0x7f54bc00dc30 comp rx=0 tx=0).stop 2024-09-21T02:58:25.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.409+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 shutdown_connections 2024-09-21T02:58:25.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.409+0000 7f54cdb86640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f54b00b6540 0x7f54b00b8a50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.409+0000 7f54cdb86640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54c8108d20 0x7f54c819bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.409+0000 7f54cdb86640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54c81083d0 0x7f54c819b720 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.409+0000 7f54cdb86640 1 --2- 172.21.15.29:0/727222345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54c8102ae0 0x7f54c819b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.409+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 >> 172.21.15.29:0/727222345 conn(0x7f54c8079760 msgr2=0x7f54c81044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.409+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 shutdown_connections 2024-09-21T02:58:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:25.409+0000 7f54cdb86640 1 -- 172.21.15.29:0/727222345 wait complete. 2024-09-21T02:58:25.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.54 MB/sec execute 242 sec latency 19499.849 ms 2024-09-21T02:58:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:25 smithi066 ceph-mon[33436]: pgmap v968: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 138 KiB/s wr, 40 op/s 2024-09-21T02:58:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:25 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/727222345' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:25 smithi190 ceph-mon[33871]: pgmap v968: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 138 KiB/s wr, 40 op/s 2024-09-21T02:58:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:25 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/727222345' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:25 smithi029 ceph-mon[31006]: pgmap v968: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 138 KiB/s wr, 40 op/s 2024-09-21T02:58:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:25 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/727222345' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:26.265 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 7776ac5f --release --await' 2024-09-21T02:58:26.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 7776ac5f --release --await 2024-09-21T02:58:26.535 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:58:26.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37803 5.52 MB/sec execute 243 sec latency 20499.921 ms 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 -- 172.21.15.29:0/24581650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 msgr2=0x7ff50c112520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 --2- 172.21.15.29:0/24581650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 0x7ff50c112520 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff504009ca0 tx=0x7ff50402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 -- 172.21.15.29:0/24581650 shutdown_connections 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 --2- 172.21.15.29:0/24581650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 0x7ff50c112520 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 --2- 172.21.15.29:0/24581650 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff50c076100 0x7ff50c10fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 --2- 172.21.15.29:0/24581650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff50c0757b0 0x7ff50c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 -- 172.21.15.29:0/24581650 >> 172.21.15.29:0/24581650 conn(0x7ff50c0ff590 msgr2=0x7ff50c1019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 -- 172.21.15.29:0/24581650 shutdown_connections 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.167+0000 7ff513be5640 1 -- 172.21.15.29:0/24581650 wait complete. 2024-09-21T02:58:27.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.168+0000 7ff513be5640 1 Processor -- start 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.168+0000 7ff513be5640 1 -- start start 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff513be5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff50c0757b0 0x7ff50c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff513be5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff50c076100 0x7ff50c19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff513be5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 0x7ff50c1a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff513be5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c115b90 con 0x7ff50c1100e0 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff513be5640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c115a10 con 0x7ff50c0757b0 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff513be5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c115d10 con 0x7ff50c076100 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 0x7ff50c1a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 0x7ff50c1a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48972/0 (socket says 172.21.15.29:48972) 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 -- 172.21.15.29:0/1457258391 learned_addr learned my addr 172.21.15.29:0/1457258391 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff511180640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff50c076100 0x7ff50c19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff511981640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff50c0757b0 0x7ff50c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 -- 172.21.15.29:0/1457258391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff50c076100 msgr2=0x7ff50c19cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff50c076100 0x7ff50c19cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 -- 172.21.15.29:0/1457258391 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff50c0757b0 msgr2=0x7ff50c19c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff50c0757b0 0x7ff50c19c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 -- 172.21.15.29:0/1457258391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50c1a2530 con 0x7ff50c1100e0 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.169+0000 7ff512182640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 0x7ff50c1a1e00 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff504005cc0 tx=0x7ff504032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.170+0000 7ff502ffd640 1 -- 172.21.15.29:0/1457258391 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504032e00 con 0x7ff50c1100e0 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.170+0000 7ff513be5640 1 -- 172.21.15.29:0/1457258391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50c1a27c0 con 0x7ff50c1100e0 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.170+0000 7ff502ffd640 1 -- 172.21.15.29:0/1457258391 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5040044d0 con 0x7ff50c1100e0 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.170+0000 7ff502ffd640 1 -- 172.21.15.29:0/1457258391 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504030b70 con 0x7ff50c1100e0 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.170+0000 7ff513be5640 1 -- 172.21.15.29:0/1457258391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff50c1a8f80 con 0x7ff50c1100e0 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.172+0000 7ff51a159740 1 Processor -- start 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.172+0000 7ff502ffd640 1 -- 172.21.15.29:0/1457258391 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff504004680 con 0x7ff50c1100e0 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff51a159740 1 -- start start 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff51a159740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf7c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff51a159740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf7ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff51a159740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf7cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff51a159740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5608ece67dc0 con 0x5608ecbb5840 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff51a159740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5608ecef41f0 con 0x5608ecbad660 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff51a159740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5608ecee3620 con 0x5608ecbd6520 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff511180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf7cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff511981640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf7ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff511180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf7cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48976/0 (socket says 172.21.15.29:48976) 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.173+0000 7ff511180640 1 -- 172.21.15.29:0/5463189 learned_addr learned my addr 172.21.15.29:0/5463189 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff502ffd640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4f80b6510 0x7ff4f80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff511180640 1 --2- 172.21.15.29:0/5463189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf7c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff511981640 1 -- 172.21.15.29:0/5463189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 msgr2=0x5608ecf7c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff511981640 1 --2- 172.21.15.29:0/5463189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf7c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff511981640 1 -- 172.21.15.29:0/5463189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 msgr2=0x5608ecf7cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff511981640 1 --2- 172.21.15.29:0/5463189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf7cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff511981640 1 -- 172.21.15.29:0/5463189 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608eceabfd0 con 0x5608ecbad660 2024-09-21T02:58:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff512182640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4f80b6510 0x7ff4f80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff502ffd640 1 -- 172.21.15.29:0/1457258391 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff504108900 con 0x7ff50c1100e0 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff511981640 1 --2- 172.21.15.29:0/5463189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf7ca50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff50800bbe0 tx=0x7ff50800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95cbc4381251c562 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff4f7fff640 1 -- 172.21.15.29:0/5463189 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508015070 con 0x5608ecbad660 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff512182640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4f80b6510 0x7ff4f80b8a20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff5040cace0 tx=0x7ff50402a8c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff4f7fff640 1 -- 172.21.15.29:0/5463189 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5080054b0 con 0x5608ecbad660 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.174+0000 7ff4f7fff640 1 -- 172.21.15.29:0/5463189 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508004e60 con 0x5608ecbad660 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 -- 172.21.15.29:0/5463189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 msgr2=0x5608ecf7ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 --2- 172.21.15.29:0/5463189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf7ca50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff50800bbe0 tx=0x7ff50800f730 comp rx=0 tx=0).stop 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 -- 172.21.15.29:0/5463189 shutdown_connections 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 --2- 172.21.15.29:0/5463189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf7cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 --2- 172.21.15.29:0/5463189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf7ca50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 --2- 172.21.15.29:0/5463189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf7c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 -- 172.21.15.29:0/5463189 >> 172.21.15.29:0/5463189 conn(0x5608ecbca910 msgr2=0x5608ecf7a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 -- 172.21.15.29:0/5463189 shutdown_connections 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.175+0000 7ff51a159740 1 -- 172.21.15.29:0/5463189 wait complete. 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 1 -- start start 2024-09-21T02:58:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 1 Processor -- start 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf87c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf88160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf93e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5608ece7e300 con 0x5608ecbb5840 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5608ecee7430 con 0x5608ecbad660 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5608ecd44bb0 con 0x5608ecbd6520 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.176+0000 7ff51a159740 10 client.0 _ll_register_callbacks cb 0x5608ece98120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf88160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf88160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48984/0 (socket says 172.21.15.29:48984) 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 -- 172.21.15.29:0/1763627053 learned_addr learned my addr 172.21.15.29:0/1763627053 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511180640 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf87c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511180640 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf93e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 -- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 msgr2=0x5608ecf93e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf93e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 -- 172.21.15.29:0/1763627053 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 msgr2=0x5608ecf87c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf87c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 -- 172.21.15.29:0/1763627053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608ece6d780 con 0x5608ecbb5840 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511180640 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf93e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff511981640 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf88160 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff50800bf30 tx=0x7ff50800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87d91f93601fb1e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff4f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff4f57fa640 1 -- 172.21.15.29:0/1763627053 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508028060 con 0x5608ecbb5840 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff4f57fa640 1 -- 172.21.15.29:0/1763627053 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff508004be0 con 0x5608ecbb5840 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff51a159740 1 -- 172.21.15.29:0/1763627053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5080054b0 con 0x5608ecbb5840 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.177+0000 7ff4f57fa640 1 -- 172.21.15.29:0/1763627053 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff50802f050 con 0x5608ecbb5840 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff51a159740 10 client.18045 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff51a159740 1 -- 172.21.15.29:0/1763627053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5608ecc41b00 con 0x5608ecbb5840 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff4f57fa640 1 -- 172.21.15.29:0/1763627053 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5080054b0 con 0x5608ecbb5840 2024-09-21T02:58:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff4f57fa640 1 -- 172.21.15.29:0/1763627053 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff5080118b0 con 0x5608ecbb5840 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff51a159740 10 client.18045 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff51a159740 10 client.18045 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff51a159740 20 client.18045 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff51a159740 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608ecb94ea0 0x5608ecf99e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff51a159740 4 client.18045 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff51a159740 1 -- 172.21.15.29:0/1763627053 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608ecef6140 con 0x5608ecb94ea0 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.178+0000 7ff511180640 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608ecb94ea0 0x5608ecf99e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.179+0000 7ff4f57fa640 10 client.18045 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.179+0000 7ff511180640 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608ecb94ea0 0x5608ecf99e00 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faa84a6892bccef4 server_cookie=319e53e55cb36709 in_seq=0 out_seq=0 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff4f57fa640 1 -- 172.21.15.29:0/1763627053 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5608ecef6140 con 0x5608ecb94ea0 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff4f57fa640 10 client.18045 handle_command_reply: tid=0 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 client.18045 shutdown 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 -- 172.21.15.29:0/1763627053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 msgr2=0x5608ecf88160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf88160 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff50800bf30 tx=0x7ff50800bf60 comp rx=0 tx=0).stop 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 -- 172.21.15.29:0/1763627053 shutdown_connections 2024-09-21T02:58:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf93e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608ecb94ea0 0x5608ecf99e00 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbb5840 0x5608ecf88160 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 --2- 172.21.15.29:0/1763627053 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbad660 0x5608ecf87c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 -- 172.21.15.29:0/1763627053 >> 172.21.15.29:0/1763627053 conn(0x5608ecbca910 msgr2=0x5608ecf7f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff4f57fa640 0 client.18045 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 -- 172.21.15.29:0/1763627053 shutdown_connections 2024-09-21T02:58:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 1 -- 172.21.15.29:0/1763627053 wait complete. 2024-09-21T02:58:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.180+0000 7ff51a159740 20 client.18045 trim_cache size 0 max 16384 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff51a159740 1 Processor -- start 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff51a159740 1 -- start start 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff51a159740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf83350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff51a159740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbb5840 0x5608ecf7df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff51a159740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbd6520 0x5608ecf864a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff51a159740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c1b2b40 con 0x5608ecbad660 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff51a159740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c1b4680 con 0x5608ecbd6520 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff51a159740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c1b3210 con 0x5608ecbb5840 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf83350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf83350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48988/0 (socket says 172.21.15.29:48988) 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511180640 1 -- 172.21.15.29:0/2271798733 learned_addr learned my addr 172.21.15.29:0/2271798733 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511981640 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbb5840 0x5608ecf7df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511180640 1 -- 172.21.15.29:0/2271798733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbb5840 msgr2=0x5608ecf7df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511180640 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbb5840 0x5608ecf7df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511180640 1 -- 172.21.15.29:0/2271798733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbd6520 msgr2=0x5608ecf864a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511180640 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbd6520 0x5608ecf864a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511180640 1 -- 172.21.15.29:0/2271798733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608ecf81d00 con 0x5608ecbad660 2024-09-21T02:58:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.198+0000 7ff511981640 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbb5840 0x5608ecf7df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:58:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.199+0000 7ff511180640 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf83350 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b620 tx=0x7ff4fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60520e875528f311 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.199+0000 7ff4f77fe640 1 -- 172.21.15.29:0/2271798733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc005020 con 0x5608ecbad660 2024-09-21T02:58:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.199+0000 7ff4f77fe640 1 -- 172.21.15.29:0/2271798733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4fc0047a0 con 0x5608ecbad660 2024-09-21T02:58:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.199+0000 7ff4f77fe640 1 -- 172.21.15.29:0/2271798733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc005650 con 0x5608ecbad660 2024-09-21T02:58:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 -- 172.21.15.29:0/2271798733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 msgr2=0x5608ecf83350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf83350 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b620 tx=0x7ff4fc0046c0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 -- 172.21.15.29:0/2271798733 shutdown_connections 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbd6520 0x5608ecf864a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbb5840 0x5608ecf7df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 --2- 172.21.15.29:0/2271798733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf83350 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 -- 172.21.15.29:0/2271798733 >> 172.21.15.29:0/2271798733 conn(0x5608ecbca910 msgr2=0x5608ecf879d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 -- 172.21.15.29:0/2271798733 shutdown_connections 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 -- 172.21.15.29:0/2271798733 wait complete. 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 -- start start 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.200+0000 7ff51a159740 1 Processor -- start 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff51a159740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf9d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff51a159740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbb5840 0x5608ecf9b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff51a159740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf9bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff51a159740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c1b3fb0 con 0x5608ecbad660 2024-09-21T02:58:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff51a159740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c1b38e0 con 0x5608ecbb5840 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff51a159740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c1b0b20 con 0x5608ecbd6520 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff51a159740 10 client.0 _ll_register_callbacks cb 0x7ff4fc00b6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf9d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf9d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49002/0 (socket says 172.21.15.29:49002) 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 -- 172.21.15.29:0/2837891075 learned_addr learned my addr 172.21.15.29:0/2837891075 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511981640 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbb5840 0x5608ecf9b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf9bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 -- 172.21.15.29:0/2837891075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 msgr2=0x5608ecf9bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf9bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 -- 172.21.15.29:0/2837891075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbb5840 msgr2=0x5608ecf9b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbb5840 0x5608ecf9b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 -- 172.21.15.29:0/2837891075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608eccb8960 con 0x5608ecbad660 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff511180640 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf9d4f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b3d0 tx=0x7ff4fc003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cc8e1243d5e87b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff4f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.201+0000 7ff4f57fa640 1 -- 172.21.15.29:0/2837891075 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0052a0 con 0x5608ecbad660 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff4f57fa640 1 -- 172.21.15.29:0/2837891075 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff4fc005440 con 0x5608ecbad660 2024-09-21T02:58:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 1 -- 172.21.15.29:0/2837891075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4fc0047a0 con 0x5608ecbad660 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff4f57fa640 1 -- 172.21.15.29:0/2837891075 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff4fc038050 con 0x5608ecbad660 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 10 client.18051 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 1 -- 172.21.15.29:0/2837891075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5608ed0051a0 con 0x5608ecbad660 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff4f57fa640 1 -- 172.21.15.29:0/2837891075 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4fc0047a0 con 0x5608ecbad660 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff4f57fa640 1 -- 172.21.15.29:0/2837891075 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff4fc020470 con 0x5608ecbad660 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 10 client.18051 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 10 client.18051 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 20 client.18051 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608ecb6a250 0x5608ed0077e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 4 client.18051 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7776ac5f", "release": true, "await": true}] 2024-09-21T02:58:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff51a159740 1 -- 172.21.15.29:0/2837891075 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7776ac5f", "release": true, "await": true}) -- 0x5608ecef1bf0 con 0x5608ecb6a250 2024-09-21T02:58:27.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.202+0000 7ff511981640 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608ecb6a250 0x5608ed0077e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:27.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.203+0000 7ff4f57fa640 10 client.18051 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:27.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.203+0000 7ff511981640 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608ecb6a250 0x5608ed0077e0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb09e66341c4a25f server_cookie=bed76ea6e7e84238 in_seq=0 out_seq=0 2024-09-21T02:58:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.284+0000 7ff4f57fa640 1 -- 172.21.15.29:0/2837891075 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5608ecef1bf0 con 0x5608ecb6a250 2024-09-21T02:58:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.284+0000 7ff4f57fa640 10 client.18051 handle_command_reply: tid=0 2024-09-21T02:58:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.284+0000 7ff51a159740 1 client.18051 shutdown 2024-09-21T02:58:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 -- 172.21.15.29:0/2837891075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 msgr2=0x5608ecf9d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf9d4f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b3d0 tx=0x7ff4fc003f90 comp rx=0 tx=0).stop 2024-09-21T02:58:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 -- 172.21.15.29:0/2837891075 shutdown_connections 2024-09-21T02:58:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5608ecbd6520 0x5608ecf9bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5608ecbb5840 0x5608ecf9b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5608ecb6a250 0x5608ed0077e0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 --2- 172.21.15.29:0/2837891075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608ecbad660 0x5608ecf9d4f0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 -- 172.21.15.29:0/2837891075 >> 172.21.15.29:0/2837891075 conn(0x5608ecbca910 msgr2=0x5608ecf879d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff4f57fa640 0 client.18051 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 -- 172.21.15.29:0/2837891075 shutdown_connections 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.285+0000 7ff51a159740 1 -- 172.21.15.29:0/2837891075 wait complete. 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.286+0000 7ff51a159740 20 client.18051 trim_cache size 0 max 16384 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 76, 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout: "7776ac5f": { 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 76, 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:58:27.288 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.2, 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:27.289 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:58:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.286+0000 7ff513be5640 1 -- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4f80b6510 msgr2=0x7ff4f80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.286+0000 7ff513be5640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4f80b6510 0x7ff4f80b8a20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff5040cace0 tx=0x7ff50402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.286+0000 7ff513be5640 1 -- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 msgr2=0x7ff50c1a1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.286+0000 7ff513be5640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 0x7ff50c1a1e00 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff504005cc0 tx=0x7ff504032a10 comp rx=0 tx=0).stop 2024-09-21T02:58:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.286+0000 7ff513be5640 1 -- 172.21.15.29:0/1457258391 shutdown_connections 2024-09-21T02:58:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.287+0000 7ff513be5640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff4f80b6510 0x7ff4f80b8a20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.287+0000 7ff513be5640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c1100e0 0x7ff50c1a1e00 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.287+0000 7ff513be5640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff50c076100 0x7ff50c19cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.287+0000 7ff513be5640 1 --2- 172.21.15.29:0/1457258391 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff50c0757b0 0x7ff50c19c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.287+0000 7ff513be5640 1 -- 172.21.15.29:0/1457258391 >> 172.21.15.29:0/1457258391 conn(0x7ff50c0ff590 msgr2=0x7ff50c1122d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.287+0000 7ff513be5640 1 -- 172.21.15.29:0/1457258391 shutdown_connections 2024-09-21T02:58:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:27.287+0000 7ff513be5640 1 -- 172.21.15.29:0/1457258391 wait complete. 2024-09-21T02:58:27.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37884 5.51 MB/sec execute 244 sec latency 21003.409 ms 2024-09-21T02:58:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:27 smithi066 ceph-mon[33436]: pgmap v969: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 KiB/s wr, 17 op/s 2024-09-21T02:58:28.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:27 smithi029 ceph-mon[31006]: pgmap v969: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 KiB/s wr, 17 op/s 2024-09-21T02:58:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:27 smithi190 ceph-mon[33871]: pgmap v969: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 KiB/s wr, 17 op/s 2024-09-21T02:58:28.154 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7776ac5f', total seconds elapsed: 26.0 2024-09-21T02:58:28.154 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.15 2024-09-21T02:58:28.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:58:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:58:28.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38025 5.54 MB/sec execute 245 sec latency 90.995 ms 2024-09-21T02:58:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.048+0000 7f8701c8b640 1 -- 172.21.15.29:0/3179238203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc109500 msgr2=0x7f86fc109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.048+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3179238203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc109500 0x7f86fc109910 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009ca0 tx=0x7f86f002a8b0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.048+0000 7f8701c8b640 1 -- 172.21.15.29:0/3179238203 shutdown_connections 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.048+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3179238203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86fc109e50 0x7f86fc110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.048+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3179238203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc109500 0x7f86fc109910 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.048+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3179238203 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86fc101fc0 0x7f86fc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.048+0000 7f8701c8b640 1 -- 172.21.15.29:0/3179238203 >> 172.21.15.29:0/3179238203 conn(0x7f86fc079f90 msgr2=0x7f86fc07a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.049+0000 7f8701c8b640 1 -- 172.21.15.29:0/3179238203 shutdown_connections 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.049+0000 7f8701c8b640 1 -- 172.21.15.29:0/3179238203 wait complete. 2024-09-21T02:58:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.049+0000 7f8701c8b640 1 Processor -- start 2024-09-21T02:58:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f8701c8b640 1 -- start start 2024-09-21T02:58:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f8701c8b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc101fc0 0x7f86fc1a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f8701c8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86fc109500 0x7f86fc1a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f8701c8b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86fc109e50 0x7f86fc1ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f8701c8b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc1139e0 con 0x7f86fc109500 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f8701c8b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc113860 con 0x7f86fc109e50 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f8701c8b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc113b60 con 0x7f86fc101fc0 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f86fb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc101fc0 0x7f86fc1a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f86fbfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86fc109e50 0x7f86fc1ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f86fb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc101fc0 0x7f86fc1a16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44218/0 (socket says 172.21.15.29:44218) 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f86faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86fc109500 0x7f86fc1a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.050+0000 7f86fb7fe640 1 -- 172.21.15.29:0/3008441863 learned_addr learned my addr 172.21.15.29:0/3008441863 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.051+0000 7f86fb7fe640 1 -- 172.21.15.29:0/3008441863 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86fc109e50 msgr2=0x7f86fc1ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.051+0000 7f86fb7fe640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86fc109e50 0x7f86fc1ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.051+0000 7f86fb7fe640 1 -- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86fc109500 msgr2=0x7f86fc1a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.051+0000 7f86fb7fe640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86fc109500 0x7f86fc1a1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.051+0000 7f86fb7fe640 1 -- 172.21.15.29:0/3008441863 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc1acc80 con 0x7f86fc101fc0 2024-09-21T02:58:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.051+0000 7f86faffd640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86fc109500 0x7f86fc1a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:58:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.051+0000 7f86fb7fe640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc101fc0 0x7f86fc1a16c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00b4a0 tx=0x7f86ec00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.051+0000 7f86f8ff9640 1 -- 172.21.15.29:0/3008441863 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec014070 con 0x7f86fc101fc0 2024-09-21T02:58:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.052+0000 7f86f8ff9640 1 -- 172.21.15.29:0/3008441863 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86ec010aa0 con 0x7f86fc101fc0 2024-09-21T02:58:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.052+0000 7f86f8ff9640 1 -- 172.21.15.29:0/3008441863 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec017b70 con 0x7f86fc101fc0 2024-09-21T02:58:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.052+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86fc1ace50 con 0x7f86fc101fc0 2024-09-21T02:58:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.052+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86fc1ad330 con 0x7f86fc101fc0 2024-09-21T02:58:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.053+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86fc1023d0 con 0x7f86fc101fc0 2024-09-21T02:58:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.055+0000 7f86f8ff9640 1 -- 172.21.15.29:0/3008441863 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f86ec017070 con 0x7f86fc101fc0 2024-09-21T02:58:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.056+0000 7f86f8ff9640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f86e40b65c0 0x7f86e40b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.057+0000 7f86f8ff9640 1 -- 172.21.15.29:0/3008441863 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f86ec0e6750 con 0x7f86fc101fc0 2024-09-21T02:58:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.057+0000 7f86faffd640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f86e40b65c0 0x7f86e40b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.057+0000 7f86faffd640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f86e40b65c0 0x7f86e40b8ad0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f86f0000f80 tx=0x7f86f0030f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.059+0000 7f86f8ff9640 1 -- 172.21.15.29:0/3008441863 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f86ec0b1060 con 0x7f86fc101fc0 2024-09-21T02:58:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.405+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86fc10fe40 con 0x7f86fc101fc0 2024-09-21T02:58:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:58:29.409 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:58:29.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.406+0000 7f86f8ff9640 1 -- 172.21.15.29:0/3008441863 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f86fc10fe40 con 0x7f86fc101fc0 2024-09-21T02:58:29.409 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:58:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.412+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f86e40b65c0 msgr2=0x7f86e40b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.412+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f86e40b65c0 0x7f86e40b8ad0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f86f0000f80 tx=0x7f86f0030f00 comp rx=0 tx=0).stop 2024-09-21T02:58:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.412+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc101fc0 msgr2=0x7f86fc1a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.412+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc101fc0 0x7f86fc1a16c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00b4a0 tx=0x7f86ec00b960 comp rx=0 tx=0).stop 2024-09-21T02:58:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.413+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 shutdown_connections 2024-09-21T02:58:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.413+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f86e40b65c0 0x7f86e40b8ad0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.413+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86fc109e50 0x7f86fc1ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.413+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86fc109500 0x7f86fc1a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.413+0000 7f8701c8b640 1 --2- 172.21.15.29:0/3008441863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc101fc0 0x7f86fc1a16c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:29.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.413+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 >> 172.21.15.29:0/3008441863 conn(0x7f86fc079f90 msgr2=0x7f86fc107310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:29.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.414+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 shutdown_connections 2024-09-21T02:58:29.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:29.414+0000 7f8701c8b640 1 -- 172.21.15.29:0/3008441863 wait complete. 2024-09-21T02:58:29.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38163 5.55 MB/sec execute 246 sec latency 86.460 ms 2024-09-21T02:58:30.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:29 smithi066 ceph-mon[33436]: pgmap v970: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 30 KiB/s wr, 22 op/s 2024-09-21T02:58:30.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:29 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3008441863' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:29 smithi190 ceph-mon[33871]: pgmap v970: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 30 KiB/s wr, 22 op/s 2024-09-21T02:58:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:29 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3008441863' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:29 smithi029 ceph-mon[31006]: pgmap v970: 129 pgs: 129 active+clean; 284 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 30 KiB/s wr, 22 op/s 2024-09-21T02:58:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:29 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3008441863' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:30.263 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 142.15 --await-for 45.0' 2024-09-21T02:58:30.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 142.15 --await-for 45.0 2024-09-21T02:58:30.626 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:58:30.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38318 5.56 MB/sec execute 247 sec latency 120.023 ms 2024-09-21T02:58:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 -- 172.21.15.29:0/1676135767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d01083b0 msgr2=0x7f48d01087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 --2- 172.21.15.29:0/1676135767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d01083b0 0x7f48d01087c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009900 tx=0x7f48b802a3f0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 -- 172.21.15.29:0/1676135767 shutdown_connections 2024-09-21T02:58:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 --2- 172.21.15.29:0/1676135767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48d0108d00 0x7f48d010f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 --2- 172.21.15.29:0/1676135767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d01083b0 0x7f48d01087c0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 --2- 172.21.15.29:0/1676135767 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48d0102a70 0x7f48d0102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 -- 172.21.15.29:0/1676135767 >> 172.21.15.29:0/1676135767 conn(0x7f48d0079830 msgr2=0x7f48d0079c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 -- 172.21.15.29:0/1676135767 shutdown_connections 2024-09-21T02:58:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.228+0000 7f48d640d640 1 -- 172.21.15.29:0/1676135767 wait complete. 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d640d640 1 Processor -- start 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d640d640 1 -- start start 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d640d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48d0102a70 0x7f48d01134a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d640d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d01083b0 0x7f48d0113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d640d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48d0108d00 0x7f48d011de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d640d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f48d0111720 con 0x7f48d01083b0 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d640d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01115a0 con 0x7f48d0102a70 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d640d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01118a0 con 0x7f48d0108d00 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48cffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48d0102a70 0x7f48d01134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48cffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48d0102a70 0x7f48d01134a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:41382/0 (socket says 172.21.15.29:41382) 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48cffff640 1 -- 172.21.15.29:0/1403543834 learned_addr learned my addr 172.21.15.29:0/1403543834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48c77fe640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d01083b0 0x7f48d0113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.231+0000 7f48d49aa640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48d0108d00 0x7f48d011de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cffff640 1 -- 172.21.15.29:0/1403543834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48d0108d00 msgr2=0x7f48d011de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cffff640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48d0108d00 0x7f48d011de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cffff640 1 -- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d01083b0 msgr2=0x7f48d0113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cffff640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d01083b0 0x7f48d0113a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cffff640 1 -- 172.21.15.29:0/1403543834 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d011e5a0 con 0x7f48d0102a70 2024-09-21T02:58:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cffff640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48d0102a70 0x7f48d01134a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f48c000ebc0 tx=0x7f48c000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cdffb640 1 -- 172.21.15.29:0/1403543834 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0010630 con 0x7f48d0102a70 2024-09-21T02:58:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cdffb640 1 -- 172.21.15.29:0/1403543834 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48c0010c60 con 0x7f48d0102a70 2024-09-21T02:58:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48cdffb640 1 -- 172.21.15.29:0/1403543834 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0018b00 con 0x7f48d0102a70 2024-09-21T02:58:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.232+0000 7f48d640d640 1 -- 172.21.15.29:0/1403543834 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d011e830 con 0x7f48d0102a70 2024-09-21T02:58:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.233+0000 7f48d640d640 1 -- 172.21.15.29:0/1403543834 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d0118ce0 con 0x7f48d0102a70 2024-09-21T02:58:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.236+0000 7f48cdffb640 1 -- 172.21.15.29:0/1403543834 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f48c0018ca0 con 0x7f48d0102a70 2024-09-21T02:58:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.237+0000 7f48cdffb640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48bc0b6590 0x7f48bc0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.237+0000 7f48c77fe640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48bc0b6590 0x7f48bc0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.237+0000 7f48cdffb640 1 -- 172.21.15.29:0/1403543834 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f48c00e7c70 con 0x7f48d0102a70 2024-09-21T02:58:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.238+0000 7f48c77fe640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48bc0b6590 0x7f48bc0b8aa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f48b8005940 tx=0x7f48b80058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:31.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.243+0000 7f48dc981740 1 Processor -- start 2024-09-21T02:58:31.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.243+0000 7f48dc981740 1 -- start start 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.243+0000 7f48dc981740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0b8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.243+0000 7f48dc981740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.243+0000 7f48dc981740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 0x55f00a0b98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.243+0000 7f48dc981740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f00a02fbb0 con 0x55f009d12850 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48dc981740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f00a01efe0 con 0x55f009ce9990 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48dc981740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f00a022df0 con 0x55f009cf1b70 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0b8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0b8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49032/0 (socket says 172.21.15.29:49032) 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48cffff640 1 -- 172.21.15.29:0/901115197 learned_addr learned my addr 172.21.15.29:0/901115197 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48c77fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48d49aa640 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 0x55f00a0b98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48c77fe640 1 -- 172.21.15.29:0/901115197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 msgr2=0x55f00a0b98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:31.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48c77fe640 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 0x55f00a0b98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48c77fe640 1 -- 172.21.15.29:0/901115197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 msgr2=0x55f00a0b8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:31.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48c77fe640 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0b8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48c77fe640 1 -- 172.21.15.29:0/901115197 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f009d8cba0 con 0x55f009ce9990 2024-09-21T02:58:31.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.244+0000 7f48cffff640 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0b8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:58:31.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.245+0000 7f48c77fe640 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0b93b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f48b802a910 tx=0x7f48b803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f65a5ba1f30631a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.245+0000 7f48c67fc640 1 -- 172.21.15.29:0/901115197 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b804c070 con 0x55f009ce9990 2024-09-21T02:58:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.245+0000 7f48c67fc640 1 -- 172.21.15.29:0/901115197 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48b80035f0 con 0x55f009ce9990 2024-09-21T02:58:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.245+0000 7f48c67fc640 1 -- 172.21.15.29:0/901115197 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8035040 con 0x55f009ce9990 2024-09-21T02:58:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.246+0000 7f48dc981740 1 -- 172.21.15.29:0/901115197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 msgr2=0x55f00a0b93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.246+0000 7f48dc981740 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0b93b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f48b802a910 tx=0x7f48b803d000 comp rx=0 tx=0).stop 2024-09-21T02:58:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.246+0000 7f48dc981740 1 -- 172.21.15.29:0/901115197 shutdown_connections 2024-09-21T02:58:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.246+0000 7f48dc981740 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 0x55f00a0b98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.246+0000 7f48dc981740 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0b93b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.247+0000 7f48dc981740 1 --2- 172.21.15.29:0/901115197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0b8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.247+0000 7f48dc981740 1 -- 172.21.15.29:0/901115197 >> 172.21.15.29:0/901115197 conn(0x55f009d06c40 msgr2=0x55f00a0b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.247+0000 7f48dc981740 1 -- 172.21.15.29:0/901115197 shutdown_connections 2024-09-21T02:58:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.247+0000 7f48dc981740 1 -- 172.21.15.29:0/901115197 wait complete. 2024-09-21T02:58:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.248+0000 7f48dc981740 1 -- start start 2024-09-21T02:58:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.248+0000 7f48dc981740 1 Processor -- start 2024-09-21T02:58:31.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48dc981740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48dc981740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 0x55f00a0c5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48dc981740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48cffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48cffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:41402/0 (socket says 172.21.15.29:41402) 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48cffff640 1 -- 172.21.15.29:0/4216751521 learned_addr learned my addr 172.21.15.29:0/4216751521 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48c77fe640 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 0x55f00a0c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f009e807b0 con 0x55f009d12850 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.249+0000 7f48d49aa640 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f009fb9e90 con 0x55f009ce9990 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f00a0c75f0 con 0x55f009cf1b70 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48dc981740 10 client.0 _ll_register_callbacks cb 0x55f00a01c710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48cffff640 1 -- 172.21.15.29:0/4216751521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 msgr2=0x55f00a0c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48cffff640 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 0x55f00a0c5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48cffff640 1 -- 172.21.15.29:0/4216751521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 msgr2=0x55f00a0ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48cffff640 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48cffff640 1 -- 172.21.15.29:0/4216751521 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f00a0caef0 con 0x55f009ce9990 2024-09-21T02:58:31.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b80035f0 con 0x55f009ce9990 2024-09-21T02:58:31.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48cffff640 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0c4b60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f48c000eb90 tx=0x7f48c00ad270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b82601a52dd6f46c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:31.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48af7fe640 10 client.27389 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:58:31.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.250+0000 7f48af7fe640 1 -- 172.21.15.29:0/4216751521 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c00ad590 con 0x55f009ce9990 2024-09-21T02:58:31.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48af7fe640 1 -- 172.21.15.29:0/4216751521 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f48c00b2180 con 0x55f009ce9990 2024-09-21T02:58:31.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48af7fe640 1 -- 172.21.15.29:0/4216751521 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f48c00ae180 con 0x55f009ce9990 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48dc981740 10 client.27389 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({fsmap=37}) -- 0x55f00a0d37c0 con 0x55f009ce9990 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48af7fe640 1 -- 172.21.15.29:0/4216751521 <== mon.2 v2:172.21.15.66:3300/0 4 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f48c00a93d0 con 0x55f009ce9990 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48dc981740 10 client.27389 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48dc981740 10 client.27389 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48dc981740 20 client.27389 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48dc981740 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f009cd11d0 0x55f00a0d6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48dc981740 4 client.27389 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.251+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f00a031b00 con 0x55f009cd11d0 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.252+0000 7f48c77fe640 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f009cd11d0 0x55f00a0d6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.252+0000 7f48af7fe640 10 client.27389 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.252+0000 7f48c77fe640 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f009cd11d0 0x55f00a0d6c10 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=195b8aaa2c6bb073 server_cookie=f681fa93c4cf34e in_seq=0 out_seq=0 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48af7fe640 1 -- 172.21.15.29:0/4216751521 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f00a031b00 con 0x55f009cd11d0 2024-09-21T02:58:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48af7fe640 10 client.27389 handle_command_reply: tid=0 2024-09-21T02:58:31.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 client.27389 shutdown 2024-09-21T02:58:31.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 msgr2=0x55f00a0c4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0c4b60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f48c000eb90 tx=0x7f48c00ad270 comp rx=0 tx=0).stop 2024-09-21T02:58:31.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 shutdown_connections 2024-09-21T02:58:31.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f009cd11d0 0x55f00a0d6c10 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009cf1b70 0x55f00a0c5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 --2- 172.21.15.29:0/4216751521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0c4b60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 >> 172.21.15.29:0/4216751521 conn(0x55f009d06c40 msgr2=0x55f00a0bc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:31.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.253+0000 7f48af7fe640 0 client.27389 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:31.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.254+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 shutdown_connections 2024-09-21T02:58:31.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.254+0000 7f48dc981740 1 -- 172.21.15.29:0/4216751521 wait complete. 2024-09-21T02:58:31.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.254+0000 7f48dc981740 20 client.27389 trim_cache size 0 max 16384 2024-09-21T02:58:31.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.282+0000 7f48dc981740 1 Processor -- start 2024-09-21T02:58:31.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.282+0000 7f48dc981740 1 -- start start 2024-09-21T02:58:31.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.282+0000 7f48dc981740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009ce9990 0x55f00a0c6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.282+0000 7f48dc981740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009cf1b70 0x55f00a0c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.282+0000 7f48dc981740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0bdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.282+0000 7f48dc981740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01bf820 con 0x55f009d12850 2024-09-21T02:58:31.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.282+0000 7f48dc981740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01c1360 con 0x55f009cf1b70 2024-09-21T02:58:31.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.282+0000 7f48dc981740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01bfef0 con 0x55f009ce9990 2024-09-21T02:58:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0bdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48cffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009ce9990 0x55f00a0c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0bdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49052/0 (socket says 172.21.15.29:49052) 2024-09-21T02:58:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 -- 172.21.15.29:0/4224877740 learned_addr learned my addr 172.21.15.29:0/4224877740 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48c77fe640 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009cf1b70 0x55f00a0c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 -- 172.21.15.29:0/4224877740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009ce9990 msgr2=0x55f00a0c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009ce9990 0x55f00a0c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 -- 172.21.15.29:0/4224877740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009cf1b70 msgr2=0x55f00a0c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:31.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009cf1b70 0x55f00a0c41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 -- 172.21.15.29:0/4224877740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f00a0c5c20 con 0x55f009d12850 2024-09-21T02:58:31.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48c77fe640 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009cf1b70 0x55f00a0c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:58:31.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.283+0000 7f48d49aa640 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0bdf50 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009e70 tx=0x7f48c8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0395fb24cb77c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:31.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.284+0000 7f48c5ffb640 1 -- 172.21.15.29:0/4224877740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8004020 con 0x55f009d12850 2024-09-21T02:58:31.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.284+0000 7f48c5ffb640 1 -- 172.21.15.29:0/4224877740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48c80041c0 con 0x55f009d12850 2024-09-21T02:58:31.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.284+0000 7f48c5ffb640 1 -- 172.21.15.29:0/4224877740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c800f6a0 con 0x55f009d12850 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.285+0000 7f48dc981740 1 -- 172.21.15.29:0/4224877740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 msgr2=0x55f00a0bdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.285+0000 7f48dc981740 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0bdf50 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009e70 tx=0x7f48c8005f90 comp rx=0 tx=0).stop 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.285+0000 7f48dc981740 1 -- 172.21.15.29:0/4224877740 shutdown_connections 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.285+0000 7f48dc981740 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009d12850 0x55f00a0bdf50 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.285+0000 7f48dc981740 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009cf1b70 0x55f00a0c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.285+0000 7f48dc981740 1 --2- 172.21.15.29:0/4224877740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009ce9990 0x55f00a0c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.285+0000 7f48dc981740 1 -- 172.21.15.29:0/4224877740 >> 172.21.15.29:0/4224877740 conn(0x55f009d06c40 msgr2=0x55f00a0b6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.285+0000 7f48dc981740 1 -- 172.21.15.29:0/4224877740 shutdown_connections 2024-09-21T02:58:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.286+0000 7f48dc981740 1 -- 172.21.15.29:0/4224877740 wait complete. 2024-09-21T02:58:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.286+0000 7f48dc981740 1 -- start start 2024-09-21T02:58:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.286+0000 7f48dc981740 1 Processor -- start 2024-09-21T02:58:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48dc981740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48dc981740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009cf1b70 0x55f00a0ce910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48dc981740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009d12850 0x55f00a0cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48cffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009cf1b70 0x55f00a0ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48cffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0d0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:41418/0 (socket says 172.21.15.29:41418) 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009cf1b70 0x55f00a0ce910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49064/0 (socket says 172.21.15.29:49064) 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48cffff640 1 -- 172.21.15.29:0/3555514119 learned_addr learned my addr 172.21.15.29:0/3555514119 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01c0c90 con 0x55f009cf1b70 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48d49aa640 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009d12850 0x55f00a0cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01c05c0 con 0x55f009ce9990 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01bd800 con 0x55f009d12850 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.287+0000 7f48dc981740 10 client.0 _ll_register_callbacks cb 0x7f48c00ad4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48cffff640 1 -- 172.21.15.29:0/3555514119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009d12850 msgr2=0x55f00a0cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48cffff640 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009d12850 0x55f00a0cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48cffff640 1 -- 172.21.15.29:0/3555514119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009cf1b70 msgr2=0x55f00a0ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48cffff640 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009cf1b70 0x55f00a0ce910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48cffff640 1 -- 172.21.15.29:0/3555514119 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f00a0cf390 con 0x55f009ce9990 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48cffff640 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0d0ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f48c00a98c0 tx=0x7f48c001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43daf574a8e7a26 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:58:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48af7fe640 1 -- 172.21.15.29:0/3555514119 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c00cb070 con 0x55f009ce9990 2024-09-21T02:58:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48af7fe640 1 -- 172.21.15.29:0/3555514119 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f48c00ac950 con 0x55f009ce9990 2024-09-21T02:58:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48c80041c0 con 0x55f009ce9990 2024-09-21T02:58:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.288+0000 7f48af7fe640 1 -- 172.21.15.29:0/3555514119 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f48c00ad1f0 con 0x55f009ce9990 2024-09-21T02:58:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48dc981740 10 client.27395 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f00a142200 con 0x55f009ce9990 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48af7fe640 1 -- 172.21.15.29:0/3555514119 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f48c80041c0 con 0x55f009ce9990 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48af7fe640 1 -- 172.21.15.29:0/3555514119 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f48c00b36f0 con 0x55f009ce9990 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48dc981740 10 client.27395 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48dc981740 10 client.27395 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48dc981740 20 client.27395 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48dc981740 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f009ca6580 0x55f00a144950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48dc981740 4 client.27395 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.15, "await_for": 45.0}] 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.15, "await_for": 45.0}) -- 0x55f00a02d5b0 con 0x55f009ca6580 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.289+0000 7f48c77fe640 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f009ca6580 0x55f00a144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.290+0000 7f48af7fe640 10 client.27395 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.290+0000 7f48c77fe640 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f009ca6580 0x55f00a144950 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1011a827ae5d819 server_cookie=c2576c11e9958964 in_seq=0 out_seq=0 2024-09-21T02:58:31.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38420 5.57 MB/sec execute 248 sec latency 430.225 ms 2024-09-21T02:58:31.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48af7fe640 1 -- 172.21.15.29:0/3555514119 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f00a02d5b0 con 0x55f009ca6580 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48af7fe640 10 client.27395 handle_command_reply: tid=0 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 client.27395 shutdown 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 msgr2=0x55f00a0d0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0d0ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f48c00a98c0 tx=0x7f48c001a040 comp rx=0 tx=0).stop 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 shutdown_connections 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f009d12850 0x55f00a0cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f009ca6580 0x55f00a144950 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f009cf1b70 0x55f00a0ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 --2- 172.21.15.29:0/3555514119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f009ce9990 0x55f00a0d0ea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48af7fe640 0 client.27395 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 >> 172.21.15.29:0/3555514119 conn(0x55f009d06c40 msgr2=0x55f00a0b6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 shutdown_connections 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 1 -- 172.21.15.29:0/3555514119 wait complete. 2024-09-21T02:58:31.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.817+0000 7f48dc981740 20 client.27395 trim_cache size 0 max 16384 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 78, 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "a998ab4c": { 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 78, 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:58:31.820 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.2, 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:31.821 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 -- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48bc0b6590 msgr2=0x7f48bc0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48bc0b6590 0x7f48bc0b8aa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f48b8005940 tx=0x7f48b80058d0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 -- 172.21.15.29:0/1403543834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48d0102a70 msgr2=0x7f48d01134a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48d0102a70 0x7f48d01134a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f48c000ebc0 tx=0x7f48c000c6a0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 -- 172.21.15.29:0/1403543834 shutdown_connections 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48bc0b6590 0x7f48bc0b8aa0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48d0108d00 0x7f48d011de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d01083b0 0x7f48d0113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 --2- 172.21.15.29:0/1403543834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48d0102a70 0x7f48d01134a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 -- 172.21.15.29:0/1403543834 >> 172.21.15.29:0/1403543834 conn(0x7f48d0079830 msgr2=0x7f48d01064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 -- 172.21.15.29:0/1403543834 shutdown_connections 2024-09-21T02:58:31.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:31.818+0000 7f48d640d640 1 -- 172.21.15.29:0/1403543834 wait complete. 2024-09-21T02:58:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:31 smithi066 ceph-mon[33436]: pgmap v971: 129 pgs: 129 active+clean; 286 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 277 KiB/s wr, 27 op/s 2024-09-21T02:58:32.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:31 smithi190 ceph-mon[33871]: pgmap v971: 129 pgs: 129 active+clean; 286 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 277 KiB/s wr, 27 op/s 2024-09-21T02:58:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:31 smithi029 ceph-mon[31006]: pgmap v971: 129 pgs: 129 active+clean; 286 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 277 KiB/s wr, 27 op/s 2024-09-21T02:58:32.731 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.55 MB/sec execute 249 sec latency 1430.311 ms 2024-09-21T02:58:32.855 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a998ab4c', quiesce took 4.7 seconds. Will release after: 17.45 2024-09-21T02:58:33.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.52 MB/sec execute 250 sec latency 2430.433 ms 2024-09-21T02:58:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:33 smithi190 ceph-mon[33871]: pgmap v972: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 649 KiB/s wr, 73 op/s 2024-09-21T02:58:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:33 smithi029 ceph-mon[31006]: pgmap v972: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 649 KiB/s wr, 73 op/s 2024-09-21T02:58:34.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:33 smithi066 ceph-mon[33436]: pgmap v972: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 649 KiB/s wr, 73 op/s 2024-09-21T02:58:34.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.50 MB/sec execute 251 sec latency 3430.505 ms 2024-09-21T02:58:35.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.48 MB/sec execute 252 sec latency 4430.606 ms 2024-09-21T02:58:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:35 smithi029 ceph-mon[31006]: pgmap v973: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1006 KiB/s wr, 105 op/s 2024-09-21T02:58:36.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 -- 172.21.15.29:0/2055529546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103ff0 msgr2=0x7f929410a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 --2- 172.21.15.29:0/2055529546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103ff0 0x7f929410a990 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f928c009ca0 tx=0x7f928c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 -- 172.21.15.29:0/2055529546 shutdown_connections 2024-09-21T02:58:36.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 --2- 172.21.15.29:0/2055529546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103ff0 0x7f929410a990 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 --2- 172.21.15.29:0/2055529546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9294103650 0x7f9294103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 --2- 172.21.15.29:0/2055529546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92941025d0 0x7f9294102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 -- 172.21.15.29:0/2055529546 >> 172.21.15.29:0/2055529546 conn(0x7f92940fdf30 msgr2=0x7f92941003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:36.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 -- 172.21.15.29:0/2055529546 shutdown_connections 2024-09-21T02:58:36.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.088+0000 7f929a18e640 1 -- 172.21.15.29:0/2055529546 wait complete. 2024-09-21T02:58:36.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.089+0000 7f929a18e640 1 Processor -- start 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f929a18e640 1 -- start start 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f929a18e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92941025d0 0x7f9294199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f929a18e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103650 0x7f929419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f929a18e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9294103ff0 0x7f929419b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f929a18e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f929410e070 con 0x7f9294103650 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f929a18e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f929410def0 con 0x7f92941025d0 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f929a18e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f929410e1f0 con 0x7f9294103ff0 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9292ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103650 0x7f929419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9292ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103650 0x7f929419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60790/0 (socket says 172.21.15.29:60790) 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9292ffd640 1 -- 172.21.15.29:0/3102914140 learned_addr learned my addr 172.21.15.29:0/3102914140 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f92937fe640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92941025d0 0x7f9294199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9293fff640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9294103ff0 0x7f929419b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9292ffd640 1 -- 172.21.15.29:0/3102914140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9294103ff0 msgr2=0x7f929419b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9294103ff0 0x7f929419b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9292ffd640 1 -- 172.21.15.29:0/3102914140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92941025d0 msgr2=0x7f9294199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92941025d0 0x7f9294199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.090+0000 7f9292ffd640 1 -- 172.21.15.29:0/3102914140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929419b800 con 0x7f9294103650 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.091+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103650 0x7f929419a4c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f928800d950 tx=0x7f928800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.091+0000 7f9290ff9640 1 -- 172.21.15.29:0/3102914140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288017070 con 0x7f9294103650 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.091+0000 7f929a18e640 1 -- 172.21.15.29:0/3102914140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929419baf0 con 0x7f9294103650 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.091+0000 7f9290ff9640 1 -- 172.21.15.29:0/3102914140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f928800de80 con 0x7f9294103650 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.091+0000 7f9290ff9640 1 -- 172.21.15.29:0/3102914140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928800f620 con 0x7f9294103650 2024-09-21T02:58:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.091+0000 7f929a18e640 1 -- 172.21.15.29:0/3102914140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92941a2270 con 0x7f9294103650 2024-09-21T02:58:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.093+0000 7f92a11db740 1 Processor -- start 2024-09-21T02:58:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92a11db740 1 -- start start 2024-09-21T02:58:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9290ff9640 1 -- 172.21.15.29:0/3102914140 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9288009c30 con 0x7f9294103650 2024-09-21T02:58:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92a11db740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988ad0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92a11db740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 0x559988ad08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92a11db740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887034d0 0x559988ad1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92a11db740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559988a3e6f0 con 0x5599887034d0 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92a11db740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x559988a2db20 con 0x5599886fb2f0 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92a11db740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559988a31930 con 0x5599887241b0 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9292ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887034d0 0x559988ad1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9292ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887034d0 0x559988ad1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60794/0 (socket says 172.21.15.29:60794) 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9292ffd640 1 -- 172.21.15.29:0/99021557 learned_addr learned my addr 172.21.15.29:0/99021557 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9293fff640 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988ad0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92937fe640 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 0x559988ad08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9292ffd640 1 -- 172.21.15.29:0/99021557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 msgr2=0x559988ad0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9292ffd640 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988ad0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9292ffd640 1 -- 172.21.15.29:0/99021557 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 msgr2=0x559988ad08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9290ff9640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f927c0b6510 0x7f927c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9292ffd640 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 0x559988ad08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f9292ffd640 1 -- 172.21.15.29:0/99021557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599889f3530 con 0x5599887034d0 2024-09-21T02:58:36.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.094+0000 7f92937fe640 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 0x559988ad08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:36.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.095+0000 7f9293fff640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f927c0b6510 0x7f927c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.095+0000 7f9292ffd640 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887034d0 0x559988ad1420 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f92880b4740 tx=0x7f92880ad200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be7dd087721feb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:36.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.095+0000 7f9290ff9640 1 -- 172.21.15.29:0/3102914140 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f92880e7200 con 0x7f9294103650 2024-09-21T02:58:36.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.095+0000 7f9279ffb640 1 -- 172.21.15.29:0/99021557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288020070 con 0x5599887034d0 2024-09-21T02:58:36.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.095+0000 7f9279ffb640 1 -- 172.21.15.29:0/99021557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92880be570 con 0x5599887034d0 2024-09-21T02:58:36.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.095+0000 7f9279ffb640 1 -- 172.21.15.29:0/99021557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92880ac960 con 0x5599887034d0 2024-09-21T02:58:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 -- 172.21.15.29:0/99021557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887034d0 msgr2=0x559988ad1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f9293fff640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f927c0b6510 0x7f927c0b8a20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f928c009a60 tx=0x7f928c031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887034d0 0x559988ad1420 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f92880b4740 tx=0x7f92880ad200 comp rx=0 tx=0).stop 2024-09-21T02:58:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 -- 172.21.15.29:0/99021557 shutdown_connections 2024-09-21T02:58:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887034d0 0x559988ad1420 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 0x559988ad08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 --2- 172.21.15.29:0/99021557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988ad0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 -- 172.21.15.29:0/99021557 >> 172.21.15.29:0/99021557 conn(0x559988718300 msgr2=0x559988ac78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 -- 172.21.15.29:0/99021557 shutdown_connections 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 -- 172.21.15.29:0/99021557 wait complete. 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.096+0000 7f92a11db740 1 -- start start 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92a11db740 1 Processor -- start 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92a11db740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ad9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92a11db740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ada100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92a11db740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988adac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92a11db740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559988965ab0 con 0x5599886fb2f0 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92a11db740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5599889d3730 con 0x5599887034d0 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92a11db740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559988ad1ee0 con 0x5599887241b0 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92a11db740 10 client.0 _ll_register_callbacks cb 0x559988a276b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ad9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ad9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60798/0 (socket says 172.21.15.29:60798) 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92937fe640 1 -- 172.21.15.29:0/3561321731 learned_addr learned my addr 172.21.15.29:0/3561321731 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ada100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f9293fff640 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988adac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92937fe640 1 -- 172.21.15.29:0/3561321731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 msgr2=0x559988adac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92937fe640 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988adac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92937fe640 1 -- 172.21.15.29:0/3561321731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 msgr2=0x559988ada100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92937fe640 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ada100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f92937fe640 1 -- 172.21.15.29:0/3561321731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55998878f790 con 0x5599886fb2f0 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.097+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ada100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92937fe640 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ad9a40 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f92840127c0 tx=0x7f9284012c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3dd3732913487f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f9272ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f9272ffd640 1 -- 172.21.15.29:0/3561321731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928401f060 con 0x5599886fb2f0 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f9272ffd640 1 -- 172.21.15.29:0/3561321731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9284010dc0 con 0x5599886fb2f0 2024-09-21T02:58:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 1 -- 172.21.15.29:0/3561321731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559988a40640 con 0x5599886fb2f0 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f9272ffd640 1 -- 172.21.15.29:0/3561321731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9284026050 con 0x5599886fb2f0 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 10 client.18087 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 1 -- 172.21.15.29:0/3561321731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559988808620 con 0x5599886fb2f0 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f9272ffd640 1 -- 172.21.15.29:0/3561321731 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x559988a40640 con 0x5599886fb2f0 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f9272ffd640 1 -- 172.21.15.29:0/3561321731 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f928400d3d0 con 0x5599886fb2f0 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 10 client.18087 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 10 client.18087 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 20 client.18087 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5599886e2d80 0x559988ae71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 4 client.18087 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.098+0000 7f92a11db740 1 -- 172.21.15.29:0/3561321731 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559988a3c0f0 con 0x5599886e2d80 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.099+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5599886e2d80 0x559988ae71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.099+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5599886e2d80 0x559988ae71d0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b5b404d7a6aa78 server_cookie=bfa7ad8a3448db51 in_seq=0 out_seq=0 2024-09-21T02:58:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.099+0000 7f9272ffd640 10 client.18087 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f9272ffd640 1 -- 172.21.15.29:0/3561321731 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559988a3c0f0 con 0x5599886e2d80 2024-09-21T02:58:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f9272ffd640 10 client.18087 handle_command_reply: tid=0 2024-09-21T02:58:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 client.18087 shutdown 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 -- 172.21.15.29:0/3561321731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 msgr2=0x559988ad9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ad9a40 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f92840127c0 tx=0x7f9284012c80 comp rx=0 tx=0).stop 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 -- 172.21.15.29:0/3561321731 shutdown_connections 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988adac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ada100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5599886e2d80 0x559988ae71d0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 --2- 172.21.15.29:0/3561321731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ad9a40 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f9272ffd640 0 client.18087 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.100+0000 7f92a11db740 1 -- 172.21.15.29:0/3561321731 >> 172.21.15.29:0/3561321731 conn(0x559988718300 msgr2=0x559988ac5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.101+0000 7f92a11db740 1 -- 172.21.15.29:0/3561321731 shutdown_connections 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.101+0000 7f92a11db740 1 -- 172.21.15.29:0/3561321731 wait complete. 2024-09-21T02:58:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.101+0000 7f92a11db740 20 client.18087 trim_cache size 0 max 16384 2024-09-21T02:58:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.115+0000 7f92a11db740 1 Processor -- start 2024-09-21T02:58:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.115+0000 7f92a11db740 1 -- start start 2024-09-21T02:58:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.115+0000 7f92a11db740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 0x559988ac66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.115+0000 7f92a11db740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887034d0 0x559988ae1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.115+0000 7f92a11db740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887241b0 0x559988ac6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.115+0000 7f92a11db740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92941aa3d0 con 0x5599887241b0 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.115+0000 7f92a11db740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f92941abf10 con 0x5599886fb2f0 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.115+0000 7f92a11db740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f92941aaaa0 con 0x5599887034d0 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887241b0 0x559988ac6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887241b0 0x559988ac6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60806/0 (socket says 172.21.15.29:60806) 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 -- 172.21.15.29:0/3073959183 learned_addr learned my addr 172.21.15.29:0/3073959183 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 -- 172.21.15.29:0/3073959183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887034d0 msgr2=0x559988ae1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 --2- 172.21.15.29:0/3073959183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887034d0 0x559988ae1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 -- 172.21.15.29:0/3073959183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 msgr2=0x559988ac66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 --2- 172.21.15.29:0/3073959183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 0x559988ac66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 -- 172.21.15.29:0/3073959183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559988adafe0 con 0x5599887241b0 2024-09-21T02:58:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f9293fff640 1 --2- 172.21.15.29:0/3073959183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887241b0 0x559988ac6c30 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f928c031d40 tx=0x7f928c041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9047eaac01e23942 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f92797fa640 1 -- 172.21.15.29:0/3073959183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c003a30 con 0x5599887241b0 2024-09-21T02:58:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f92797fa640 1 -- 172.21.15.29:0/3073959183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f928c036050 con 0x5599887241b0 2024-09-21T02:58:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.116+0000 7f92797fa640 1 -- 172.21.15.29:0/3073959183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c0305c0 con 0x5599887241b0 2024-09-21T02:58:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 -- 172.21.15.29:0/3073959183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887241b0 msgr2=0x559988ac6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 --2- 172.21.15.29:0/3073959183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887241b0 0x559988ac6c30 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f928c031d40 tx=0x7f928c041000 comp rx=0 tx=0).stop 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 -- 172.21.15.29:0/3073959183 shutdown_connections 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 --2- 172.21.15.29:0/3073959183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599887241b0 0x559988ac6c30 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 --2- 172.21.15.29:0/3073959183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887034d0 0x559988ae1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 --2- 172.21.15.29:0/3073959183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599886fb2f0 0x559988ac66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 -- 172.21.15.29:0/3073959183 >> 172.21.15.29:0/3073959183 conn(0x559988718300 msgr2=0x559988ad8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 -- 172.21.15.29:0/3073959183 shutdown_connections 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 -- 172.21.15.29:0/3073959183 wait complete. 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.117+0000 7f92a11db740 1 -- start start 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92a11db740 1 Processor -- start 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92a11db740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ae1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92a11db740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ae05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92a11db740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988ae0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92a11db740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92941ab840 con 0x5599886fb2f0 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92a11db740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f92941ab170 con 0x5599887034d0 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92a11db740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f92941a83b0 con 0x5599887241b0 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92a11db740 10 client.0 _ll_register_callbacks cb 0x7f928400a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ae1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ae1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60820/0 (socket says 172.21.15.29:60820) 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 -- 172.21.15.29:0/3951914373 learned_addr learned my addr 172.21.15.29:0/3951914373 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ae05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f9293fff640 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988ae0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 -- 172.21.15.29:0/3951914373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 msgr2=0x559988ae0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988ae0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 -- 172.21.15.29:0/3951914373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 msgr2=0x559988ae05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ae05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 -- 172.21.15.29:0/3951914373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559988ae13a0 con 0x5599886fb2f0 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ae05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:58:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f92937fe640 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ae1900 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f928400d7a0 tx=0x7f928400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d6babbffa26db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.118+0000 7f9272ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f9272ffd640 1 -- 172.21.15.29:0/3951914373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928402e020 con 0x5599886fb2f0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f9272ffd640 1 -- 172.21.15.29:0/3951914373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f928400d540 con 0x5599886fb2f0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 1 -- 172.21.15.29:0/3951914373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f928c036050 con 0x5599886fb2f0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f9272ffd640 1 -- 172.21.15.29:0/3951914373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f928401f020 con 0x5599886fb2f0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 10 client.18093 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 1 -- 172.21.15.29:0/3951914373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559988b51c70 con 0x5599886fb2f0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f9272ffd640 1 -- 172.21.15.29:0/3951914373 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f928c036050 con 0x5599886fb2f0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f9272ffd640 1 -- 172.21.15.29:0/3951914373 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f9284010c50 con 0x5599886fb2f0 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 10 client.18093 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 10 client.18093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 20 client.18093 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5599886b81c0 0x559988b546e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 4 client.18093 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:58:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.119+0000 7f92a11db740 1 -- 172.21.15.29:0/3951914373 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559988a39520 con 0x5599886b81c0 2024-09-21T02:58:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.120+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5599886b81c0 0x559988b546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.120+0000 7f9292ffd640 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5599886b81c0 0x559988b546e0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8f9b0c41da525c4 server_cookie=2f6c227d8ae8e4b8 in_seq=0 out_seq=0 2024-09-21T02:58:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.120+0000 7f9272ffd640 10 client.18093 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:36.125 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:35 smithi190 ceph-mon[33871]: pgmap v973: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1006 KiB/s wr, 105 op/s 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f9272ffd640 1 -- 172.21.15.29:0/3951914373 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559988a39520 con 0x5599886b81c0 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f9272ffd640 10 client.18093 handle_command_reply: tid=0 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 client.18093 shutdown 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 -- 172.21.15.29:0/3951914373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 msgr2=0x559988ae1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ae1900 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f928400d7a0 tx=0x7f928400af90 comp rx=0 tx=0).stop 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 -- 172.21.15.29:0/3951914373 shutdown_connections 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599887241b0 0x559988ae0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5599887034d0 0x559988ae05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5599886b81c0 0x559988b546e0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 --2- 172.21.15.29:0/3951914373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599886fb2f0 0x559988ae1900 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f9272ffd640 0 client.18093 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 -- 172.21.15.29:0/3951914373 >> 172.21.15.29:0/3951914373 conn(0x559988718300 msgr2=0x559988ad42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 -- 172.21.15.29:0/3951914373 shutdown_connections 2024-09-21T02:58:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 1 -- 172.21.15.29:0/3951914373 wait complete. 2024-09-21T02:58:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.214+0000 7f92a11db740 20 client.18093 trim_cache size 0 max 16384 2024-09-21T02:58:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:58:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:58:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:58:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 -- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f927c0b6510 msgr2=0x7f927c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f927c0b6510 0x7f927c0b8a20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f928c009a60 tx=0x7f928c031000 comp rx=0 tx=0).stop 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 -- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103650 msgr2=0x7f929419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103650 0x7f929419a4c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f928800d950 tx=0x7f928800dd50 comp rx=0 tx=0).stop 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 -- 172.21.15.29:0/3102914140 shutdown_connections 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f927c0b6510 0x7f927c0b8a20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9294103ff0 0x7f929419b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9294103650 0x7f929419a4c0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 --2- 172.21.15.29:0/3102914140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92941025d0 0x7f9294199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 -- 172.21.15.29:0/3102914140 >> 172.21.15.29:0/3102914140 conn(0x7f92940fdf30 msgr2=0x7f929406b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 -- 172.21.15.29:0/3102914140 shutdown_connections 2024-09-21T02:58:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:36.215+0000 7f929a18e640 1 -- 172.21.15.29:0/3102914140 wait complete. 2024-09-21T02:58:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:35 smithi066 ceph-mon[33436]: pgmap v973: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1006 KiB/s wr, 105 op/s 2024-09-21T02:58:36.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.46 MB/sec execute 253 sec latency 5430.736 ms 2024-09-21T02:58:37.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.44 MB/sec execute 254 sec latency 6430.861 ms 2024-09-21T02:58:38.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:37 smithi190 ceph-mon[33871]: pgmap v974: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-09-21T02:58:38.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:37 smithi029 ceph-mon[31006]: pgmap v974: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-09-21T02:58:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:37 smithi066 ceph-mon[33436]: pgmap v974: 129 pgs: 129 active+clean; 287 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-09-21T02:58:38.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.42 MB/sec execute 255 sec latency 7430.992 ms 2024-09-21T02:58:39.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.39 MB/sec execute 256 sec latency 8431.124 ms 2024-09-21T02:58:40.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:39 smithi190 ceph-mon[33871]: pgmap v975: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.0 MiB/s wr, 131 op/s 2024-09-21T02:58:40.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:39 smithi029 ceph-mon[31006]: pgmap v975: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.0 MiB/s wr, 131 op/s 2024-09-21T02:58:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:39 smithi066 ceph-mon[33436]: pgmap v975: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.0 MiB/s wr, 131 op/s 2024-09-21T02:58:40.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.37 MB/sec execute 257 sec latency 9431.257 ms 2024-09-21T02:58:41.732 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.35 MB/sec execute 258 sec latency 10431.366 ms 2024-09-21T02:58:42.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:41 smithi190 ceph-mon[33871]: pgmap v976: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-09-21T02:58:42.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:41 smithi029 ceph-mon[31006]: pgmap v976: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-09-21T02:58:42.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:41 smithi066 ceph-mon[33436]: pgmap v976: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-09-21T02:58:42.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.33 MB/sec execute 259 sec latency 11431.473 ms 2024-09-21T02:58:43.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:42 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:58:43.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:42 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:58:43.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:42 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:58:43.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.31 MB/sec execute 260 sec latency 12431.591 ms 2024-09-21T02:58:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:43 smithi190 ceph-mon[33871]: pgmap v977: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 874 KiB/s wr, 129 op/s 2024-09-21T02:58:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:43 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:43 smithi029 ceph-mon[31006]: pgmap v977: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 874 KiB/s wr, 129 op/s 2024-09-21T02:58:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:43 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:43 smithi066 ceph-mon[33436]: pgmap v977: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 874 KiB/s wr, 129 op/s 2024-09-21T02:58:44.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:43 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:44.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.29 MB/sec execute 261 sec latency 13431.684 ms 2024-09-21T02:58:45.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.27 MB/sec execute 262 sec latency 14431.749 ms 2024-09-21T02:58:46.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:45 smithi190 ceph-mon[33871]: pgmap v978: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 503 KiB/s wr, 84 op/s 2024-09-21T02:58:46.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:45 smithi029 ceph-mon[31006]: pgmap v978: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 503 KiB/s wr, 84 op/s 2024-09-21T02:58:46.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:45 smithi066 ceph-mon[33436]: pgmap v978: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 503 KiB/s wr, 84 op/s 2024-09-21T02:58:46.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.25 MB/sec execute 263 sec latency 15431.834 ms 2024-09-21T02:58:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:58:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:58:47.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:58:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:58:47.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:58:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:58:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:58:47.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.23 MB/sec execute 264 sec latency 16431.883 ms 2024-09-21T02:58:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:47 smithi190 ceph-mon[33871]: pgmap v979: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 144 KiB/s wr, 51 op/s 2024-09-21T02:58:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:47 smithi029 ceph-mon[31006]: pgmap v979: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 144 KiB/s wr, 51 op/s 2024-09-21T02:58:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:47 smithi066 ceph-mon[33436]: pgmap v979: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 144 KiB/s wr, 51 op/s 2024-09-21T02:58:48.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.21 MB/sec execute 265 sec latency 17431.985 ms 2024-09-21T02:58:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.327+0000 7f810a789640 1 -- 172.21.15.29:0/732964626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8104103ff0 msgr2=0x7f810410a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.327+0000 7f810a789640 1 --2- 172.21.15.29:0/732964626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8104103ff0 0x7f810410a990 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009ca0 tx=0x7f80fc02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.327+0000 7f810a789640 1 -- 172.21.15.29:0/732964626 shutdown_connections 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.327+0000 7f810a789640 1 --2- 172.21.15.29:0/732964626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8104103ff0 0x7f810410a990 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.327+0000 7f810a789640 1 --2- 172.21.15.29:0/732964626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8104103650 0x7f8104103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.327+0000 7f810a789640 1 --2- 172.21.15.29:0/732964626 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f81041025d0 0x7f8104102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.328+0000 7f810a789640 1 -- 172.21.15.29:0/732964626 >> 172.21.15.29:0/732964626 conn(0x7f81040fdf30 msgr2=0x7f81041003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.328+0000 7f810a789640 1 -- 172.21.15.29:0/732964626 shutdown_connections 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.328+0000 7f810a789640 1 -- 172.21.15.29:0/732964626 wait complete. 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.328+0000 7f810a789640 1 Processor -- start 2024-09-21T02:58:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.328+0000 7f810a789640 1 -- start start 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f810a789640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81041025d0 0x7f810406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f810a789640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8104103650 0x7f810419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f810a789640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8104103ff0 0x7f81041a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f810a789640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f810410e120 con 0x7f81041025d0 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f810a789640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f810410dfa0 con 0x7f8104103650 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f810a789640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f810410e2a0 con 0x7f8104103ff0 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81041025d0 0x7f810406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81041025d0 0x7f810406c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47174/0 (socket says 172.21.15.29:47174) 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 -- 172.21.15.29:0/2404056551 learned_addr learned my addr 172.21.15.29:0/2404056551 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8108d26640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8104103ff0 0x7f81041a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f81037fe640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8104103650 0x7f810419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 -- 172.21.15.29:0/2404056551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8104103ff0 msgr2=0x7f81041a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8104103ff0 0x7f81041a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 -- 172.21.15.29:0/2404056551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8104103650 msgr2=0x7f810419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8104103650 0x7f810419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 -- 172.21.15.29:0/2404056551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81041a18d0 con 0x7f81041025d0 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.329+0000 7f8103fff640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81041025d0 0x7f810406c1f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f80f400d950 tx=0x7f80f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.330+0000 7f81017fa640 1 -- 172.21.15.29:0/2404056551 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4017070 con 0x7f81041025d0 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.330+0000 7f81017fa640 1 -- 172.21.15.29:0/2404056551 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80f4009c30 con 0x7f81041025d0 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.330+0000 7f810a789640 1 -- 172.21.15.29:0/2404056551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81041a1bc0 con 0x7f81041025d0 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.330+0000 7f81017fa640 1 -- 172.21.15.29:0/2404056551 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f400f620 con 0x7f81041025d0 2024-09-21T02:58:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.330+0000 7f810a789640 1 -- 172.21.15.29:0/2404056551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81041a2110 con 0x7f81041025d0 2024-09-21T02:58:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.332+0000 7f81017fa640 1 -- 172.21.15.29:0/2404056551 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f80f4010460 con 0x7f81041025d0 2024-09-21T02:58:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.332+0000 7f81117d1740 1 Processor -- start 2024-09-21T02:58:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81117d1740 1 -- start start 2024-09-21T02:58:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81117d1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80eb61b0 0x55cd8125de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81117d1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 0x55cd8125e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81117d1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd8125f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81117d1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cd811d06f0 con 0x55cd80e954d0 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81117d1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55cd811bfb20 con 0x55cd80eb61b0 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81117d1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cd811c3930 con 0x55cd80e8d2f0 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd8125f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd8125f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47190/0 (socket says 172.21.15.29:47190) 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f81037fe640 1 -- 172.21.15.29:0/2680738794 learned_addr learned my addr 172.21.15.29:0/2680738794 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f8103fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 0x55cd8125e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.333+0000 7f8108d26640 1 --2- 172.21.15.29:0/2680738794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80eb61b0 0x55cd8125de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f81037fe640 1 -- 172.21.15.29:0/2680738794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 msgr2=0x55cd8125e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f81037fe640 1 --2- 172.21.15.29:0/2680738794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 0x55cd8125e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f81037fe640 1 -- 172.21.15.29:0/2680738794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80eb61b0 msgr2=0x55cd8125de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f81037fe640 1 --2- 172.21.15.29:0/2680738794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80eb61b0 0x55cd8125de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f81037fe640 1 -- 172.21.15.29:0/2680738794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd81185530 con 0x55cd80e954d0 2024-09-21T02:58:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f81017fa640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f80ec0b6510 0x7f80ec0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f81037fe640 1 --2- 172.21.15.29:0/2680738794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd8125f010 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f80f800bcf0 tx=0x7f80f800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e16069e17c27cd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f8108d26640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f80ec0b6510 0x7f80ec0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f81017fa640 1 -- 172.21.15.29:0/2404056551 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f80f40e7370 con 0x7f81041025d0 2024-09-21T02:58:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f80ea7fc640 1 -- 172.21.15.29:0/2680738794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8015070 con 0x55cd80e954d0 2024-09-21T02:58:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f80ea7fc640 1 -- 172.21.15.29:0/2680738794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80f801a8f0 con 0x55cd80e954d0 2024-09-21T02:58:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.334+0000 7f80ea7fc640 1 -- 172.21.15.29:0/2680738794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8011880 con 0x55cd80e954d0 2024-09-21T02:58:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 -- 172.21.15.29:0/2680738794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 msgr2=0x55cd8125f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f8108d26640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f80ec0b6510 0x7f80ec0b8a20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f80fc005e30 tx=0x7f80fc031980 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 --2- 172.21.15.29:0/2680738794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd8125f010 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f80f800bcf0 tx=0x7f80f800f730 comp rx=0 tx=0).stop 2024-09-21T02:58:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 -- 172.21.15.29:0/2680738794 shutdown_connections 2024-09-21T02:58:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 --2- 172.21.15.29:0/2680738794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd8125f010 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 --2- 172.21.15.29:0/2680738794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 0x55cd8125e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 --2- 172.21.15.29:0/2680738794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80eb61b0 0x55cd8125de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 -- 172.21.15.29:0/2680738794 >> 172.21.15.29:0/2680738794 conn(0x55cd80eaa300 msgr2=0x55cd812598e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 -- 172.21.15.29:0/2680738794 shutdown_connections 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 -- 172.21.15.29:0/2680738794 wait complete. 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.335+0000 7f81117d1740 1 -- start start 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81117d1740 1 Processor -- start 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81117d1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 0x55cd81267600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81117d1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e954d0 0x55cd81267cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81117d1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd81268830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81117d1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cd810f7ab0 con 0x55cd80eb61b0 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81117d1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55cd81165730 con 0x55cd80e954d0 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81117d1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cd8125fb30 con 0x55cd80e8d2f0 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81117d1740 10 client.0 _ll_register_callbacks cb 0x55cd811b96b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f8108d26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd81268830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f81037fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e954d0 0x55cd81267cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f8108d26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd81268830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47206/0 (socket says 172.21.15.29:47206) 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f8108d26640 1 -- 172.21.15.29:0/201500422 learned_addr learned my addr 172.21.15.29:0/201500422 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.336+0000 7f8103fff640 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 0x55cd81267600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f8108d26640 1 -- 172.21.15.29:0/201500422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 msgr2=0x55cd81267600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f8108d26640 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 0x55cd81267600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f8108d26640 1 -- 172.21.15.29:0/201500422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e954d0 msgr2=0x55cd81267cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f8108d26640 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e954d0 0x55cd81267cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f8108d26640 1 -- 172.21.15.29:0/201500422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd80f21790 con 0x55cd80eb61b0 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f81037fe640 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e954d0 0x55cd81267cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f8108d26640 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd81268830 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f80fc035000 tx=0x7f80fc02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fed1e5571435c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f80e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f80e37fe640 1 -- 172.21.15.29:0/201500422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc0098c0 con 0x55cd80eb61b0 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f80e37fe640 1 -- 172.21.15.29:0/201500422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f80fc004250 con 0x55cd80eb61b0 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f81117d1740 1 -- 172.21.15.29:0/201500422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80f801a8f0 con 0x55cd80eb61b0 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f80e37fe640 1 -- 172.21.15.29:0/201500422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f80fc004450 con 0x55cd80eb61b0 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f81117d1740 10 client.18105 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.337+0000 7f81117d1740 1 -- 172.21.15.29:0/201500422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cd80f9a620 con 0x55cd80eb61b0 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f80e37fe640 1 -- 172.21.15.29:0/201500422 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80f801a8f0 con 0x55cd80eb61b0 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f80e37fe640 1 -- 172.21.15.29:0/201500422 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f80fc051070 con 0x55cd80eb61b0 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f81117d1740 10 client.18105 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f81117d1740 10 client.18105 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f81117d1740 20 client.18105 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f81117d1740 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cd80e74d80 0x55cd81274e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f81117d1740 4 client.18105 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f81117d1740 1 -- 172.21.15.29:0/201500422 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd811d2640 con 0x55cd80e74d80 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.338+0000 7f8103fff640 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cd80e74d80 0x55cd81274e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.339+0000 7f80e37fe640 10 client.18105 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.339+0000 7f8103fff640 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cd80e74d80 0x55cd81274e90 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5bd6849d52078f1 server_cookie=d618e1b61eb4ccb5 in_seq=0 out_seq=0 2024-09-21T02:58:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.339+0000 7f80e37fe640 1 -- 172.21.15.29:0/201500422 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cd811d2640 con 0x55cd80e74d80 2024-09-21T02:58:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.339+0000 7f80e37fe640 10 client.18105 handle_command_reply: tid=0 2024-09-21T02:58:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.339+0000 7f81117d1740 1 client.18105 shutdown 2024-09-21T02:58:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 -- 172.21.15.29:0/201500422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 msgr2=0x55cd81268830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd81268830 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f80fc035000 tx=0x7f80fc02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 -- 172.21.15.29:0/201500422 shutdown_connections 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cd80e74d80 0x55cd81274e90 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd81268830 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e954d0 0x55cd81267cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f80e37fe640 0 client.18105 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 --2- 172.21.15.29:0/201500422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e8d2f0 0x55cd81267600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 -- 172.21.15.29:0/201500422 >> 172.21.15.29:0/201500422 conn(0x55cd80eaa300 msgr2=0x55cd81257ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 -- 172.21.15.29:0/201500422 shutdown_connections 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 1 -- 172.21.15.29:0/201500422 wait complete. 2024-09-21T02:58:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.340+0000 7f81117d1740 20 client.18105 trim_cache size 0 max 16384 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81117d1740 1 Processor -- start 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81117d1740 1 -- start start 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81117d1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8125ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81117d1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd81259220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81117d1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80eb61b0 0x55cd812545c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81117d1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f81041ac4d0 con 0x55cd80e954d0 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81117d1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f81041ae010 con 0x55cd80e8d2f0 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81117d1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f81041acba0 con 0x55cd80eb61b0 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd81259220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd81259220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47220/0 (socket says 172.21.15.29:47220) 2024-09-21T02:58:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81037fe640 1 -- 172.21.15.29:0/516152432 learned_addr learned my addr 172.21.15.29:0/516152432 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f8108d26640 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80eb61b0 0x55cd812545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f8103fff640 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8125ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81037fe640 1 -- 172.21.15.29:0/516152432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80eb61b0 msgr2=0x55cd812545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81037fe640 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80eb61b0 0x55cd812545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81037fe640 1 -- 172.21.15.29:0/516152432 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 msgr2=0x55cd8125ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81037fe640 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8125ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.355+0000 7f81037fe640 1 -- 172.21.15.29:0/516152432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd81258fd0 con 0x55cd80e954d0 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.356+0000 7f81037fe640 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd81259220 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f80f800bf70 tx=0x7f80f800bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1ca6677f717b11 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.356+0000 7f80e9ffb640 1 -- 172.21.15.29:0/516152432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8029070 con 0x55cd80e954d0 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.356+0000 7f80e9ffb640 1 -- 172.21.15.29:0/516152432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80f80054d0 con 0x55cd80e954d0 2024-09-21T02:58:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.356+0000 7f80e9ffb640 1 -- 172.21.15.29:0/516152432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8015050 con 0x55cd80e954d0 2024-09-21T02:58:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.356+0000 7f81117d1740 1 -- 172.21.15.29:0/516152432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 msgr2=0x55cd81259220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.356+0000 7f81117d1740 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd81259220 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f80f800bf70 tx=0x7f80f800bfa0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.356+0000 7f81117d1740 1 -- 172.21.15.29:0/516152432 shutdown_connections 2024-09-21T02:58:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.356+0000 7f81117d1740 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80eb61b0 0x55cd812545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80e954d0 0x55cd81259220 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 --2- 172.21.15.29:0/516152432 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8125ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 -- 172.21.15.29:0/516152432 >> 172.21.15.29:0/516152432 conn(0x55cd80eaa300 msgr2=0x55cd81268ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 -- 172.21.15.29:0/516152432 shutdown_connections 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 -- 172.21.15.29:0/516152432 wait complete. 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 -- start start 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 Processor -- start 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8126c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e954d0 0x55cd8126ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd8126d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f81041ad940 con 0x55cd80eb61b0 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f81041ad270 con 0x55cd80e8d2f0 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f81041aa4b0 con 0x55cd80e954d0 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.357+0000 7f81117d1740 10 client.0 _ll_register_callbacks cb 0x7f80fc032f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8108d26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd8126d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8108d26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd8126d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47230/0 (socket says 172.21.15.29:47230) 2024-09-21T02:58:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8108d26640 1 -- 172.21.15.29:0/224702970 learned_addr learned my addr 172.21.15.29:0/224702970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8103fff640 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8126c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f81037fe640 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e954d0 0x55cd8126ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8103fff640 1 -- 172.21.15.29:0/224702970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e954d0 msgr2=0x55cd8126ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8103fff640 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e954d0 0x55cd8126ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8103fff640 1 -- 172.21.15.29:0/224702970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 msgr2=0x55cd8126d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8103fff640 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd8126d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8103fff640 1 -- 172.21.15.29:0/224702970 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd8126e140 con 0x55cd80e8d2f0 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8108d26640 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd8126d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f8103fff640 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8126c7b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80f40e4b40 tx=0x7f80f40eab30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1cb6145d79cdec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f80e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f80e37fe640 1 -- 172.21.15.29:0/224702970 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4020040 con 0x55cd80e8d2f0 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f81117d1740 1 -- 172.21.15.29:0/224702970 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80f80054d0 con 0x55cd80e8d2f0 2024-09-21T02:58:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.358+0000 7f80e37fe640 1 -- 172.21.15.29:0/224702970 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f80f40a8ad0 con 0x55cd80e8d2f0 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f80e37fe640 1 -- 172.21.15.29:0/224702970 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f80f40acc90 con 0x55cd80e8d2f0 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81117d1740 10 client.27422 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81117d1740 1 -- 172.21.15.29:0/224702970 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cd8126f4a0 con 0x55cd80e8d2f0 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f80e37fe640 1 -- 172.21.15.29:0/224702970 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80f80054d0 con 0x55cd80e8d2f0 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f80e37fe640 1 -- 172.21.15.29:0/224702970 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f80f40a8cd0 con 0x55cd80e8d2f0 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81117d1740 10 client.27422 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81117d1740 10 client.27422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81117d1740 20 client.27422 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81117d1740 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cd80e4a1c0 0x55cd812e1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81117d1740 4 client.27422 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81117d1740 1 -- 172.21.15.29:0/224702970 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd811ce0f0 con 0x55cd80e4a1c0 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.359+0000 7f81037fe640 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cd80e4a1c0 0x55cd812e1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.360+0000 7f81037fe640 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cd80e4a1c0 0x55cd812e1a00 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5091b473d976dec6 server_cookie=d133dd790356570e in_seq=0 out_seq=0 2024-09-21T02:58:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.360+0000 7f80e37fe640 10 client.27422 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.396+0000 7f80e37fe640 1 -- 172.21.15.29:0/224702970 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd811ce0f0 con 0x55cd80e4a1c0 2024-09-21T02:58:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.396+0000 7f80e37fe640 10 client.27422 handle_command_reply: tid=0 2024-09-21T02:58:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.396+0000 7f81117d1740 1 client.27422 shutdown 2024-09-21T02:58:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.396+0000 7f81117d1740 1 -- 172.21.15.29:0/224702970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 msgr2=0x55cd8126c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.396+0000 7f81117d1740 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8126c7b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80f40e4b40 tx=0x7f80f40eab30 comp rx=0 tx=0).stop 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 1 -- 172.21.15.29:0/224702970 shutdown_connections 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cd80e4a1c0 0x55cd812e1a00 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd80eb61b0 0x55cd8126d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd80e954d0 0x55cd8126ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 1 --2- 172.21.15.29:0/224702970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cd80e8d2f0 0x55cd8126c7b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f80e37fe640 0 client.27422 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 1 -- 172.21.15.29:0/224702970 >> 172.21.15.29:0/224702970 conn(0x55cd80eaa300 msgr2=0x55cd81259200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 1 -- 172.21.15.29:0/224702970 shutdown_connections 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 1 -- 172.21.15.29:0/224702970 wait complete. 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f81117d1740 20 client.27422 trim_cache size 0 max 16384 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:58:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:58:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:58:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:58:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 -- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f80ec0b6510 msgr2=0x7f80ec0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f80ec0b6510 0x7f80ec0b8a20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f80fc005e30 tx=0x7f80fc031980 comp rx=0 tx=0).stop 2024-09-21T02:58:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 -- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81041025d0 msgr2=0x7f810406c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81041025d0 0x7f810406c1f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f80f400d950 tx=0x7f80f400de10 comp rx=0 tx=0).stop 2024-09-21T02:58:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 -- 172.21.15.29:0/2404056551 shutdown_connections 2024-09-21T02:58:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f80ec0b6510 0x7f80ec0b8a20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8104103ff0 0x7f81041a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8104103650 0x7f810419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 --2- 172.21.15.29:0/2404056551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81041025d0 0x7f810406c1f0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.397+0000 7f810a789640 1 -- 172.21.15.29:0/2404056551 >> 172.21.15.29:0/2404056551 conn(0x7f81040fdf30 msgr2=0x7f8104106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.398+0000 7f810a789640 1 -- 172.21.15.29:0/2404056551 shutdown_connections 2024-09-21T02:58:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:58:49.398+0000 7f810a789640 1 -- 172.21.15.29:0/2404056551 wait complete. 2024-09-21T02:58:49.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.19 MB/sec execute 266 sec latency 18432.075 ms 2024-09-21T02:58:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:49 smithi190 ceph-mon[33871]: pgmap v980: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 110 KiB/s wr, 41 op/s 2024-09-21T02:58:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:49 smithi029 ceph-mon[31006]: pgmap v980: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 110 KiB/s wr, 41 op/s 2024-09-21T02:58:50.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:49 smithi066 ceph-mon[33436]: pgmap v980: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 110 KiB/s wr, 41 op/s 2024-09-21T02:58:50.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:58:50.581 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:58:50.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.17 MB/sec execute 267 sec latency 19432.184 ms 2024-09-21T02:58:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.102+0000 7f414b7a5640 1 -- 172.21.15.29:0/3248773304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 msgr2=0x7f4144104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.102+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3248773304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 0x7f4144104dc0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f4140009900 tx=0x7f414002a410 comp rx=0 tx=0).stop 2024-09-21T02:58:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.102+0000 7f414b7a5640 1 -- 172.21.15.29:0/3248773304 shutdown_connections 2024-09-21T02:58:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.102+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3248773304 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f414410de10 0x7f4144110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.102+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3248773304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4144105300 0x7f414410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.102+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3248773304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 0x7f4144104dc0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.102+0000 7f414b7a5640 1 -- 172.21.15.29:0/3248773304 >> 172.21.15.29:0/3248773304 conn(0x7f4144079730 msgr2=0x7f4144079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.102+0000 7f414b7a5640 1 -- 172.21.15.29:0/3248773304 shutdown_connections 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.103+0000 7f414b7a5640 1 -- 172.21.15.29:0/3248773304 wait complete. 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.103+0000 7f414b7a5640 1 Processor -- start 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.103+0000 7f414b7a5640 1 -- start start 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f414b7a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 0x7f4144104100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f414b7a5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4144105300 0x7f41441026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f414b7a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f414410de10 0x7f4144102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f414b7a5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4144113970 con 0x7f41441049b0 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f414b7a5640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f41441137f0 con 0x7f4144105300 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f414b7a5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4144113af0 con 0x7f414410de10 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 0x7f4144104100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 0x7f4144104100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47252/0 (socket says 172.21.15.29:47252) 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4148d40640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4144105300 0x7f41441026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 -- 172.21.15.29:0/3331798135 learned_addr learned my addr 172.21.15.29:0/3331798135 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149d42640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f414410de10 0x7f4144102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 -- 172.21.15.29:0/3331798135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f414410de10 msgr2=0x7f4144102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f414410de10 0x7f4144102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 -- 172.21.15.29:0/3331798135 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4144105300 msgr2=0x7f41441026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4144105300 0x7f41441026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 -- 172.21.15.29:0/3331798135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41441033c0 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.104+0000 7f4149541640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 0x7f4144104100 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f414002a930 tx=0x7f4140003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.105+0000 7f413a7fc640 1 -- 172.21.15.29:0/3331798135 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414003f070 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.105+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41441acd60 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.105+0000 7f413a7fc640 1 -- 172.21.15.29:0/3331798135 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4140003e10 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.105+0000 7f413a7fc640 1 -- 172.21.15.29:0/3331798135 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4140035030 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.105+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41441ad1c0 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.106+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4144104dc0 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.108+0000 7f413a7fc640 1 -- 172.21.15.29:0/3331798135 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4140042070 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.108+0000 7f413a7fc640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41300b6510 0x7f41300b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.108+0000 7f413a7fc640 1 -- 172.21.15.29:0/3331798135 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4140107de0 con 0x7f41441049b0 2024-09-21T02:58:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.108+0000 7f4148d40640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41300b6510 0x7f41300b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:51.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.109+0000 7f4148d40640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41300b6510 0x7f41300b8a20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f414407a0b0 tx=0x7f413400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:51.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.113+0000 7f413a7fc640 1 -- 172.21.15.29:0/3331798135 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f41400d26d0 con 0x7f41441049b0 2024-09-21T02:58:51.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.357+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f414402e640 con 0x7f41441049b0 2024-09-21T02:58:51.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.358+0000 7f413a7fc640 1 -- 172.21.15.29:0/3331798135 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f41400d7610 con 0x7f41441049b0 2024-09-21T02:58:51.359 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T02:58:51.360 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T02:58:51.360 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41300b6510 msgr2=0x7f41300b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41300b6510 0x7f41300b8a20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f414407a0b0 tx=0x7f413400b390 comp rx=0 tx=0).stop 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 msgr2=0x7f4144104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 0x7f4144104100 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f414002a930 tx=0x7f4140003c20 comp rx=0 tx=0).stop 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 shutdown_connections 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f41300b6510 0x7f41300b8a20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f414410de10 0x7f4144102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4144105300 0x7f41441026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 --2- 172.21.15.29:0/3331798135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41441049b0 0x7f4144104100 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 >> 172.21.15.29:0/3331798135 conn(0x7f4144079730 msgr2=0x7f41441008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 shutdown_connections 2024-09-21T02:58:51.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:51.361+0000 7f414b7a5640 1 -- 172.21.15.29:0/3331798135 wait complete. 2024-09-21T02:58:51.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.15 MB/sec execute 268 sec latency 20432.229 ms 2024-09-21T02:58:52.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:51 smithi190 ceph-mon[33871]: pgmap v981: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 76 KiB/s wr, 21 op/s 2024-09-21T02:58:52.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:51 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3331798135' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:52.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:51 smithi029 ceph-mon[31006]: pgmap v981: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 76 KiB/s wr, 21 op/s 2024-09-21T02:58:52.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:51 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3331798135' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:52.193 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id a998ab4c --release --await' 2024-09-21T02:58:52.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id a998ab4c --release --await 2024-09-21T02:58:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:51 smithi066 ceph-mon[33436]: pgmap v981: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 76 KiB/s wr, 21 op/s 2024-09-21T02:58:52.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:51 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3331798135' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:58:52.454 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:58:52.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38436 5.13 MB/sec execute 269 sec latency 21432.261 ms 2024-09-21T02:58:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 -- 172.21.15.29:0/1613279790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 msgr2=0x7feda810f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 --2- 172.21.15.29:0/1613279790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 0x7feda810f560 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7feda0009ca0 tx=0x7feda002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 -- 172.21.15.29:0/1613279790 shutdown_connections 2024-09-21T02:58:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 --2- 172.21.15.29:0/1613279790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 0x7feda810f560 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 --2- 172.21.15.29:0/1613279790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda81083e0 0x7feda81087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 --2- 172.21.15.29:0/1613279790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feda8102aa0 0x7feda8102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 -- 172.21.15.29:0/1613279790 >> 172.21.15.29:0/1613279790 conn(0x7feda8079750 msgr2=0x7feda8079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 -- 172.21.15.29:0/1613279790 shutdown_connections 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.015+0000 7fedaf183640 1 -- 172.21.15.29:0/1613279790 wait complete. 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedaf183640 1 Processor -- start 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedaf183640 1 -- start start 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedaf183640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda8102aa0 0x7feda8101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedaf183640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feda81083e0 0x7feda80ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedaf183640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 0x7feda80ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedaf183640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feda81128f0 con 0x7feda8108d30 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedaf183640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feda8112770 con 0x7feda81083e0 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedaf183640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feda8112a70 con 0x7feda8102aa0 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedad720640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 0x7feda80ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedad720640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 0x7feda80ffbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47278/0 (socket says 172.21.15.29:47278) 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.016+0000 7fedad720640 1 -- 172.21.15.29:0/4155834050 learned_addr learned my addr 172.21.15.29:0/4155834050 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedacf1f640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda8102aa0 0x7feda8101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7feda7fff640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feda81083e0 0x7feda80ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedad720640 1 -- 172.21.15.29:0/4155834050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda8102aa0 msgr2=0x7feda8101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedad720640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda8102aa0 0x7feda8101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedad720640 1 -- 172.21.15.29:0/4155834050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feda81083e0 msgr2=0x7feda80ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedad720640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feda81083e0 0x7feda80ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedad720640 1 -- 172.21.15.29:0/4155834050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda8100340 con 0x7feda8108d30 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedacf1f640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda8102aa0 0x7feda8101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7feda7fff640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feda81083e0 0x7feda80ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedad720640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 0x7feda80ffbb0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7feda0005cc0 tx=0x7feda0032c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7feda5ffb640 1 -- 172.21.15.29:0/4155834050 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0004040 con 0x7feda8108d30 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7feda5ffb640 1 -- 172.21.15.29:0/4155834050 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feda0004670 con 0x7feda8108d30 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedaf183640 1 -- 172.21.15.29:0/4155834050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda81accd0 con 0x7feda8108d30 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7feda5ffb640 1 -- 172.21.15.29:0/4155834050 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0030cc0 con 0x7feda8108d30 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.017+0000 7fedaf183640 1 -- 172.21.15.29:0/4155834050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda81ad180 con 0x7feda8108d30 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7fedb56f7740 1 Processor -- start 2024-09-21T02:58:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7feda5ffb640 1 -- 172.21.15.29:0/4155834050 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7feda00041e0 con 0x7feda8108d30 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7fedb56f7740 1 -- start start 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ecb0520 0x55f77f056510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ec87660 0x55f77f056a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 0x55f77f056f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7fedb56f7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f77ef41dc0 con 0x55f77ecb0520 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7fedb56f7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f77efce1f0 con 0x55f77ec87660 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7fedb56f7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f77efbd620 con 0x55f77ec8f840 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.020+0000 7feda7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ecb0520 0x55f77f056510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ecb0520 0x55f77f056510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47286/0 (socket says 172.21.15.29:47286) 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda7fff640 1 -- 172.21.15.29:0/1380572625 learned_addr learned my addr 172.21.15.29:0/1380572625 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7fedacf1f640 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ec87660 0x55f77f056a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda5ffb640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed900b6510 0x7fed900b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda7fff640 1 -- 172.21.15.29:0/1380572625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 msgr2=0x55f77f056f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda7fff640 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 0x55f77f056f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda7fff640 1 -- 172.21.15.29:0/1380572625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ec87660 msgr2=0x55f77f056a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda7fff640 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ec87660 0x55f77f056a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda7fff640 1 -- 172.21.15.29:0/1380572625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f77ef85fd0 con 0x55f77ecb0520 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7fedacf1f640 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ec87660 0x55f77f056a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7fedad720640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed900b6510 0x7fed900b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda7fff640 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ecb0520 0x55f77f056510 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00bc30 tx=0x7fed9c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e50834ff67845df server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7feda5ffb640 1 -- 172.21.15.29:0/4155834050 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7feda0108b40 con 0x7feda8108d30 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7fed8effd640 1 -- 172.21.15.29:0/1380572625 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c005220 con 0x55f77ecb0520 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7fed8effd640 1 -- 172.21.15.29:0/1380572625 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed9c004c30 con 0x55f77ecb0520 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.021+0000 7fed8effd640 1 -- 172.21.15.29:0/1380572625 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0116d0 con 0x55f77ecb0520 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 -- 172.21.15.29:0/1380572625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ecb0520 msgr2=0x55f77f056510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedad720640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed900b6510 0x7fed900b8a20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7feda00cbbc0 tx=0x7feda009bd90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ecb0520 0x55f77f056510 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00bc30 tx=0x7fed9c00f730 comp rx=0 tx=0).stop 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 -- 172.21.15.29:0/1380572625 shutdown_connections 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 0x55f77f056f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ec87660 0x55f77f056a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1380572625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ecb0520 0x55f77f056510 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 -- 172.21.15.29:0/1380572625 >> 172.21.15.29:0/1380572625 conn(0x55f77eca4910 msgr2=0x55f77f054400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 -- 172.21.15.29:0/1380572625 shutdown_connections 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.022+0000 7fedb56f7740 1 -- 172.21.15.29:0/1380572625 wait complete. 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.023+0000 7fedb56f7740 1 -- start start 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.023+0000 7fedb56f7740 1 Processor -- start 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.023+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec87660 0x55f77f061bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.023+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 0x55f77f0620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.023+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f06de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.023+0000 7fedb56f7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f77ef58300 con 0x55f77ec87660 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.023+0000 7fedb56f7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55f77efc1430 con 0x55f77ecb0520 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.023+0000 7fedb56f7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f77ee1ebb0 con 0x55f77ec8f840 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7fedb56f7740 10 client.0 _ll_register_callbacks cb 0x55f77ef72120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec87660 0x55f77f061bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec87660 0x55f77f061bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47294/0 (socket says 172.21.15.29:47294) 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 -- 172.21.15.29:0/1177450844 learned_addr learned my addr 172.21.15.29:0/1177450844 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7fedacf1f640 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 0x55f77f0620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 -- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 msgr2=0x55f77f0620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 0x55f77f0620f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 -- 172.21.15.29:0/1177450844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 msgr2=0x55f77f06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f06de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 -- 172.21.15.29:0/1177450844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f77ef47780 con 0x55f77ec87660 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7fedacf1f640 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 0x55f77f0620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7feda7fff640 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec87660 0x55f77f061bb0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fed9c005660 tx=0x7fed9c00b310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273f6033468c934c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7fed87fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7fed87fff640 1 -- 172.21.15.29:0/1177450844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c011670 con 0x55f77ec87660 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7fed87fff640 1 -- 172.21.15.29:0/1177450844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fed9c011810 con 0x55f77ec87660 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.024+0000 7fedb56f7740 1 -- 172.21.15.29:0/1177450844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed9c004c30 con 0x55f77ec87660 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fed87fff640 1 -- 172.21.15.29:0/1177450844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fed9c032050 con 0x55f77ec87660 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedb56f7740 10 client.18129 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedb56f7740 1 -- 172.21.15.29:0/1177450844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f77ed1bb00 con 0x55f77ec87660 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fed87fff640 1 -- 172.21.15.29:0/1177450844 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed9c004c30 con 0x55f77ec87660 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fed87fff640 1 -- 172.21.15.29:0/1177450844 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fed9c027020 con 0x55f77ec87660 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedb56f7740 10 client.18129 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedb56f7740 10 client.18129 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedb56f7740 20 client.18129 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f77ec6eea0 0x55f77f073ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedb56f7740 4 client.18129 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:58:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedb56f7740 1 -- 172.21.15.29:0/1177450844 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f77efd0140 con 0x55f77ec6eea0 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.025+0000 7fedacf1f640 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f77ec6eea0 0x55f77f073ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.026+0000 7fedacf1f640 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f77ec6eea0 0x55f77f073ed0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=127c6aa9828be61c server_cookie=60ec80918a294a39 in_seq=0 out_seq=0 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.026+0000 7fed87fff640 10 client.18129 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fed87fff640 1 -- 172.21.15.29:0/1177450844 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f77efd0140 con 0x55f77ec6eea0 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fed87fff640 10 client.18129 handle_command_reply: tid=0 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 client.18129 shutdown 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 -- 172.21.15.29:0/1177450844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec87660 msgr2=0x55f77f061bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec87660 0x55f77f061bb0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fed9c005660 tx=0x7fed9c00b310 comp rx=0 tx=0).stop 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 -- 172.21.15.29:0/1177450844 shutdown_connections 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f06de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec8f840 0x55f77f0620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f77ec6eea0 0x55f77f073ed0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 --2- 172.21.15.29:0/1177450844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec87660 0x55f77f061bb0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 -- 172.21.15.29:0/1177450844 >> 172.21.15.29:0/1177450844 conn(0x55f77eca4910 msgr2=0x55f77f0596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fed87fff640 0 client.18129 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 -- 172.21.15.29:0/1177450844 shutdown_connections 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 1 -- 172.21.15.29:0/1177450844 wait complete. 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.027+0000 7fedb56f7740 20 client.18129 trim_cache size 0 max 16384 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.044+0000 7fedb56f7740 1 Processor -- start 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.044+0000 7fedb56f7740 1 -- start start 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 0x55f77f06ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f0630d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f05eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedb56f7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feda81b6ed0 con 0x55f77ec8f840 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedb56f7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feda81b8a10 con 0x55f77ecb0520 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedb56f7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feda81b75a0 con 0x55f77ec87660 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f0630d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f0630d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47308/0 (socket says 172.21.15.29:47308) 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 -- 172.21.15.29:0/3374571266 learned_addr learned my addr 172.21.15.29:0/3374571266 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7feda7fff640 1 --2- 172.21.15.29:0/3374571266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f05eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 -- 172.21.15.29:0/3374571266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 msgr2=0x55f77f06ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 --2- 172.21.15.29:0/3374571266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 0x55f77f06ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 -- 172.21.15.29:0/3374571266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 msgr2=0x55f77f05eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 --2- 172.21.15.29:0/3374571266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f05eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 -- 172.21.15.29:0/3374571266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f77ed92960 con 0x55f77ec8f840 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fedacf1f640 1 --2- 172.21.15.29:0/3374571266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f0630d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7feda819e240 tx=0x7fed9800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0104d8cdc68a59c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fed8e7fc640 1 -- 172.21.15.29:0/3374571266 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed98004220 con 0x55f77ec8f840 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fed8e7fc640 1 -- 172.21.15.29:0/3374571266 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed980043c0 con 0x55f77ec8f840 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.045+0000 7fed8e7fc640 1 -- 172.21.15.29:0/3374571266 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed980108c0 con 0x55f77ec8f840 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 -- 172.21.15.29:0/3374571266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 msgr2=0x55f77f0630d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3374571266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f0630d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7feda819e240 tx=0x7fed9800c490 comp rx=0 tx=0).stop 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 -- 172.21.15.29:0/3374571266 shutdown_connections 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3374571266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f05eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3374571266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f0630d0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3374571266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 0x55f77f06ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 -- 172.21.15.29:0/3374571266 >> 172.21.15.29:0/3374571266 conn(0x55f77eca4910 msgr2=0x55f77f059fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 -- 172.21.15.29:0/3374571266 shutdown_connections 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.046+0000 7fedb56f7740 1 -- 172.21.15.29:0/3374571266 wait complete. 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 1 -- start start 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 1 Processor -- start 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 0x55f77f0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f075d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f0d7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feda81b8340 con 0x55f77ec8f840 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feda81b7c70 con 0x55f77ecb0520 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feda81b4eb0 con 0x55f77ec87660 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedb56f7740 10 client.0 _ll_register_callbacks cb 0x7fed9c00bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedacf1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedacf1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47324/0 (socket says 172.21.15.29:47324) 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7fedacf1f640 1 -- 172.21.15.29:0/3540700847 learned_addr learned my addr 172.21.15.29:0/3540700847 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.047+0000 7feda7fff640 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f0d7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7feda7fff640 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 0x55f77f0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedacf1f640 1 -- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 msgr2=0x55f77f0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedacf1f640 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 0x55f77f0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedacf1f640 1 -- 172.21.15.29:0/3540700847 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 msgr2=0x55f77f0d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedacf1f640 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f0d7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedacf1f640 1 -- 172.21.15.29:0/3540700847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f77f0d7c90 con 0x55f77ec8f840 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7feda7fff640 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 0x55f77f0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedacf1f640 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f075d80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fed9800cce0 tx=0x7fed9800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54415aaccc4c1a08 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fed87fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fed87fff640 1 -- 172.21.15.29:0/3540700847 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed980038c0 con 0x55f77ec8f840 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fed87fff640 1 -- 172.21.15.29:0/3540700847 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fed98003a60 con 0x55f77ec8f840 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedb56f7740 1 -- 172.21.15.29:0/3540700847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed980043c0 con 0x55f77ec8f840 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fed87fff640 1 -- 172.21.15.29:0/3540700847 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fed98003c60 con 0x55f77ec8f840 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedb56f7740 10 client.18135 fetch_fsmap learned FSMap version 37 2024-09-21T02:58:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fedb56f7740 1 -- 172.21.15.29:0/3540700847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f77f0defd0 con 0x55f77ec8f840 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.048+0000 7fed87fff640 1 -- 172.21.15.29:0/3540700847 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed980043c0 con 0x55f77ec8f840 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7fed87fff640 1 -- 172.21.15.29:0/3540700847 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fed9802e420 con 0x55f77ec8f840 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7fedb56f7740 10 client.18135 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7fedb56f7740 10 client.18135 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7fedb56f7740 20 client.18135 populate_metadata read hostname 'smithi029' 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f77ec44250 0x55f77f0e1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7fedb56f7740 4 client.18135 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a998ab4c", "release": true, "await": true}] 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7fedb56f7740 1 -- 172.21.15.29:0/3540700847 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a998ab4c", "release": true, "await": true}) -- 0x55f77efcbbf0 con 0x55f77ec44250 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7feda7fff640 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f77ec44250 0x55f77f0e1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7fed87fff640 10 client.18135 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.049+0000 7feda7fff640 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f77ec44250 0x55f77f0e1620 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c6c16a8e6d9636f server_cookie=81e4e2e12c4508d2 in_seq=0 out_seq=0 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fed87fff640 1 -- 172.21.15.29:0/3540700847 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f77efcbbf0 con 0x55f77ec44250 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fed87fff640 10 client.18135 handle_command_reply: tid=0 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 client.18135 shutdown 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 -- 172.21.15.29:0/3540700847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 msgr2=0x55f77f075d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f075d80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fed9800cce0 tx=0x7fed9800cb70 comp rx=0 tx=0).stop 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 -- 172.21.15.29:0/3540700847 shutdown_connections 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55f77ecb0520 0x55f77f0d7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55f77ec44250 0x55f77f0e1620 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f77ec8f840 0x55f77f075d80 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 --2- 172.21.15.29:0/3540700847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f77ec87660 0x55f77f0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 -- 172.21.15.29:0/3540700847 >> 172.21.15.29:0/3540700847 conn(0x55f77eca4910 msgr2=0x55f77f059fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:53.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fed87fff640 0 client.18135 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:58:53.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 -- 172.21.15.29:0/3540700847 shutdown_connections 2024-09-21T02:58:53.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.127+0000 7fedb56f7740 1 -- 172.21.15.29:0/3540700847 wait complete. 2024-09-21T02:58:53.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.128+0000 7fedb56f7740 20 client.18135 trim_cache size 0 max 16384 2024-09-21T02:58:53.130 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T02:58:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T02:58:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T02:58:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 80, 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "a998ab4c": { 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 80, 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.2, 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T02:58:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T02:58:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T02:58:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T02:58:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T02:58:53.132 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.128+0000 7fedaf183640 1 -- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed900b6510 msgr2=0x7fed900b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.128+0000 7fedaf183640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed900b6510 0x7fed900b8a20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7feda00cbbc0 tx=0x7feda009bd90 comp rx=0 tx=0).stop 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.128+0000 7fedaf183640 1 -- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 msgr2=0x7feda80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.128+0000 7fedaf183640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 0x7feda80ffbb0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7feda0005cc0 tx=0x7feda0032c40 comp rx=0 tx=0).stop 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.129+0000 7fedaf183640 1 -- 172.21.15.29:0/4155834050 shutdown_connections 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.129+0000 7fedaf183640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fed900b6510 0x7fed900b8a20 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.129+0000 7fedaf183640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda8108d30 0x7feda80ffbb0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.129+0000 7fedaf183640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feda81083e0 0x7feda80ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.129+0000 7fedaf183640 1 --2- 172.21.15.29:0/4155834050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda8102aa0 0x7feda8101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.129+0000 7fedaf183640 1 -- 172.21.15.29:0/4155834050 >> 172.21.15.29:0/4155834050 conn(0x7feda8079750 msgr2=0x7feda8101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.129+0000 7fedaf183640 1 -- 172.21.15.29:0/4155834050 shutdown_connections 2024-09-21T02:58:53.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T02:58:53.129+0000 7fedaf183640 1 -- 172.21.15.29:0/4155834050 wait complete. 2024-09-21T02:58:53.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38523 5.12 MB/sec execute 270 sec latency 21771.077 ms 2024-09-21T02:58:54.008 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:53 smithi029 ceph-mon[31006]: pgmap v982: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-09-21T02:58:54.009 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a998ab4c', total seconds elapsed: 25.9 2024-09-21T02:58:54.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:53 smithi190 ceph-mon[33871]: pgmap v982: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-09-21T02:58:54.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:53 smithi066 ceph-mon[33436]: pgmap v982: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-09-21T02:58:54.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38668 5.11 MB/sec execute 271 sec latency 72.654 ms 2024-09-21T02:58:55.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38824 5.11 MB/sec execute 272 sec latency 87.098 ms 2024-09-21T02:58:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:55 smithi190 ceph-mon[33871]: pgmap v983: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 66 KiB/s wr, 20 op/s 2024-09-21T02:58:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:55 smithi029 ceph-mon[31006]: pgmap v983: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 66 KiB/s wr, 20 op/s 2024-09-21T02:58:56.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:55 smithi066 ceph-mon[33436]: pgmap v983: 129 pgs: 129 active+clean; 287 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 66 KiB/s wr, 20 op/s 2024-09-21T02:58:56.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38968 5.11 MB/sec execute 273 sec latency 73.672 ms 2024-09-21T02:58:57.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39122 5.11 MB/sec execute 274 sec latency 107.381 ms 2024-09-21T02:58:58.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:57 smithi190 ceph-mon[33871]: pgmap v984: 129 pgs: 129 active+clean; 288 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 200 KiB/s wr, 27 op/s 2024-09-21T02:58:58.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:57 smithi029 ceph-mon[31006]: pgmap v984: 129 pgs: 129 active+clean; 288 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 200 KiB/s wr, 27 op/s 2024-09-21T02:58:58.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:57 smithi066 ceph-mon[33436]: pgmap v984: 129 pgs: 129 active+clean; 288 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 200 KiB/s wr, 27 op/s 2024-09-21T02:58:58.733 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39255 5.11 MB/sec execute 275 sec latency 100.705 ms 2024-09-21T02:58:59.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39353 5.11 MB/sec execute 276 sec latency 463.378 ms 2024-09-21T02:59:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:58:59 smithi190 ceph-mon[33871]: pgmap v985: 129 pgs: 129 active+clean; 289 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 677 KiB/s wr, 83 op/s 2024-09-21T02:59:00.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:58:59 smithi029 ceph-mon[31006]: pgmap v985: 129 pgs: 129 active+clean; 289 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 677 KiB/s wr, 83 op/s 2024-09-21T02:59:00.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:58:59 smithi066 ceph-mon[33436]: pgmap v985: 129 pgs: 129 active+clean; 289 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 677 KiB/s wr, 83 op/s 2024-09-21T02:59:00.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39552 5.15 MB/sec execute 277 sec latency 90.203 ms 2024-09-21T02:59:01.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39740 5.17 MB/sec execute 278 sec latency 123.262 ms 2024-09-21T02:59:02.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:01 smithi190 ceph-mon[33871]: pgmap v986: 129 pgs: 129 active+clean; 289 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 989 KiB/s wr, 99 op/s 2024-09-21T02:59:02.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:01 smithi029 ceph-mon[31006]: pgmap v986: 129 pgs: 129 active+clean; 289 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 989 KiB/s wr, 99 op/s 2024-09-21T02:59:02.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:01 smithi066 ceph-mon[33436]: pgmap v986: 129 pgs: 129 active+clean; 289 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 989 KiB/s wr, 99 op/s 2024-09-21T02:59:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 -- 172.21.15.29:0/1157566434 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810ce00 msgr2=0x7fa4c810d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/1157566434 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810ce00 0x7fa4c810d290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009900 tx=0x7fa4bc02a3f0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 -- 172.21.15.29:0/1157566434 shutdown_connections 2024-09-21T02:59:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/1157566434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c810d9d0 0x7fa4c81163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/1157566434 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810ce00 0x7fa4c810d290 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/1157566434 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4c810a3c0 0x7fa4c810c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 -- 172.21.15.29:0/1157566434 >> 172.21.15.29:0/1157566434 conn(0x7fa4c8070c70 msgr2=0x7fa4c8073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:02.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 -- 172.21.15.29:0/1157566434 shutdown_connections 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.541+0000 7fa4cde9b640 1 -- 172.21.15.29:0/1157566434 wait complete. 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4cde9b640 1 Processor -- start 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4cde9b640 1 -- start start 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4cde9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c810a3c0 0x7fa4c810f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4cde9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4c810ce00 0x7fa4c810f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4cde9b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810d9d0 0x7fa4c8110330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4cde9b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c8119c50 con 0x7fa4c810a3c0 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4cde9b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c8119ad0 con 0x7fa4c810d9d0 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4cde9b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c8119dd0 con 0x7fa4c810ce00 2024-09-21T02:59:02.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4c7fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810d9d0 0x7fa4c8110330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4c6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4c810ce00 0x7fa4c810f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4c7fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810d9d0 0x7fa4c8110330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42908/0 (socket says 172.21.15.29:42908) 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4c7fff640 1 -- 172.21.15.29:0/3897545472 learned_addr learned my addr 172.21.15.29:0/3897545472 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.542+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c810a3c0 0x7fa4c810f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c7fff640 1 -- 172.21.15.29:0/3897545472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4c810ce00 msgr2=0x7fa4c810f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c7fff640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4c810ce00 0x7fa4c810f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c7fff640 1 -- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c810a3c0 msgr2=0x7fa4c810f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c7fff640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c810a3c0 0x7fa4c810f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c7fff640 1 -- 172.21.15.29:0/3897545472 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c8110b50 con 0x7fa4c810d9d0 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c7fff640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810d9d0 0x7fa4c8110330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800e9b0 tx=0x7fa4b800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c4ff9640 1 -- 172.21.15.29:0/3897545472 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b800cd80 con 0x7fa4c810d9d0 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4cde9b640 1 -- 172.21.15.29:0/3897545472 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c8110e40 con 0x7fa4c810d9d0 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c4ff9640 1 -- 172.21.15.29:0/3897545472 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4b80035f0 con 0x7fa4c810d9d0 2024-09-21T02:59:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4c4ff9640 1 -- 172.21.15.29:0/3897545472 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8010690 con 0x7fa4c810d9d0 2024-09-21T02:59:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.543+0000 7fa4cde9b640 1 -- 172.21.15.29:0/3897545472 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c81155a0 con 0x7fa4c810d9d0 2024-09-21T02:59:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.546+0000 7fa4d4e12740 1 Processor -- start 2024-09-21T02:59:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4c4ff9640 1 -- 172.21.15.29:0/3897545472 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fa4b8010830 con 0x7fa4c810d9d0 2024-09-21T02:59:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4d4e12740 1 -- start start 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 0x560038ce1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038ce3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560038c546f0 con 0x5600389112f0 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560038c43b20 con 0x5600389194d0 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560038c47930 con 0x56003893a1b0 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38396/0 (socket says 172.21.15.29:38396) 2024-09-21T02:59:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4c77fe640 1 -- 172.21.15.29:0/1187080290 learned_addr learned my addr 172.21.15.29:0/1187080290 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.547+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038ce3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.548+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 0x560038ce1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.548+0000 7fa4c77fe640 1 -- 172.21.15.29:0/1187080290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 msgr2=0x560038ce1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.548+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 0x560038ce1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.548+0000 7fa4c77fe640 1 -- 172.21.15.29:0/1187080290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 msgr2=0x560038ce3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.548+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038ce3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.548+0000 7fa4c77fe640 1 -- 172.21.15.29:0/1187080290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560038c09530 con 0x5600389112f0 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.549+0000 7fa4c4ff9640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4b00b6510 0x7fa4b00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.549+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce24c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000a4b0 tx=0x7fa4c00096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a270b0cc9f29e4d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.549+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4b00b6510 0x7fa4b00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.549+0000 7fa4c4ff9640 1 -- 172.21.15.29:0/3897545472 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa4b80e79e0 con 0x7fa4c810d9d0 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.549+0000 7fa4adffb640 1 -- 172.21.15.29:0/1187080290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0018020 con 0x5600389112f0 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.549+0000 7fa4adffb640 1 -- 172.21.15.29:0/1187080290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4c0014ca0 con 0x5600389112f0 2024-09-21T02:59:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.549+0000 7fa4adffb640 1 -- 172.21.15.29:0/1187080290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0013bf0 con 0x5600389112f0 2024-09-21T02:59:02.552 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1187080290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 msgr2=0x560038ce24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.552 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce24c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000a4b0 tx=0x7fa4c00096d0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.552 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4b00b6510 0x7fa4b00b8a20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009230 tx=0x7fa4bc009640 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:02.552 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1187080290 shutdown_connections 2024-09-21T02:59:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038ce3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce24c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1187080290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 0x560038ce1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1187080290 >> 172.21.15.29:0/1187080290 conn(0x56003892e300 msgr2=0x560038cdd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1187080290 shutdown_connections 2024-09-21T02:59:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.550+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1187080290 wait complete. 2024-09-21T02:59:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.551+0000 7fa4d4e12740 1 -- start start 2024-09-21T02:59:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.551+0000 7fa4d4e12740 1 Processor -- start 2024-09-21T02:59:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.551+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ceb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.551+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cebd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.551+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56003893a1b0 0x560038cec8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560038b7bab0 con 0x5600389112f0 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560038be9730 con 0x56003893a1b0 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560038ce3b30 con 0x5600389194d0 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ceb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ceb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38398/0 (socket says 172.21.15.29:38398) 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 -- 172.21.15.29:0/3045881232 learned_addr learned my addr 172.21.15.29:0/3045881232 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4d4e12740 10 client.0 _ll_register_callbacks cb 0x560038c3d6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c7fff640 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56003893a1b0 0x560038cec8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 -- 172.21.15.29:0/3045881232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 msgr2=0x560038cebd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cebd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 -- 172.21.15.29:0/3045881232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56003893a1b0 msgr2=0x560038cec8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56003893a1b0 0x560038cec8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 -- 172.21.15.29:0/3045881232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600389a5790 con 0x5600389112f0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ceb670 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0014c70 tx=0x7fa4c0013650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd07ec1db221705b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.552+0000 7fa4a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/3045881232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0013460 con 0x5600389112f0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4d4e12740 1 -- 172.21.15.29:0/3045881232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c0014ca0 con 0x5600389112f0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/3045881232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa4c0013b70 con 0x5600389112f0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/3045881232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa4c0013d70 con 0x5600389112f0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4d4e12740 10 client.18147 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4d4e12740 1 -- 172.21.15.29:0/3045881232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560038a1e620 con 0x5600389112f0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/3045881232 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4c0014ca0 con 0x5600389112f0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/3045881232 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fa4c001d430 con 0x5600389112f0 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4d4e12740 10 client.18147 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4d4e12740 10 client.18147 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4d4e12740 20 client.18147 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.553+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5600388f8d80 0x560038cf8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.554+0000 7fa4d4e12740 4 client.18147 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:59:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.554+0000 7fa4d4e12740 1 -- 172.21.15.29:0/3045881232 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560038c56640 con 0x5600388f8d80 2024-09-21T02:59:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.554+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5600388f8d80 0x560038cf8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.554+0000 7fa4a6ffd640 10 client.18147 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.554+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5600388f8d80 0x560038cf8a70 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=156819bbf542fbf4 server_cookie=e6f36879d0d61ea7 in_seq=0 out_seq=0 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/3045881232 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560038c56640 con 0x5600388f8d80 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4a6ffd640 10 client.18147 handle_command_reply: tid=0 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 client.18147 shutdown 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 -- 172.21.15.29:0/3045881232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 msgr2=0x560038ceb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ceb670 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0014c70 tx=0x7fa4c0013650 comp rx=0 tx=0).stop 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 -- 172.21.15.29:0/3045881232 shutdown_connections 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56003893a1b0 0x560038cec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cebd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5600388f8d80 0x560038cf8a70 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/3045881232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ceb670 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 -- 172.21.15.29:0/3045881232 >> 172.21.15.29:0/3045881232 conn(0x56003892e300 msgr2=0x560038cdbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4a6ffd640 0 client.18147 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 -- 172.21.15.29:0/3045881232 shutdown_connections 2024-09-21T02:59:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 1 -- 172.21.15.29:0/3045881232 wait complete. 2024-09-21T02:59:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.557+0000 7fa4d4e12740 20 client.18147 trim_cache size 0 max 16384 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.578+0000 7fa4d4e12740 1 Processor -- start 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.578+0000 7fa4d4e12740 1 -- start start 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038cf3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 0x560038ce2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c81c3c50 con 0x5600389112f0 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c81c5790 con 0x5600389194d0 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c81c4320 con 0x56003893a1b0 2024-09-21T02:59:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038cf3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038cf3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42926/0 (socket says 172.21.15.29:42926) 2024-09-21T02:59:02.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 -- 172.21.15.29:0/728179325 learned_addr learned my addr 172.21.15.29:0/728179325 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:02.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c7fff640 1 --2- 172.21.15.29:0/728179325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 0x560038ce2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 -- 172.21.15.29:0/728179325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 msgr2=0x560038ce2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/728179325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 0x560038ce2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 -- 172.21.15.29:0/728179325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 msgr2=0x560038ce1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/728179325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 -- 172.21.15.29:0/728179325 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560038cecc10 con 0x5600389194d0 2024-09-21T02:59:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.579+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/728179325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038cf3780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009900 tx=0x7fa4bc03f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c598b5a8432fcfc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.580+0000 7fa4ad7fa640 1 -- 172.21.15.29:0/728179325 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc035040 con 0x5600389194d0 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.580+0000 7fa4ad7fa640 1 -- 172.21.15.29:0/728179325 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4bc003d30 con 0x5600389194d0 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.580+0000 7fa4ad7fa640 1 -- 172.21.15.29:0/728179325 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc04b410 con 0x5600389194d0 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 -- 172.21.15.29:0/728179325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 msgr2=0x560038cf3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/728179325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038cf3780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009900 tx=0x7fa4bc03f070 comp rx=0 tx=0).stop 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 -- 172.21.15.29:0/728179325 shutdown_connections 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/728179325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56003893a1b0 0x560038ce2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/728179325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389194d0 0x560038cf3780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/728179325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5600389112f0 0x560038ce1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 -- 172.21.15.29:0/728179325 >> 172.21.15.29:0/728179325 conn(0x56003892e300 msgr2=0x560038cdc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 -- 172.21.15.29:0/728179325 shutdown_connections 2024-09-21T02:59:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 -- 172.21.15.29:0/728179325 wait complete. 2024-09-21T02:59:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.581+0000 7fa4d4e12740 1 -- start start 2024-09-21T02:59:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4d4e12740 1 Processor -- start 2024-09-21T02:59:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389112f0 0x560038cf05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cf0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4d4e12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56003893a1b0 0x560038cf1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c81c50c0 con 0x56003893a1b0 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c81c49f0 con 0x5600389112f0 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4d4e12740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c81c1c30 con 0x5600389194d0 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4d4e12740 10 client.0 _ll_register_callbacks cb 0x7fa4bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56003893a1b0 0x560038cf1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56003893a1b0 0x560038cf1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38404/0 (socket says 172.21.15.29:38404) 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4c7fff640 1 -- 172.21.15.29:0/1989286247 learned_addr learned my addr 172.21.15.29:0/1989286247 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cf0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.582+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389112f0 0x560038cf05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4c7fff640 1 -- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 msgr2=0x560038cf0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4c7fff640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cf0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4c7fff640 1 -- 172.21.15.29:0/1989286247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389112f0 msgr2=0x560038cf05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4c7fff640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389112f0 0x560038cf05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4c7fff640 1 -- 172.21.15.29:0/1989286247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560038cf1ef0 con 0x56003893a1b0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4c6ffd640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cf0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389112f0 0x560038cf05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4c7fff640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56003893a1b0 0x560038cf1790 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80a9fb0 tx=0x7fa4b80095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6a31dd4437204d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/1989286247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80d2070 con 0x56003893a1b0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/1989286247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa4b8010d20 con 0x56003893a1b0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1989286247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4bc003d30 con 0x56003893a1b0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/1989286247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa4b80b22d0 con 0x56003893a1b0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4d4e12740 10 client.18153 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1989286247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560038cf32b0 con 0x56003893a1b0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.583+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/1989286247 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4bc003d30 con 0x56003893a1b0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/1989286247 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fa4b80ac4c0 con 0x56003893a1b0 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4d4e12740 10 client.18153 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4d4e12740 10 client.18153 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4d4e12740 20 client.18153 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5600388ce1c0 0x560038d653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4d4e12740 4 client.18153 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1989286247 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560038c520f0 con 0x5600388ce1c0 2024-09-21T02:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5600388ce1c0 0x560038d653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4a6ffd640 10 client.18153 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.584+0000 7fa4c77fe640 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5600388ce1c0 0x560038d653f0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e87ffc75b332db16 server_cookie=bea7dd4582c35758 in_seq=0 out_seq=0 2024-09-21T02:59:02.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39882 5.18 MB/sec execute 279 sec latency 170.262 ms 2024-09-21T02:59:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4a6ffd640 1 -- 172.21.15.29:0/1989286247 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560038c520f0 con 0x5600388ce1c0 2024-09-21T02:59:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4a6ffd640 10 client.18153 handle_command_reply: tid=0 2024-09-21T02:59:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 client.18153 shutdown 2024-09-21T02:59:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1989286247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56003893a1b0 msgr2=0x560038cf1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56003893a1b0 0x560038cf1790 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80a9fb0 tx=0x7fa4b80095b0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1989286247 shutdown_connections 2024-09-21T02:59:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5600388ce1c0 0x560038d653f0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56003893a1b0 0x560038cf1790 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600389194d0 0x560038cf0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4a6ffd640 0 client.18153 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 --2- 172.21.15.29:0/1989286247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5600389112f0 0x560038cf05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1989286247 >> 172.21.15.29:0/1989286247 conn(0x56003892e300 msgr2=0x560038cdc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1989286247 shutdown_connections 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.762+0000 7fa4d4e12740 1 -- 172.21.15.29:0/1989286247 wait complete. 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4d4e12740 20 client.18153 trim_cache size 0 max 16384 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:59:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:59:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:59:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 -- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4b00b6510 msgr2=0x7fa4b00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4b00b6510 0x7fa4b00b8a20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009230 tx=0x7fa4bc009640 comp rx=0 tx=0).stop 2024-09-21T02:59:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 -- 172.21.15.29:0/3897545472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810d9d0 msgr2=0x7fa4c8110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810d9d0 0x7fa4c8110330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800e9b0 tx=0x7fa4b800ee70 comp rx=0 tx=0).stop 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 -- 172.21.15.29:0/3897545472 shutdown_connections 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa4b00b6510 0x7fa4b00b8a20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4c810d9d0 0x7fa4c8110330 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4c810ce00 0x7fa4c810f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 --2- 172.21.15.29:0/3897545472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c810a3c0 0x7fa4c810f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.763+0000 7fa4cde9b640 1 -- 172.21.15.29:0/3897545472 >> 172.21.15.29:0/3897545472 conn(0x7fa4c8070c70 msgr2=0x7fa4c819f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.764+0000 7fa4cde9b640 1 -- 172.21.15.29:0/3897545472 shutdown_connections 2024-09-21T02:59:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:02.764+0000 7fa4cde9b640 1 -- 172.21.15.29:0/3897545472 wait complete. 2024-09-21T02:59:03.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40062 5.21 MB/sec execute 280 sec latency 196.641 ms 2024-09-21T02:59:04.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:03 smithi190 ceph-mon[33871]: pgmap v987: 129 pgs: 129 active+clean; 292 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.6 MiB/s wr, 122 op/s 2024-09-21T02:59:04.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:03 smithi029 ceph-mon[31006]: pgmap v987: 129 pgs: 129 active+clean; 292 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.6 MiB/s wr, 122 op/s 2024-09-21T02:59:04.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:03 smithi066 ceph-mon[33436]: pgmap v987: 129 pgs: 129 active+clean; 292 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.6 MiB/s wr, 122 op/s 2024-09-21T02:59:04.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40212 5.24 MB/sec execute 281 sec latency 88.185 ms 2024-09-21T02:59:05.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40341 5.25 MB/sec execute 282 sec latency 99.115 ms 2024-09-21T02:59:06.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:05 smithi190 ceph-mon[33871]: pgmap v988: 129 pgs: 129 active+clean; 299 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 2.8 MiB/s wr, 210 op/s 2024-09-21T02:59:06.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:05 smithi029 ceph-mon[31006]: pgmap v988: 129 pgs: 129 active+clean; 299 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 2.8 MiB/s wr, 210 op/s 2024-09-21T02:59:06.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:05 smithi066 ceph-mon[33436]: pgmap v988: 129 pgs: 129 active+clean; 299 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 2.8 MiB/s wr, 210 op/s 2024-09-21T02:59:06.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40513 5.29 MB/sec execute 283 sec latency 107.471 ms 2024-09-21T02:59:07.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40667 5.30 MB/sec execute 284 sec latency 108.763 ms 2024-09-21T02:59:08.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:07 smithi190 ceph-mon[33871]: pgmap v989: 129 pgs: 129 active+clean; 300 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 3.2 MiB/s wr, 238 op/s 2024-09-21T02:59:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:07 smithi029 ceph-mon[31006]: pgmap v989: 129 pgs: 129 active+clean; 300 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 3.2 MiB/s wr, 238 op/s 2024-09-21T02:59:08.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:07 smithi066 ceph-mon[33436]: pgmap v989: 129 pgs: 129 active+clean; 300 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 3.2 MiB/s wr, 238 op/s 2024-09-21T02:59:08.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40760 5.29 MB/sec execute 285 sec latency 87.206 ms 2024-09-21T02:59:09.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40898 5.31 MB/sec execute 286 sec latency 97.931 ms 2024-09-21T02:59:10.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:09 smithi029 ceph-mon[31006]: pgmap v990: 129 pgs: 129 active+clean; 301 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 3.9 MiB/s wr, 288 op/s 2024-09-21T02:59:10.123 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:09 smithi190 ceph-mon[33871]: pgmap v990: 129 pgs: 129 active+clean; 301 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 3.9 MiB/s wr, 288 op/s 2024-09-21T02:59:10.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:09 smithi066 ceph-mon[33436]: pgmap v990: 129 pgs: 129 active+clean; 301 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 3.9 MiB/s wr, 288 op/s 2024-09-21T02:59:10.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41021 5.31 MB/sec execute 287 sec latency 94.636 ms 2024-09-21T02:59:11.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41156 5.32 MB/sec execute 288 sec latency 119.427 ms 2024-09-21T02:59:12.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:11 smithi190 ceph-mon[33871]: pgmap v991: 129 pgs: 129 active+clean; 304 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-09-21T02:59:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:11 smithi029 ceph-mon[31006]: pgmap v991: 129 pgs: 129 active+clean; 304 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-09-21T02:59:12.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:11 smithi066 ceph-mon[33436]: pgmap v991: 129 pgs: 129 active+clean; 304 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-09-21T02:59:12.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41277 5.31 MB/sec execute 289 sec latency 122.858 ms 2024-09-21T02:59:13.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41417 5.30 MB/sec execute 290 sec latency 96.198 ms 2024-09-21T02:59:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:13 smithi190 ceph-mon[33871]: pgmap v992: 129 pgs: 129 active+clean; 307 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 4.5 MiB/s wr, 264 op/s 2024-09-21T02:59:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:13 smithi029 ceph-mon[31006]: pgmap v992: 129 pgs: 129 active+clean; 307 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 4.5 MiB/s wr, 264 op/s 2024-09-21T02:59:14.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:13 smithi066 ceph-mon[33436]: pgmap v992: 129 pgs: 129 active+clean; 307 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 4.5 MiB/s wr, 264 op/s 2024-09-21T02:59:14.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41546 5.29 MB/sec execute 291 sec latency 120.022 ms 2024-09-21T02:59:15.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41698 5.31 MB/sec execute 292 sec latency 124.997 ms 2024-09-21T02:59:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/792273895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f681068b0 msgr2=0x7f9f68108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/792273895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f681068b0 0x7f9f68108cd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009900 tx=0x7f9f5c02a410 comp rx=0 tx=0).stop 2024-09-21T02:59:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/792273895 shutdown_connections 2024-09-21T02:59:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/792273895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68109410 0x7f9f6810b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/792273895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f681068b0 0x7f9f68108cd0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/792273895 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f6806c450 0x7f9f68106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/792273895 >> 172.21.15.29:0/792273895 conn(0x7f9f680fdf30 msgr2=0x7f9f681003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:15.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/792273895 shutdown_connections 2024-09-21T02:59:15.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.904+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/792273895 wait complete. 2024-09-21T02:59:15.908 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.905+0000 7f9f6dcbd640 1 Processor -- start 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.905+0000 7f9f6dcbd640 1 -- start start 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f6dcbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f6806c450 0x7f9f68197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f6dcbd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f681068b0 0x7f9f681982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f6dcbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68109410 0x7f9f681a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f6dcbd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6810f1d0 con 0x7f9f6806c450 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f6dcbd640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6810f050 con 0x7f9f681068b0 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f6dcbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6810f350 con 0x7f9f68109410 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f6806c450 0x7f9f68197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f6806c450 0x7f9f68197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36716/0 (socket says 172.21.15.29:36716) 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f677fe640 1 -- 172.21.15.29:0/2452717747 learned_addr learned my addr 172.21.15.29:0/2452717747 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f67fff640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68109410 0x7f9f681a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.906+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f681068b0 0x7f9f681982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.907+0000 7f9f67fff640 1 -- 172.21.15.29:0/2452717747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f681068b0 msgr2=0x7f9f681982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.907+0000 7f9f67fff640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f681068b0 0x7f9f681982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.907+0000 7f9f67fff640 1 -- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f6806c450 msgr2=0x7f9f68197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.907+0000 7f9f67fff640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f6806c450 0x7f9f68197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.907+0000 7f9f67fff640 1 -- 172.21.15.29:0/2452717747 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f681a1ef0 con 0x7f9f68109410 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.907+0000 7f9f677fe640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f6806c450 0x7f9f68197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.907+0000 7f9f67fff640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68109410 0x7f9f681a16d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009fd0 tx=0x7f9f6000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.907+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2452717747 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000ce20 con 0x7f9f68109410 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.908+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/2452717747 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f681a21e0 con 0x7f9f68109410 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.908+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/2452717747 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f681a88b0 con 0x7f9f68109410 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.908+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2452717747 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f60005020 con 0x7f9f68109410 2024-09-21T02:59:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.908+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2452717747 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60010690 con 0x7f9f68109410 2024-09-21T02:59:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.911+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2452717747 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f9f60004a40 con 0x7f9f68109410 2024-09-21T02:59:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.911+0000 7f9f74c2b740 1 Processor -- start 2024-09-21T02:59:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f74c2b740 1 -- start start 2024-09-21T02:59:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 0x560408142e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x5604081434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x560408144010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5604080b56f0 con 0x560407d722f0 2024-09-21T02:59:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5604080a4b20 con 0x560407d7a4d0 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5604080a8930 con 0x560407d9b1b0 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f64ff9640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f500b6590 0x7f9f500b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x5604081434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x5604081434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36732/0 (socket says 172.21.15.29:36732) 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f677fe640 1 -- 172.21.15.29:0/3705333147 learned_addr learned my addr 172.21.15.29:0/3705333147 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x560408144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 0x560408142e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.912+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2452717747 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9f600e7ba0 con 0x7f9f68109410 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f67fff640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f500b6590 0x7f9f500b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f66ffd640 1 -- 172.21.15.29:0/3705333147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 msgr2=0x560408142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 0x560408142e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f66ffd640 1 -- 172.21.15.29:0/3705333147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 msgr2=0x5604081434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x5604081434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f66ffd640 1 -- 172.21.15.29:0/3705333147 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56040806a530 con 0x560407d7a4d0 2024-09-21T02:59:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f677fe640 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x5604081434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:59:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x560408144010 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009230 tx=0x7f9f5c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c167eea3972bfca3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.913+0000 7f9f67fff640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f500b6590 0x7f9f500b8aa0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9f600e5690 tx=0x7f9f60005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.914+0000 7f9f4dffb640 1 -- 172.21.15.29:0/3705333147 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c035040 con 0x560407d7a4d0 2024-09-21T02:59:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.914+0000 7f9f4dffb640 1 -- 172.21.15.29:0/3705333147 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f5c004030 con 0x560407d7a4d0 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.914+0000 7f9f4dffb640 1 -- 172.21.15.29:0/3705333147 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c003850 con 0x560407d7a4d0 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 -- 172.21.15.29:0/3705333147 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 msgr2=0x560408144010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x560408144010 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009230 tx=0x7f9f5c005fd0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 -- 172.21.15.29:0/3705333147 shutdown_connections 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x560408144010 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x5604081434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/3705333147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 0x560408142e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 -- 172.21.15.29:0/3705333147 >> 172.21.15.29:0/3705333147 conn(0x560407d8f300 msgr2=0x56040813e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 -- 172.21.15.29:0/3705333147 shutdown_connections 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 -- 172.21.15.29:0/3705333147 wait complete. 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 -- start start 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.915+0000 7f9f74c2b740 1 Processor -- start 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x56040814c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x56040814ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 0x56040814d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560407fdcab0 con 0x560407d722f0 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56040804a730 con 0x560407d7a4d0 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560408144ad0 con 0x560407d9b1b0 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f66ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x56040814c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f66ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x56040814c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36748/0 (socket says 172.21.15.29:36748) 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f66ffd640 1 -- 172.21.15.29:0/240637250 learned_addr learned my addr 172.21.15.29:0/240637250 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f74c2b740 10 client.0 _ll_register_callbacks cb 0x56040809e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f677fe640 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x56040814ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.916+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 0x56040814d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f66ffd640 1 -- 172.21.15.29:0/240637250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 msgr2=0x56040814d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 0x56040814d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f66ffd640 1 -- 172.21.15.29:0/240637250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 msgr2=0x56040814ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x56040814ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f66ffd640 1 -- 172.21.15.29:0/240637250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560407e06790 con 0x560407d722f0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f677fe640 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x56040814ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x56040814c600 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c005ab0 tx=0x7f9f5c031420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aba08474c05135d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f46ffd640 1 -- 172.21.15.29:0/240637250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c052060 con 0x560407d722f0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f46ffd640 1 -- 172.21.15.29:0/240637250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9f5c0037f0 con 0x560407d722f0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f74c2b740 1 -- 172.21.15.29:0/240637250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f5c004030 con 0x560407d722f0 2024-09-21T02:59:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f46ffd640 1 -- 172.21.15.29:0/240637250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9f5c059050 con 0x560407d722f0 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f74c2b740 10 client.18165 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.917+0000 7f9f74c2b740 1 -- 172.21.15.29:0/240637250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560407e7f620 con 0x560407d722f0 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f46ffd640 1 -- 172.21.15.29:0/240637250 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f5c004030 con 0x560407d722f0 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f46ffd640 1 -- 172.21.15.29:0/240637250 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f9f5c04b970 con 0x560407d722f0 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f74c2b740 10 client.18165 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f74c2b740 10 client.18165 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f74c2b740 20 client.18165 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560407d59d80 0x5604081598f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f74c2b740 4 client.18165 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f74c2b740 1 -- 172.21.15.29:0/240637250 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604080b7640 con 0x560407d59d80 2024-09-21T02:59:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.918+0000 7f9f677fe640 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560407d59d80 0x5604081598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.919+0000 7f9f46ffd640 10 client.18165 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.919+0000 7f9f677fe640 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560407d59d80 0x5604081598f0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5744bfb1682c295 server_cookie=f491f0b1605e34d0 in_seq=0 out_seq=0 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.919+0000 7f9f46ffd640 1 -- 172.21.15.29:0/240637250 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5604080b7640 con 0x560407d59d80 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.919+0000 7f9f46ffd640 10 client.18165 handle_command_reply: tid=0 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.919+0000 7f9f74c2b740 1 client.18165 shutdown 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 -- 172.21.15.29:0/240637250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 msgr2=0x56040814c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x56040814c600 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c005ab0 tx=0x7f9f5c031420 comp rx=0 tx=0).stop 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 -- 172.21.15.29:0/240637250 shutdown_connections 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d9b1b0 0x56040814d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d7a4d0 0x56040814ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560407d59d80 0x5604081598f0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/240637250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d722f0 0x56040814c600 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f46ffd640 0 client.18165 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 -- 172.21.15.29:0/240637250 >> 172.21.15.29:0/240637250 conn(0x560407d8f300 msgr2=0x56040813cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 -- 172.21.15.29:0/240637250 shutdown_connections 2024-09-21T02:59:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 1 -- 172.21.15.29:0/240637250 wait complete. 2024-09-21T02:59:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.920+0000 7f9f74c2b740 20 client.18165 trim_cache size 0 max 16384 2024-09-21T02:59:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.946+0000 7f9f74c2b740 1 Processor -- start 2024-09-21T02:59:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.946+0000 7f9f74c2b740 1 -- start start 2024-09-21T02:59:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d7a4d0 0x5604081395c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d9b1b0 0x56040813d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d2f1c0 0x56040813db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5400fb50 con 0x560407d9b1b0 2024-09-21T02:59:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9f54011690 con 0x560407d2f1c0 2024-09-21T02:59:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9f54010220 con 0x560407d7a4d0 2024-09-21T02:59:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d9b1b0 0x56040813d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d9b1b0 0x56040813d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36758/0 (socket says 172.21.15.29:36758) 2024-09-21T02:59:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f677fe640 1 -- 172.21.15.29:0/611484859 learned_addr learned my addr 172.21.15.29:0/611484859 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d7a4d0 0x5604081395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d2f1c0 0x56040813db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f677fe640 1 -- 172.21.15.29:0/611484859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d7a4d0 msgr2=0x5604081395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f677fe640 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d7a4d0 0x5604081395c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f677fe640 1 -- 172.21.15.29:0/611484859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d2f1c0 msgr2=0x56040813db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f677fe640 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d2f1c0 0x56040813db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.947+0000 7f9f677fe640 1 -- 172.21.15.29:0/611484859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56040814d520 con 0x560407d9b1b0 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.948+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d2f1c0 0x56040813db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.948+0000 7f9f677fe640 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d9b1b0 0x56040813d650 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f9f580096e0 tx=0x7f9f58005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becf638cc7e6beac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.948+0000 7f9f4d7fa640 1 -- 172.21.15.29:0/611484859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58004030 con 0x560407d9b1b0 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.948+0000 7f9f4d7fa640 1 -- 172.21.15.29:0/611484859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f58005020 con 0x560407d9b1b0 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.948+0000 7f9f4d7fa640 1 -- 172.21.15.29:0/611484859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58004b20 con 0x560407d9b1b0 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.949+0000 7f9f74c2b740 1 -- 172.21.15.29:0/611484859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d9b1b0 msgr2=0x56040813d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.949+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d9b1b0 0x56040813d650 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f9f580096e0 tx=0x7f9f58005f90 comp rx=0 tx=0).stop 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.949+0000 7f9f74c2b740 1 -- 172.21.15.29:0/611484859 shutdown_connections 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.949+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d2f1c0 0x56040813db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.949+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d9b1b0 0x56040813d650 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.949+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/611484859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d7a4d0 0x5604081395c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.949+0000 7f9f74c2b740 1 -- 172.21.15.29:0/611484859 >> 172.21.15.29:0/611484859 conn(0x560407d8f300 msgr2=0x56040813e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.950+0000 7f9f74c2b740 1 -- 172.21.15.29:0/611484859 shutdown_connections 2024-09-21T02:59:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.950+0000 7f9f74c2b740 1 -- 172.21.15.29:0/611484859 wait complete. 2024-09-21T02:59:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.950+0000 7f9f74c2b740 1 -- start start 2024-09-21T02:59:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.950+0000 7f9f74c2b740 1 Processor -- start 2024-09-21T02:59:15.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d2f1c0 0x56040814fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d7a4d0 0x5604081500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f74c2b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d9b1b0 0x560408150c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f54010fc0 con 0x560407d7a4d0 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f9f540108f0 con 0x560407d9b1b0 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f74c2b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5400db30 con 0x560407d2f1c0 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f74c2b740 10 client.0 _ll_register_callbacks cb 0x7f9f5c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d7a4d0 0x5604081500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d7a4d0 0x5604081500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36766/0 (socket says 172.21.15.29:36766) 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 -- 172.21.15.29:0/1108171535 learned_addr learned my addr 172.21.15.29:0/1108171535 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d2f1c0 0x56040814fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d9b1b0 0x560408150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 -- 172.21.15.29:0/1108171535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d2f1c0 msgr2=0x56040814fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d2f1c0 0x56040814fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 -- 172.21.15.29:0/1108171535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d9b1b0 msgr2=0x560408150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d9b1b0 0x560408150c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 -- 172.21.15.29:0/1108171535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560408151390 con 0x560407d7a4d0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f677fe640 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d7a4d0 0x5604081500c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800ffc0 tx=0x7f9f580051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fcd8e7b2178073 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.951+0000 7f9f46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.952+0000 7f9f46ffd640 1 -- 172.21.15.29:0/1108171535 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58004240 con 0x560407d7a4d0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.952+0000 7f9f74c2b740 1 -- 172.21.15.29:0/1108171535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f58005020 con 0x560407d7a4d0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.952+0000 7f9f46ffd640 1 -- 172.21.15.29:0/1108171535 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f9f580043e0 con 0x560407d7a4d0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.952+0000 7f9f46ffd640 1 -- 172.21.15.29:0/1108171535 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9f58036050 con 0x560407d7a4d0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.952+0000 7f9f74c2b740 10 client.18171 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.952+0000 7f9f74c2b740 1 -- 172.21.15.29:0/1108171535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560408154730 con 0x560407d7a4d0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.952+0000 7f9f46ffd640 1 -- 172.21.15.29:0/1108171535 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f58005020 con 0x560407d7a4d0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.953+0000 7f9f46ffd640 1 -- 172.21.15.29:0/1108171535 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f9f580038f0 con 0x560407d7a4d0 2024-09-21T02:59:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.953+0000 7f9f74c2b740 10 client.18171 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.953+0000 7f9f74c2b740 10 client.18171 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.953+0000 7f9f74c2b740 20 client.18171 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.953+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5604081c44e0 0x5604081c6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.953+0000 7f9f74c2b740 4 client.18171 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:59:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.953+0000 7f9f74c2b740 1 -- 172.21.15.29:0/1108171535 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5604080b30f0 con 0x5604081c44e0 2024-09-21T02:59:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.953+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5604081c44e0 0x5604081c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.954+0000 7f9f46ffd640 10 client.18171 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:15.954+0000 7f9f66ffd640 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5604081c44e0 0x5604081c6900 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8cd8cce9a7dfc2 server_cookie=490d668ff95ca728 in_seq=0 out_seq=0 2024-09-21T02:59:16.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:15 smithi190 ceph-mon[33871]: pgmap v993: 129 pgs: 129 active+clean; 308 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 4.6 MiB/s wr, 279 op/s 2024-09-21T02:59:16.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:15 smithi029 ceph-mon[31006]: pgmap v993: 129 pgs: 129 active+clean; 308 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 4.6 MiB/s wr, 279 op/s 2024-09-21T02:59:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f46ffd640 1 -- 172.21.15.29:0/1108171535 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5604080b30f0 con 0x5604081c44e0 2024-09-21T02:59:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f46ffd640 10 client.18171 handle_command_reply: tid=0 2024-09-21T02:59:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 client.18171 shutdown 2024-09-21T02:59:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 -- 172.21.15.29:0/1108171535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d7a4d0 msgr2=0x5604081500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d7a4d0 0x5604081500c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800ffc0 tx=0x7f9f580051d0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 -- 172.21.15.29:0/1108171535 shutdown_connections 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560407d9b1b0 0x560408150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5604081c44e0 0x5604081c6900 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560407d7a4d0 0x5604081500c0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f46ffd640 0 client.18171 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 --2- 172.21.15.29:0/1108171535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560407d2f1c0 0x56040814fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 -- 172.21.15.29:0/1108171535 >> 172.21.15.29:0/1108171535 conn(0x560407d8f300 msgr2=0x5604081395c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 -- 172.21.15.29:0/1108171535 shutdown_connections 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 1 -- 172.21.15.29:0/1108171535 wait complete. 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.159+0000 7f9f74c2b740 20 client.18171 trim_cache size 0 max 16384 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:59:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:59:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:59:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:59:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f500b6590 msgr2=0x7f9f500b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f500b6590 0x7f9f500b8aa0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9f600e5690 tx=0x7f9f60005f70 comp rx=0 tx=0).stop 2024-09-21T02:59:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/2452717747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68109410 msgr2=0x7f9f681a16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68109410 0x7f9f681a16d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009fd0 tx=0x7f9f6000ef10 comp rx=0 tx=0).stop 2024-09-21T02:59:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/2452717747 shutdown_connections 2024-09-21T02:59:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f9f500b6590 0x7f9f500b8aa0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68109410 0x7f9f681a16d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9f681068b0 0x7f9f681982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 --2- 172.21.15.29:0/2452717747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f6806c450 0x7f9f68197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/2452717747 >> 172.21.15.29:0/2452717747 conn(0x7f9f680fdf30 msgr2=0x7f9f68104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/2452717747 shutdown_connections 2024-09-21T02:59:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:16.160+0000 7f9f6dcbd640 1 -- 172.21.15.29:0/2452717747 wait complete. 2024-09-21T02:59:16.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:15 smithi066 ceph-mon[33436]: pgmap v993: 129 pgs: 129 active+clean; 308 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 4.6 MiB/s wr, 279 op/s 2024-09-21T02:59:16.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41810 5.30 MB/sec execute 293 sec latency 235.990 ms 2024-09-21T02:59:17.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41978 5.32 MB/sec execute 294 sec latency 151.677 ms 2024-09-21T02:59:18.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:17 smithi190 ceph-mon[33871]: pgmap v994: 129 pgs: 129 active+clean; 305 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 3.6 MiB/s wr, 223 op/s 2024-09-21T02:59:18.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:17 smithi029 ceph-mon[31006]: pgmap v994: 129 pgs: 129 active+clean; 305 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 3.6 MiB/s wr, 223 op/s 2024-09-21T02:59:18.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:17 smithi066 ceph-mon[33436]: pgmap v994: 129 pgs: 129 active+clean; 305 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 3.6 MiB/s wr, 223 op/s 2024-09-21T02:59:18.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42135 5.35 MB/sec execute 295 sec latency 146.897 ms 2024-09-21T02:59:19.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42210 5.34 MB/sec execute 296 sec latency 338.579 ms 2024-09-21T02:59:20.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:19 smithi190 ceph-mon[33871]: pgmap v995: 129 pgs: 129 active+clean; 299 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 3.9 MiB/s wr, 289 op/s 2024-09-21T02:59:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:19 smithi029 ceph-mon[31006]: pgmap v995: 129 pgs: 129 active+clean; 299 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 3.9 MiB/s wr, 289 op/s 2024-09-21T02:59:20.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:19 smithi066 ceph-mon[33436]: pgmap v995: 129 pgs: 129 active+clean; 299 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 3.9 MiB/s wr, 289 op/s 2024-09-21T02:59:20.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42347 5.34 MB/sec execute 297 sec latency 99.180 ms 2024-09-21T02:59:21.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42530 5.36 MB/sec execute 298 sec latency 117.070 ms 2024-09-21T02:59:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:21 smithi190 ceph-mon[33871]: pgmap v996: 129 pgs: 129 active+clean; 296 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 3.5 MiB/s wr, 293 op/s 2024-09-21T02:59:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:21 smithi029 ceph-mon[31006]: pgmap v996: 129 pgs: 129 active+clean; 296 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 3.5 MiB/s wr, 293 op/s 2024-09-21T02:59:22.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:21 smithi066 ceph-mon[33436]: pgmap v996: 129 pgs: 129 active+clean; 296 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 3.5 MiB/s wr, 293 op/s 2024-09-21T02:59:22.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42665 5.35 MB/sec execute 299 sec latency 82.374 ms 2024-09-21T02:59:23.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42815 5.35 MB/sec execute 300 sec latency 116.893 ms 2024-09-21T02:59:24.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:23 smithi190 ceph-mon[33871]: pgmap v997: 129 pgs: 129 active+clean; 294 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 3.4 MiB/s wr, 310 op/s 2024-09-21T02:59:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:23 smithi029 ceph-mon[31006]: pgmap v997: 129 pgs: 129 active+clean; 294 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 3.4 MiB/s wr, 310 op/s 2024-09-21T02:59:24.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:23 smithi066 ceph-mon[33436]: pgmap v997: 129 pgs: 129 active+clean; 294 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 3.4 MiB/s wr, 310 op/s 2024-09-21T02:59:24.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42957 5.35 MB/sec execute 301 sec latency 228.038 ms 2024-09-21T02:59:25.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43120 5.36 MB/sec execute 302 sec latency 112.127 ms 2024-09-21T02:59:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:25 smithi190 ceph-mon[33871]: pgmap v998: 129 pgs: 129 active+clean; 296 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 3.3 MiB/s wr, 336 op/s 2024-09-21T02:59:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:25 smithi029 ceph-mon[31006]: pgmap v998: 129 pgs: 129 active+clean; 296 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 3.3 MiB/s wr, 336 op/s 2024-09-21T02:59:26.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:25 smithi066 ceph-mon[33436]: pgmap v998: 129 pgs: 129 active+clean; 296 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 3.3 MiB/s wr, 336 op/s 2024-09-21T02:59:26.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43247 5.35 MB/sec execute 303 sec latency 104.154 ms 2024-09-21T02:59:27.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43402 5.37 MB/sec execute 304 sec latency 91.835 ms 2024-09-21T02:59:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:27 smithi190 ceph-mon[33871]: pgmap v999: 129 pgs: 129 active+clean; 298 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.9 MiB/s wr, 326 op/s 2024-09-21T02:59:28.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:27 smithi029 ceph-mon[31006]: pgmap v999: 129 pgs: 129 active+clean; 298 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.9 MiB/s wr, 326 op/s 2024-09-21T02:59:28.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:27 smithi066 ceph-mon[33436]: pgmap v999: 129 pgs: 129 active+clean; 298 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.9 MiB/s wr, 326 op/s 2024-09-21T02:59:28.734 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43588 5.40 MB/sec execute 305 sec latency 89.751 ms 2024-09-21T02:59:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 -- 172.21.15.29:0/1419056544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 msgr2=0x7f4ebc10b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 --2- 172.21.15.29:0/1419056544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 0x7f4ebc10b8f0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009c90 tx=0x7f4eb402a860 comp rx=0 tx=0).stop 2024-09-21T02:59:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 -- 172.21.15.29:0/1419056544 shutdown_connections 2024-09-21T02:59:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 --2- 172.21.15.29:0/1419056544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 0x7f4ebc10b8f0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 --2- 172.21.15.29:0/1419056544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc106880 0x7f4ebc108ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 --2- 172.21.15.29:0/1419056544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ebc06b6b0 0x7f4ebc106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 -- 172.21.15.29:0/1419056544 >> 172.21.15.29:0/1419056544 conn(0x7f4ebc0fdee0 msgr2=0x7f4ebc100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 -- 172.21.15.29:0/1419056544 shutdown_connections 2024-09-21T02:59:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.315+0000 7f4ec124d640 1 -- 172.21.15.29:0/1419056544 wait complete. 2024-09-21T02:59:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.316+0000 7f4ec124d640 1 Processor -- start 2024-09-21T02:59:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ec124d640 1 -- start start 2024-09-21T02:59:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ec124d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc06b6b0 0x7f4ebc199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ec124d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ebc106880 0x7f4ebc19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ec124d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 0x7f4ebc19b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ec124d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc10f1b0 con 0x7f4ebc1093e0 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ec124d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc10f030 con 0x7f4ebc106880 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ec124d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc10f330 con 0x7f4ebc06b6b0 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ebad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc06b6b0 0x7f4ebc199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ebb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 0x7f4ebc19b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ebad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc06b6b0 0x7f4ebc199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43040/0 (socket says 172.21.15.29:43040) 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ebb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 0x7f4ebc19b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47944/0 (socket says 172.21.15.29:47944) 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4ebad9d640 1 -- 172.21.15.29:0/4027121074 learned_addr learned my addr 172.21.15.29:0/4027121074 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4eb259c640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ebc106880 0x7f4ebc19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4eb259c640 1 -- 172.21.15.29:0/4027121074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc06b6b0 msgr2=0x7f4ebc199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4eb259c640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc06b6b0 0x7f4ebc199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4eb259c640 1 -- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 msgr2=0x7f4ebc19b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4eb259c640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 0x7f4ebc19b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.317+0000 7f4eb259c640 1 -- 172.21.15.29:0/4027121074 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc19b8b0 con 0x7f4ebc106880 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.318+0000 7f4eb259c640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ebc106880 0x7f4ebc19a4e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4ea400e9c0 tx=0x7f4ea400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.318+0000 7f4eb8d99640 1 -- 172.21.15.29:0/4027121074 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400cd80 con 0x7f4ebc106880 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.318+0000 7f4eb8d99640 1 -- 172.21.15.29:0/4027121074 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ea40035f0 con 0x7f4ebc106880 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.318+0000 7f4eb8d99640 1 -- 172.21.15.29:0/4027121074 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4010690 con 0x7f4ebc106880 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.318+0000 7f4ec124d640 1 -- 172.21.15.29:0/4027121074 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ebc19bba0 con 0x7f4ebc106880 2024-09-21T02:59:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.318+0000 7f4ec124d640 1 -- 172.21.15.29:0/4027121074 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ebc102300 con 0x7f4ebc106880 2024-09-21T02:59:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.321+0000 7f4ec81bc740 1 Processor -- start 2024-09-21T02:59:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.321+0000 7f4ec81bc740 1 -- start start 2024-09-21T02:59:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4bde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0ed2f0 0x556aee4be4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0f54d0 0x556aee4bf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556aee4306f0 con 0x556aee1161b0 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556aee41fb20 con 0x556aee0f54d0 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556aee423930 con 0x556aee0ed2f0 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4eb8d99640 1 -- 172.21.15.29:0/4027121074 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4ea4004000 con 0x7f4ebc106880 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4bde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4bde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47960/0 (socket says 172.21.15.29:47960) 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebb59e640 1 -- 172.21.15.29:0/1218241144 learned_addr learned my addr 172.21.15.29:0/1218241144 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4eb259c640 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0f54d0 0x556aee4bf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0ed2f0 0x556aee4be4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebb59e640 1 -- 172.21.15.29:0/1218241144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0ed2f0 msgr2=0x556aee4be4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebb59e640 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0ed2f0 0x556aee4be4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebb59e640 1 -- 172.21.15.29:0/1218241144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0f54d0 msgr2=0x556aee4bf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebb59e640 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0f54d0 0x556aee4bf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.322+0000 7f4ebb59e640 1 -- 172.21.15.29:0/1218241144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556aee3e5530 con 0x556aee1161b0 2024-09-21T02:59:29.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.323+0000 7f4ebb59e640 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4bde50 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009c90 tx=0x7f4eb40093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e60a2dffb7d8d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.323+0000 7f4eb159a640 1 -- 172.21.15.29:0/1218241144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb404b070 con 0x556aee1161b0 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.323+0000 7f4eb159a640 1 -- 172.21.15.29:0/1218241144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4eb4036040 con 0x556aee1161b0 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.323+0000 7f4eb159a640 1 -- 172.21.15.29:0/1218241144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4003c00 con 0x556aee1161b0 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4eb8d99640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ea80b6590 0x7f4ea80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ea80b6590 0x7f4ea80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1218241144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 msgr2=0x556aee4bde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4bde50 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009c90 tx=0x7f4eb40093d0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4eb8d99640 1 -- 172.21.15.29:0/4027121074 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4ea40e7850 con 0x7f4ebc106880 2024-09-21T02:59:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1218241144 shutdown_connections 2024-09-21T02:59:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0f54d0 0x556aee4bf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0ed2f0 0x556aee4be4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/1218241144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4bde50 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.324+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1218241144 >> 172.21.15.29:0/1218241144 conn(0x556aee10a300 msgr2=0x556aee4b98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.325+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1218241144 shutdown_connections 2024-09-21T02:59:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.325+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1218241144 wait complete. 2024-09-21T02:59:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.325+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ea80b6590 0x7f4ea80b8aa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009e50 tx=0x7f4eac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.325+0000 7f4ec81bc740 1 -- start start 2024-09-21T02:59:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.325+0000 7f4ec81bc740 1 Processor -- start 2024-09-21T02:59:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4c7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556aee357ab0 con 0x556aee1161b0 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x556aee3c5730 con 0x556aee0ed2f0 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556aee4bfad0 con 0x556aee0f54d0 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebad9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4c87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47976/0 (socket says 172.21.15.29:47976) 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebad9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4c7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42634/0 (socket says 172.21.15.29:42634) 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebb59e640 1 -- 172.21.15.29:0/553950708 learned_addr learned my addr 172.21.15.29:0/553950708 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4eb259c640 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ec81bc740 10 client.0 _ll_register_callbacks cb 0x556aee4196b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebb59e640 1 -- 172.21.15.29:0/553950708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 msgr2=0x556aee4c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebb59e640 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebb59e640 1 -- 172.21.15.29:0/553950708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 msgr2=0x556aee4c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebb59e640 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.326+0000 7f4ebb59e640 1 -- 172.21.15.29:0/553950708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556aee181790 con 0x556aee1161b0 2024-09-21T02:59:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.327+0000 7f4ebb59e640 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4c87c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40011e0 tx=0x7f4eb40098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5e6c83aed50510 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.327+0000 7f4ea27fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:59:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.327+0000 7f4ea27fc640 1 -- 172.21.15.29:0/553950708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb404b040 con 0x556aee1161b0 2024-09-21T02:59:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.327+0000 7f4ea27fc640 1 -- 172.21.15.29:0/553950708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4eb4004a90 con 0x556aee1161b0 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.327+0000 7f4ec81bc740 1 -- 172.21.15.29:0/553950708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb4036040 con 0x556aee1161b0 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.327+0000 7f4ea27fc640 1 -- 172.21.15.29:0/553950708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4eb405d050 con 0x556aee1161b0 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.327+0000 7f4ec81bc740 10 client.18183 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.327+0000 7f4ec81bc740 1 -- 172.21.15.29:0/553950708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556aee1fa620 con 0x556aee1161b0 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.328+0000 7f4ea27fc640 1 -- 172.21.15.29:0/553950708 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4eb4036040 con 0x556aee1161b0 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.328+0000 7f4ea27fc640 1 -- 172.21.15.29:0/553950708 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f4eb404e070 con 0x556aee1161b0 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.328+0000 7f4ec81bc740 10 client.18183 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.328+0000 7f4ec81bc740 10 client.18183 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.328+0000 7f4ec81bc740 20 client.18183 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.328+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556aee0d4d80 0x556aee4d4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.328+0000 7f4ec81bc740 4 client.18183 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:59:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.328+0000 7f4ec81bc740 1 -- 172.21.15.29:0/553950708 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556aee432640 con 0x556aee0d4d80 2024-09-21T02:59:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.329+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556aee0d4d80 0x556aee4d4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.329+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556aee0d4d80 0x556aee4d4d90 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64670fb6c2d69d7c server_cookie=7dfdf27ade7ed209 in_seq=0 out_seq=0 2024-09-21T02:59:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.329+0000 7f4ea27fc640 10 client.18183 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.330+0000 7f4ea27fc640 1 -- 172.21.15.29:0/553950708 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556aee432640 con 0x556aee0d4d80 2024-09-21T02:59:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.330+0000 7f4ea27fc640 10 client.18183 handle_command_reply: tid=0 2024-09-21T02:59:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.330+0000 7f4ec81bc740 1 client.18183 shutdown 2024-09-21T02:59:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.330+0000 7f4ec81bc740 1 -- 172.21.15.29:0/553950708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 msgr2=0x556aee4c87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4c87c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40011e0 tx=0x7f4eb40098f0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 -- 172.21.15.29:0/553950708 shutdown_connections 2024-09-21T02:59:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556aee0d4d80 0x556aee4d4d90 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4c87c0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/553950708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 -- 172.21.15.29:0/553950708 >> 172.21.15.29:0/553950708 conn(0x556aee10a300 msgr2=0x556aee4b7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ea27fc640 0 client.18183 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 -- 172.21.15.29:0/553950708 shutdown_connections 2024-09-21T02:59:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.331+0000 7f4ec81bc740 1 -- 172.21.15.29:0/553950708 wait complete. 2024-09-21T02:59:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.332+0000 7f4ec81bc740 20 client.18183 trim_cache size 0 max 16384 2024-09-21T02:59:29.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.355+0000 7f4ec81bc740 1 Processor -- start 2024-09-21T02:59:29.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.355+0000 7f4ec81bc740 1 -- start start 2024-09-21T02:59:29.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee0ed2f0 0x556aee4b9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4c6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee1161b0 0x556aee4bec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1aa2a0 con 0x556aee0ed2f0 2024-09-21T02:59:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1abde0 con 0x556aee1161b0 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1aa970 con 0x556aee0f54d0 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee0ed2f0 0x556aee4b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee0ed2f0 0x556aee4b9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47986/0 (socket says 172.21.15.29:47986) 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebad9d640 1 -- 172.21.15.29:0/1331509152 learned_addr learned my addr 172.21.15.29:0/1331509152 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4eb259c640 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebb59e640 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee1161b0 0x556aee4bec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebad9d640 1 -- 172.21.15.29:0/1331509152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 msgr2=0x556aee4c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebad9d640 1 -- 172.21.15.29:0/1331509152 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee1161b0 msgr2=0x556aee4bec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee1161b0 0x556aee4bec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.356+0000 7f4ebad9d640 1 -- 172.21.15.29:0/1331509152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556aee4b4f50 con 0x556aee0ed2f0 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.357+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee0ed2f0 0x556aee4b9220 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4eac01d620 tx=0x7f4eac01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2439be6885212c2a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.357+0000 7f4eb0d99640 1 -- 172.21.15.29:0/1331509152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac01dd30 con 0x556aee0ed2f0 2024-09-21T02:59:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.357+0000 7f4eb0d99640 1 -- 172.21.15.29:0/1331509152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4eac0035f0 con 0x556aee0ed2f0 2024-09-21T02:59:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.357+0000 7f4eb0d99640 1 -- 172.21.15.29:0/1331509152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac02b690 con 0x556aee0ed2f0 2024-09-21T02:59:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1331509152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee0ed2f0 msgr2=0x556aee4b9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee0ed2f0 0x556aee4b9220 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4eac01d620 tx=0x7f4eac01b000 comp rx=0 tx=0).stop 2024-09-21T02:59:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1331509152 shutdown_connections 2024-09-21T02:59:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee1161b0 0x556aee4bec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/1331509152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee0ed2f0 0x556aee4b9220 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1331509152 >> 172.21.15.29:0/1331509152 conn(0x556aee10a300 msgr2=0x556aee4d4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1331509152 shutdown_connections 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.358+0000 7f4ec81bc740 1 -- 172.21.15.29:0/1331509152 wait complete. 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.359+0000 7f4ec81bc740 1 -- start start 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.359+0000 7f4ec81bc740 1 Processor -- start 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.359+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.359+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.359+0000 7f4ec81bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4cd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.359+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1ab710 con 0x556aee1161b0 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.359+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1ab040 con 0x556aee0ed2f0 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.359+0000 7f4ec81bc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1a8280 con 0x556aee0f54d0 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42654/0 (socket says 172.21.15.29:42654) 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 -- 172.21.15.29:0/481296098 learned_addr learned my addr 172.21.15.29:0/481296098 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ec81bc740 10 client.0 _ll_register_callbacks cb 0x7f4eb4049070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4eb259c640 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebb59e640 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 -- 172.21.15.29:0/481296098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 msgr2=0x556aee4ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 -- 172.21.15.29:0/481296098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 msgr2=0x556aee4cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 -- 172.21.15.29:0/481296098 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556aee4cdf60 con 0x556aee0ed2f0 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ebad9d640 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4cc620 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4eac01dfd0 tx=0x7f4eac003dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b44a05effe724d7f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ea27fc640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.360+0000 7f4ea27fc640 1 -- 172.21.15.29:0/481296098 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac0039a0 con 0x556aee0ed2f0 2024-09-21T02:59:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.361+0000 7f4ea27fc640 1 -- 172.21.15.29:0/481296098 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f4eac004000 con 0x556aee0ed2f0 2024-09-21T02:59:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.361+0000 7f4ec81bc740 1 -- 172.21.15.29:0/481296098 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eac0035f0 con 0x556aee0ed2f0 2024-09-21T02:59:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.361+0000 7f4ea27fc640 1 -- 172.21.15.29:0/481296098 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4eac003b40 con 0x556aee0ed2f0 2024-09-21T02:59:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.361+0000 7f4ec81bc740 10 client.27488 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.361+0000 7f4ec81bc740 1 -- 172.21.15.29:0/481296098 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556aee4cf260 con 0x556aee0ed2f0 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.361+0000 7f4ea27fc640 1 -- 172.21.15.29:0/481296098 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4eac0035f0 con 0x556aee0ed2f0 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.362+0000 7f4ea27fc640 1 -- 172.21.15.29:0/481296098 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f4eac038070 con 0x556aee0ed2f0 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.362+0000 7f4ec81bc740 10 client.27488 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.362+0000 7f4ec81bc740 10 client.27488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.362+0000 7f4ec81bc740 20 client.27488 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.362+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556aee0aa1c0 0x556aee5418d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.363+0000 7f4ec81bc740 4 client.27488 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.363+0000 7f4ec81bc740 1 -- 172.21.15.29:0/481296098 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556aee42e0f0 con 0x556aee0aa1c0 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.363+0000 7f4eb259c640 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556aee0aa1c0 0x556aee5418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.364+0000 7f4ea27fc640 10 client.27488 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.363+0000 7f4eb259c640 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556aee0aa1c0 0x556aee5418d0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=870505d31970fe08 server_cookie=32189f7294b66548 in_seq=0 out_seq=0 2024-09-21T02:59:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ea27fc640 1 -- 172.21.15.29:0/481296098 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556aee42e0f0 con 0x556aee0aa1c0 2024-09-21T02:59:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ea27fc640 10 client.27488 handle_command_reply: tid=0 2024-09-21T02:59:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 client.27488 shutdown 2024-09-21T02:59:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 -- 172.21.15.29:0/481296098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 msgr2=0x556aee4cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4cc620 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4eac01dfd0 tx=0x7f4eac003dd0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 -- 172.21.15.29:0/481296098 shutdown_connections 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556aee0aa1c0 0x556aee5418d0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556aee1161b0 0x556aee4cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556aee0f54d0 0x556aee4ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 --2- 172.21.15.29:0/481296098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556aee0ed2f0 0x556aee4cc620 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 -- 172.21.15.29:0/481296098 >> 172.21.15.29:0/481296098 conn(0x556aee10a300 msgr2=0x556aee4c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ea27fc640 0 client.27488 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 -- 172.21.15.29:0/481296098 shutdown_connections 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.546+0000 7f4ec81bc740 1 -- 172.21.15.29:0/481296098 wait complete. 2024-09-21T02:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec81bc740 20 client.27488 trim_cache size 0 max 16384 2024-09-21T02:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 -- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ea80b6590 msgr2=0x7f4ea80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ea80b6590 0x7f4ea80b8aa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009e50 tx=0x7f4eac005f90 comp rx=0 tx=0).stop 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 -- 172.21.15.29:0/4027121074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ebc106880 msgr2=0x7f4ebc19a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ebc106880 0x7f4ebc19a4e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4ea400e9c0 tx=0x7f4ea400ee80 comp rx=0 tx=0).stop 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 -- 172.21.15.29:0/4027121074 shutdown_connections 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4ea80b6590 0x7f4ea80b8aa0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc1093e0 0x7f4ebc19b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ebc106880 0x7f4ebc19a4e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 --2- 172.21.15.29:0/4027121074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc06b6b0 0x7f4ebc199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.547+0000 7f4ec124d640 1 -- 172.21.15.29:0/4027121074 >> 172.21.15.29:0/4027121074 conn(0x7f4ebc0fdee0 msgr2=0x7f4ebc104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.548+0000 7f4ec124d640 1 -- 172.21.15.29:0/4027121074 shutdown_connections 2024-09-21T02:59:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:29.548+0000 7f4ec124d640 1 -- 172.21.15.29:0/4027121074 wait complete. 2024-09-21T02:59:29.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43743 5.41 MB/sec execute 306 sec latency 227.922 ms 2024-09-21T02:59:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:29 smithi190 ceph-mon[33871]: pgmap v1000: 129 pgs: 129 active+clean; 300 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.2 MiB/s wr, 337 op/s 2024-09-21T02:59:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:29 smithi029 ceph-mon[31006]: pgmap v1000: 129 pgs: 129 active+clean; 300 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.2 MiB/s wr, 337 op/s 2024-09-21T02:59:30.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:29 smithi066 ceph-mon[33436]: pgmap v1000: 129 pgs: 129 active+clean; 300 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.2 MiB/s wr, 337 op/s 2024-09-21T02:59:30.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43947 5.43 MB/sec execute 307 sec latency 126.361 ms 2024-09-21T02:59:31.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44058 5.43 MB/sec execute 308 sec latency 98.668 ms 2024-09-21T02:59:32.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:31 smithi190 ceph-mon[33871]: pgmap v1001: 129 pgs: 129 active+clean; 303 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 3.0 MiB/s wr, 259 op/s 2024-09-21T02:59:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:31 smithi029 ceph-mon[31006]: pgmap v1001: 129 pgs: 129 active+clean; 303 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 3.0 MiB/s wr, 259 op/s 2024-09-21T02:59:32.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:31 smithi066 ceph-mon[33436]: pgmap v1001: 129 pgs: 129 active+clean; 303 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 3.0 MiB/s wr, 259 op/s 2024-09-21T02:59:32.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44220 5.45 MB/sec execute 309 sec latency 88.554 ms 2024-09-21T02:59:33.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44366 5.47 MB/sec execute 310 sec latency 215.025 ms 2024-09-21T02:59:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:33 smithi029 ceph-mon[31006]: pgmap v1002: 129 pgs: 129 active+clean; 302 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-09-21T02:59:34.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:33 smithi066 ceph-mon[33436]: pgmap v1002: 129 pgs: 129 active+clean; 302 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-09-21T02:59:34.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:33 smithi190 ceph-mon[33871]: pgmap v1002: 129 pgs: 129 active+clean; 302 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 3.4 MiB/s wr, 246 op/s 2024-09-21T02:59:34.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44500 5.50 MB/sec execute 311 sec latency 161.200 ms 2024-09-21T02:59:35.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44680 5.52 MB/sec execute 312 sec latency 91.224 ms 2024-09-21T02:59:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:35 smithi029 ceph-mon[31006]: pgmap v1003: 129 pgs: 129 active+clean; 303 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 3.9 MiB/s wr, 296 op/s 2024-09-21T02:59:36.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:35 smithi066 ceph-mon[33436]: pgmap v1003: 129 pgs: 129 active+clean; 303 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 3.9 MiB/s wr, 296 op/s 2024-09-21T02:59:36.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:35 smithi190 ceph-mon[33871]: pgmap v1003: 129 pgs: 129 active+clean; 303 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 3.9 MiB/s wr, 296 op/s 2024-09-21T02:59:36.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44801 5.52 MB/sec execute 313 sec latency 101.807 ms 2024-09-21T02:59:37.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44929 5.52 MB/sec execute 314 sec latency 83.716 ms 2024-09-21T02:59:38.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:37 smithi029 ceph-mon[31006]: pgmap v1004: 129 pgs: 129 active+clean; 308 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:59:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:37 smithi066 ceph-mon[33436]: pgmap v1004: 129 pgs: 129 active+clean; 308 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:59:38.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:37 smithi190 ceph-mon[33871]: pgmap v1004: 129 pgs: 129 active+clean; 308 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 3.9 MiB/s wr, 273 op/s 2024-09-21T02:59:38.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45066 5.52 MB/sec execute 315 sec latency 85.372 ms 2024-09-21T02:59:39.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45182 5.52 MB/sec execute 316 sec latency 123.595 ms 2024-09-21T02:59:40.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:39 smithi029 ceph-mon[31006]: pgmap v1005: 129 pgs: 129 active+clean; 310 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 4.3 MiB/s wr, 286 op/s 2024-09-21T02:59:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:39 smithi066 ceph-mon[33436]: pgmap v1005: 129 pgs: 129 active+clean; 310 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 4.3 MiB/s wr, 286 op/s 2024-09-21T02:59:40.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:39 smithi190 ceph-mon[33871]: pgmap v1005: 129 pgs: 129 active+clean; 310 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 4.3 MiB/s wr, 286 op/s 2024-09-21T02:59:40.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45328 5.52 MB/sec execute 317 sec latency 88.061 ms 2024-09-21T02:59:41.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45479 5.53 MB/sec execute 318 sec latency 277.182 ms 2024-09-21T02:59:42.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:41 smithi066 ceph-mon[33436]: pgmap v1006: 129 pgs: 129 active+clean; 311 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-09-21T02:59:42.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:41 smithi190 ceph-mon[33871]: pgmap v1006: 129 pgs: 129 active+clean; 311 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-09-21T02:59:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:41 smithi029 ceph-mon[31006]: pgmap v1006: 129 pgs: 129 active+clean; 311 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-09-21T02:59:42.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45607 5.53 MB/sec execute 319 sec latency 149.155 ms 2024-09-21T02:59:42.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 -- 172.21.15.29:0/1036210415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4788102520 msgr2=0x7f4788102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:42.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 --2- 172.21.15.29:0/1036210415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4788102520 0x7f4788102930 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f477c009900 tx=0x7f477c02a410 comp rx=0 tx=0).stop 2024-09-21T02:59:42.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 -- 172.21.15.29:0/1036210415 shutdown_connections 2024-09-21T02:59:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 --2- 172.21.15.29:0/1036210415 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4788102ec0 0x7f4788103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 --2- 172.21.15.29:0/1036210415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4788102520 0x7f4788102930 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 --2- 172.21.15.29:0/1036210415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4788101710 0x7f4788101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 -- 172.21.15.29:0/1036210415 >> 172.21.15.29:0/1036210415 conn(0x7f47880fd270 msgr2=0x7f47880ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 -- 172.21.15.29:0/1036210415 shutdown_connections 2024-09-21T02:59:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 -- 172.21.15.29:0/1036210415 wait complete. 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.944+0000 7f478e04d640 1 Processor -- start 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f478e04d640 1 -- start start 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f478e04d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4788101710 0x7f478819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f478e04d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4788102520 0x7f478819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f478e04d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4788102ec0 0x7f478819d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f478e04d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f47881113f0 con 0x7f4788101710 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f478e04d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4788111270 con 0x7f4788102ec0 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f478e04d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4788111570 con 0x7f4788102520 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f4786ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4788102520 0x7f478819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f4786ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4788102520 0x7f478819c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40158/0 (socket says 172.21.15.29:40158) 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f4786ffd640 1 -- 172.21.15.29:0/2001177459 learned_addr learned my addr 172.21.15.29:0/2001177459 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f47877fe640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4788101710 0x7f478819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.945+0000 7f4787fff640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4788102ec0 0x7f478819d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4787fff640 1 -- 172.21.15.29:0/2001177459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4788102520 msgr2=0x7f478819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4787fff640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4788102520 0x7f478819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4787fff640 1 -- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4788101710 msgr2=0x7f478819c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4787fff640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4788101710 0x7f478819c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4787fff640 1 -- 172.21.15.29:0/2001177459 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f478819dab0 con 0x7f4788102ec0 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4786ffd640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4788102520 0x7f478819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4787fff640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4788102ec0 0x7f478819d290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4780009fd0 tx=0x7f478000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4784ff9640 1 -- 172.21.15.29:0/2001177459 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478000ce20 con 0x7f4788102ec0 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f478e04d640 1 -- 172.21.15.29:0/2001177459 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f478819dda0 con 0x7f4788102ec0 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4784ff9640 1 -- 172.21.15.29:0/2001177459 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4780005020 con 0x7f4788102ec0 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f478e04d640 1 -- 172.21.15.29:0/2001177459 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47881a4490 con 0x7f4788102ec0 2024-09-21T02:59:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.946+0000 7f4784ff9640 1 -- 172.21.15.29:0/2001177459 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4780010690 con 0x7f4788102ec0 2024-09-21T02:59:42.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.949+0000 7f4784ff9640 1 -- 172.21.15.29:0/2001177459 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4780010830 con 0x7f4788102ec0 2024-09-21T02:59:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.950+0000 7f4784ff9640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47700b6590 0x7f47700b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.950+0000 7f47877fe640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47700b6590 0x7f47700b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:42.950+0000 7f4784ff9640 1 -- 172.21.15.29:0/2001177459 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f47800e8b60 con 0x7f4788102ec0 2024-09-21T02:59:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.061+0000 7f47877fe640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47700b6590 0x7f47700b8aa0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4778009bd0 tx=0x7f4778005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.062+0000 7f4794fc6740 1 Processor -- start 2024-09-21T02:59:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.062+0000 7f4794fc6740 1 -- start start 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a4659901b0 0x55a465d37e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a46596f4d0 0x55a465d39010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4794fc6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a465caa6f0 con 0x55a46596f4d0 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4794fc6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a465c99b20 con 0x55a4659672f0 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4794fc6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a465c9d930 con 0x55a4659901b0 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a46596f4d0 0x55a465d39010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4786ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f47877fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a4659901b0 0x55a465d37e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a46596f4d0 0x55a465d39010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58812/0 (socket says 172.21.15.29:58812) 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4787fff640 1 -- 172.21.15.29:0/1431439122 learned_addr learned my addr 172.21.15.29:0/1431439122 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4786ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:42566/0 (socket says 172.21.15.29:42566) 2024-09-21T02:59:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4787fff640 1 -- 172.21.15.29:0/1431439122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a4659901b0 msgr2=0x55a465d37e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4787fff640 1 --2- 172.21.15.29:0/1431439122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a4659901b0 0x55a465d37e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4787fff640 1 -- 172.21.15.29:0/1431439122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 msgr2=0x55a465d384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4787fff640 1 --2- 172.21.15.29:0/1431439122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4787fff640 1 -- 172.21.15.29:0/1431439122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a465c5f530 con 0x55a46596f4d0 2024-09-21T02:59:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.063+0000 7f4786ffd640 1 --2- 172.21.15.29:0/1431439122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.064+0000 7f4787fff640 1 --2- 172.21.15.29:0/1431439122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a46596f4d0 0x55a465d39010 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f47800aaf40 tx=0x7f47800ae0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902891f2ee6be8a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.064+0000 7f476dffb640 1 -- 172.21.15.29:0/1431439122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47800b3070 con 0x55a46596f4d0 2024-09-21T02:59:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.064+0000 7f476dffb640 1 -- 172.21.15.29:0/1431439122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47800b3210 con 0x55a46596f4d0 2024-09-21T02:59:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.064+0000 7f476dffb640 1 -- 172.21.15.29:0/1431439122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4780010b20 con 0x55a46596f4d0 2024-09-21T02:59:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 -- 172.21.15.29:0/1431439122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a46596f4d0 msgr2=0x55a465d39010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 --2- 172.21.15.29:0/1431439122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a46596f4d0 0x55a465d39010 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f47800aaf40 tx=0x7f47800ae0e0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 -- 172.21.15.29:0/1431439122 shutdown_connections 2024-09-21T02:59:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 --2- 172.21.15.29:0/1431439122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a46596f4d0 0x55a465d39010 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 --2- 172.21.15.29:0/1431439122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 --2- 172.21.15.29:0/1431439122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a4659901b0 0x55a465d37e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 -- 172.21.15.29:0/1431439122 >> 172.21.15.29:0/1431439122 conn(0x55a465984300 msgr2=0x55a465d338e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 -- 172.21.15.29:0/1431439122 shutdown_connections 2024-09-21T02:59:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.065+0000 7f4794fc6740 1 -- 172.21.15.29:0/1431439122 wait complete. 2024-09-21T02:59:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.066+0000 7f4794fc6740 1 -- start start 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.066+0000 7f4794fc6740 1 Processor -- start 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d415e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d41ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f47877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f47877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58822/0 (socket says 172.21.15.29:58822) 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f47877fe640 1 -- 172.21.15.29:0/913394070 learned_addr learned my addr 172.21.15.29:0/913394070 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f4794fc6740 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d42810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a465bd1ab0 con 0x55a4659672f0 2024-09-21T02:59:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55a465c3f730 con 0x55a4659901b0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a465d39b30 con 0x55a46596f4d0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.067+0000 7f4786ffd640 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d41ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f4787fff640 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d42810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f4786ffd640 1 -- 172.21.15.29:0/913394070 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 msgr2=0x55a465d42810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f4786ffd640 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d42810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f4794fc6740 10 client.0 _ll_register_callbacks cb 0x55a465c936b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f4786ffd640 1 -- 172.21.15.29:0/913394070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 msgr2=0x55a465d415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f4786ffd640 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d415e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f4786ffd640 1 -- 172.21.15.29:0/913394070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4659fb790 con 0x55a46596f4d0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f47877fe640 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.068+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47800b3210 con 0x55a46596f4d0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.069+0000 7f4786ffd640 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d41ca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f477c0057c0 tx=0x7f477c031c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33899824cea1a1dc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.069+0000 7f4766ffd640 10 client.27664 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:59:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.069+0000 7f4766ffd640 1 -- 172.21.15.29:0/913394070 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c035040 con 0x55a46596f4d0 2024-09-21T02:59:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.069+0000 7f4766ffd640 1 -- 172.21.15.29:0/913394070 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f477c004450 con 0x55a46596f4d0 2024-09-21T02:59:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.069+0000 7f4766ffd640 1 -- 172.21.15.29:0/913394070 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f477c0035f0 con 0x55a46596f4d0 2024-09-21T02:59:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.069+0000 7f4794fc6740 10 client.27664 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.069+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=37}) -- 0x55a465a74620 con 0x55a46596f4d0 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.069+0000 7f4766ffd640 1 -- 172.21.15.29:0/913394070 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f477c003790 con 0x55a46596f4d0 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f4794fc6740 10 client.27664 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f4794fc6740 10 client.27664 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f4794fc6740 20 client.27664 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f4794fc6740 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a46594ed80 0x55a465d4eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f4794fc6740 4 client.27664 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a465cac640 con 0x55a46594ed80 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f47877fe640 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a46594ed80 0x55a465d4eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f47877fe640 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a46594ed80 0x55a465d4eb60 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e86f79c3bb8bfd91 server_cookie=336b9d56ee9fb831 in_seq=0 out_seq=0 2024-09-21T02:59:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.070+0000 7f4766ffd640 10 client.27664 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4766ffd640 1 -- 172.21.15.29:0/913394070 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a465cac640 con 0x55a46594ed80 2024-09-21T02:59:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4766ffd640 10 client.27664 handle_command_reply: tid=0 2024-09-21T02:59:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 client.27664 shutdown 2024-09-21T02:59:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 msgr2=0x55a465d41ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d41ca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f477c0057c0 tx=0x7f477c031c40 comp rx=0 tx=0).stop 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 shutdown_connections 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d42810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d41ca0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a46594ed80 0x55a465d4eb60 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 --2- 172.21.15.29:0/913394070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4766ffd640 0 client.27664 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 >> 172.21.15.29:0/913394070 conn(0x55a465984300 msgr2=0x55a465d31ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 shutdown_connections 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 1 -- 172.21.15.29:0/913394070 wait complete. 2024-09-21T02:59:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.072+0000 7f4794fc6740 20 client.27664 trim_cache size 0 max 16384 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.095+0000 7f4794fc6740 1 Processor -- start 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.095+0000 7f4794fc6740 1 -- start start 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.095+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d40200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.095+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d33220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.095+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d37df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.095+0000 7f4794fc6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f47881aa220 con 0x55a4659672f0 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.095+0000 7f4794fc6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f47881abd60 con 0x55a4659901b0 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.095+0000 7f4794fc6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f47881aa8f0 con 0x55a46596f4d0 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f47877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d40200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4786ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d33220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f47877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d40200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58838/0 (socket says 172.21.15.29:58838) 2024-09-21T02:59:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f47877fe640 1 -- 172.21.15.29:0/519098757 learned_addr learned my addr 172.21.15.29:0/519098757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4787fff640 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d37df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4787fff640 1 -- 172.21.15.29:0/519098757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 msgr2=0x55a465d33220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4787fff640 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d33220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4787fff640 1 -- 172.21.15.29:0/519098757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 msgr2=0x55a465d40200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4787fff640 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d40200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4787fff640 1 -- 172.21.15.29:0/519098757 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a465d2f080 con 0x55a4659901b0 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4786ffd640 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d33220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f47877fe640 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d40200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.096+0000 7f4787fff640 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d37df0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f47800aaf40 tx=0x7f478001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fb9010c5706227e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.097+0000 7f476d7fa640 1 -- 172.21.15.29:0/519098757 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47800c8070 con 0x55a4659901b0 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.097+0000 7f476d7fa640 1 -- 172.21.15.29:0/519098757 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47800b3450 con 0x55a4659901b0 2024-09-21T02:59:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.097+0000 7f476d7fa640 1 -- 172.21.15.29:0/519098757 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47800b3040 con 0x55a4659901b0 2024-09-21T02:59:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 -- 172.21.15.29:0/519098757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 msgr2=0x55a465d37df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d37df0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f47800aaf40 tx=0x7f478001f000 comp rx=0 tx=0).stop 2024-09-21T02:59:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 -- 172.21.15.29:0/519098757 shutdown_connections 2024-09-21T02:59:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659901b0 0x55a465d37df0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d33220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 --2- 172.21.15.29:0/519098757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659672f0 0x55a465d40200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 -- 172.21.15.29:0/519098757 >> 172.21.15.29:0/519098757 conn(0x55a465984300 msgr2=0x55a465d3faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 -- 172.21.15.29:0/519098757 shutdown_connections 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 -- 172.21.15.29:0/519098757 wait complete. 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.098+0000 7f4794fc6740 1 -- start start 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4794fc6740 1 Processor -- start 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d48450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d48990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4794fc6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659901b0 0x55a465d48ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4794fc6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f47881ab690 con 0x55a4659901b0 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4794fc6740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f47881aafc0 con 0x55a4659672f0 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4794fc6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f47881a8220 con 0x55a46596f4d0 2024-09-21T02:59:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4794fc6740 10 client.0 _ll_register_callbacks cb 0x7f477c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4786ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d48990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659901b0 0x55a465d48ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f47877fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d48450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4786ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d48990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40184/0 (socket says 172.21.15.29:40184) 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4786ffd640 1 -- 172.21.15.29:0/4199515418 learned_addr learned my addr 172.21.15.29:0/4199515418 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4786ffd640 1 -- 172.21.15.29:0/4199515418 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 msgr2=0x55a465d48450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4786ffd640 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d48450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4786ffd640 1 -- 172.21.15.29:0/4199515418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659901b0 msgr2=0x55a465d48ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4786ffd640 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659901b0 0x55a465d48ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4786ffd640 1 -- 172.21.15.29:0/4199515418 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a465d47210 con 0x55a46596f4d0 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f47877fe640 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d48450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.099+0000 7f4787fff640 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659901b0 0x55a465d48ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.100+0000 7f4786ffd640 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d48990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f477c009a20 tx=0x7f477c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d2be9f7c76da464 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.100+0000 7f4766ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T02:59:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.100+0000 7f4766ffd640 1 -- 172.21.15.29:0/4199515418 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c05f060 con 0x55a46596f4d0 2024-09-21T02:59:43.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.100+0000 7f4766ffd640 1 -- 172.21.15.29:0/4199515418 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f477c004250 con 0x55a46596f4d0 2024-09-21T02:59:43.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.100+0000 7f4794fc6740 1 -- 172.21.15.29:0/4199515418 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a465ca80f0 con 0x55a46596f4d0 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.100+0000 7f4766ffd640 1 -- 172.21.15.29:0/4199515418 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f477c035040 con 0x55a46596f4d0 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.100+0000 7f4794fc6740 10 client.27670 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4794fc6740 1 -- 172.21.15.29:0/4199515418 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a465db99b0 con 0x55a46596f4d0 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4766ffd640 1 -- 172.21.15.29:0/4199515418 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55a465ca80f0 con 0x55a46596f4d0 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4766ffd640 1 -- 172.21.15.29:0/4199515418 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f477c03cad0 con 0x55a46596f4d0 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4794fc6740 10 client.27670 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4794fc6740 10 client.27670 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4794fc6740 20 client.27670 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4794fc6740 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a4659241c0 0x55a465dbc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4794fc6740 4 client.27670 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f4794fc6740 1 -- 172.21.15.29:0/4199515418 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a465ca5520 con 0x55a4659241c0 2024-09-21T02:59:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.101+0000 7f47877fe640 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a4659241c0 0x55a465dbc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.102+0000 7f4766ffd640 10 client.27670 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.102+0000 7f47877fe640 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a4659241c0 0x55a465dbc370 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a3845dec0076723 server_cookie=44b1fee270e2e209 in_seq=0 out_seq=0 2024-09-21T02:59:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4766ffd640 1 -- 172.21.15.29:0/4199515418 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a465ca5520 con 0x55a4659241c0 2024-09-21T02:59:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4766ffd640 10 client.27670 handle_command_reply: tid=0 2024-09-21T02:59:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 client.27670 shutdown 2024-09-21T02:59:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 -- 172.21.15.29:0/4199515418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 msgr2=0x55a465d48990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d48990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f477c009a20 tx=0x7f477c009260 comp rx=0 tx=0).stop 2024-09-21T02:59:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 -- 172.21.15.29:0/4199515418 shutdown_connections 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55a4659241c0 0x55a465dbc370 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a4659901b0 0x55a465d48ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a46596f4d0 0x55a465d48990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 --2- 172.21.15.29:0/4199515418 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55a4659672f0 0x55a465d48450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4766ffd640 0 client.27670 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 -- 172.21.15.29:0/4199515418 >> 172.21.15.29:0/4199515418 conn(0x55a465984300 msgr2=0x55a465d415e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.302+0000 7f4794fc6740 1 -- 172.21.15.29:0/4199515418 shutdown_connections 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f4794fc6740 1 -- 172.21.15.29:0/4199515418 wait complete. 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f4794fc6740 20 client.27670 trim_cache size 0 max 16384 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:59:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:59:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:59:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:59:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 -- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47700b6590 msgr2=0x7f47700b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47700b6590 0x7f47700b8aa0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4778009bd0 tx=0x7f4778005f90 comp rx=0 tx=0).stop 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 -- 172.21.15.29:0/2001177459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4788102ec0 msgr2=0x7f478819d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4788102ec0 0x7f478819d290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4780009fd0 tx=0x7f478000ef10 comp rx=0 tx=0).stop 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 -- 172.21.15.29:0/2001177459 shutdown_connections 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f47700b6590 0x7f47700b8aa0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4788102ec0 0x7f478819d290 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4788102520 0x7f478819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 --2- 172.21.15.29:0/2001177459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4788101710 0x7f478819c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.303+0000 7f478e04d640 1 -- 172.21.15.29:0/2001177459 >> 172.21.15.29:0/2001177459 conn(0x7f47880fd270 msgr2=0x7f478810b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.304+0000 7f478e04d640 1 -- 172.21.15.29:0/2001177459 shutdown_connections 2024-09-21T02:59:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:43.304+0000 7f478e04d640 1 -- 172.21.15.29:0/2001177459 wait complete. 2024-09-21T02:59:43.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45715 5.51 MB/sec execute 320 sec latency 231.381 ms 2024-09-21T02:59:44.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:43 smithi066 ceph-mon[33436]: pgmap v1007: 129 pgs: 129 active+clean; 312 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 4.1 MiB/s wr, 270 op/s 2024-09-21T02:59:44.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:43 smithi190 ceph-mon[33871]: pgmap v1007: 129 pgs: 129 active+clean; 312 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 4.1 MiB/s wr, 270 op/s 2024-09-21T02:59:44.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:43 smithi029 ceph-mon[31006]: pgmap v1007: 129 pgs: 129 active+clean; 312 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 4.1 MiB/s wr, 270 op/s 2024-09-21T02:59:44.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45834 5.52 MB/sec execute 321 sec latency 218.501 ms 2024-09-21T02:59:45.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45995 5.53 MB/sec execute 322 sec latency 97.519 ms 2024-09-21T02:59:46.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:45 smithi066 ceph-mon[33436]: pgmap v1008: 129 pgs: 129 active+clean; 311 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 4.3 MiB/s wr, 299 op/s 2024-09-21T02:59:46.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:45 smithi190 ceph-mon[33871]: pgmap v1008: 129 pgs: 129 active+clean; 311 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 4.3 MiB/s wr, 299 op/s 2024-09-21T02:59:46.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:45 smithi029 ceph-mon[31006]: pgmap v1008: 129 pgs: 129 active+clean; 311 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 4.3 MiB/s wr, 299 op/s 2024-09-21T02:59:46.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46181 5.57 MB/sec execute 323 sec latency 120.511 ms 2024-09-21T02:59:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:59:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:46 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:59:47.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:46 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:59:47.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:46 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:47.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46320 5.57 MB/sec execute 324 sec latency 111.735 ms 2024-09-21T02:59:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: pgmap v1009: 129 pgs: 129 active+clean; 308 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 3.7 MiB/s wr, 266 op/s 2024-09-21T02:59:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:59:48.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:59:48.261 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:47 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: pgmap v1009: 129 pgs: 129 active+clean; 308 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 3.7 MiB/s wr, 266 op/s 2024-09-21T02:59:48.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:59:48.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:59:48.375 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:47 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: pgmap v1009: 129 pgs: 129 active+clean; 308 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 3.7 MiB/s wr, 266 op/s 2024-09-21T02:59:48.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:59:48.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:59:48.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:47 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T02:59:48.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46437 5.56 MB/sec execute 325 sec latency 188.529 ms 2024-09-21T02:59:49.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:49 smithi190 ceph-mon[33871]: pgmap v1010: 129 pgs: 129 active+clean; 304 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.7 MiB/s wr, 301 op/s 2024-09-21T02:59:49.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:49 smithi029 ceph-mon[31006]: pgmap v1010: 129 pgs: 129 active+clean; 304 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.7 MiB/s wr, 301 op/s 2024-09-21T02:59:49.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46612 5.58 MB/sec execute 326 sec latency 90.085 ms 2024-09-21T02:59:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:49 smithi066 ceph-mon[33436]: pgmap v1010: 129 pgs: 129 active+clean; 304 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.7 MiB/s wr, 301 op/s 2024-09-21T02:59:50.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46770 5.58 MB/sec execute 327 sec latency 131.230 ms 2024-09-21T02:59:51.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46912 5.58 MB/sec execute 328 sec latency 114.240 ms 2024-09-21T02:59:51.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:51 smithi066 ceph-mon[33436]: pgmap v1011: 129 pgs: 129 active+clean; 303 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-09-21T02:59:51.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:51 smithi190 ceph-mon[33871]: pgmap v1011: 129 pgs: 129 active+clean; 303 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-09-21T02:59:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:51 smithi029 ceph-mon[31006]: pgmap v1011: 129 pgs: 129 active+clean; 303 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-09-21T02:59:52.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47076 5.58 MB/sec execute 329 sec latency 99.426 ms 2024-09-21T02:59:53.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47209 5.57 MB/sec execute 330 sec latency 84.224 ms 2024-09-21T02:59:53.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:53 smithi066 ceph-mon[33436]: pgmap v1012: 129 pgs: 129 active+clean; 304 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-09-21T02:59:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:53 smithi190 ceph-mon[33871]: pgmap v1012: 129 pgs: 129 active+clean; 304 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-09-21T02:59:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:53 smithi029 ceph-mon[31006]: pgmap v1012: 129 pgs: 129 active+clean; 304 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-09-21T02:59:54.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47347 5.57 MB/sec execute 331 sec latency 114.927 ms 2024-09-21T02:59:55.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47507 5.57 MB/sec execute 332 sec latency 92.763 ms 2024-09-21T02:59:55.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:55 smithi066 ceph-mon[33436]: pgmap v1013: 129 pgs: 129 active+clean; 303 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-09-21T02:59:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:55 smithi190 ceph-mon[33871]: pgmap v1013: 129 pgs: 129 active+clean; 303 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-09-21T02:59:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:55 smithi029 ceph-mon[31006]: pgmap v1013: 129 pgs: 129 active+clean; 303 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 -- 172.21.15.29:0/1372023880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75801036c0 msgr2=0x7f7580103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 --2- 172.21.15.29:0/1372023880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75801036c0 0x7f7580103b30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f757c009860 tx=0x7f757c02a340 comp rx=0 tx=0).stop 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 -- 172.21.15.29:0/1372023880 shutdown_connections 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 --2- 172.21.15.29:0/1372023880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75801050e0 0x7f758010ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 --2- 172.21.15.29:0/1372023880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7580104740 0x7f7580104b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 --2- 172.21.15.29:0/1372023880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75801036c0 0x7f7580103b30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 -- 172.21.15.29:0/1372023880 >> 172.21.15.29:0/1372023880 conn(0x7f75800fefe0 msgr2=0x7f75801014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 -- 172.21.15.29:0/1372023880 shutdown_connections 2024-09-21T02:59:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.444+0000 7f75877ed640 1 -- 172.21.15.29:0/1372023880 wait complete. 2024-09-21T02:59:56.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.445+0000 7f75877ed640 1 Processor -- start 2024-09-21T02:59:56.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.445+0000 7f75877ed640 1 -- start start 2024-09-21T02:59:56.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.445+0000 7f75877ed640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75801036c0 0x7f7580199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.445+0000 7f75877ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7580104740 0x7f758019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.445+0000 7f75877ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75801050e0 0x7f758019b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.445+0000 7f75877ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f758010f190 con 0x7f7580104740 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.445+0000 7f75877ed640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f758010f010 con 0x7f75801036c0 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.445+0000 7f75877ed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f758010f310 con 0x7f75801050e0 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585589640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75801036c0 0x7f7580199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7584d88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7580104740 0x7f758019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585589640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75801036c0 0x7f7580199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:60620/0 (socket says 172.21.15.29:60620) 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585589640 1 -- 172.21.15.29:0/3377132300 learned_addr learned my addr 172.21.15.29:0/3377132300 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7584d88640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7580104740 0x7f758019a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38696/0 (socket says 172.21.15.29:38696) 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585d8a640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75801050e0 0x7f758019b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585589640 1 -- 172.21.15.29:0/3377132300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75801050e0 msgr2=0x7f758019b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585589640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75801050e0 0x7f758019b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585589640 1 -- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7580104740 msgr2=0x7f758019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585589640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7580104740 0x7f758019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7585589640 1 -- 172.21.15.29:0/3377132300 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758019b880 con 0x7f75801036c0 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.446+0000 7f7584d88640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7580104740 0x7f758019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.447+0000 7f7585589640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75801036c0 0x7f7580199e40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f757c02a930 tx=0x7f757c02ad70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.447+0000 7f75767fc640 1 -- 172.21.15.29:0/3377132300 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c040030 con 0x7f75801036c0 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.447+0000 7f75877ed640 1 -- 172.21.15.29:0/3377132300 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f758019bb10 con 0x7f75801036c0 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.447+0000 7f75767fc640 1 -- 172.21.15.29:0/3377132300 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f757c032bc0 con 0x7f75801036c0 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.447+0000 7f75767fc640 1 -- 172.21.15.29:0/3377132300 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c037040 con 0x7f75801036c0 2024-09-21T02:59:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.447+0000 7f75877ed640 1 -- 172.21.15.29:0/3377132300 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75801a2260 con 0x7f75801036c0 2024-09-21T02:59:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.450+0000 7f75767fc640 1 -- 172.21.15.29:0/3377132300 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f757c043070 con 0x7f75801036c0 2024-09-21T02:59:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f758e830740 1 Processor -- start 2024-09-21T02:59:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f758e830740 1 -- start start 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f758e830740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebe98e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f758e830740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebe994c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f758e830740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebe9a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f758e830740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ebebe0b6f0 con 0x55ebebad04d0 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f758e830740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55ebebdfab20 con 0x55ebebac82f0 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f758e830740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ebebdfe930 con 0x55ebebaf11b0 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f7585d8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebe98e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f7585d8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebe98e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56514/0 (socket says 172.21.15.29:56514) 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f7585d8a640 1 -- 172.21.15.29:0/2181900889 learned_addr learned my addr 172.21.15.29:0/2181900889 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f7584d88640 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebe9a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f75767fc640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f756c0ba9a0 0x7f756c0bceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.451+0000 7f7585589640 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebe994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f7585d8a640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f756c0ba9a0 0x7f756c0bceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f75767fc640 1 -- 172.21.15.29:0/3377132300 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f757c1072a0 con 0x7f75801036c0 2024-09-21T02:59:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f7584d88640 1 -- 172.21.15.29:0/2181900889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 msgr2=0x55ebebe98e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f7584d88640 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebe98e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f7584d88640 1 -- 172.21.15.29:0/2181900889 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 msgr2=0x55ebebe994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f7584d88640 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebe994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f7584d88640 1 -- 172.21.15.29:0/2181900889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebebdc0530 con 0x55ebebad04d0 2024-09-21T02:59:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f7585d8a640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f756c0ba9a0 0x7f756c0bceb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f757800edc0 tx=0x7f7578004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.452+0000 7f7584d88640 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebe9a010 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f7570009fd0 tx=0x7f75700118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e01b0d56a335a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.453+0000 7f756b7fe640 1 -- 172.21.15.29:0/2181900889 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570004030 con 0x55ebebad04d0 2024-09-21T02:59:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.453+0000 7f756b7fe640 1 -- 172.21.15.29:0/2181900889 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7570005020 con 0x55ebebad04d0 2024-09-21T02:59:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.453+0000 7f756b7fe640 1 -- 172.21.15.29:0/2181900889 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570004a80 con 0x55ebebad04d0 2024-09-21T02:59:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 -- 172.21.15.29:0/2181900889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 msgr2=0x55ebebe9a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebe9a010 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f7570009fd0 tx=0x7f75700118d0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 -- 172.21.15.29:0/2181900889 shutdown_connections 2024-09-21T02:59:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebe9a010 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebe994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 --2- 172.21.15.29:0/2181900889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebe98e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 -- 172.21.15.29:0/2181900889 >> 172.21.15.29:0/2181900889 conn(0x55ebebae5300 msgr2=0x55ebebe948e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 -- 172.21.15.29:0/2181900889 shutdown_connections 2024-09-21T02:59:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 -- 172.21.15.29:0/2181900889 wait complete. 2024-09-21T02:59:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.454+0000 7f758e830740 1 -- start start 2024-09-21T02:59:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f758e830740 1 Processor -- start 2024-09-21T02:59:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f758e830740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebeaad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f758e830740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebeab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f758e830740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebeabfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f758e830740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ebebd32ab0 con 0x55ebebad04d0 2024-09-21T02:59:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f758e830740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55ebebda0730 con 0x55ebebac82f0 2024-09-21T02:59:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f758e830740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ebebe9ab30 con 0x55ebebaf11b0 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f7584d88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebeab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f7584d88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebeab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38716/0 (socket says 172.21.15.29:38716) 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f7584d88640 1 -- 172.21.15.29:0/1496457880 learned_addr learned my addr 172.21.15.29:0/1496457880 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f7585d8a640 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebeabfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f7585589640 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebeaad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.455+0000 7f758e830740 10 client.0 _ll_register_callbacks cb 0x55ebebdf46b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7584d88640 1 -- 172.21.15.29:0/1496457880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 msgr2=0x55ebebeabfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7584d88640 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebeabfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7584d88640 1 -- 172.21.15.29:0/1496457880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 msgr2=0x55ebebeaad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7584d88640 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebeaad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7584d88640 1 -- 172.21.15.29:0/1496457880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebebb5c790 con 0x55ebebad04d0 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7585589640 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebeaad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7584d88640 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebeab440 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f757001b040 tx=0x7f7570004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13b1fd49c64479f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7568ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7568ff9640 1 -- 172.21.15.29:0/1496457880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570028020 con 0x55ebebad04d0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f758e830740 1 -- 172.21.15.29:0/1496457880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7570005020 con 0x55ebebad04d0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.456+0000 7f7568ff9640 1 -- 172.21.15.29:0/1496457880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7570016040 con 0x55ebebad04d0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f7568ff9640 1 -- 172.21.15.29:0/1496457880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7570033050 con 0x55ebebad04d0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f758e830740 10 client.18219 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f758e830740 1 -- 172.21.15.29:0/1496457880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ebebbd5620 con 0x55ebebad04d0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f7568ff9640 1 -- 172.21.15.29:0/1496457880 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7570005020 con 0x55ebebad04d0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f7568ff9640 1 -- 172.21.15.29:0/1496457880 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f7570003960 con 0x55ebebad04d0 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f758e830740 10 client.18219 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f758e830740 10 client.18219 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f758e830740 20 client.18219 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f758e830740 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ebebaafd80 0x55ebebeb82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f758e830740 4 client.18219 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:59:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.457+0000 7f758e830740 1 -- 172.21.15.29:0/1496457880 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebebe0d640 con 0x55ebebaafd80 2024-09-21T02:59:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.458+0000 7f7585589640 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ebebaafd80 0x55ebebeb82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.458+0000 7f7568ff9640 10 client.18219 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.458+0000 7f7585589640 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ebebaafd80 0x55ebebeb82c0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1605bf914eecab4 server_cookie=ab62281544e5120e in_seq=0 out_seq=0 2024-09-21T02:59:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.459+0000 7f7568ff9640 1 -- 172.21.15.29:0/1496457880 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ebebe0d640 con 0x55ebebaafd80 2024-09-21T02:59:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.459+0000 7f7568ff9640 10 client.18219 handle_command_reply: tid=0 2024-09-21T02:59:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.459+0000 7f758e830740 1 client.18219 shutdown 2024-09-21T02:59:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 -- 172.21.15.29:0/1496457880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 msgr2=0x55ebebeab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebeab440 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f757001b040 tx=0x7f7570004270 comp rx=0 tx=0).stop 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 -- 172.21.15.29:0/1496457880 shutdown_connections 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaf11b0 0x55ebebeabfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ebebaafd80 0x55ebebeb82c0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebad04d0 0x55ebebeab440 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f7568ff9640 0 client.18219 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 --2- 172.21.15.29:0/1496457880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebac82f0 0x55ebebeaad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 -- 172.21.15.29:0/1496457880 >> 172.21.15.29:0/1496457880 conn(0x55ebebae5300 msgr2=0x55ebebe92ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 -- 172.21.15.29:0/1496457880 shutdown_connections 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 1 -- 172.21.15.29:0/1496457880 wait complete. 2024-09-21T02:59:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.460+0000 7f758e830740 20 client.18219 trim_cache size 0 max 16384 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.482+0000 7f758e830740 1 Processor -- start 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.482+0000 7f758e830740 1 -- start start 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.482+0000 7f758e830740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebaafd80 0x55ebebea0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.482+0000 7f758e830740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebea1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.482+0000 7f758e830740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebad04d0 0x55ebebe98df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.482+0000 7f758e830740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f75801aa3a0 con 0x55ebebac82f0 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.482+0000 7f758e830740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f75801abee0 con 0x55ebebaafd80 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.482+0000 7f758e830740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f75801aaa70 con 0x55ebebad04d0 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebea1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebea1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38722/0 (socket says 172.21.15.29:38722) 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 -- 172.21.15.29:0/4273968908 learned_addr learned my addr 172.21.15.29:0/4273968908 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7585589640 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebaafd80 0x55ebebea0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7585d8a640 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebad04d0 0x55ebebe98df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 -- 172.21.15.29:0/4273968908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebad04d0 msgr2=0x55ebebe98df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebad04d0 0x55ebebe98df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 -- 172.21.15.29:0/4273968908 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebaafd80 msgr2=0x55ebebea0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebaafd80 0x55ebebea0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 -- 172.21.15.29:0/4273968908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebebe900a0 con 0x55ebebac82f0 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f7584d88640 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebea1000 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f757000a010 tx=0x7f7570005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b860e9a2e6d838 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.483+0000 7f756affd640 1 -- 172.21.15.29:0/4273968908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570028040 con 0x55ebebac82f0 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.484+0000 7f756affd640 1 -- 172.21.15.29:0/4273968908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7570005690 con 0x55ebebac82f0 2024-09-21T02:59:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.484+0000 7f756affd640 1 -- 172.21.15.29:0/4273968908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570003850 con 0x55ebebac82f0 2024-09-21T02:59:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 -- 172.21.15.29:0/4273968908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 msgr2=0x55ebebea1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebea1000 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f757000a010 tx=0x7f7570005b90 comp rx=0 tx=0).stop 2024-09-21T02:59:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 -- 172.21.15.29:0/4273968908 shutdown_connections 2024-09-21T02:59:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebad04d0 0x55ebebe98df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebea1000 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 --2- 172.21.15.29:0/4273968908 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebaafd80 0x55ebebea0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 -- 172.21.15.29:0/4273968908 >> 172.21.15.29:0/4273968908 conn(0x55ebebae5300 msgr2=0x55ebebea1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 -- 172.21.15.29:0/4273968908 shutdown_connections 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 -- 172.21.15.29:0/4273968908 wait complete. 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 -- start start 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.485+0000 7f758e830740 1 Processor -- start 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f758e830740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaafd80 0x55ebebeafcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f758e830740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebeb0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f758e830740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebad04d0 0x55ebebeb0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f758e830740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f75801ab810 con 0x55ebebac82f0 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f758e830740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f75801ab140 con 0x55ebebad04d0 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f758e830740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f75801a8380 con 0x55ebebaafd80 2024-09-21T02:59:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f758e830740 10 client.0 _ll_register_callbacks cb 0x7f757c0c9060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f7584d88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebeb0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f7584d88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebeb0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38732/0 (socket says 172.21.15.29:38732) 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f7584d88640 1 -- 172.21.15.29:0/1532509747 learned_addr learned my addr 172.21.15.29:0/1532509747 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f7585589640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaafd80 0x55ebebeafcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.486+0000 7f7585d8a640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebad04d0 0x55ebebeb0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7584d88640 1 -- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaafd80 msgr2=0x55ebebeafcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7584d88640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaafd80 0x55ebebeafcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7584d88640 1 -- 172.21.15.29:0/1532509747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebad04d0 msgr2=0x55ebebeb0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7584d88640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebad04d0 0x55ebebeb0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7584d88640 1 -- 172.21.15.29:0/1532509747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebebeb1640 con 0x55ebebac82f0 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7585d8a640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebad04d0 0x55ebebeb0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7585589640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaafd80 0x55ebebeafcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7584d88640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebeb0370 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f7570003f40 tx=0x7f7570009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e15d52cfe780a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:59:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7568ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T02:59:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7568ff9640 1 -- 172.21.15.29:0/1532509747 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757003d040 con 0x55ebebac82f0 2024-09-21T02:59:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7568ff9640 1 -- 172.21.15.29:0/1532509747 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f757000f960 con 0x55ebebac82f0 2024-09-21T02:59:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f758e830740 1 -- 172.21.15.29:0/1532509747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ebebe090f0 con 0x55ebebac82f0 2024-09-21T02:59:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f7568ff9640 1 -- 172.21.15.29:0/1532509747 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f757004f050 con 0x55ebebac82f0 2024-09-21T02:59:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f758e830740 10 client.18225 fetch_fsmap learned FSMap version 37 2024-09-21T02:59:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.487+0000 7f758e830740 1 -- 172.21.15.29:0/1532509747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ebebeb2930 con 0x55ebebac82f0 2024-09-21T02:59:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f7568ff9640 1 -- 172.21.15.29:0/1532509747 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55ebebe090f0 con 0x55ebebac82f0 2024-09-21T02:59:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f7568ff9640 1 -- 172.21.15.29:0/1532509747 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f7570040070 con 0x55ebebac82f0 2024-09-21T02:59:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f758e830740 10 client.18225 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T02:59:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f758e830740 10 client.18225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T02:59:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f758e830740 20 client.18225 populate_metadata read hostname 'smithi029' 2024-09-21T02:59:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f758e830740 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ebebaf11b0 0x55ebebf250c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:59:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f758e830740 4 client.18225 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T02:59:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f758e830740 1 -- 172.21.15.29:0/1532509747 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ebebe06520 con 0x55ebebaf11b0 2024-09-21T02:59:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.488+0000 7f7585589640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ebebaf11b0 0x55ebebf250c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:59:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.489+0000 7f7568ff9640 10 client.18225 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.489+0000 7f7585589640 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ebebaf11b0 0x55ebebf250c0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c320355be2c649f server_cookie=1e99f2469f7abe91 in_seq=0 out_seq=0 2024-09-21T02:59:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f7568ff9640 1 -- 172.21.15.29:0/1532509747 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ebebe06520 con 0x55ebebaf11b0 2024-09-21T02:59:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f7568ff9640 10 client.18225 handle_command_reply: tid=0 2024-09-21T02:59:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f758e830740 1 client.18225 shutdown 2024-09-21T02:59:56.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f758e830740 1 -- 172.21.15.29:0/1532509747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 msgr2=0x55ebebeb0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f758e830740 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebeb0370 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f7570003f40 tx=0x7f7570009b50 comp rx=0 tx=0).stop 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f758e830740 1 -- 172.21.15.29:0/1532509747 shutdown_connections 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f758e830740 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55ebebad04d0 0x55ebebeb0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f758e830740 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55ebebaf11b0 0x55ebebf250c0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f758e830740 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebebac82f0 0x55ebebeb0370 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.681+0000 7f758e830740 1 --2- 172.21.15.29:0/1532509747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebebaafd80 0x55ebebeafcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.682+0000 7f758e830740 1 -- 172.21.15.29:0/1532509747 >> 172.21.15.29:0/1532509747 conn(0x55ebebae5300 msgr2=0x55ebebe98df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.682+0000 7f7568ff9640 0 client.18225 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.682+0000 7f758e830740 1 -- 172.21.15.29:0/1532509747 shutdown_connections 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.682+0000 7f758e830740 1 -- 172.21.15.29:0/1532509747 wait complete. 2024-09-21T02:59:56.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.682+0000 7f758e830740 20 client.18225 trim_cache size 0 max 16384 2024-09-21T02:59:56.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T02:59:56.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T02:59:56.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T02:59:56.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T02:59:56.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 -- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f756c0ba9a0 msgr2=0x7f756c0bceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f756c0ba9a0 0x7f756c0bceb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f757800edc0 tx=0x7f7578004ab0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 -- 172.21.15.29:0/3377132300 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75801036c0 msgr2=0x7f7580199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75801036c0 0x7f7580199e40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f757c02a930 tx=0x7f757c02ad70 comp rx=0 tx=0).stop 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 -- 172.21.15.29:0/3377132300 shutdown_connections 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f756c0ba9a0 0x7f756c0bceb0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75801050e0 0x7f758019b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7580104740 0x7f758019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 --2- 172.21.15.29:0/3377132300 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75801036c0 0x7f7580199e40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.683+0000 7f75877ed640 1 -- 172.21.15.29:0/3377132300 >> 172.21.15.29:0/3377132300 conn(0x7f75800fefe0 msgr2=0x7f7580107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.684+0000 7f75877ed640 1 -- 172.21.15.29:0/3377132300 shutdown_connections 2024-09-21T02:59:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T02:59:56.684+0000 7f75877ed640 1 -- 172.21.15.29:0/3377132300 wait complete. 2024-09-21T02:59:56.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47690 5.60 MB/sec execute 333 sec latency 221.307 ms 2024-09-21T02:59:57.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47824 5.61 MB/sec execute 334 sec latency 187.237 ms 2024-09-21T02:59:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:57 smithi066 ceph-mon[33436]: pgmap v1014: 129 pgs: 129 active+clean; 304 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.1 MiB/s wr, 283 op/s 2024-09-21T02:59:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:57 smithi190 ceph-mon[33871]: pgmap v1014: 129 pgs: 129 active+clean; 304 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.1 MiB/s wr, 283 op/s 2024-09-21T02:59:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:57 smithi029 ceph-mon[31006]: pgmap v1014: 129 pgs: 129 active+clean; 304 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.1 MiB/s wr, 283 op/s 2024-09-21T02:59:58.735 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48025 5.62 MB/sec execute 335 sec latency 103.629 ms 2024-09-21T02:59:59.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48139 5.63 MB/sec execute 336 sec latency 264.006 ms 2024-09-21T02:59:59.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 02:59:59 smithi066 ceph-mon[33436]: pgmap v1015: 129 pgs: 129 active+clean; 303 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 3.3 MiB/s wr, 286 op/s 2024-09-21T02:59:59.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 02:59:59 smithi190 ceph-mon[33871]: pgmap v1015: 129 pgs: 129 active+clean; 303 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 3.3 MiB/s wr, 286 op/s 2024-09-21T02:59:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 02:59:59 smithi029 ceph-mon[31006]: pgmap v1015: 129 pgs: 129 active+clean; 303 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 3.3 MiB/s wr, 286 op/s 2024-09-21T03:00:00.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48283 5.65 MB/sec execute 337 sec latency 271.795 ms 2024-09-21T03:00:00.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:00 smithi066 ceph-mon[33436]: overall HEALTH_OK 2024-09-21T03:00:00.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:00 smithi190 ceph-mon[33871]: overall HEALTH_OK 2024-09-21T03:00:00.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:00 smithi029 ceph-mon[31006]: overall HEALTH_OK 2024-09-21T03:00:01.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48423 5.66 MB/sec execute 338 sec latency 97.532 ms 2024-09-21T03:00:01.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:01 smithi066 ceph-mon[33436]: pgmap v1016: 129 pgs: 129 active+clean; 302 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.3 MiB/s wr, 249 op/s 2024-09-21T03:00:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:01 smithi190 ceph-mon[33871]: pgmap v1016: 129 pgs: 129 active+clean; 302 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.3 MiB/s wr, 249 op/s 2024-09-21T03:00:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:01 smithi029 ceph-mon[31006]: pgmap v1016: 129 pgs: 129 active+clean; 302 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.3 MiB/s wr, 249 op/s 2024-09-21T03:00:02.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48607 5.70 MB/sec execute 339 sec latency 161.794 ms 2024-09-21T03:00:03.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48751 5.71 MB/sec execute 340 sec latency 97.346 ms 2024-09-21T03:00:03.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:03 smithi066 ceph-mon[33436]: pgmap v1017: 129 pgs: 129 active+clean; 309 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 3.8 MiB/s wr, 290 op/s 2024-09-21T03:00:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:03 smithi190 ceph-mon[33871]: pgmap v1017: 129 pgs: 129 active+clean; 309 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 3.8 MiB/s wr, 290 op/s 2024-09-21T03:00:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:03 smithi029 ceph-mon[31006]: pgmap v1017: 129 pgs: 129 active+clean; 309 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 3.8 MiB/s wr, 290 op/s 2024-09-21T03:00:04.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48873 5.72 MB/sec execute 341 sec latency 171.661 ms 2024-09-21T03:00:05.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48977 5.72 MB/sec execute 342 sec latency 223.498 ms 2024-09-21T03:00:05.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:05 smithi066 ceph-mon[33436]: pgmap v1018: 129 pgs: 129 active+clean; 313 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 4.0 MiB/s wr, 253 op/s 2024-09-21T03:00:05.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:05 smithi190 ceph-mon[33871]: pgmap v1018: 129 pgs: 129 active+clean; 313 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 4.0 MiB/s wr, 253 op/s 2024-09-21T03:00:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:05 smithi029 ceph-mon[31006]: pgmap v1018: 129 pgs: 129 active+clean; 313 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 4.0 MiB/s wr, 253 op/s 2024-09-21T03:00:06.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49100 5.71 MB/sec execute 343 sec latency 160.521 ms 2024-09-21T03:00:07.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49216 5.71 MB/sec execute 344 sec latency 111.793 ms 2024-09-21T03:00:07.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:07 smithi066 ceph-mon[33436]: pgmap v1019: 129 pgs: 129 active+clean; 317 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 4.1 MiB/s wr, 268 op/s 2024-09-21T03:00:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:07 smithi190 ceph-mon[33871]: pgmap v1019: 129 pgs: 129 active+clean; 317 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 4.1 MiB/s wr, 268 op/s 2024-09-21T03:00:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:07 smithi029 ceph-mon[31006]: pgmap v1019: 129 pgs: 129 active+clean; 317 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 4.1 MiB/s wr, 268 op/s 2024-09-21T03:00:08.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49352 5.71 MB/sec execute 345 sec latency 124.991 ms 2024-09-21T03:00:09.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49490 5.70 MB/sec execute 346 sec latency 118.006 ms 2024-09-21T03:00:09.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:09 smithi066 ceph-mon[33436]: pgmap v1020: 129 pgs: 129 active+clean; 320 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 4.3 MiB/s wr, 248 op/s 2024-09-21T03:00:09.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 -- 172.21.15.29:0/487087461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc06c070 msgr2=0x7f17cc106350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 --2- 172.21.15.29:0/487087461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc06c070 0x7f17cc106350 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009900 tx=0x7f17bc02a410 comp rx=0 tx=0).stop 2024-09-21T03:00:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 -- 172.21.15.29:0/487087461 shutdown_connections 2024-09-21T03:00:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 --2- 172.21.15.29:0/487087461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17cc1093f0 0x7f17cc10b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 --2- 172.21.15.29:0/487087461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17cc106890 0x7f17cc108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 --2- 172.21.15.29:0/487087461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc06c070 0x7f17cc106350 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 -- 172.21.15.29:0/487087461 >> 172.21.15.29:0/487087461 conn(0x7f17cc0fdf30 msgr2=0x7f17cc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 -- 172.21.15.29:0/487087461 shutdown_connections 2024-09-21T03:00:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.813+0000 7f17d2830640 1 -- 172.21.15.29:0/487087461 wait complete. 2024-09-21T03:00:09.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.814+0000 7f17d2830640 1 Processor -- start 2024-09-21T03:00:09.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.814+0000 7f17d2830640 1 -- start start 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d2830640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17cc06c070 0x7f17cc19c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d2830640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17cc106890 0x7f17cc19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d2830640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc1093f0 0x7f17cc19d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d2830640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc111430 con 0x7f17cc1093f0 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d2830640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc1112b0 con 0x7f17cc106890 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d2830640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc1115b0 con 0x7f17cc06c070 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d0dcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc1093f0 0x7f17cc19d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17cc06c070 0x7f17cc19c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d0dcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc1093f0 0x7f17cc19d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53494/0 (socket says 172.21.15.29:53494) 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17cc106890 0x7f17cc19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17cc06c070 0x7f17cc19c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45960/0 (socket says 172.21.15.29:45960) 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d0dcd640 1 -- 172.21.15.29:0/3472112616 learned_addr learned my addr 172.21.15.29:0/3472112616 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d0dcd640 1 -- 172.21.15.29:0/3472112616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17cc06c070 msgr2=0x7f17cc19c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d0dcd640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17cc06c070 0x7f17cc19c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d0dcd640 1 -- 172.21.15.29:0/3472112616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17cc106890 msgr2=0x7f17cc19c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d0dcd640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17cc106890 0x7f17cc19c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17d0dcd640 1 -- 172.21.15.29:0/3472112616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc19dae0 con 0x7f17cc1093f0 2024-09-21T03:00:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17cc106890 0x7f17cc19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.815+0000 7f17cbfff640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17cc06c070 0x7f17cc19c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.816+0000 7f17d0dcd640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc1093f0 0x7f17cc19d2c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f17c400eb10 tx=0x7f17c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.816+0000 7f17c97fa640 1 -- 172.21.15.29:0/3472112616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c4004b20 con 0x7f17cc1093f0 2024-09-21T03:00:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.816+0000 7f17d2830640 1 -- 172.21.15.29:0/3472112616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17cc19ddd0 con 0x7f17cc1093f0 2024-09-21T03:00:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.816+0000 7f17d2830640 1 -- 172.21.15.29:0/3472112616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17cc1a44c0 con 0x7f17cc1093f0 2024-09-21T03:00:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.816+0000 7f17c97fa640 1 -- 172.21.15.29:0/3472112616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17c4004cc0 con 0x7f17cc1093f0 2024-09-21T03:00:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.816+0000 7f17c97fa640 1 -- 172.21.15.29:0/3472112616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c4010690 con 0x7f17cc1093f0 2024-09-21T03:00:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.819+0000 7f17d987a740 1 Processor -- start 2024-09-21T03:00:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.819+0000 7f17c97fa640 1 -- 172.21.15.29:0/3472112616 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f17c4010830 con 0x7f17cc1093f0 2024-09-21T03:00:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.819+0000 7f17d987a740 1 -- start start 2024-09-21T03:00:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17d987a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69c21b0 0x5640e6d69e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17d987a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17d987a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d6b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17d987a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5640e6cdc6f0 con 0x5640e69c21b0 2024-09-21T03:00:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17d987a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5640e6ccbb20 con 0x5640e69a14d0 2024-09-21T03:00:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17d987a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5640e6ccf930 con 0x5640e69992f0 2024-09-21T03:00:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69c21b0 0x5640e6d69e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69c21b0 0x5640e6d69e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53496/0 (socket says 172.21.15.29:53496) 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cb7fe640 1 -- 172.21.15.29:0/3549571351 learned_addr learned my addr 172.21.15.29:0/3549571351 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/3549571351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d6b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cbfff640 1 -- 172.21.15.29:0/3549571351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 msgr2=0x5640e6d6b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cbfff640 1 --2- 172.21.15.29:0/3549571351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d6b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cbfff640 1 -- 172.21.15.29:0/3549571351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69c21b0 msgr2=0x5640e6d69e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cbfff640 1 --2- 172.21.15.29:0/3549571351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69c21b0 0x5640e6d69e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17cbfff640 1 -- 172.21.15.29:0/3549571351 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e6c91530 con 0x5640e69992f0 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.820+0000 7f17c97fa640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17b40b6510 0x7f17b40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.821+0000 7f17cbfff640 1 --2- 172.21.15.29:0/3549571351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6a4c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f17bc02a930 tx=0x7f17bc0043f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcb7bba4d2251f96 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.821+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17b40b6510 0x7f17b40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.821+0000 7f17c97fa640 1 -- 172.21.15.29:0/3472112616 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f17c40e7b00 con 0x7f17cc1093f0 2024-09-21T03:00:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.821+0000 7f17b27fc640 1 -- 172.21.15.29:0/3549571351 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc035030 con 0x5640e69992f0 2024-09-21T03:00:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.821+0000 7f17b27fc640 1 -- 172.21.15.29:0/3549571351 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17bc040070 con 0x5640e69992f0 2024-09-21T03:00:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.821+0000 7f17b27fc640 1 -- 172.21.15.29:0/3549571351 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc031a60 con 0x5640e69992f0 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 -- 172.21.15.29:0/3549571351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 msgr2=0x5640e6d6a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17b40b6510 0x7f17b40b8a20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f17c0011c20 tx=0x7f17c0009b70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 --2- 172.21.15.29:0/3549571351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6a4c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f17bc02a930 tx=0x7f17bc0043f0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 -- 172.21.15.29:0/3549571351 shutdown_connections 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 --2- 172.21.15.29:0/3549571351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d6b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 --2- 172.21.15.29:0/3549571351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6a4c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 --2- 172.21.15.29:0/3549571351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69c21b0 0x5640e6d69e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 -- 172.21.15.29:0/3549571351 >> 172.21.15.29:0/3549571351 conn(0x5640e69b6300 msgr2=0x5640e6d658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 -- 172.21.15.29:0/3549571351 shutdown_connections 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.822+0000 7f17d987a740 1 -- 172.21.15.29:0/3549571351 wait complete. 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 1 -- start start 2024-09-21T03:00:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 1 Processor -- start 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d73560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69a14d0 0x5640e6d73c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 0x5640e6d74790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5640e6c03ab0 con 0x5640e69992f0 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5640e6c71730 con 0x5640e69c21b0 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5640e6d6bad0 con 0x5640e69a14d0 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.823+0000 7f17d987a740 10 client.0 _ll_register_callbacks cb 0x5640e6cc56b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69a14d0 0x5640e6d73c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d73560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69a14d0 0x5640e6d73c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45986/0 (socket says 172.21.15.29:45986) 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 -- 172.21.15.29:0/2227750412 learned_addr learned my addr 172.21.15.29:0/2227750412 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17d0dcd640 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 0x5640e6d74790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 -- 172.21.15.29:0/2227750412 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 msgr2=0x5640e6d74790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 0x5640e6d74790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 -- 172.21.15.29:0/2227750412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 msgr2=0x5640e6d73560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d73560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 -- 172.21.15.29:0/2227750412 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e6a2d790 con 0x5640e69a14d0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d73560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69a14d0 0x5640e6d73c20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f17c0021960 tx=0x7f17c0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73a864d59a482518 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.824+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2227750412 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0004260 con 0x5640e69a14d0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 1 -- 172.21.15.29:0/2227750412 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17bc040070 con 0x5640e69a14d0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2227750412 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f17c001a070 con 0x5640e69a14d0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2227750412 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f17c0034050 con 0x5640e69a14d0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 10 client.27700 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 1 -- 172.21.15.29:0/2227750412 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5640e6aa6620 con 0x5640e69a14d0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2227750412 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17bc040070 con 0x5640e69a14d0 2024-09-21T03:00:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2227750412 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f17c001d070 con 0x5640e69a14d0 2024-09-21T03:00:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 10 client.27700 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 10 client.27700 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:00:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 20 client.27700 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640e6980d80 0x5640e6d80db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 4 client.27700 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:00:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.825+0000 7f17d987a740 1 -- 172.21.15.29:0/2227750412 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640e6cde640 con 0x5640e6980d80 2024-09-21T03:00:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.826+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640e6980d80 0x5640e6d80db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.826+0000 7f17ab7fe640 10 client.27700 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.826+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640e6980d80 0x5640e6d80db0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44cd90b19477f4fb server_cookie=45b50527c7fb578 in_seq=0 out_seq=0 2024-09-21T03:00:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.828+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2227750412 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5640e6cde640 con 0x5640e6980d80 2024-09-21T03:00:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.828+0000 7f17ab7fe640 10 client.27700 handle_command_reply: tid=0 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.828+0000 7f17d987a740 1 client.27700 shutdown 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.828+0000 7f17d987a740 1 -- 172.21.15.29:0/2227750412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69a14d0 msgr2=0x5640e6d73c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.828+0000 7f17d987a740 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69a14d0 0x5640e6d73c20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f17c0021960 tx=0x7f17c0004730 comp rx=0 tx=0).stop 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 1 -- 172.21.15.29:0/2227750412 shutdown_connections 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 0x5640e6d74790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69a14d0 0x5640e6d73c20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640e6980d80 0x5640e6d80db0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 1 --2- 172.21.15.29:0/2227750412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d73560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 1 -- 172.21.15.29:0/2227750412 >> 172.21.15.29:0/2227750412 conn(0x5640e69b6300 msgr2=0x5640e6d63ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17ab7fe640 0 client.27700 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 1 -- 172.21.15.29:0/2227750412 shutdown_connections 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 1 -- 172.21.15.29:0/2227750412 wait complete. 2024-09-21T03:00:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.829+0000 7f17d987a740 20 client.27700 trim_cache size 0 max 16384 2024-09-21T03:00:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d987a740 1 Processor -- start 2024-09-21T03:00:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d987a740 1 -- start start 2024-09-21T03:00:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d987a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d987a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69a14d0 0x5640e6d72760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d987a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 0x5640e6d605c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d987a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17b800fb50 con 0x5640e69a14d0 2024-09-21T03:00:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d987a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f17b8011690 con 0x5640e69c21b0 2024-09-21T03:00:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d987a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17b8010220 con 0x5640e69992f0 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69a14d0 0x5640e6d72760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69a14d0 0x5640e6d72760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53518/0 (socket says 172.21.15.29:53518) 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17cb7fe640 1 -- 172.21.15.29:0/2782593538 learned_addr learned my addr 172.21.15.29:0/2782593538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.850+0000 7f17d0dcd640 1 --2- 172.21.15.29:0/2782593538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 0x5640e6d605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17cbfff640 1 -- 172.21.15.29:0/2782593538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 msgr2=0x5640e6d605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2782593538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 0x5640e6d605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17cbfff640 1 -- 172.21.15.29:0/2782593538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69a14d0 msgr2=0x5640e6d72760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2782593538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69a14d0 0x5640e6d72760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17cbfff640 1 -- 172.21.15.29:0/2782593538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e6d6bc40 con 0x5640e69992f0 2024-09-21T03:00:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2782593538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6ace0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f17bc005760 tx=0x7f17bc047040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3178e18b28957a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17b1ffb640 1 -- 172.21.15.29:0/2782593538 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc035040 con 0x5640e69992f0 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17b1ffb640 1 -- 172.21.15.29:0/2782593538 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17bc040070 con 0x5640e69992f0 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.851+0000 7f17b1ffb640 1 -- 172.21.15.29:0/2782593538 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc003e70 con 0x5640e69992f0 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.852+0000 7f17d987a740 1 -- 172.21.15.29:0/2782593538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 msgr2=0x5640e6d6ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.852+0000 7f17d987a740 1 --2- 172.21.15.29:0/2782593538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6ace0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f17bc005760 tx=0x7f17bc047040 comp rx=0 tx=0).stop 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 -- 172.21.15.29:0/2782593538 shutdown_connections 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 --2- 172.21.15.29:0/2782593538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69c21b0 0x5640e6d605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 --2- 172.21.15.29:0/2782593538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69a14d0 0x5640e6d72760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 --2- 172.21.15.29:0/2782593538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69992f0 0x5640e6d6ace0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 -- 172.21.15.29:0/2782593538 >> 172.21.15.29:0/2782593538 conn(0x5640e69b6300 msgr2=0x5640e6d65220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 -- 172.21.15.29:0/2782593538 shutdown_connections 2024-09-21T03:00:09.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 -- 172.21.15.29:0/2782593538 wait complete. 2024-09-21T03:00:09.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 -- start start 2024-09-21T03:00:09.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.853+0000 7f17d987a740 1 Processor -- start 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17d987a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d77dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17d987a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d78480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17d987a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69c21b0 0x5640e6d78ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d77dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d78480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17cb7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d78480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:56992/0 (socket says 172.21.15.29:56992) 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d77dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53534/0 (socket says 172.21.15.29:53534) 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17cb7fe640 1 -- 172.21.15.29:0/2662520933 learned_addr learned my addr 172.21.15.29:0/2662520933 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17b8010fc0 con 0x5640e69992f0 2024-09-21T03:00:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f17b80108f0 con 0x5640e69a14d0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.854+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17b800db30 con 0x5640e69c21b0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17d987a740 10 client.0 _ll_register_callbacks cb 0x7f17c00217f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17d0dcd640 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69c21b0 0x5640e6d78ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17cbfff640 1 -- 172.21.15.29:0/2662520933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69c21b0 msgr2=0x5640e6d78ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69c21b0 0x5640e6d78ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17cbfff640 1 -- 172.21.15.29:0/2662520933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 msgr2=0x5640e6d78480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d78480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17cbfff640 1 -- 172.21.15.29:0/2662520933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e6d79750 con 0x5640e69992f0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d78480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.855+0000 7f17cbfff640 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d77dc0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009850 tx=0x7f17bc031730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d9e3c6448b7988 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.856+0000 7f17ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.856+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2662520933 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc035070 con 0x5640e69992f0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.856+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2662520933 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f17bc003e00 con 0x5640e69992f0 2024-09-21T03:00:09.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.856+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17bc040070 con 0x5640e69992f0 2024-09-21T03:00:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.856+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2662520933 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f17bc06c050 con 0x5640e69992f0 2024-09-21T03:00:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.856+0000 7f17d987a740 10 client.18243 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.856+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5640e6d7aa40 con 0x5640e69992f0 2024-09-21T03:00:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.857+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2662520933 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17bc040070 con 0x5640e69992f0 2024-09-21T03:00:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.857+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2662520933 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f17bc045670 con 0x5640e69992f0 2024-09-21T03:00:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.857+0000 7f17d987a740 10 client.18243 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.857+0000 7f17d987a740 10 client.18243 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:00:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.857+0000 7f17d987a740 20 client.18243 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.857+0000 7f17d987a740 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640e69561c0 0x5640e6d7af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.857+0000 7f17d987a740 4 client.18243 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:00:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.857+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640e6cda0f0 con 0x5640e69561c0 2024-09-21T03:00:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.858+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640e69561c0 0x5640e6d7af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.860+0000 7f17cb7fe640 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640e69561c0 0x5640e6d7af30 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72eecad16ad3a07 server_cookie=99fe97034ae0db04 in_seq=0 out_seq=0 2024-09-21T03:00:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:09.861+0000 7f17ab7fe640 10 client.18243 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:09 smithi190 ceph-mon[33871]: pgmap v1020: 129 pgs: 129 active+clean; 320 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 4.3 MiB/s wr, 248 op/s 2024-09-21T03:00:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:09 smithi029 ceph-mon[31006]: pgmap v1020: 129 pgs: 129 active+clean; 320 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 4.3 MiB/s wr, 248 op/s 2024-09-21T03:00:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.153+0000 7f17ab7fe640 1 -- 172.21.15.29:0/2662520933 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640e6cda0f0 con 0x5640e69561c0 2024-09-21T03:00:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.153+0000 7f17ab7fe640 10 client.18243 handle_command_reply: tid=0 2024-09-21T03:00:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.153+0000 7f17d987a740 1 client.18243 shutdown 2024-09-21T03:00:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 msgr2=0x5640e6d77dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d77dc0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009850 tx=0x7f17bc031730 comp rx=0 tx=0).stop 2024-09-21T03:00:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 shutdown_connections 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640e69c21b0 0x5640e6d78ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5640e69a14d0 0x5640e6d78480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5640e69561c0 0x5640e6d7af30 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 --2- 172.21.15.29:0/2662520933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640e69992f0 0x5640e6d77dc0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 >> 172.21.15.29:0/2662520933 conn(0x5640e69b6300 msgr2=0x5640e6d6ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17ab7fe640 0 client.18243 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 shutdown_connections 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 1 -- 172.21.15.29:0/2662520933 wait complete. 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.154+0000 7f17d987a740 20 client.18243 trim_cache size 0 max 16384 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:00:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:00:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:00:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:00:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 -- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17b40b6510 msgr2=0x7f17b40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17b40b6510 0x7f17b40b8a20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f17c0011c20 tx=0x7f17c0009b70 comp rx=0 tx=0).stop 2024-09-21T03:00:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 -- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc1093f0 msgr2=0x7f17cc19d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc1093f0 0x7f17cc19d2c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f17c400eb10 tx=0x7f17c400c2d0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 -- 172.21.15.29:0/3472112616 shutdown_connections 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17b40b6510 0x7f17b40b8a20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc1093f0 0x7f17cc19d2c0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17cc106890 0x7f17cc19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 --2- 172.21.15.29:0/3472112616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17cc06c070 0x7f17cc19c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 -- 172.21.15.29:0/3472112616 >> 172.21.15.29:0/3472112616 conn(0x7f17cc0fdf30 msgr2=0x7f17cc104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.155+0000 7f17d2830640 1 -- 172.21.15.29:0/3472112616 shutdown_connections 2024-09-21T03:00:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:10.156+0000 7f17d2830640 1 -- 172.21.15.29:0/3472112616 wait complete. 2024-09-21T03:00:10.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49607 5.71 MB/sec execute 347 sec latency 242.561 ms 2024-09-21T03:00:11.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49728 5.70 MB/sec execute 348 sec latency 125.938 ms 2024-09-21T03:00:11.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:11 smithi066 ceph-mon[33436]: pgmap v1021: 129 pgs: 129 active+clean; 319 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 4.2 MiB/s wr, 203 op/s 2024-09-21T03:00:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:11 smithi190 ceph-mon[33871]: pgmap v1021: 129 pgs: 129 active+clean; 319 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 4.2 MiB/s wr, 203 op/s 2024-09-21T03:00:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:11 smithi029 ceph-mon[31006]: pgmap v1021: 129 pgs: 129 active+clean; 319 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 4.2 MiB/s wr, 203 op/s 2024-09-21T03:00:12.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49877 5.70 MB/sec execute 349 sec latency 98.174 ms 2024-09-21T03:00:13.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50021 5.69 MB/sec execute 350 sec latency 104.796 ms 2024-09-21T03:00:13.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:13 smithi066 ceph-mon[33436]: pgmap v1022: 129 pgs: 129 active+clean; 316 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 4.2 MiB/s wr, 280 op/s 2024-09-21T03:00:13.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:13 smithi190 ceph-mon[33871]: pgmap v1022: 129 pgs: 129 active+clean; 316 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 4.2 MiB/s wr, 280 op/s 2024-09-21T03:00:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:13 smithi029 ceph-mon[31006]: pgmap v1022: 129 pgs: 129 active+clean; 316 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 4.2 MiB/s wr, 280 op/s 2024-09-21T03:00:14.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50191 5.72 MB/sec execute 351 sec latency 129.566 ms 2024-09-21T03:00:15.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50354 5.73 MB/sec execute 352 sec latency 101.827 ms 2024-09-21T03:00:15.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:15 smithi066 ceph-mon[33436]: pgmap v1023: 129 pgs: 129 active+clean; 308 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 3.9 MiB/s wr, 322 op/s 2024-09-21T03:00:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:15 smithi190 ceph-mon[33871]: pgmap v1023: 129 pgs: 129 active+clean; 308 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 3.9 MiB/s wr, 322 op/s 2024-09-21T03:00:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:15 smithi029 ceph-mon[31006]: pgmap v1023: 129 pgs: 129 active+clean; 308 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 3.9 MiB/s wr, 322 op/s 2024-09-21T03:00:16.281 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.3 2024-09-21T03:00:16.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:00:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:00:16.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50500 5.74 MB/sec execute 353 sec latency 147.715 ms 2024-09-21T03:00:17.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:17 smithi029 ceph-mon[31006]: pgmap v1024: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 3.6 MiB/s wr, 339 op/s 2024-09-21T03:00:17.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50671 5.74 MB/sec execute 354 sec latency 148.528 ms 2024-09-21T03:00:17.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:17 smithi066 ceph-mon[33436]: pgmap v1024: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 3.6 MiB/s wr, 339 op/s 2024-09-21T03:00:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:17 smithi190 ceph-mon[33871]: pgmap v1024: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 3.6 MiB/s wr, 339 op/s 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 -- 172.21.15.29:0/1618335604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064104880 msgr2=0x7fd064104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 --2- 172.21.15.29:0/1618335604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064104880 0x7fd064104c90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fd054009900 tx=0x7fd05402a3f0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 -- 172.21.15.29:0/1618335604 shutdown_connections 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 --2- 172.21.15.29:0/1618335604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd06410de30 0x7fd064110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 --2- 172.21.15.29:0/1618335604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0641052b0 0x7fd06410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 --2- 172.21.15.29:0/1618335604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd064104880 0x7fd064104c90 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 -- 172.21.15.29:0/1618335604 >> 172.21.15.29:0/1618335604 conn(0x7fd064079730 msgr2=0x7fd064079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 -- 172.21.15.29:0/1618335604 shutdown_connections 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.040+0000 7fd069337640 1 -- 172.21.15.29:0/1618335604 wait complete. 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.041+0000 7fd069337640 1 Processor -- start 2024-09-21T03:00:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.041+0000 7fd069337640 1 -- start start 2024-09-21T03:00:18.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd069337640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd064104880 0x7fd06419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:18.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd069337640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0641052b0 0x7fd06419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:18.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd069337640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd06410de30 0x7fd0641a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:18.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd069337640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd064113970 con 0x7fd0641052b0 2024-09-21T03:00:18.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd062ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd064104880 0x7fd06419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd069337640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd0641137f0 con 0x7fd064104880 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd069337640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd064113af0 con 0x7fd06410de30 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd062ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd064104880 0x7fd06419c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:55290/0 (socket says 172.21.15.29:55290) 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd062ffd640 1 -- 172.21.15.29:0/4131888424 learned_addr learned my addr 172.21.15.29:0/4131888424 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.042+0000 7fd0637fe640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd06410de30 0x7fd0641a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd062ffd640 1 -- 172.21.15.29:0/4131888424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd06410de30 msgr2=0x7fd0641a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd062ffd640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd06410de30 0x7fd0641a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd0627fc640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0641052b0 0x7fd06419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd062ffd640 1 -- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0641052b0 msgr2=0x7fd06419cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd062ffd640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0641052b0 0x7fd06419cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd062ffd640 1 -- 172.21.15.29:0/4131888424 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0641a2550 con 0x7fd064104880 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd0627fc640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0641052b0 0x7fd06419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd062ffd640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd064104880 0x7fd06419c5e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd05402a910 tx=0x7fd054003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd04bfff640 1 -- 172.21.15.29:0/4131888424 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05403f070 con 0x7fd064104880 2024-09-21T03:00:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd04bfff640 1 -- 172.21.15.29:0/4131888424 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd05403baf0 con 0x7fd064104880 2024-09-21T03:00:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.043+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0641a27e0 con 0x7fd064104880 2024-09-21T03:00:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.044+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0641a8fa0 con 0x7fd064104880 2024-09-21T03:00:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.044+0000 7fd04bfff640 1 -- 172.21.15.29:0/4131888424 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054035030 con 0x7fd064104880 2024-09-21T03:00:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.045+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd064104c90 con 0x7fd064104880 2024-09-21T03:00:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.046+0000 7fd04bfff640 1 -- 172.21.15.29:0/4131888424 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd054042070 con 0x7fd064104880 2024-09-21T03:00:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.047+0000 7fd04bfff640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd04c0b6590 0x7fd04c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.047+0000 7fd0627fc640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd04c0b6590 0x7fd04c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.048+0000 7fd04bfff640 1 -- 172.21.15.29:0/4131888424 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd054107c20 con 0x7fd064104880 2024-09-21T03:00:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.048+0000 7fd0627fc640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd04c0b6590 0x7fd04c0b8aa0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd06410fb90 tx=0x7fd058005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:18.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.052+0000 7fd04bfff640 1 -- 172.21.15.29:0/4131888424 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd0540ce130 con 0x7fd064104880 2024-09-21T03:00:18.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.396+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd06410b3e0 con 0x7fd064104880 2024-09-21T03:00:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.397+0000 7fd04bfff640 1 -- 172.21.15.29:0/4131888424 <== mon.2 v2:172.21.15.66:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7fd0540d2510 con 0x7fd064104880 2024-09-21T03:00:18.398 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:00:18.398 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:00:18.399 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T03:00:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd04c0b6590 msgr2=0x7fd04c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd04c0b6590 0x7fd04c0b8aa0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd06410fb90 tx=0x7fd058005f90 comp rx=0 tx=0).stop 2024-09-21T03:00:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd064104880 msgr2=0x7fd06419c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd064104880 0x7fd06419c5e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd05402a910 tx=0x7fd054003b90 comp rx=0 tx=0).stop 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 shutdown_connections 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd04c0b6590 0x7fd04c0b8aa0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd06410de30 0x7fd0641a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0641052b0 0x7fd06419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 --2- 172.21.15.29:0/4131888424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd064104880 0x7fd06419c5e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 >> 172.21.15.29:0/4131888424 conn(0x7fd064079730 msgr2=0x7fd064100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 shutdown_connections 2024-09-21T03:00:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:18.400+0000 7fd069337640 1 -- 172.21.15.29:0/4131888424 wait complete. 2024-09-21T03:00:18.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50789 5.74 MB/sec execute 355 sec latency 109.376 ms 2024-09-21T03:00:18.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:18 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4131888424' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:00:18.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:18 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4131888424' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:00:18.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:18 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4131888424' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:00:19.626 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 142.3 --await-for 45.0' 2024-09-21T03:00:19.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 142.3 --await-for 45.0 2024-09-21T03:00:19.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50925 5.74 MB/sec execute 356 sec latency 83.451 ms 2024-09-21T03:00:19.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:19 smithi066 ceph-mon[33436]: pgmap v1025: 129 pgs: 129 active+clean; 305 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.3 MiB/s wr, 323 op/s 2024-09-21T03:00:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:19 smithi190 ceph-mon[33871]: pgmap v1025: 129 pgs: 129 active+clean; 305 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.3 MiB/s wr, 323 op/s 2024-09-21T03:00:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:19 smithi029 ceph-mon[31006]: pgmap v1025: 129 pgs: 129 active+clean; 305 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.3 MiB/s wr, 323 op/s 2024-09-21T03:00:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:00:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.716+0000 7f1484a46640 1 -- 172.21.15.29:0/442318574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14801083d0 msgr2=0x7f14801087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.716+0000 7f1484a46640 1 --2- 172.21.15.29:0/442318574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14801083d0 0x7f14801087e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1474009900 tx=0x7f147402a3f0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.716+0000 7f1484a46640 1 -- 172.21.15.29:0/442318574 shutdown_connections 2024-09-21T03:00:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.716+0000 7f1484a46640 1 --2- 172.21.15.29:0/442318574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1480108d20 0x7f148010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.716+0000 7f1484a46640 1 --2- 172.21.15.29:0/442318574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14801083d0 0x7f14801087e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.716+0000 7f1484a46640 1 --2- 172.21.15.29:0/442318574 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1480102a90 0x7f1480102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.716+0000 7f1484a46640 1 -- 172.21.15.29:0/442318574 >> 172.21.15.29:0/442318574 conn(0x7f1480079820 msgr2=0x7f1480079c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:20.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.717+0000 7f1484a46640 1 -- 172.21.15.29:0/442318574 shutdown_connections 2024-09-21T03:00:20.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.717+0000 7f1484a46640 1 -- 172.21.15.29:0/442318574 wait complete. 2024-09-21T03:00:20.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.718+0000 7f1484a46640 1 Processor -- start 2024-09-21T03:00:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.718+0000 7f1484a46640 1 -- start start 2024-09-21T03:00:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f1484a46640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480102a90 0x7f1480110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f1484a46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14801083d0 0x7f1480110570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f1484a46640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1480108d20 0x7f1480110ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f1484a46640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14801139b0 con 0x7f14801083d0 2024-09-21T03:00:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f1484a46640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1480113830 con 0x7f1480108d20 2024-09-21T03:00:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f1484a46640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1480113b30 con 0x7f1480102a90 2024-09-21T03:00:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f147dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14801083d0 0x7f1480110570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f147e59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480102a90 0x7f1480110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f147e59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480102a90 0x7f1480110030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:35904/0 (socket says 172.21.15.29:35904) 2024-09-21T03:00:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f147e59c640 1 -- 172.21.15.29:0/2381359668 learned_addr learned my addr 172.21.15.29:0/2381359668 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.719+0000 7f147ed9d640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1480108d20 0x7f1480110ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f147e59c640 1 -- 172.21.15.29:0/2381359668 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1480108d20 msgr2=0x7f1480110ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f147e59c640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1480108d20 0x7f1480110ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f147e59c640 1 -- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14801083d0 msgr2=0x7f1480110570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f147e59c640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14801083d0 0x7f1480110570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f147e59c640 1 -- 172.21.15.29:0/2381359668 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14801a8720 con 0x7f1480102a90 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f147e59c640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480102a90 0x7f1480110030 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f147000c890 tx=0x7f147000cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f146f7fe640 1 -- 172.21.15.29:0/2381359668 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470007bb0 con 0x7f1480102a90 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f146f7fe640 1 -- 172.21.15.29:0/2381359668 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1470007d50 con 0x7f1480102a90 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f146f7fe640 1 -- 172.21.15.29:0/2381359668 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470004de0 con 0x7f1480102a90 2024-09-21T03:00:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.720+0000 7f1484a46640 1 -- 172.21.15.29:0/2381359668 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14801a8a10 con 0x7f1480102a90 2024-09-21T03:00:20.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.721+0000 7f1484a46640 1 -- 172.21.15.29:0/2381359668 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14801a8e70 con 0x7f1480102a90 2024-09-21T03:00:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.724+0000 7f146f7fe640 1 -- 172.21.15.29:0/2381359668 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f1470004fc0 con 0x7f1480102a90 2024-09-21T03:00:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.725+0000 7f146f7fe640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f14680b6660 0x7f14680b8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.725+0000 7f146f7fe640 1 -- 172.21.15.29:0/2381359668 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f14700e7290 con 0x7f1480102a90 2024-09-21T03:00:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.725+0000 7f147dd9b640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f14680b6660 0x7f14680b8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.726+0000 7f147dd9b640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f14680b6660 0x7f14680b8b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1474005940 tx=0x7f14740058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:20.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51080 5.76 MB/sec execute 357 sec latency 117.609 ms 2024-09-21T03:00:20.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.740+0000 7f148afba740 1 Processor -- start 2024-09-21T03:00:20.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f148afba740 1 -- start start 2024-09-21T03:00:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f148afba740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f148afba740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefacb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f148afba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefad080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f148afba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b5fef22b90 con 0x55b5febe4b60 2024-09-21T03:00:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f148afba740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b5fef11fc0 con 0x55b5febdc980 2024-09-21T03:00:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f148afba740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b5fef15dd0 con 0x55b5fec05840 2024-09-21T03:00:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f147e59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f147dd9b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefacb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f147dd9b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefacb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:55308/0 (socket says 172.21.15.29:55308) 2024-09-21T03:00:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f147e59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefac600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:35920/0 (socket says 172.21.15.29:35920) 2024-09-21T03:00:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f147dd9b640 1 -- 172.21.15.29:0/1320399772 learned_addr learned my addr 172.21.15.29:0/1320399772 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.741+0000 7f147ed9d640 1 --2- 172.21.15.29:0/1320399772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.742+0000 7f147dd9b640 1 -- 172.21.15.29:0/1320399772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 msgr2=0x55b5fefac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.742+0000 7f147dd9b640 1 --2- 172.21.15.29:0/1320399772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.742+0000 7f147dd9b640 1 -- 172.21.15.29:0/1320399772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 msgr2=0x55b5fefad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.742+0000 7f147dd9b640 1 --2- 172.21.15.29:0/1320399772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.742+0000 7f147dd9b640 1 -- 172.21.15.29:0/1320399772 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5fee99060 con 0x55b5febdc980 2024-09-21T03:00:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.742+0000 7f147dd9b640 1 --2- 172.21.15.29:0/1320399772 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefacb40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f14740096c0 tx=0x7f147403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84e503b51b1b052e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.742+0000 7f146cff9640 1 -- 172.21.15.29:0/1320399772 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147404c070 con 0x55b5febdc980 2024-09-21T03:00:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.743+0000 7f146cff9640 1 -- 172.21.15.29:0/1320399772 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f14740035f0 con 0x55b5febdc980 2024-09-21T03:00:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.743+0000 7f146cff9640 1 -- 172.21.15.29:0/1320399772 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1474035040 con 0x55b5febdc980 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 -- 172.21.15.29:0/1320399772 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 msgr2=0x55b5fefacb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 --2- 172.21.15.29:0/1320399772 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefacb40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f14740096c0 tx=0x7f147403d000 comp rx=0 tx=0).stop 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 -- 172.21.15.29:0/1320399772 shutdown_connections 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 --2- 172.21.15.29:0/1320399772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 --2- 172.21.15.29:0/1320399772 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefacb40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 --2- 172.21.15.29:0/1320399772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 -- 172.21.15.29:0/1320399772 >> 172.21.15.29:0/1320399772 conn(0x55b5febf9990 msgr2=0x55b5fefaa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 -- 172.21.15.29:0/1320399772 shutdown_connections 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 -- 172.21.15.29:0/1320399772 wait complete. 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.744+0000 7f148afba740 1 -- start start 2024-09-21T03:00:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f148afba740 1 Processor -- start 2024-09-21T03:00:20.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f148afba740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefb81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f148afba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefb8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f148afba740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefc4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f148afba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b5fed737a0 con 0x55b5febe4b60 2024-09-21T03:00:20.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f148afba740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b5feeb70a0 con 0x55b5febdc980 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f148afba740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b5fefbad80 con 0x55b5fec05840 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f148afba740 10 client.0 _ll_register_callbacks cb 0x55b5fee943c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f147dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefb8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f147dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefb8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45830/0 (socket says 172.21.15.29:45830) 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f147dd9b640 1 -- 172.21.15.29:0/12765098 learned_addr learned my addr 172.21.15.29:0/12765098 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.745+0000 7f147e59c640 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefb81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f147ed9d640 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefc4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f147e59c640 1 -- 172.21.15.29:0/12765098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 msgr2=0x55b5fefc4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f147e59c640 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefc4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f147e59c640 1 -- 172.21.15.29:0/12765098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 msgr2=0x55b5fefb8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f147e59c640 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefb8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f147e59c640 1 -- 172.21.15.29:0/12765098 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5feea5880 con 0x55b5febdc980 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f147dd9b640 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefb8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:20.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f147e59c640 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefb81e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f14700acfd0 tx=0x7f14700acaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298c679cdb9bf84d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f1465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f1465ffb640 1 -- 172.21.15.29:0/12765098 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147001d040 con 0x55b5febdc980 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f1465ffb640 1 -- 172.21.15.29:0/12765098 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f14700d4470 con 0x55b5febdc980 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f148afba740 1 -- 172.21.15.29:0/12765098 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14740035f0 con 0x55b5febdc980 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f1465ffb640 1 -- 172.21.15.29:0/12765098 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f14700b1b20 con 0x55b5febdc980 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f148afba740 10 client.27557 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.746+0000 7f148afba740 1 -- 172.21.15.29:0/12765098 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b5fec7fb90 con 0x55b5febdc980 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f1465ffb640 1 -- 172.21.15.29:0/12765098 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f14740035f0 con 0x55b5febdc980 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f1465ffb640 1 -- 172.21.15.29:0/12765098 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f1470098260 con 0x55b5febdc980 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f148afba740 10 client.27557 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f148afba740 10 client.27557 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f148afba740 20 client.27557 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f148afba740 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5febc41c0 0x55b5fefd28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f148afba740 4 client.27557 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:00:20.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f148afba740 1 -- 172.21.15.29:0/12765098 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5fef24ae0 con 0x55b5febc41c0 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.747+0000 7f147dd9b640 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5febc41c0 0x55b5fefd28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.748+0000 7f1465ffb640 10 client.27557 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.748+0000 7f147dd9b640 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5febc41c0 0x55b5fefd28a0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59f41bc7eac8c2ae server_cookie=e0a3a41d2c4bfff1 in_seq=0 out_seq=0 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f1465ffb640 1 -- 172.21.15.29:0/12765098 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b5fef24ae0 con 0x55b5febc41c0 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f1465ffb640 10 client.27557 handle_command_reply: tid=0 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 client.27557 shutdown 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 -- 172.21.15.29:0/12765098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 msgr2=0x55b5fefb81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefb81e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f14700acfd0 tx=0x7f14700acaa0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 -- 172.21.15.29:0/12765098 shutdown_connections 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5fec05840 0x55b5fefc4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5febc41c0 0x55b5fefd28a0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febe4b60 0x55b5fefb8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 --2- 172.21.15.29:0/12765098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febdc980 0x55b5fefb81e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f1465ffb640 0 client.27557 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.749+0000 7f148afba740 1 -- 172.21.15.29:0/12765098 >> 172.21.15.29:0/12765098 conn(0x55b5febf9990 msgr2=0x55b5fefaf790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.750+0000 7f148afba740 1 -- 172.21.15.29:0/12765098 shutdown_connections 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.750+0000 7f148afba740 1 -- 172.21.15.29:0/12765098 wait complete. 2024-09-21T03:00:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.750+0000 7f148afba740 20 client.27557 trim_cache size 0 max 16384 2024-09-21T03:00:20.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f148afba740 1 Processor -- start 2024-09-21T03:00:20.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f148afba740 1 -- start start 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f148afba740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febc41c0 0x55b5fefc3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f148afba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefb51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f148afba740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febe4b60 0x55b5fefb5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f147dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefb51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f147dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefb51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45836/0 (socket says 172.21.15.29:45836) 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f147e59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febc41c0 0x55b5fefc3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f147e59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febc41c0 0x55b5fefc3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:35948/0 (socket says 172.21.15.29:35948) 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f147dd9b640 1 -- 172.21.15.29:0/877604978 learned_addr learned my addr 172.21.15.29:0/877604978 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f148afba740 1 -- 172.21.15.29:0/877604978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14801ae760 con 0x55b5febdc980 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f148afba740 1 -- 172.21.15.29:0/877604978 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f14801b02a0 con 0x55b5febe4b60 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f148afba740 1 -- 172.21.15.29:0/877604978 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f14801aee30 con 0x55b5febc41c0 2024-09-21T03:00:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.780+0000 7f147ed9d640 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febe4b60 0x55b5fefb5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.781+0000 7f147e59c640 1 -- 172.21.15.29:0/877604978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febe4b60 msgr2=0x55b5fefb5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:20.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.781+0000 7f147e59c640 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febe4b60 0x55b5fefb5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.781+0000 7f147e59c640 1 -- 172.21.15.29:0/877604978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 msgr2=0x55b5fefb51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:20.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.781+0000 7f147e59c640 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefb51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.781+0000 7f147e59c640 1 -- 172.21.15.29:0/877604978 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5fefb3870 con 0x55b5febc41c0 2024-09-21T03:00:20.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.781+0000 7f147dd9b640 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefb51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:20.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.781+0000 7f147e59c640 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febc41c0 0x55b5fefc3f70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f14700e6010 tx=0x7f14700a9010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84ba6faf450d2946 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:20.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.781+0000 7f1467fff640 1 -- 172.21.15.29:0/877604978 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147001d040 con 0x55b5febc41c0 2024-09-21T03:00:20.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.782+0000 7f1467fff640 1 -- 172.21.15.29:0/877604978 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f14700054c0 con 0x55b5febc41c0 2024-09-21T03:00:20.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.782+0000 7f1467fff640 1 -- 172.21.15.29:0/877604978 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14700ace10 con 0x55b5febc41c0 2024-09-21T03:00:20.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 -- 172.21.15.29:0/877604978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febc41c0 msgr2=0x55b5fefc3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:20.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febc41c0 0x55b5fefc3f70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f14700e6010 tx=0x7f14700a9010 comp rx=0 tx=0).stop 2024-09-21T03:00:20.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 -- 172.21.15.29:0/877604978 shutdown_connections 2024-09-21T03:00:20.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febe4b60 0x55b5fefb5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefb51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 --2- 172.21.15.29:0/877604978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febc41c0 0x55b5fefc3f70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 -- 172.21.15.29:0/877604978 >> 172.21.15.29:0/877604978 conn(0x55b5febf9990 msgr2=0x55b5fefb1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 -- 172.21.15.29:0/877604978 shutdown_connections 2024-09-21T03:00:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 -- 172.21.15.29:0/877604978 wait complete. 2024-09-21T03:00:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.783+0000 7f148afba740 1 -- start start 2024-09-21T03:00:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f148afba740 1 Processor -- start 2024-09-21T03:00:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f148afba740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febc41c0 0x55b5fefc0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f148afba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefbcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f147e59c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febc41c0 0x55b5fefc0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f148afba740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febe4b60 0x55b5fefbd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f148afba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14801afbd0 con 0x55b5febdc980 2024-09-21T03:00:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f148afba740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f14801af500 con 0x55b5febc41c0 2024-09-21T03:00:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f147e59c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febc41c0 0x55b5fefc0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:55346/0 (socket says 172.21.15.29:55346) 2024-09-21T03:00:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f14801ac740 con 0x55b5febe4b60 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f147e59c640 1 -- 172.21.15.29:0/871985149 learned_addr learned my addr 172.21.15.29:0/871985149 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f147dd9b640 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefbcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.784+0000 7f148afba740 10 client.0 _ll_register_callbacks cb 0x7f1470007f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.785+0000 7f147ed9d640 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febe4b60 0x55b5fefbd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.785+0000 7f147dd9b640 1 -- 172.21.15.29:0/871985149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febe4b60 msgr2=0x55b5fefbd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.785+0000 7f147dd9b640 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febe4b60 0x55b5fefbd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.785+0000 7f147dd9b640 1 -- 172.21.15.29:0/871985149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febc41c0 msgr2=0x55b5fefc0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.785+0000 7f147dd9b640 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febc41c0 0x55b5fefc0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.785+0000 7f147dd9b640 1 -- 172.21.15.29:0/871985149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5fefbdb10 con 0x55b5febdc980 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.785+0000 7f147dd9b640 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefbcd50 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f1474009900 tx=0x7f1474004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d6452cf5565e0f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.785+0000 7f1465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:20.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.786+0000 7f1465ffb640 1 -- 172.21.15.29:0/871985149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1474040040 con 0x55b5febdc980 2024-09-21T03:00:20.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.786+0000 7f1465ffb640 1 -- 172.21.15.29:0/871985149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f14740037f0 con 0x55b5febdc980 2024-09-21T03:00:20.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.786+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14700054c0 con 0x55b5febdc980 2024-09-21T03:00:20.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.787+0000 7f1465ffb640 1 -- 172.21.15.29:0/871985149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f147404c040 con 0x55b5febdc980 2024-09-21T03:00:20.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.787+0000 7f148afba740 10 client.18264 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:20.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.787+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b5ff03e2c0 con 0x55b5febdc980 2024-09-21T03:00:20.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.787+0000 7f1465ffb640 1 -- 172.21.15.29:0/871985149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f14700054c0 con 0x55b5febdc980 2024-09-21T03:00:20.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.788+0000 7f1465ffb640 1 -- 172.21.15.29:0/871985149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f1474064aa0 con 0x55b5febdc980 2024-09-21T03:00:20.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.788+0000 7f148afba740 10 client.18264 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:20.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.788+0000 7f148afba740 10 client.18264 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:00:20.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.788+0000 7f148afba740 20 client.18264 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:20.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.788+0000 7f148afba740 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5fec05840 0x55b5ff040690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:20.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.788+0000 7f148afba740 4 client.18264 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.3, "await_for": 45.0}] 2024-09-21T03:00:20.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.788+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.3, "await_for": 45.0}) -- 0x55b5fef20590 con 0x55b5fec05840 2024-09-21T03:00:20.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.788+0000 7f147e59c640 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5fec05840 0x55b5ff040690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:20.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.789+0000 7f147e59c640 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5fec05840 0x55b5ff040690 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aaf5c8e330e6c30 server_cookie=b63e3a6685e4ef7 in_seq=0 out_seq=0 2024-09-21T03:00:20.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:20.789+0000 7f1465ffb640 10 client.18264 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.305+0000 7f1465ffb640 1 -- 172.21.15.29:0/871985149 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b5fef20590 con 0x55b5fec05840 2024-09-21T03:00:21.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.305+0000 7f1465ffb640 10 client.18264 handle_command_reply: tid=0 2024-09-21T03:00:21.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.305+0000 7f148afba740 1 client.18264 shutdown 2024-09-21T03:00:21.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 msgr2=0x55b5fefbcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:21.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefbcd50 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f1474009900 tx=0x7f1474004350 comp rx=0 tx=0).stop 2024-09-21T03:00:21.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 shutdown_connections 2024-09-21T03:00:21.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5febe4b60 0x55b5fefbd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b5fec05840 0x55b5ff040690 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b5febdc980 0x55b5fefbcd50 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f1465ffb640 0 client.18264 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 --2- 172.21.15.29:0/871985149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b5febc41c0 0x55b5fefc0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 >> 172.21.15.29:0/871985149 conn(0x55b5febf9990 msgr2=0x55b5fefad630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.306+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 shutdown_connections 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.307+0000 7f148afba740 1 -- 172.21.15.29:0/871985149 wait complete. 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.307+0000 7f148afba740 20 client.18264 trim_cache size 0 max 16384 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:00:21.310 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 82, 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "63042810": { 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 82, 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.3, 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:00:21.311 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:21.312 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:00:21.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 -- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f14680b6660 msgr2=0x7f14680b8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:21.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f14680b6660 0x7f14680b8b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1474005940 tx=0x7f14740058d0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 -- 172.21.15.29:0/2381359668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480102a90 msgr2=0x7f1480110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:21.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480102a90 0x7f1480110030 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f147000c890 tx=0x7f147000cd50 comp rx=0 tx=0).stop 2024-09-21T03:00:21.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 -- 172.21.15.29:0/2381359668 shutdown_connections 2024-09-21T03:00:21.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f14680b6660 0x7f14680b8b70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1480108d20 0x7f1480110ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14801083d0 0x7f1480110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 --2- 172.21.15.29:0/2381359668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480102a90 0x7f1480110030 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:21.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 -- 172.21.15.29:0/2381359668 >> 172.21.15.29:0/2381359668 conn(0x7f1480079820 msgr2=0x7f14801065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:21.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 -- 172.21.15.29:0/2381359668 shutdown_connections 2024-09-21T03:00:21.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:21.309+0000 7f1484a46640 1 -- 172.21.15.29:0/2381359668 wait complete. 2024-09-21T03:00:21.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:21 smithi029 ceph-mon[31006]: pgmap v1026: 129 pgs: 129 active+clean; 309 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-09-21T03:00:21.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.74 MB/sec execute 358 sec latency 903.698 ms 2024-09-21T03:00:21.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:21 smithi066 ceph-mon[33436]: pgmap v1026: 129 pgs: 129 active+clean; 309 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-09-21T03:00:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:21 smithi190 ceph-mon[33871]: pgmap v1026: 129 pgs: 129 active+clean; 309 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-09-21T03:00:22.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.72 MB/sec execute 359 sec latency 1903.761 ms 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 -- 172.21.15.29:0/1732531021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c98103fc0 msgr2=0x7f3c9810a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/1732531021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c98103fc0 0x7f3c9810a960 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009c90 tx=0x7f3c9002a860 comp rx=0 tx=0).stop 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 -- 172.21.15.29:0/1732531021 shutdown_connections 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/1732531021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c98103fc0 0x7f3c9810a960 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/1732531021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c98103620 0x7f3c98103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/1732531021 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c981025a0 0x7f3c98102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 -- 172.21.15.29:0/1732531021 >> 172.21.15.29:0/1732531021 conn(0x7f3c980fdee0 msgr2=0x7f3c98100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 -- 172.21.15.29:0/1732531021 shutdown_connections 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.258+0000 7f3c9f27e640 1 -- 172.21.15.29:0/1732531021 wait complete. 2024-09-21T03:00:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9f27e640 1 Processor -- start 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9f27e640 1 -- start start 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9f27e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c981025a0 0x7f3c98199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9f27e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c98103620 0x7f3c9819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9f27e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c98103fc0 0x7f3c9819b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9f27e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9810e0b0 con 0x7f3c981025a0 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9f27e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9810df30 con 0x7f3c98103620 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9f27e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9810e230 con 0x7f3c98103fc0 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9d01a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c981025a0 0x7f3c98199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9d01a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c981025a0 0x7f3c98199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45862/0 (socket says 172.21.15.29:45862) 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9d01a640 1 -- 172.21.15.29:0/4012675359 learned_addr learned my addr 172.21.15.29:0/4012675359 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9c819640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c98103620 0x7f3c9819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c98103fc0 0x7f3c9819b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9d01a640 1 -- 172.21.15.29:0/4012675359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c98103fc0 msgr2=0x7f3c9819b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9d01a640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c98103fc0 0x7f3c9819b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9d01a640 1 -- 172.21.15.29:0/4012675359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c98103620 msgr2=0x7f3c9819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.259+0000 7f3c9d01a640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c98103620 0x7f3c9819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.260+0000 7f3c9d01a640 1 -- 172.21.15.29:0/4012675359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9819b8a0 con 0x7f3c981025a0 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.260+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c98103fc0 0x7f3c9819b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.260+0000 7f3c9d01a640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c981025a0 0x7f3c98199e60 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00e9f0 tx=0x7f3c8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.260+0000 7f3c8a7fc640 1 -- 172.21.15.29:0/4012675359 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c00cd80 con 0x7f3c981025a0 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.260+0000 7f3c8a7fc640 1 -- 172.21.15.29:0/4012675359 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c8c0108c0 con 0x7f3c981025a0 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.260+0000 7f3c9f27e640 1 -- 172.21.15.29:0/4012675359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9819bb90 con 0x7f3c981025a0 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.260+0000 7f3c8a7fc640 1 -- 172.21.15.29:0/4012675359 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c0187c0 con 0x7f3c981025a0 2024-09-21T03:00:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.260+0000 7f3c9f27e640 1 -- 172.21.15.29:0/4012675359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c981a2280 con 0x7f3c981025a0 2024-09-21T03:00:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3ca61ea740 1 Processor -- start 2024-09-21T03:00:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3ca61ea740 1 -- start start 2024-09-21T03:00:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384007ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcae2f0 0x56384007f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c8a7fc640 1 -- 172.21.15.29:0/4012675359 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f3c8c010430 con 0x7f3c981025a0 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x563840080010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56383fff16f0 con 0x56383fcd71b0 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56383ffe0b20 con 0x56383fcae2f0 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56383ffe4930 con 0x56383fcb64d0 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d81b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384007ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d81b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384007ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45870/0 (socket says 172.21.15.29:45870) 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d81b640 1 -- 172.21.15.29:0/1628441818 learned_addr learned my addr 172.21.15.29:0/1628441818 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9c819640 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x563840080010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d01a640 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcae2f0 0x56384007f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d81b640 1 -- 172.21.15.29:0/1628441818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 msgr2=0x563840080010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x563840080010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d81b640 1 -- 172.21.15.29:0/1628441818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcae2f0 msgr2=0x56384007f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcae2f0 0x56384007f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d81b640 1 -- 172.21.15.29:0/1628441818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56383ffa6530 con 0x56383fcd71b0 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.263+0000 7f3c9d01a640 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcae2f0 0x56384007f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.264+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384007ee50 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f3c90031040 tx=0x7f3c900093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2309621f6dece132 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.264+0000 7f3c837fe640 1 -- 172.21.15.29:0/1628441818 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90004a90 con 0x56383fcd71b0 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.264+0000 7f3c837fe640 1 -- 172.21.15.29:0/1628441818 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c9003f070 con 0x56383fcd71b0 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.264+0000 7f3c837fe640 1 -- 172.21.15.29:0/1628441818 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90004050 con 0x56383fcd71b0 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.264+0000 7f3c8a7fc640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3c840b6510 0x7f3c840b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.264+0000 7f3c9d01a640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3c840b6510 0x7f3c840b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.264+0000 7f3c8a7fc640 1 -- 172.21.15.29:0/4012675359 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3c8c0e7aa0 con 0x7f3c981025a0 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1628441818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 msgr2=0x56384007ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3c9d01a640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3c840b6510 0x7f3c840b8a20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00c6a0 tx=0x7f3c8c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384007ee50 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f3c90031040 tx=0x7f3c900093d0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1628441818 shutdown_connections 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x563840080010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcae2f0 0x56384007f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1628441818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384007ee50 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1628441818 >> 172.21.15.29:0/1628441818 conn(0x56383fccb300 msgr2=0x56384007a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1628441818 shutdown_connections 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1628441818 wait complete. 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 -- start start 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.265+0000 7f3ca61ea740 1 Processor -- start 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x5638400885d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcb64d0 0x563840088c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 0x563840089800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56383ff18ab0 con 0x56383fcb64d0 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56383ff86730 con 0x56383fcd71b0 2024-09-21T03:00:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563840080ad0 con 0x56383fcae2f0 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3ca61ea740 10 client.0 _ll_register_callbacks cb 0x56383ffda6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcb64d0 0x563840088c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcb64d0 0x563840088c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45872/0 (socket says 172.21.15.29:45872) 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 -- 172.21.15.29:0/4267653771 learned_addr learned my addr 172.21.15.29:0/4267653771 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x5638400885d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 0x563840089800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 -- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 msgr2=0x5638400885d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x5638400885d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 -- 172.21.15.29:0/4267653771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 msgr2=0x563840089800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 0x563840089800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 -- 172.21.15.29:0/4267653771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56383fd42790 con 0x56383fcb64d0 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x5638400885d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:00:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c9c819640 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcb64d0 0x563840088c90 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400ba60 tx=0x7f3c9400bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfb4a24b7983354 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c80ff9640 1 -- 172.21.15.29:0/4267653771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94005020 con 0x56383fcb64d0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3c80ff9640 1 -- 172.21.15.29:0/4267653771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3c940051c0 con 0x56383fcb64d0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.266+0000 7f3ca61ea740 1 -- 172.21.15.29:0/4267653771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c9003f070 con 0x56383fcb64d0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3c80ff9640 1 -- 172.21.15.29:0/4267653771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3c94024050 con 0x56383fcb64d0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3ca61ea740 10 client.18276 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3ca61ea740 1 -- 172.21.15.29:0/4267653771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56383fdbb620 con 0x56383fcb64d0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3c80ff9640 1 -- 172.21.15.29:0/4267653771 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c9003f070 con 0x56383fcb64d0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3c80ff9640 1 -- 172.21.15.29:0/4267653771 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f3c9401b470 con 0x56383fcb64d0 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3ca61ea740 10 client.18276 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3ca61ea740 10 client.18276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3ca61ea740 20 client.18276 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56383fc95d80 0x563840095da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3ca61ea740 4 client.18276 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:00:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3ca61ea740 1 -- 172.21.15.29:0/4267653771 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56383fff3640 con 0x56383fc95d80 2024-09-21T03:00:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.267+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56383fc95d80 0x563840095da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.268+0000 7f3c80ff9640 10 client.18276 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.268+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56383fc95d80 0x563840095da0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c03afd6c40bedae3 server_cookie=a99c6f7814455bdb in_seq=0 out_seq=0 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3c80ff9640 1 -- 172.21.15.29:0/4267653771 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56383fff3640 con 0x56383fc95d80 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3c80ff9640 10 client.18276 handle_command_reply: tid=0 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 client.18276 shutdown 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 -- 172.21.15.29:0/4267653771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcb64d0 msgr2=0x563840088c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcb64d0 0x563840088c90 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400ba60 tx=0x7f3c9400bf20 comp rx=0 tx=0).stop 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 -- 172.21.15.29:0/4267653771 shutdown_connections 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 0x563840089800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56383fc95d80 0x563840095da0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcb64d0 0x563840088c90 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/4267653771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x5638400885d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 -- 172.21.15.29:0/4267653771 >> 172.21.15.29:0/4267653771 conn(0x56383fccb300 msgr2=0x56384007fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3c80ff9640 0 client.18276 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 -- 172.21.15.29:0/4267653771 shutdown_connections 2024-09-21T03:00:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 1 -- 172.21.15.29:0/4267653771 wait complete. 2024-09-21T03:00:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.269+0000 7f3ca61ea740 20 client.18276 trim_cache size 0 max 16384 2024-09-21T03:00:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3ca61ea740 1 Processor -- start 2024-09-21T03:00:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3ca61ea740 1 -- start start 2024-09-21T03:00:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcae2f0 0x563840083950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x5638400795b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 0x563840080f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3c981ac580 con 0x56383fcae2f0 2024-09-21T03:00:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3c981ae0c0 con 0x56383fcd71b0 2024-09-21T03:00:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3c981acc50 con 0x56383fcb64d0 2024-09-21T03:00:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcae2f0 0x563840083950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcae2f0 0x563840083950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45880/0 (socket says 172.21.15.29:45880) 2024-09-21T03:00:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 -- 172.21.15.29:0/2977459414 learned_addr learned my addr 172.21.15.29:0/2977459414 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9c819640 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x5638400795b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 -- 172.21.15.29:0/2977459414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 msgr2=0x5638400795b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x5638400795b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 -- 172.21.15.29:0/2977459414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 msgr2=0x563840080f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 0x563840080f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 -- 172.21.15.29:0/2977459414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384007f8c0 con 0x56383fcae2f0 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9c819640 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x5638400795b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.284+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcae2f0 0x563840083950 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009440 tx=0x7f3c90003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0509f06d8bb1e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3c82ffd640 1 -- 172.21.15.29:0/2977459414 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90031c80 con 0x56383fcae2f0 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3c82ffd640 1 -- 172.21.15.29:0/2977459414 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c9003f070 con 0x56383fcae2f0 2024-09-21T03:00:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3c82ffd640 1 -- 172.21.15.29:0/2977459414 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90030ba0 con 0x56383fcae2f0 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3ca61ea740 1 -- 172.21.15.29:0/2977459414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcae2f0 msgr2=0x563840083950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcae2f0 0x563840083950 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009440 tx=0x7f3c90003a50 comp rx=0 tx=0).stop 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3ca61ea740 1 -- 172.21.15.29:0/2977459414 shutdown_connections 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcd71b0 0x563840080f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcb64d0 0x5638400795b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/2977459414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcae2f0 0x563840083950 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.285+0000 7f3ca61ea740 1 -- 172.21.15.29:0/2977459414 >> 172.21.15.29:0/2977459414 conn(0x56383fccb300 msgr2=0x563840078ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 -- 172.21.15.29:0/2977459414 shutdown_connections 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 -- 172.21.15.29:0/2977459414 wait complete. 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 -- start start 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 Processor -- start 2024-09-21T03:00:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x56384008d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcb64d0 0x56384008e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384008ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3c981ad9f0 con 0x56383fcd71b0 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3c981ad320 con 0x56383fcb64d0 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3c981aa560 con 0x56383fcae2f0 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3ca61ea740 10 client.0 _ll_register_callbacks cb 0x7f3c940042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3c9d81b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384008ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3c9d81b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384008ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45886/0 (socket says 172.21.15.29:45886) 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3c9d81b640 1 -- 172.21.15.29:0/1226833493 learned_addr learned my addr 172.21.15.29:0/1226833493 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.286+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x56384008d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c9c819640 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcb64d0 0x56384008e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c9d81b640 1 -- 172.21.15.29:0/1226833493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 msgr2=0x56384008d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x56384008d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c9d81b640 1 -- 172.21.15.29:0/1226833493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcb64d0 msgr2=0x56384008e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcb64d0 0x56384008e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c9d81b640 1 -- 172.21.15.29:0/1226833493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56384008f350 con 0x56383fcd71b0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c9d81b640 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384008ebf0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009680 tx=0x7f3c90033040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc26db16855b4a50 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c80ff9640 1 -- 172.21.15.29:0/1226833493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90030920 con 0x56383fcd71b0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c80ff9640 1 -- 172.21.15.29:0/1226833493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f3c90030ac0 con 0x56383fcd71b0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1226833493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c9003f070 con 0x56383fcd71b0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c80ff9640 1 -- 172.21.15.29:0/1226833493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3c90031a80 con 0x56383fcd71b0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3ca61ea740 10 client.18282 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1226833493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5638400905b0 con 0x56383fcd71b0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.287+0000 7f3c80ff9640 1 -- 172.21.15.29:0/1226833493 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c9003f070 con 0x56383fcd71b0 2024-09-21T03:00:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3c80ff9640 1 -- 172.21.15.29:0/1226833493 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f3c90066020 con 0x56383fcd71b0 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3ca61ea740 10 client.18282 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3ca61ea740 10 client.18282 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3ca61ea740 20 client.18282 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56383fc6b1c0 0x563840102770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3ca61ea740 4 client.18282 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1226833493 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56383ffef0f0 con 0x56383fc6b1c0 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3c9c819640 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56383fc6b1c0 0x563840102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3c80ff9640 10 client.18282 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.288+0000 7f3c9c819640 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56383fc6b1c0 0x563840102770 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbdda696ba543a8b server_cookie=924e8666797baebf in_seq=0 out_seq=0 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3c80ff9640 1 -- 172.21.15.29:0/1226833493 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56383ffef0f0 con 0x56383fc6b1c0 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3c80ff9640 10 client.18282 handle_command_reply: tid=0 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 client.18282 shutdown 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1226833493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 msgr2=0x56384008ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384008ebf0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009680 tx=0x7f3c90033040 comp rx=0 tx=0).stop 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1226833493 shutdown_connections 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56383fc6b1c0 0x563840102770 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3c80ff9640 0 client.18282 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56383fcd71b0 0x56384008ebf0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56383fcb64d0 0x56384008e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 --2- 172.21.15.29:0/1226833493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56383fcae2f0 0x56384008d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1226833493 >> 172.21.15.29:0/1226833493 conn(0x56383fccb300 msgr2=0x56384009a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1226833493 shutdown_connections 2024-09-21T03:00:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 1 -- 172.21.15.29:0/1226833493 wait complete. 2024-09-21T03:00:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.424+0000 7f3ca61ea740 20 client.18282 trim_cache size 0 max 16384 2024-09-21T03:00:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:00:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:00:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:00:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:00:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 -- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3c840b6510 msgr2=0x7f3c840b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3c840b6510 0x7f3c840b8a20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00c6a0 tx=0x7f3c8c000bc0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 -- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c981025a0 msgr2=0x7f3c98199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c981025a0 0x7f3c98199e60 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00e9f0 tx=0x7f3c8c00eeb0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 -- 172.21.15.29:0/4012675359 shutdown_connections 2024-09-21T03:00:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3c840b6510 0x7f3c840b8a20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c98103fc0 0x7f3c9819b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c98103620 0x7f3c9819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 --2- 172.21.15.29:0/4012675359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c981025a0 0x7f3c98199e60 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 -- 172.21.15.29:0/4012675359 >> 172.21.15.29:0/4012675359 conn(0x7f3c980fdee0 msgr2=0x7f3c9806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 -- 172.21.15.29:0/4012675359 shutdown_connections 2024-09-21T03:00:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:23.425+0000 7f3c9f27e640 1 -- 172.21.15.29:0/4012675359 wait complete. 2024-09-21T03:00:23.573 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '63042810', quiesce took 7.3 seconds. Will release after: 15.0 2024-09-21T03:00:23.736 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.71 MB/sec execute 360 sec latency 2903.847 ms 2024-09-21T03:00:23.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:23 smithi066 ceph-mon[33436]: pgmap v1027: 129 pgs: 129 active+clean; 310 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-09-21T03:00:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:23 smithi190 ceph-mon[33871]: pgmap v1027: 129 pgs: 129 active+clean; 310 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-09-21T03:00:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:23 smithi029 ceph-mon[31006]: pgmap v1027: 129 pgs: 129 active+clean; 310 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-09-21T03:00:24.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.69 MB/sec execute 361 sec latency 3903.966 ms 2024-09-21T03:00:25.737 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.68 MB/sec execute 362 sec latency 4904.035 ms 2024-09-21T03:00:25.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:25 smithi066 ceph-mon[33436]: pgmap v1028: 129 pgs: 129 active+clean; 311 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 2.8 MiB/s wr, 298 op/s 2024-09-21T03:00:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:25 smithi190 ceph-mon[33871]: pgmap v1028: 129 pgs: 129 active+clean; 311 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 2.8 MiB/s wr, 298 op/s 2024-09-21T03:00:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:25 smithi029 ceph-mon[31006]: pgmap v1028: 129 pgs: 129 active+clean; 311 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 2.8 MiB/s wr, 298 op/s 2024-09-21T03:00:26.737 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.66 MB/sec execute 363 sec latency 5904.155 ms 2024-09-21T03:00:27.737 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.65 MB/sec execute 364 sec latency 6904.310 ms 2024-09-21T03:00:27.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:27 smithi066 ceph-mon[33436]: pgmap v1029: 129 pgs: 129 active+clean; 311 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2024-09-21T03:00:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:27 smithi190 ceph-mon[33871]: pgmap v1029: 129 pgs: 129 active+clean; 311 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2024-09-21T03:00:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:27 smithi029 ceph-mon[31006]: pgmap v1029: 129 pgs: 129 active+clean; 311 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2024-09-21T03:00:28.737 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.63 MB/sec execute 365 sec latency 7904.442 ms 2024-09-21T03:00:29.737 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.62 MB/sec execute 366 sec latency 8904.572 ms 2024-09-21T03:00:29.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:29 smithi066 ceph-mon[33436]: pgmap v1030: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.7 MiB/s wr, 193 op/s 2024-09-21T03:00:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:29 smithi190 ceph-mon[33871]: pgmap v1030: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.7 MiB/s wr, 193 op/s 2024-09-21T03:00:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:29 smithi029 ceph-mon[31006]: pgmap v1030: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.7 MiB/s wr, 193 op/s 2024-09-21T03:00:30.737 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.60 MB/sec execute 367 sec latency 9904.682 ms 2024-09-21T03:00:31.737 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.58 MB/sec execute 368 sec latency 10904.800 ms 2024-09-21T03:00:31.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:31 smithi066 ceph-mon[33436]: pgmap v1031: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-09-21T03:00:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:31 smithi190 ceph-mon[33871]: pgmap v1031: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-09-21T03:00:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:31 smithi029 ceph-mon[31006]: pgmap v1031: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-09-21T03:00:32.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.57 MB/sec execute 369 sec latency 11904.912 ms 2024-09-21T03:00:33.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.55 MB/sec execute 370 sec latency 12905.061 ms 2024-09-21T03:00:33.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:33 smithi066 ceph-mon[33436]: pgmap v1032: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 603 KiB/s wr, 126 op/s 2024-09-21T03:00:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:33 smithi190 ceph-mon[33871]: pgmap v1032: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 603 KiB/s wr, 126 op/s 2024-09-21T03:00:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:33 smithi029 ceph-mon[31006]: pgmap v1032: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 603 KiB/s wr, 126 op/s 2024-09-21T03:00:34.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.54 MB/sec execute 371 sec latency 13905.520 ms 2024-09-21T03:00:35.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.52 MB/sec execute 372 sec latency 14905.596 ms 2024-09-21T03:00:35.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:35 smithi066 ceph-mon[33436]: pgmap v1033: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 353 KiB/s wr, 97 op/s 2024-09-21T03:00:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:35 smithi190 ceph-mon[33871]: pgmap v1033: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 353 KiB/s wr, 97 op/s 2024-09-21T03:00:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:35 smithi029 ceph-mon[31006]: pgmap v1033: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 353 KiB/s wr, 97 op/s 2024-09-21T03:00:36.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 -- 172.21.15.29:0/1388434630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d641025a0 msgr2=0x7f7d64102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/1388434630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d641025a0 0x7f7d64102a10 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7d54009900 tx=0x7f7d5402a410 comp rx=0 tx=0).stop 2024-09-21T03:00:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 -- 172.21.15.29:0/1388434630 shutdown_connections 2024-09-21T03:00:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/1388434630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d64103fc0 0x7f7d6410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/1388434630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d64103620 0x7f7d64103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/1388434630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d641025a0 0x7f7d64102a10 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 -- 172.21.15.29:0/1388434630 >> 172.21.15.29:0/1388434630 conn(0x7f7d640fdee0 msgr2=0x7f7d64100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 -- 172.21.15.29:0/1388434630 shutdown_connections 2024-09-21T03:00:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.526+0000 7f7d68ff1640 1 -- 172.21.15.29:0/1388434630 wait complete. 2024-09-21T03:00:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d68ff1640 1 Processor -- start 2024-09-21T03:00:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d68ff1640 1 -- start start 2024-09-21T03:00:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d68ff1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d641025a0 0x7f7d6406b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d68ff1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d64103620 0x7f7d6419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d68ff1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d64103fc0 0x7f7d641a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d68ff1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6410e080 con 0x7f7d64103fc0 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d68ff1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6410df00 con 0x7f7d64103620 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d68ff1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6410e200 con 0x7f7d641025a0 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d62d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d64103fc0 0x7f7d641a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d62d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d64103fc0 0x7f7d641a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60212/0 (socket says 172.21.15.29:60212) 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d62d9d640 1 -- 172.21.15.29:0/2464116899 learned_addr learned my addr 172.21.15.29:0/2464116899 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d64103620 0x7f7d6419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d6259c640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d641025a0 0x7f7d6406b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d62d9d640 1 -- 172.21.15.29:0/2464116899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d641025a0 msgr2=0x7f7d6406b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d641025a0 0x7f7d6406b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d62d9d640 1 -- 172.21.15.29:0/2464116899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d64103620 msgr2=0x7f7d6419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.527+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d64103620 0x7f7d6419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.528+0000 7f7d62d9d640 1 -- 172.21.15.29:0/2464116899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d641a1950 con 0x7f7d64103fc0 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.528+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d64103fc0 0x7f7d641a1130 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00c970 tx=0x7f7d5c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.528+0000 7f7d537fe640 1 -- 172.21.15.29:0/2464116899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c007bb0 con 0x7f7d64103fc0 2024-09-21T03:00:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.528+0000 7f7d537fe640 1 -- 172.21.15.29:0/2464116899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d5c007d50 con 0x7f7d64103fc0 2024-09-21T03:00:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.528+0000 7f7d68ff1640 1 -- 172.21.15.29:0/2464116899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d641a1c40 con 0x7f7d64103fc0 2024-09-21T03:00:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.528+0000 7f7d537fe640 1 -- 172.21.15.29:0/2464116899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c0055c0 con 0x7f7d64103fc0 2024-09-21T03:00:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.528+0000 7f7d68ff1640 1 -- 172.21.15.29:0/2464116899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d641a2190 con 0x7f7d64103fc0 2024-09-21T03:00:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.530+0000 7f7d7003d740 1 Processor -- start 2024-09-21T03:00:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.532+0000 7f7d537fe640 1 -- 172.21.15.29:0/2464116899 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f7d5c00f450 con 0x7f7d64103fc0 2024-09-21T03:00:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.532+0000 7f7d7003d740 1 -- start start 2024-09-21T03:00:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.532+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 0x55d467d61e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.532+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679912f0 0x55d467d624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.532+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d63010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.532+0000 7f7d7003d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d467cd46f0 con 0x55d4679912f0 2024-09-21T03:00:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.532+0000 7f7d7003d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d467cc3b20 con 0x55d4679994d0 2024-09-21T03:00:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.532+0000 7f7d7003d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d467cc7930 con 0x55d4679ba1b0 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679912f0 0x55d467d624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679912f0 0x55d467d624c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60222/0 (socket says 172.21.15.29:60222) 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 -- 172.21.15.29:0/352842369 learned_addr learned my addr 172.21.15.29:0/352842369 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d63010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 0x55d467d61e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 -- 172.21.15.29:0/352842369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 msgr2=0x55d467d61e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 0x55d467d61e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 -- 172.21.15.29:0/352842369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 msgr2=0x55d467d63010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d63010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 -- 172.21.15.29:0/352842369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d467c89530 con 0x55d4679912f0 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d537fe640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7d4c0b6510 0x7f7d4c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d6259c640 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679912f0 0x55d467d624c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7d540096c0 tx=0x7f7d54031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85892d27cbfe8d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7d4c0b6510 0x7f7d4c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d537fe640 1 -- 172.21.15.29:0/2464116899 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7d5c0e72f0 con 0x7f7d64103fc0 2024-09-21T03:00:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.533+0000 7f7d50ff9640 1 -- 172.21.15.29:0/352842369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d54035030 con 0x55d4679912f0 2024-09-21T03:00:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d50ff9640 1 -- 172.21.15.29:0/352842369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d540044c0 con 0x55d4679912f0 2024-09-21T03:00:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d50ff9640 1 -- 172.21.15.29:0/352842369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d54003b80 con 0x55d4679912f0 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d7003d740 1 -- 172.21.15.29:0/352842369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679912f0 msgr2=0x55d467d624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7d4c0b6510 0x7f7d4c0b8a20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009fd0 tx=0x7f7d580099c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d7003d740 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679912f0 0x55d467d624c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7d540096c0 tx=0x7f7d54031650 comp rx=0 tx=0).stop 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d7003d740 1 -- 172.21.15.29:0/352842369 shutdown_connections 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d7003d740 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d63010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d7003d740 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679912f0 0x55d467d624c0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d7003d740 1 --2- 172.21.15.29:0/352842369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 0x55d467d61e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.534+0000 7f7d7003d740 1 -- 172.21.15.29:0/352842369 >> 172.21.15.29:0/352842369 conn(0x55d4679ae300 msgr2=0x55d467d5d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 -- 172.21.15.29:0/352842369 shutdown_connections 2024-09-21T03:00:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 -- 172.21.15.29:0/352842369 wait complete. 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 -- start start 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 Processor -- start 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d6b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d6bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d6c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d467bfbab0 con 0x55d4679ba1b0 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55d467c69730 con 0x55d4679994d0 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d467d63ad0 con 0x55d4679912f0 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d7003d740 10 client.0 _ll_register_callbacks cb 0x55d467cbd6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d62d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d6c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d62d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d6c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60228/0 (socket says 172.21.15.29:60228) 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.535+0000 7f7d62d9d640 1 -- 172.21.15.29:0/765042308 learned_addr learned my addr 172.21.15.29:0/765042308 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d6bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d6259c640 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d6b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d62d9d640 1 -- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 msgr2=0x55d467d6b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d6b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d62d9d640 1 -- 172.21.15.29:0/765042308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 msgr2=0x55d467d6bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d6bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d62d9d640 1 -- 172.21.15.29:0/765042308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d467a25790 con 0x55d4679ba1b0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d6259c640 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d6b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d6c700 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0ad090 tx=0x7f7d5c0b69d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e5617d77320ddf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d49ffb640 1 -- 172.21.15.29:0/765042308 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c01d040 con 0x55d4679ba1b0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d49ffb640 1 -- 172.21.15.29:0/765042308 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7d5c0abdd0 con 0x55d4679ba1b0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d7003d740 1 -- 172.21.15.29:0/765042308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d467cd6640 con 0x55d4679ba1b0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d49ffb640 1 -- 172.21.15.29:0/765042308 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7d5c0acc10 con 0x55d4679ba1b0 2024-09-21T03:00:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d7003d740 10 client.18294 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.536+0000 7f7d7003d740 1 -- 172.21.15.29:0/765042308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d467a9e620 con 0x55d4679ba1b0 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d49ffb640 1 -- 172.21.15.29:0/765042308 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55d467cd6640 con 0x55d4679ba1b0 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d49ffb640 1 -- 172.21.15.29:0/765042308 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f7d5c0abfd0 con 0x55d4679ba1b0 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d7003d740 10 client.18294 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d7003d740 10 client.18294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d7003d740 20 client.18294 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d7003d740 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d467978d80 0x55d467d78da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d7003d740 4 client.18294 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d7003d740 1 -- 172.21.15.29:0/765042308 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d467cd20f0 con 0x55d467978d80 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.537+0000 7f7d6259c640 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d467978d80 0x55d467d78da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.538+0000 7f7d49ffb640 10 client.18294 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.538+0000 7f7d6259c640 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d467978d80 0x55d467d78da0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddb56f976e903c33 server_cookie=4b31fe7dd1af1c81 in_seq=0 out_seq=0 2024-09-21T03:00:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.538+0000 7f7d49ffb640 1 -- 172.21.15.29:0/765042308 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d467cd20f0 con 0x55d467978d80 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.538+0000 7f7d49ffb640 10 client.18294 handle_command_reply: tid=0 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 client.18294 shutdown 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 -- 172.21.15.29:0/765042308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 msgr2=0x55d467d6c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d6c700 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0ad090 tx=0x7f7d5c0b69d0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 -- 172.21.15.29:0/765042308 shutdown_connections 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d467978d80 0x55d467d78da0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d6c700 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d49ffb640 0 client.18294 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d6bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 --2- 172.21.15.29:0/765042308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d6b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 -- 172.21.15.29:0/765042308 >> 172.21.15.29:0/765042308 conn(0x55d4679ae300 msgr2=0x55d467d5e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 -- 172.21.15.29:0/765042308 shutdown_connections 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 1 -- 172.21.15.29:0/765042308 wait complete. 2024-09-21T03:00:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.539+0000 7f7d7003d740 20 client.18294 trim_cache size 0 max 16384 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.553+0000 7f7d7003d740 1 Processor -- start 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.553+0000 7f7d7003d740 1 -- start start 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679912f0 0x55d467d668a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679994d0 0x55d467d585c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 0x55d467d6a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d7003d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641ac3f0 con 0x55d4679994d0 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d7003d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641adf30 con 0x55d4679912f0 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d7003d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641acac0 con 0x55d4679ba1b0 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679994d0 0x55d467d585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679994d0 0x55d467d585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60230/0 (socket says 172.21.15.29:60230) 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 -- 172.21.15.29:0/1779671134 learned_addr learned my addr 172.21.15.29:0/1779671134 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d6259c640 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679912f0 0x55d467d668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 -- 172.21.15.29:0/1779671134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 msgr2=0x55d467d6a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 0x55d467d6a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 -- 172.21.15.29:0/1779671134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679912f0 msgr2=0x55d467d668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679912f0 0x55d467d668a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 -- 172.21.15.29:0/1779671134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d467d69710 con 0x55d4679994d0 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d6259c640 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679912f0 0x55d467d668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679994d0 0x55d467d585c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f7d58010ad0 tx=0x7f7d58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943db3576c9e0165 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d4bfff640 1 -- 172.21.15.29:0/1779671134 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58023d30 con 0x55d4679994d0 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d4bfff640 1 -- 172.21.15.29:0/1779671134 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d58003e00 con 0x55d4679994d0 2024-09-21T03:00:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.554+0000 7f7d4bfff640 1 -- 172.21.15.29:0/1779671134 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5802c440 con 0x55d4679994d0 2024-09-21T03:00:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 -- 172.21.15.29:0/1779671134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679994d0 msgr2=0x55d467d585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679994d0 0x55d467d585c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f7d58010ad0 tx=0x7f7d58004a40 comp rx=0 tx=0).stop 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 -- 172.21.15.29:0/1779671134 shutdown_connections 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679ba1b0 0x55d467d6a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679994d0 0x55d467d585c0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1779671134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679912f0 0x55d467d668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 -- 172.21.15.29:0/1779671134 >> 172.21.15.29:0/1779671134 conn(0x55d4679ae300 msgr2=0x55d467d69a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 -- 172.21.15.29:0/1779671134 shutdown_connections 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.555+0000 7f7d7003d740 1 -- 172.21.15.29:0/1779671134 wait complete. 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 1 -- start start 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 1 Processor -- start 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d73250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d71f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d72450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641ad860 con 0x55d4679ba1b0 2024-09-21T03:00:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641ad190 con 0x55d4679994d0 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641aa3d0 con 0x55d4679912f0 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d7003d740 10 client.0 _ll_register_callbacks cb 0x7f7d5c0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d62d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d72450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d62d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d72450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60234/0 (socket says 172.21.15.29:60234) 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d62d9d640 1 -- 172.21.15.29:0/1509515248 learned_addr learned my addr 172.21.15.29:0/1509515248 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d6259c640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d73250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d71f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d62d9d640 1 -- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 msgr2=0x55d467d73250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d73250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d62d9d640 1 -- 172.21.15.29:0/1509515248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 msgr2=0x55d467d71f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d71f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d62d9d640 1 -- 172.21.15.29:0/1509515248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d467d72b90 con 0x55d4679ba1b0 2024-09-21T03:00:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d6259c640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d73250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.556+0000 7f7d61d9b640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d71f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d62d9d640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d72450 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0e4ac0 tx=0x7f7d5c0ac000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdd3c9a4df85c7c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d49ffb640 1 -- 172.21.15.29:0/1509515248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c01d030 con 0x55d4679ba1b0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d49ffb640 1 -- 172.21.15.29:0/1509515248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f7d5c011b00 con 0x55d4679ba1b0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 1 -- 172.21.15.29:0/1509515248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d58003e00 con 0x55d4679ba1b0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d49ffb640 1 -- 172.21.15.29:0/1509515248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7d5c0ac170 con 0x55d4679ba1b0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 10 client.18300 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 1 -- 172.21.15.29:0/1509515248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d467d71970 con 0x55d4679ba1b0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d49ffb640 1 -- 172.21.15.29:0/1509515248 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d58003e00 con 0x55d4679ba1b0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d49ffb640 1 -- 172.21.15.29:0/1509515248 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f7d5c011d20 con 0x55d4679ba1b0 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 10 client.18300 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 10 client.18300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:00:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 20 client.18300 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d46794e1c0 0x55d467de5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 4 client.18300 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:00:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.557+0000 7f7d7003d740 1 -- 172.21.15.29:0/1509515248 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d467ccf520 con 0x55d46794e1c0 2024-09-21T03:00:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.558+0000 7f7d6259c640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d46794e1c0 0x55d467de5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.558+0000 7f7d6259c640 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d46794e1c0 0x55d467de5fb0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=362e7a26f1f1ff26 server_cookie=8a5449f39e375452 in_seq=0 out_seq=0 2024-09-21T03:00:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.558+0000 7f7d49ffb640 10 client.18300 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d49ffb640 1 -- 172.21.15.29:0/1509515248 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d467ccf520 con 0x55d46794e1c0 2024-09-21T03:00:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d49ffb640 10 client.18300 handle_command_reply: tid=0 2024-09-21T03:00:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 client.18300 shutdown 2024-09-21T03:00:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 -- 172.21.15.29:0/1509515248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 msgr2=0x55d467d72450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d72450 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0e4ac0 tx=0x7f7d5c0ac000 comp rx=0 tx=0).stop 2024-09-21T03:00:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 -- 172.21.15.29:0/1509515248 shutdown_connections 2024-09-21T03:00:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55d46794e1c0 0x55d467de5fb0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d4679ba1b0 0x55d467d72450 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55d4679994d0 0x55d467d71f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d49ffb640 0 client.18300 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 --2- 172.21.15.29:0/1509515248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4679912f0 0x55d467d73250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 -- 172.21.15.29:0/1509515248 >> 172.21.15.29:0/1509515248 conn(0x55d4679ae300 msgr2=0x55d467d66670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 -- 172.21.15.29:0/1509515248 shutdown_connections 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.587+0000 7f7d7003d740 1 -- 172.21.15.29:0/1509515248 wait complete. 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d7003d740 20 client.18300 trim_cache size 0 max 16384 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:00:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:00:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:00:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 -- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7d4c0b6510 msgr2=0x7f7d4c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7d4c0b6510 0x7f7d4c0b8a20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009fd0 tx=0x7f7d580099c0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 -- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d64103fc0 msgr2=0x7f7d641a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d64103fc0 0x7f7d641a1130 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00c970 tx=0x7f7d5c00ce30 comp rx=0 tx=0).stop 2024-09-21T03:00:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 -- 172.21.15.29:0/2464116899 shutdown_connections 2024-09-21T03:00:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7d4c0b6510 0x7f7d4c0b8a20 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d64103fc0 0x7f7d641a1130 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d64103620 0x7f7d6419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 --2- 172.21.15.29:0/2464116899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d641025a0 0x7f7d6406b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 -- 172.21.15.29:0/2464116899 >> 172.21.15.29:0/2464116899 conn(0x7f7d640fdee0 msgr2=0x7f7d64106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 -- 172.21.15.29:0/2464116899 shutdown_connections 2024-09-21T03:00:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:36.588+0000 7f7d68ff1640 1 -- 172.21.15.29:0/2464116899 wait complete. 2024-09-21T03:00:36.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.51 MB/sec execute 373 sec latency 15905.650 ms 2024-09-21T03:00:37.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.49 MB/sec execute 374 sec latency 16905.734 ms 2024-09-21T03:00:37.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:37 smithi066 ceph-mon[33436]: pgmap v1034: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 41 KiB/s wr, 50 op/s 2024-09-21T03:00:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:37 smithi190 ceph-mon[33871]: pgmap v1034: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 41 KiB/s wr, 50 op/s 2024-09-21T03:00:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:37 smithi029 ceph-mon[31006]: pgmap v1034: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 41 KiB/s wr, 50 op/s 2024-09-21T03:00:38.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:00:38.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.48 MB/sec execute 375 sec latency 17905.789 ms 2024-09-21T03:00:38.851 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:00:39.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 -- 172.21.15.29:0/2517975016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c105300 msgr2=0x7fd72c10d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:39.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 --2- 172.21.15.29:0/2517975016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c105300 0x7fd72c10d850 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd714009900 tx=0x7fd71402a410 comp rx=0 tx=0).stop 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 -- 172.21.15.29:0/2517975016 shutdown_connections 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 --2- 172.21.15.29:0/2517975016 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd72c10de10 0x7fd72c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 --2- 172.21.15.29:0/2517975016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c105300 0x7fd72c10d850 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 --2- 172.21.15.29:0/2517975016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c1049b0 0x7fd72c104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 -- 172.21.15.29:0/2517975016 >> 172.21.15.29:0/2517975016 conn(0x7fd72c079730 msgr2=0x7fd72c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 -- 172.21.15.29:0/2517975016 shutdown_connections 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.366+0000 7fd731017640 1 -- 172.21.15.29:0/2517975016 wait complete. 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.367+0000 7fd731017640 1 Processor -- start 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.367+0000 7fd731017640 1 -- start start 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd731017640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd72c1049b0 0x7fd72c19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd731017640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c105300 0x7fd72c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd731017640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c10de10 0x7fd72c19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd731017640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c113980 con 0x7fd72c10de10 2024-09-21T03:00:39.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd731017640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c113800 con 0x7fd72c1049b0 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd731017640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c113b00 con 0x7fd72c105300 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c10de10 0x7fd72c19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c10de10 0x7fd72c19fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60250/0 (socket says 172.21.15.29:60250) 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 -- 172.21.15.29:0/3624245284 learned_addr learned my addr 172.21.15.29:0/3624245284 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72ad9d640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd72c1049b0 0x7fd72c19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72a59c640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c105300 0x7fd72c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 -- 172.21.15.29:0/3624245284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c105300 msgr2=0x7fd72c19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c105300 0x7fd72c19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 -- 172.21.15.29:0/3624245284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd72c1049b0 msgr2=0x7fd72c19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd72c1049b0 0x7fd72c19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 -- 172.21.15.29:0/3624245284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c1a03c0 con 0x7fd72c10de10 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72ad9d640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd72c1049b0 0x7fd72c19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.368+0000 7fd72b59e640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c10de10 0x7fd72c19fc90 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00c970 tx=0x7fd71c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.369+0000 7fd723fff640 1 -- 172.21.15.29:0/3624245284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c007bb0 con 0x7fd72c10de10 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.369+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c1a06b0 con 0x7fd72c10de10 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.369+0000 7fd723fff640 1 -- 172.21.15.29:0/3624245284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd71c007d50 con 0x7fd72c10de10 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.369+0000 7fd723fff640 1 -- 172.21.15.29:0/3624245284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c004d50 con 0x7fd72c10de10 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.369+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c1a6e10 con 0x7fd72c10de10 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.370+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72c104dc0 con 0x7fd72c10de10 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.371+0000 7fd723fff640 1 -- 172.21.15.29:0/3624245284 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd71c005000 con 0x7fd72c10de10 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.372+0000 7fd723fff640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7180b6510 0x7fd7180b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.372+0000 7fd72ad9d640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7180b6510 0x7fd7180b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.372+0000 7fd723fff640 1 -- 172.21.15.29:0/3624245284 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fd71c0e7810 con 0x7fd72c10de10 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.373+0000 7fd72ad9d640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7180b6510 0x7fd7180b8a20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd72c06bc60 tx=0x7fd724009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.375+0000 7fd723fff640 1 -- 172.21.15.29:0/3624245284 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd71c014030 con 0x7fd72c10de10 2024-09-21T03:00:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.605+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd72c19b580 con 0x7fd72c10de10 2024-09-21T03:00:39.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:39 smithi029 ceph-mon[31006]: pgmap v1035: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 35 op/s 2024-09-21T03:00:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.606+0000 7fd723fff640 1 -- 172.21.15.29:0/3624245284 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7fd72c19b580 con 0x7fd72c10de10 2024-09-21T03:00:39.608 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:00:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:00:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7180b6510 msgr2=0x7fd7180b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7180b6510 0x7fd7180b8a20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd72c06bc60 tx=0x7fd724009570 comp rx=0 tx=0).stop 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c10de10 msgr2=0x7fd72c19fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c10de10 0x7fd72c19fc90 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00c970 tx=0x7fd71c00ce30 comp rx=0 tx=0).stop 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 shutdown_connections 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7180b6510 0x7fd7180b8a20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd72c10de10 0x7fd72c19fc90 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c105300 0x7fd72c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 --2- 172.21.15.29:0/3624245284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd72c1049b0 0x7fd72c19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.608+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 >> 172.21.15.29:0/3624245284 conn(0x7fd72c079730 msgr2=0x7fd72c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.609+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 shutdown_connections 2024-09-21T03:00:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:39.609+0000 7fd731017640 1 -- 172.21.15.29:0/3624245284 wait complete. 2024-09-21T03:00:39.738 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.47 MB/sec execute 376 sec latency 18905.891 ms 2024-09-21T03:00:39.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:39 smithi066 ceph-mon[33436]: pgmap v1035: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 35 op/s 2024-09-21T03:00:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:39 smithi190 ceph-mon[33871]: pgmap v1035: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 35 op/s 2024-09-21T03:00:40.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.45 MB/sec execute 377 sec latency 19905.992 ms 2024-09-21T03:00:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:40 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3624245284' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:00:40.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:40 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3624245284' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:00:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:40 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3624245284' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:00:41.370 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 63042810 --release --await' 2024-09-21T03:00:41.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 63042810 --release --await 2024-09-21T03:00:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:41 smithi029 ceph-mon[31006]: pgmap v1036: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-21T03:00:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:00:41.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51104 5.44 MB/sec execute 378 sec latency 20906.124 ms 2024-09-21T03:00:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:41 smithi190 ceph-mon[33871]: pgmap v1036: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-21T03:00:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:41 smithi066 ceph-mon[33436]: pgmap v1036: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-21T03:00:42.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.200+0000 7f02e0263640 1 -- 172.21.15.29:0/2763014641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d81083a0 msgr2=0x7f02d81087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.200+0000 7f02e0263640 1 --2- 172.21.15.29:0/2763014641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d81083a0 0x7f02d81087b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009900 tx=0x7f02cc02a3f0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.201+0000 7f02e0263640 1 -- 172.21.15.29:0/2763014641 shutdown_connections 2024-09-21T03:00:42.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.201+0000 7f02e0263640 1 --2- 172.21.15.29:0/2763014641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8108cf0 0x7f02d810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.201+0000 7f02e0263640 1 --2- 172.21.15.29:0/2763014641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d81083a0 0x7f02d81087b0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.201+0000 7f02e0263640 1 --2- 172.21.15.29:0/2763014641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02d8102a60 0x7f02d8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.201+0000 7f02e0263640 1 -- 172.21.15.29:0/2763014641 >> 172.21.15.29:0/2763014641 conn(0x7f02d8079730 msgr2=0x7f02d8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:42.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.201+0000 7f02e0263640 1 -- 172.21.15.29:0/2763014641 shutdown_connections 2024-09-21T03:00:42.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.201+0000 7f02e0263640 1 -- 172.21.15.29:0/2763014641 wait complete. 2024-09-21T03:00:42.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.201+0000 7f02e0263640 1 Processor -- start 2024-09-21T03:00:42.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02e0263640 1 -- start start 2024-09-21T03:00:42.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02e0263640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d8102a60 0x7f02d80ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02e0263640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02d81083a0 0x7f02d80ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02e0263640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8108cf0 0x7f02d81000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02e0263640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f02d8112810 con 0x7f02d8102a60 2024-09-21T03:00:42.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02e0263640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f02d8112690 con 0x7f02d81083a0 2024-09-21T03:00:42.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02e0263640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02d8112990 con 0x7f02d8108cf0 2024-09-21T03:00:42.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02ddfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d8102a60 0x7f02d80ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02ddfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d8102a60 0x7f02d80ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60264/0 (socket says 172.21.15.29:60264) 2024-09-21T03:00:42.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02ddfff640 1 -- 172.21.15.29:0/3668587624 learned_addr learned my addr 172.21.15.29:0/3668587624 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:42.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02de800640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8108cf0 0x7f02d81000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02d81083a0 0x7f02d80ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02ddfff640 1 -- 172.21.15.29:0/3668587624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8108cf0 msgr2=0x7f02d81000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02ddfff640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8108cf0 0x7f02d81000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02ddfff640 1 -- 172.21.15.29:0/3668587624 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02d81083a0 msgr2=0x7f02d80ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02ddfff640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02d81083a0 0x7f02d80ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02ddfff640 1 -- 172.21.15.29:0/3668587624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d81a8640 con 0x7f02d8102a60 2024-09-21T03:00:42.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.202+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02d81083a0 0x7f02d80ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:42.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.203+0000 7f02ddfff640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d8102a60 0x7f02d80ff670 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f02c800c920 tx=0x7f02c800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:42.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.203+0000 7f02d6ffd640 1 -- 172.21.15.29:0/3668587624 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c8007c70 con 0x7f02d8102a60 2024-09-21T03:00:42.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.203+0000 7f02e0263640 1 -- 172.21.15.29:0/3668587624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d81a8930 con 0x7f02d8102a60 2024-09-21T03:00:42.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.203+0000 7f02d6ffd640 1 -- 172.21.15.29:0/3668587624 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02c8007e10 con 0x7f02d8102a60 2024-09-21T03:00:42.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.203+0000 7f02d6ffd640 1 -- 172.21.15.29:0/3668587624 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c80186f0 con 0x7f02d8102a60 2024-09-21T03:00:42.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.203+0000 7f02e0263640 1 -- 172.21.15.29:0/3668587624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d81a8ec0 con 0x7f02d8102a60 2024-09-21T03:00:42.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02e67d7740 1 Processor -- start 2024-09-21T03:00:42.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02e67d7740 1 -- start start 2024-09-21T03:00:42.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02d6ffd640 1 -- 172.21.15.29:0/3668587624 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f02c800f450 con 0x7f02d8102a60 2024-09-21T03:00:42.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bcaa520 0x560f1c050510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c050a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc89840 0x560f1c050f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02e67d7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560f1bf3bdc0 con 0x560f1bc81660 2024-09-21T03:00:42.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02e67d7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560f1bfc81f0 con 0x560f1bc89840 2024-09-21T03:00:42.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02e67d7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560f1bfb7620 con 0x560f1bcaa520 2024-09-21T03:00:42.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02ddfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c050a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02ddfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c050a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60270/0 (socket says 172.21.15.29:60270) 2024-09-21T03:00:42.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.206+0000 7f02ddfff640 1 -- 172.21.15.29:0/694511485 learned_addr learned my addr 172.21.15.29:0/694511485 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:42.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02de800640 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bcaa520 0x560f1c050510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc89840 0x560f1c050f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02ddfff640 1 -- 172.21.15.29:0/694511485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bcaa520 msgr2=0x560f1c050510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:42.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02ddfff640 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bcaa520 0x560f1c050510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02ddfff640 1 -- 172.21.15.29:0/694511485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc89840 msgr2=0x560f1c050f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02ddfff640 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc89840 0x560f1c050f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02ddfff640 1 -- 172.21.15.29:0/694511485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f1bf7ffd0 con 0x560f1bc81660 2024-09-21T03:00:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc89840 0x560f1c050f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:00:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02d6ffd640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02c00b6510 0x7f02c00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02de800640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02c00b6510 0x7f02c00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02ddfff640 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c050a50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f02c80edc10 tx=0x7f02c80edd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbbf230fb468812 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:42.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02d6ffd640 1 -- 172.21.15.29:0/3668587624 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f02c80e7850 con 0x7f02d8102a60 2024-09-21T03:00:42.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02bffff640 1 -- 172.21.15.29:0/694511485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c80a9cb0 con 0x560f1bc81660 2024-09-21T03:00:42.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02bffff640 1 -- 172.21.15.29:0/694511485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02c80ad170 con 0x560f1bc81660 2024-09-21T03:00:42.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.207+0000 7f02bffff640 1 -- 172.21.15.29:0/694511485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c80acdd0 con 0x560f1bc81660 2024-09-21T03:00:42.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 -- 172.21.15.29:0/694511485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 msgr2=0x560f1c050a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c050a50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f02c80edc10 tx=0x7f02c80edd00 comp rx=0 tx=0).stop 2024-09-21T03:00:42.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02de800640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02c00b6510 0x7f02c00b8a20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f02d0005fd0 tx=0x7f02d0009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:42.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 -- 172.21.15.29:0/694511485 shutdown_connections 2024-09-21T03:00:42.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc89840 0x560f1c050f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c050a50 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 --2- 172.21.15.29:0/694511485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bcaa520 0x560f1c050510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 -- 172.21.15.29:0/694511485 >> 172.21.15.29:0/694511485 conn(0x560f1bc9e910 msgr2=0x560f1c04e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:42.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 -- 172.21.15.29:0/694511485 shutdown_connections 2024-09-21T03:00:42.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.208+0000 7f02e67d7740 1 -- 172.21.15.29:0/694511485 wait complete. 2024-09-21T03:00:42.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 1 -- start start 2024-09-21T03:00:42.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 1 Processor -- start 2024-09-21T03:00:42.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 0x560f1c05bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560f1bf52300 con 0x560f1bcaa520 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x560f1bfbb430 con 0x560f1bc81660 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560f1be18bb0 con 0x560f1bc89840 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.209+0000 7f02e67d7740 10 client.0 _ll_register_callbacks cb 0x560f1bf6c120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60284/0 (socket says 172.21.15.29:60284) 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 -- 172.21.15.29:0/3989904475 learned_addr learned my addr 172.21.15.29:0/3989904475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02ddfff640 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 0x560f1c05bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 -- 172.21.15.29:0/3989904475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 msgr2=0x560f1c05c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 -- 172.21.15.29:0/3989904475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 msgr2=0x560f1c05bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 0x560f1c05bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 -- 172.21.15.29:0/3989904475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f1bf41780 con 0x560f1bcaa520 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02de800640 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05c600 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f02d0011d40 tx=0x7f02d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18286aea2a6541b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02bd7fa640 1 -- 172.21.15.29:0/3989904475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d0004b20 con 0x560f1bcaa520 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02bd7fa640 1 -- 172.21.15.29:0/3989904475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f02d0004cc0 con 0x560f1bcaa520 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.210+0000 7f02e67d7740 1 -- 172.21.15.29:0/3989904475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c80ad170 con 0x560f1bcaa520 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02bd7fa640 1 -- 172.21.15.29:0/3989904475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f02d0005020 con 0x560f1bcaa520 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02e67d7740 10 client.18318 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02e67d7740 1 -- 172.21.15.29:0/3989904475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560f1bd15b00 con 0x560f1bcaa520 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02bd7fa640 1 -- 172.21.15.29:0/3989904475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02c80ad170 con 0x560f1bcaa520 2024-09-21T03:00:42.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02bd7fa640 1 -- 172.21.15.29:0/3989904475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f02d000f040 con 0x560f1bcaa520 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02e67d7740 10 client.18318 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02e67d7740 10 client.18318 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02e67d7740 20 client.18318 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560f1bc68ea0 0x560f1c06df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02e67d7740 4 client.18318 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02e67d7740 1 -- 172.21.15.29:0/3989904475 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f1bfca140 con 0x560f1bc68ea0 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.211+0000 7f02ddfff640 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560f1bc68ea0 0x560f1c06df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.212+0000 7f02ddfff640 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560f1bc68ea0 0x560f1c06df80 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b0253ffdfcd826 server_cookie=cb3b5d201fd3ac34 in_seq=0 out_seq=0 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.212+0000 7f02bd7fa640 10 client.18318 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02bd7fa640 1 -- 172.21.15.29:0/3989904475 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560f1bfca140 con 0x560f1bc68ea0 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02bd7fa640 10 client.18318 handle_command_reply: tid=0 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 client.18318 shutdown 2024-09-21T03:00:42.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 -- 172.21.15.29:0/3989904475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 msgr2=0x560f1c05c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05c600 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f02d0011d40 tx=0x7f02d0004a40 comp rx=0 tx=0).stop 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 -- 172.21.15.29:0/3989904475 shutdown_connections 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560f1bc68ea0 0x560f1c06df80 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05c600 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3989904475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 0x560f1c05bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 -- 172.21.15.29:0/3989904475 >> 172.21.15.29:0/3989904475 conn(0x560f1bc9e910 msgr2=0x560f1c0536a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02bd7fa640 0 client.18318 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 -- 172.21.15.29:0/3989904475 shutdown_connections 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 1 -- 172.21.15.29:0/3989904475 wait complete. 2024-09-21T03:00:42.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.213+0000 7f02e67d7740 20 client.18318 trim_cache size 0 max 16384 2024-09-21T03:00:42.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02e67d7740 1 Processor -- start 2024-09-21T03:00:42.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02e67d7740 1 -- start start 2024-09-21T03:00:42.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 0x560f1c051bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02e67d7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f02d81b2c60 con 0x560f1bcaa520 2024-09-21T03:00:42.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02e67d7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f02d81b47a0 con 0x560f1bc81660 2024-09-21T03:00:42.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02e67d7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02d81b3330 con 0x560f1bc89840 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02de800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02de800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60298/0 (socket says 172.21.15.29:60298) 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02de800640 1 -- 172.21.15.29:0/3826234785 learned_addr learned my addr 172.21.15.29:0/3826234785 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.231+0000 7f02ddfff640 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 0x560f1c051bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02de800640 1 -- 172.21.15.29:0/3826234785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 msgr2=0x560f1c05c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02de800640 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02de800640 1 -- 172.21.15.29:0/3826234785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 msgr2=0x560f1c051bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02de800640 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 0x560f1c051bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02de800640 1 -- 172.21.15.29:0/3826234785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f1bd8c960 con 0x560f1bcaa520 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02de800640 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05d790 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f02d0020780 tx=0x7f02d0005c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2adaddb162b69911 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02bf7fe640 1 -- 172.21.15.29:0/3826234785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d003b070 con 0x560f1bcaa520 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02bf7fe640 1 -- 172.21.15.29:0/3826234785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02d0020420 con 0x560f1bcaa520 2024-09-21T03:00:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.232+0000 7f02bf7fe640 1 -- 172.21.15.29:0/3826234785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d0004040 con 0x560f1bcaa520 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 -- 172.21.15.29:0/3826234785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 msgr2=0x560f1c05d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05d790 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f02d0020780 tx=0x7f02d0005c10 comp rx=0 tx=0).stop 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 -- 172.21.15.29:0/3826234785 shutdown_connections 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bcaa520 0x560f1c05d790 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c05c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 --2- 172.21.15.29:0/3826234785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bc81660 0x560f1c051bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 -- 172.21.15.29:0/3826234785 >> 172.21.15.29:0/3826234785 conn(0x560f1bc9e910 msgr2=0x560f1c05d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 -- 172.21.15.29:0/3826234785 shutdown_connections 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 -- 172.21.15.29:0/3826234785 wait complete. 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 -- start start 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.233+0000 7f02e67d7740 1 Processor -- start 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c062510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c062a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02e67d7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bcaa520 0x560f1c0d7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02e67d7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f02d81b40d0 con 0x560f1bc81660 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02e67d7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f02d81b3a00 con 0x560f1bcaa520 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02e67d7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02d81b0c40 con 0x560f1bc89840 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02e67d7740 10 client.0 _ll_register_callbacks cb 0x7f02d0009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02ddfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c062510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02ddfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c062510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60300/0 (socket says 172.21.15.29:60300) 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02ddfff640 1 -- 172.21.15.29:0/2932226986 learned_addr learned my addr 172.21.15.29:0/2932226986 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02de800640 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bcaa520 0x560f1c0d7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02ddfff640 1 -- 172.21.15.29:0/2932226986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 msgr2=0x560f1c062a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02ddfff640 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02ddfff640 1 -- 172.21.15.29:0/2932226986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bcaa520 msgr2=0x560f1c0d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02ddfff640 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bcaa520 0x560f1c0d7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.234+0000 7f02ddfff640 1 -- 172.21.15.29:0/2932226986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f1c0d7e90 con 0x560f1bc81660 2024-09-21T03:00:42.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02ddfff640 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c062510 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f02c800fd80 tx=0x7f02c800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e60ec7dc136075 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02bd7fa640 1 -- 172.21.15.29:0/2932226986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c80acfb0 con 0x560f1bc81660 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02bd7fa640 1 -- 172.21.15.29:0/2932226986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f02c80ed730 con 0x560f1bc81660 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02e67d7740 1 -- 172.21.15.29:0/2932226986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02d0020420 con 0x560f1bc81660 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02bd7fa640 1 -- 172.21.15.29:0/2932226986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f02c80ad430 con 0x560f1bc81660 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02e67d7740 10 client.18324 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02e67d7740 1 -- 172.21.15.29:0/2932226986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560f1c0d90a0 con 0x560f1bc81660 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02bd7fa640 1 -- 172.21.15.29:0/2932226986 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02d0020420 con 0x560f1bc81660 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02bd7fa640 1 -- 172.21.15.29:0/2932226986 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f02c80b6bf0 con 0x560f1bc81660 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02e67d7740 10 client.18324 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02e67d7740 10 client.18324 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02e67d7740 20 client.18324 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.235+0000 7f02e67d7740 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560f1bc3e250 0x560f1c0db4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.236+0000 7f02e67d7740 4 client.18324 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "63042810", "release": true, "await": true}] 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.236+0000 7f02e67d7740 1 -- 172.21.15.29:0/2932226986 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "63042810", "release": true, "await": true}) -- 0x560f1bfc5bf0 con 0x560f1bc3e250 2024-09-21T03:00:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.236+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560f1bc3e250 0x560f1c0db4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:42.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.236+0000 7f02dd7fe640 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560f1bc3e250 0x560f1c0db4b0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3f0513fdd0eb6cc server_cookie=76e9f786194ce0f4 in_seq=0 out_seq=0 2024-09-21T03:00:42.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.236+0000 7f02bd7fa640 10 client.18324 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.298+0000 7f02bd7fa640 1 -- 172.21.15.29:0/2932226986 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560f1bfc5bf0 con 0x560f1bc3e250 2024-09-21T03:00:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.298+0000 7f02bd7fa640 10 client.18324 handle_command_reply: tid=0 2024-09-21T03:00:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.298+0000 7f02e67d7740 1 client.18324 shutdown 2024-09-21T03:00:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.298+0000 7f02e67d7740 1 -- 172.21.15.29:0/2932226986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 msgr2=0x560f1c062510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.298+0000 7f02e67d7740 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c062510 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f02c800fd80 tx=0x7f02c800cf10 comp rx=0 tx=0).stop 2024-09-21T03:00:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 1 -- 172.21.15.29:0/2932226986 shutdown_connections 2024-09-21T03:00:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x560f1bcaa520 0x560f1c0d7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1bc89840 0x560f1c062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x560f1bc3e250 0x560f1c0db4b0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 1 --2- 172.21.15.29:0/2932226986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f1bc81660 0x560f1c062510 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 1 -- 172.21.15.29:0/2932226986 >> 172.21.15.29:0/2932226986 conn(0x560f1bc9e910 msgr2=0x560f1c05d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02bd7fa640 0 client.18324 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 1 -- 172.21.15.29:0/2932226986 shutdown_connections 2024-09-21T03:00:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 1 -- 172.21.15.29:0/2932226986 wait complete. 2024-09-21T03:00:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.299+0000 7f02e67d7740 20 client.18324 trim_cache size 0 max 16384 2024-09-21T03:00:42.302 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 84, 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "63042810": { 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 84, 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.3, 2024-09-21T03:00:42.303 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 -- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02c00b6510 msgr2=0x7f02c00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02c00b6510 0x7f02c00b8a20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f02d0005fd0 tx=0x7f02d0009600 comp rx=0 tx=0).stop 2024-09-21T03:00:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 -- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d8102a60 msgr2=0x7f02d80ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d8102a60 0x7f02d80ff670 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f02c800c920 tx=0x7f02c800cde0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 -- 172.21.15.29:0/3668587624 shutdown_connections 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f02c00b6510 0x7f02c00b8a20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8108cf0 0x7f02d81000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02d81083a0 0x7f02d80ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 --2- 172.21.15.29:0/3668587624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d8102a60 0x7f02d80ff670 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 -- 172.21.15.29:0/3668587624 >> 172.21.15.29:0/3668587624 conn(0x7f02d8079730 msgr2=0x7f02d806b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 -- 172.21.15.29:0/3668587624 shutdown_connections 2024-09-21T03:00:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:00:42.301+0000 7f02e0263640 1 -- 172.21.15.29:0/3668587624 wait complete. 2024-09-21T03:00:42.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51183 5.44 MB/sec execute 379 sec latency 21488.063 ms 2024-09-21T03:00:43.163 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '63042810', total seconds elapsed: 26.9 2024-09-21T03:00:43.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51294 5.43 MB/sec execute 380 sec latency 76.757 ms 2024-09-21T03:00:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:43 smithi190 ceph-mon[33871]: pgmap v1037: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-21T03:00:43.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:43 smithi029 ceph-mon[31006]: pgmap v1037: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-21T03:00:44.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:43 smithi066 ceph-mon[33436]: pgmap v1037: 129 pgs: 129 active+clean; 307 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-21T03:00:44.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51429 5.43 MB/sec execute 381 sec latency 96.560 ms 2024-09-21T03:00:45.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51574 5.42 MB/sec execute 382 sec latency 82.875 ms 2024-09-21T03:00:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:45 smithi190 ceph-mon[33871]: pgmap v1038: 129 pgs: 129 active+clean; 308 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 265 KiB/s wr, 33 op/s 2024-09-21T03:00:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:45 smithi029 ceph-mon[31006]: pgmap v1038: 129 pgs: 129 active+clean; 308 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 265 KiB/s wr, 33 op/s 2024-09-21T03:00:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:45 smithi066 ceph-mon[33436]: pgmap v1038: 129 pgs: 129 active+clean; 308 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 265 KiB/s wr, 33 op/s 2024-09-21T03:00:46.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51778 5.46 MB/sec execute 383 sec latency 233.530 ms 2024-09-21T03:00:47.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51975 5.47 MB/sec execute 384 sec latency 95.009 ms 2024-09-21T03:00:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:47 smithi190 ceph-mon[33871]: pgmap v1039: 129 pgs: 129 active+clean; 309 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 576 KiB/s wr, 42 op/s 2024-09-21T03:00:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:47 smithi029 ceph-mon[31006]: pgmap v1039: 129 pgs: 129 active+clean; 309 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 576 KiB/s wr, 42 op/s 2024-09-21T03:00:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:47 smithi066 ceph-mon[33436]: pgmap v1039: 129 pgs: 129 active+clean; 309 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 576 KiB/s wr, 42 op/s 2024-09-21T03:00:48.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52089 5.47 MB/sec execute 385 sec latency 232.233 ms 2024-09-21T03:00:48.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:00:48.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:00:48.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:00:48.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:00:48.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:00:48.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:00:48.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:00:48.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:00:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:00:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:00:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:00:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:00:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.726+0000 7fdb39f2b640 1 -- 172.21.15.29:0/3304159529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 msgr2=0x7fdb34102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.726+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/3304159529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 0x7fdb34102a10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009900 tx=0x7fdb2402a410 comp rx=0 tx=0).stop 2024-09-21T03:00:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.726+0000 7fdb39f2b640 1 -- 172.21.15.29:0/3304159529 shutdown_connections 2024-09-21T03:00:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.726+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/3304159529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb34103fc0 0x7fdb3410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.726+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/3304159529 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb34103620 0x7fdb34103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.726+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/3304159529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 0x7fdb34102a10 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.726+0000 7fdb39f2b640 1 -- 172.21.15.29:0/3304159529 >> 172.21.15.29:0/3304159529 conn(0x7fdb340fdee0 msgr2=0x7fdb34100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:49.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.727+0000 7fdb39f2b640 1 -- 172.21.15.29:0/3304159529 shutdown_connections 2024-09-21T03:00:49.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.727+0000 7fdb39f2b640 1 -- 172.21.15.29:0/3304159529 wait complete. 2024-09-21T03:00:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.727+0000 7fdb39f2b640 1 Processor -- start 2024-09-21T03:00:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb39f2b640 1 -- start start 2024-09-21T03:00:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb39f2b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 0x7fdb34198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb39f2b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb34103620 0x7fdb34199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 0x7fdb34198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb39f2b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb34103fc0 0x7fdb34199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 0x7fdb34198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54844/0 (socket says 172.21.15.29:54844) 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb337fe640 1 -- 172.21.15.29:0/343069970 learned_addr learned my addr 172.21.15.29:0/343069970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3410e080 con 0x7fdb341025a0 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3410df00 con 0x7fdb34103fc0 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb34103620 0x7fdb34199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.728+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3410e200 con 0x7fdb34103620 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb33fff640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb34103fc0 0x7fdb34199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb32ffd640 1 -- 172.21.15.29:0/343069970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb34103fc0 msgr2=0x7fdb34199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb34103fc0 0x7fdb34199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb32ffd640 1 -- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 msgr2=0x7fdb34198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 0x7fdb34198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb32ffd640 1 -- 172.21.15.29:0/343069970 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3419a800 con 0x7fdb34103620 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb33fff640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb34103fc0 0x7fdb34199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:00:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb34103620 0x7fdb34199430 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800e9f0 tx=0x7fdb2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb30ff9640 1 -- 172.21.15.29:0/343069970 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800ccc0 con 0x7fdb34103620 2024-09-21T03:00:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb30ff9640 1 -- 172.21.15.29:0/343069970 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb280108c0 con 0x7fdb34103620 2024-09-21T03:00:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3419aaf0 con 0x7fdb34103620 2024-09-21T03:00:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb30ff9640 1 -- 172.21.15.29:0/343069970 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb280187e0 con 0x7fdb34103620 2024-09-21T03:00:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.729+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb3406adc0 con 0x7fdb34103620 2024-09-21T03:00:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb40e9b740 1 Processor -- start 2024-09-21T03:00:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb40e9b740 1 -- start start 2024-09-21T03:00:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a37e1b0 0x55e06a725e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a7264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a727010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e06a6986f0 con 0x55e06a37e1b0 2024-09-21T03:00:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e06a687b20 con 0x55e06a3552f0 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e06a68b930 con 0x55e06a35d4d0 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a37e1b0 0x55e06a725e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a37e1b0 0x55e06a725e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54852/0 (socket says 172.21.15.29:54852) 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb33fff640 1 -- 172.21.15.29:0/3425257937 learned_addr learned my addr 172.21.15.29:0/3425257937 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb30ff9640 1 -- 172.21.15.29:0/343069970 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fdb28018980 con 0x7fdb34103620 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.733+0000 7fdb337fe640 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a7264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a727010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb33fff640 1 -- 172.21.15.29:0/3425257937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 msgr2=0x55e06a727010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb33fff640 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a727010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb33fff640 1 -- 172.21.15.29:0/3425257937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 msgr2=0x55e06a7264c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb33fff640 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a7264c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb33fff640 1 -- 172.21.15.29:0/3425257937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06a64d530 con 0x55e06a37e1b0 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb337fe640 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a7264c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a727010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:00:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb33fff640 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a37e1b0 0x55e06a725e50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c010480 tx=0x7fdb2c010940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69a582ac3be45ed server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.734+0000 7fdb19ffb640 1 -- 172.21.15.29:0/3425257937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c017070 con 0x55e06a37e1b0 2024-09-21T03:00:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.735+0000 7fdb19ffb640 1 -- 172.21.15.29:0/3425257937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb2c011bf0 con 0x55e06a37e1b0 2024-09-21T03:00:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.735+0000 7fdb30ff9640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb1c0b6590 0x7fdb1c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.735+0000 7fdb19ffb640 1 -- 172.21.15.29:0/3425257937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c00ec30 con 0x55e06a37e1b0 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.735+0000 7fdb337fe640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb1c0b6590 0x7fdb1c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.735+0000 7fdb30ff9640 1 -- 172.21.15.29:0/343069970 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdb280e7a30 con 0x7fdb34103620 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.735+0000 7fdb40e9b740 1 -- 172.21.15.29:0/3425257937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a37e1b0 msgr2=0x55e06a725e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.735+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a37e1b0 0x55e06a725e50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c010480 tx=0x7fdb2c010940 comp rx=0 tx=0).stop 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 -- 172.21.15.29:0/3425257937 shutdown_connections 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a727010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a7264c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/3425257937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a37e1b0 0x55e06a725e50 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 -- 172.21.15.29:0/3425257937 >> 172.21.15.29:0/3425257937 conn(0x55e06a372300 msgr2=0x55e06a7218e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 -- 172.21.15.29:0/3425257937 shutdown_connections 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 -- 172.21.15.29:0/3425257937 wait complete. 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb337fe640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb1c0b6590 0x7fdb1c0b8aa0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdb24005760 tx=0x7fdb24031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 -- start start 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.736+0000 7fdb40e9b740 1 Processor -- start 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a72f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a35d4d0 0x55e06a72fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a37e1b0 0x55e06a730820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e06a5bfab0 con 0x55e06a35d4d0 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55e06a62d730 con 0x55e06a3552f0 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e06a727bd0 con 0x55e06a37e1b0 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb40e9b740 10 client.0 _ll_register_callbacks cb 0x55e06a6816b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb32ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a35d4d0 0x55e06a72fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb32ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a35d4d0 0x55e06a72fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54856/0 (socket says 172.21.15.29:54856) 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb33fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a37e1b0 0x55e06a730820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb32ffd640 1 -- 172.21.15.29:0/2240120191 learned_addr learned my addr 172.21.15.29:0/2240120191 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb337fe640 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a72f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb32ffd640 1 -- 172.21.15.29:0/2240120191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a37e1b0 msgr2=0x55e06a730820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a37e1b0 0x55e06a730820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb32ffd640 1 -- 172.21.15.29:0/2240120191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 msgr2=0x55e06a72f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.737+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a72f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.738+0000 7fdb32ffd640 1 -- 172.21.15.29:0/2240120191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06a3e9790 con 0x55e06a35d4d0 2024-09-21T03:00:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.738+0000 7fdb33fff640 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a37e1b0 0x55e06a730820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:49.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52194 5.47 MB/sec execute 386 sec latency 180.049 ms 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.738+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a35d4d0 0x55e06a72fd20 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fdb280ad4e0 tx=0x7fdb280a97b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2117ccfac18709 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.738+0000 7fdb12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.738+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2240120191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb280b22c0 con 0x55e06a35d4d0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.738+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2240120191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdb280a9130 con 0x55e06a35d4d0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.738+0000 7fdb40e9b740 1 -- 172.21.15.29:0/2240120191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb2c011bf0 con 0x55e06a35d4d0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.738+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2240120191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdb280b7200 con 0x55e06a35d4d0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb40e9b740 10 client.18336 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb40e9b740 1 -- 172.21.15.29:0/2240120191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e06a462620 con 0x55e06a35d4d0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2240120191 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb2c011bf0 con 0x55e06a35d4d0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2240120191 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fdb280b3650 con 0x55e06a35d4d0 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb40e9b740 10 client.18336 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb40e9b740 10 client.18336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb40e9b740 20 client.18336 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e06a33cd80 0x55e06a73ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.739+0000 7fdb40e9b740 4 client.18336 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:00:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.740+0000 7fdb40e9b740 1 -- 172.21.15.29:0/2240120191 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e06a69a640 con 0x55e06a33cd80 2024-09-21T03:00:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.740+0000 7fdb337fe640 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e06a33cd80 0x55e06a73ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.740+0000 7fdb12ffd640 10 client.18336 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.742+0000 7fdb337fe640 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e06a33cd80 0x55e06a73ccd0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c896d5c32b1682b server_cookie=853d4259dcab4e95 in_seq=0 out_seq=0 2024-09-21T03:00:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.743+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2240120191 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e06a69a640 con 0x55e06a33cd80 2024-09-21T03:00:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.743+0000 7fdb12ffd640 10 client.18336 handle_command_reply: tid=0 2024-09-21T03:00:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.743+0000 7fdb40e9b740 1 client.18336 shutdown 2024-09-21T03:00:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.743+0000 7fdb40e9b740 1 -- 172.21.15.29:0/2240120191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a35d4d0 msgr2=0x55e06a72fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.743+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a35d4d0 0x55e06a72fd20 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fdb280ad4e0 tx=0x7fdb280a97b0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.743+0000 7fdb40e9b740 1 -- 172.21.15.29:0/2240120191 shutdown_connections 2024-09-21T03:00:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.743+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a37e1b0 0x55e06a730820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.743+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e06a33cd80 0x55e06a73ccd0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.744+0000 7fdb12ffd640 0 client.18336 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.744+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a35d4d0 0x55e06a72fd20 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.744+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/2240120191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a72f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.744+0000 7fdb40e9b740 1 -- 172.21.15.29:0/2240120191 >> 172.21.15.29:0/2240120191 conn(0x55e06a372300 msgr2=0x55e06a7263b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.744+0000 7fdb40e9b740 1 -- 172.21.15.29:0/2240120191 shutdown_connections 2024-09-21T03:00:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.744+0000 7fdb40e9b740 1 -- 172.21.15.29:0/2240120191 wait complete. 2024-09-21T03:00:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.744+0000 7fdb40e9b740 20 client.18336 trim_cache size 0 max 16384 2024-09-21T03:00:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.769+0000 7fdb40e9b740 1 Processor -- start 2024-09-21T03:00:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.769+0000 7fdb40e9b740 1 -- start start 2024-09-21T03:00:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a726d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a3552f0 0x55e06a737700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a35d4d0 0x55e06a72e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb32ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a3552f0 0x55e06a737700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb32ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a3552f0 0x55e06a737700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56310/0 (socket says 172.21.15.29:56310) 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a726d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a726d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54864/0 (socket says 172.21.15.29:54864) 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb32ffd640 1 -- 172.21.15.29:0/209765911 learned_addr learned my addr 172.21.15.29:0/209765911 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb40e9b740 1 -- 172.21.15.29:0/209765911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341ae820 con 0x55e06a33cd80 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb40e9b740 1 -- 172.21.15.29:0/209765911 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341b0360 con 0x55e06a35d4d0 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb40e9b740 1 -- 172.21.15.29:0/209765911 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341aeef0 con 0x55e06a3552f0 2024-09-21T03:00:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.770+0000 7fdb33fff640 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a35d4d0 0x55e06a72e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb32ffd640 1 -- 172.21.15.29:0/209765911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a35d4d0 msgr2=0x55e06a72e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a35d4d0 0x55e06a72e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb32ffd640 1 -- 172.21.15.29:0/209765911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 msgr2=0x55e06a726d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a726d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb32ffd640 1 -- 172.21.15.29:0/209765911 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06a7272a0 con 0x55e06a3552f0 2024-09-21T03:00:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb337fe640 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a726d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:00:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a3552f0 0x55e06a737700 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdb280b6760 tx=0x7fdb280ad430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f117f5a4f3d015 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb197fa640 1 -- 172.21.15.29:0/209765911 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb280b22c0 con 0x55e06a3552f0 2024-09-21T03:00:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb197fa640 1 -- 172.21.15.29:0/209765911 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb280b67e0 con 0x55e06a3552f0 2024-09-21T03:00:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.771+0000 7fdb197fa640 1 -- 172.21.15.29:0/209765911 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb280ad5b0 con 0x55e06a3552f0 2024-09-21T03:00:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 -- 172.21.15.29:0/209765911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a3552f0 msgr2=0x55e06a737700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a3552f0 0x55e06a737700 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdb280b6760 tx=0x7fdb280ad430 comp rx=0 tx=0).stop 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 -- 172.21.15.29:0/209765911 shutdown_connections 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a35d4d0 0x55e06a72e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a3552f0 0x55e06a737700 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/209765911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a726d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 -- 172.21.15.29:0/209765911 >> 172.21.15.29:0/209765911 conn(0x55e06a372300 msgr2=0x55e06a72e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 -- 172.21.15.29:0/209765911 shutdown_connections 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 -- 172.21.15.29:0/209765911 wait complete. 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 -- start start 2024-09-21T03:00:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.773+0000 7fdb40e9b740 1 Processor -- start 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a7370f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a732fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb40e9b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a733520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341afc90 con 0x55e06a33cd80 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341af5c0 con 0x55e06a3552f0 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb40e9b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341ac800 con 0x55e06a35d4d0 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb40e9b740 10 client.0 _ll_register_callbacks cb 0x7fdb280056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a7370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb32ffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a732fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a7370f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54880/0 (socket says 172.21.15.29:54880) 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb337fe640 1 -- 172.21.15.29:0/500487676 learned_addr learned my addr 172.21.15.29:0/500487676 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.774+0000 7fdb33fff640 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a733520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb32ffd640 1 -- 172.21.15.29:0/500487676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 msgr2=0x55e06a733520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a733520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb32ffd640 1 -- 172.21.15.29:0/500487676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 msgr2=0x55e06a7370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a7370f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb32ffd640 1 -- 172.21.15.29:0/500487676 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06a733a60 con 0x55e06a3552f0 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb32ffd640 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a732fe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb280ad770 tx=0x7fdb280b6990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bafe567a340cf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb12ffd640 1 -- 172.21.15.29:0/500487676 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb280ad5b0 con 0x55e06a3552f0 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb12ffd640 1 -- 172.21.15.29:0/500487676 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fdb280aeb10 con 0x55e06a3552f0 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb40e9b740 1 -- 172.21.15.29:0/500487676 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb280b67e0 con 0x55e06a3552f0 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb12ffd640 1 -- 172.21.15.29:0/500487676 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdb280b22c0 con 0x55e06a3552f0 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb40e9b740 10 client.27623 fetch_fsmap learned FSMap version 37 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.775+0000 7fdb40e9b740 1 -- 172.21.15.29:0/500487676 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e06a7a3640 con 0x55e06a3552f0 2024-09-21T03:00:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb12ffd640 1 -- 172.21.15.29:0/500487676 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb280c8e30 con 0x55e06a3552f0 2024-09-21T03:00:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb12ffd640 1 -- 172.21.15.29:0/500487676 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fdb280b6040 con 0x55e06a3552f0 2024-09-21T03:00:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb40e9b740 10 client.27623 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:00:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb40e9b740 10 client.27623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:00:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb40e9b740 20 client.27623 populate_metadata read hostname 'smithi029' 2024-09-21T03:00:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e06a37e1b0 0x55e06a7a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:00:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb40e9b740 4 client.27623 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:00:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb40e9b740 1 -- 172.21.15.29:0/500487676 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fdb280b67e0 con 0x55e06a37e1b0 2024-09-21T03:00:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.776+0000 7fdb337fe640 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e06a37e1b0 0x55e06a7a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:00:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.777+0000 7fdb12ffd640 10 client.27623 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:49.777+0000 7fdb337fe640 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e06a37e1b0 0x55e06a7a5ce0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0c7d0f5d9b4123a server_cookie=ceabe016315d6cd7 in_seq=0 out_seq=0 2024-09-21T03:00:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:49 smithi190 ceph-mon[33871]: pgmap v1040: 129 pgs: 129 active+clean; 314 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-09-21T03:00:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:49 smithi029 ceph-mon[31006]: pgmap v1040: 129 pgs: 129 active+clean; 314 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-09-21T03:00:50.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:49 smithi066 ceph-mon[33436]: pgmap v1040: 129 pgs: 129 active+clean; 314 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-09-21T03:00:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.504+0000 7fdb12ffd640 1 -- 172.21.15.29:0/500487676 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fdb280b67e0 con 0x55e06a37e1b0 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.504+0000 7fdb12ffd640 10 client.27623 handle_command_reply: tid=0 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.504+0000 7fdb40e9b740 1 client.27623 shutdown 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 -- 172.21.15.29:0/500487676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 msgr2=0x55e06a732fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a732fe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb280ad770 tx=0x7fdb280b6990 comp rx=0 tx=0).stop 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 -- 172.21.15.29:0/500487676 shutdown_connections 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e06a35d4d0 0x55e06a733520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55e06a3552f0 0x55e06a732fe0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55e06a37e1b0 0x55e06a7a5ce0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 --2- 172.21.15.29:0/500487676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e06a33cd80 0x55e06a7370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 -- 172.21.15.29:0/500487676 >> 172.21.15.29:0/500487676 conn(0x55e06a372300 msgr2=0x55e06a71fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb12ffd640 0 client.27623 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 -- 172.21.15.29:0/500487676 shutdown_connections 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 1 -- 172.21.15.29:0/500487676 wait complete. 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.505+0000 7fdb40e9b740 20 client.27623 trim_cache size 0 max 16384 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:00:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:00:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.506+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb1c0b6590 msgr2=0x7fdb1c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.506+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb1c0b6590 0x7fdb1c0b8aa0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdb24005760 tx=0x7fdb24031400 comp rx=0 tx=0).stop 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.506+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb34103620 msgr2=0x7fdb34199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.506+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb34103620 0x7fdb34199430 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800e9f0 tx=0x7fdb2800eeb0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.506+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 shutdown_connections 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.507+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fdb1c0b6590 0x7fdb1c0b8aa0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.507+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb34103fc0 0x7fdb34199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.507+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb34103620 0x7fdb34199430 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.507+0000 7fdb39f2b640 1 --2- 172.21.15.29:0/343069970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb341025a0 0x7fdb34198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.507+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 >> 172.21.15.29:0/343069970 conn(0x7fdb340fdee0 msgr2=0x7fdb34106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.507+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 shutdown_connections 2024-09-21T03:00:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:00:50.507+0000 7fdb39f2b640 1 -- 172.21.15.29:0/343069970 wait complete. 2024-09-21T03:00:50.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52320 5.48 MB/sec execute 387 sec latency 396.770 ms 2024-09-21T03:00:51.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52471 5.49 MB/sec execute 388 sec latency 108.716 ms 2024-09-21T03:00:51.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:51 smithi190 ceph-mon[33871]: pgmap v1041: 129 pgs: 129 active+clean; 314 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-09-21T03:00:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:51 smithi029 ceph-mon[31006]: pgmap v1041: 129 pgs: 129 active+clean; 314 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-09-21T03:00:52.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:51 smithi066 ceph-mon[33436]: pgmap v1041: 129 pgs: 129 active+clean; 314 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-09-21T03:00:52.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52652 5.52 MB/sec execute 389 sec latency 271.040 ms 2024-09-21T03:00:53.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52809 5.53 MB/sec execute 390 sec latency 126.976 ms 2024-09-21T03:00:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:53 smithi190 ceph-mon[33871]: pgmap v1042: 129 pgs: 129 active+clean; 317 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 2.4 MiB/s wr, 151 op/s 2024-09-21T03:00:53.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:53 smithi029 ceph-mon[31006]: pgmap v1042: 129 pgs: 129 active+clean; 317 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 2.4 MiB/s wr, 151 op/s 2024-09-21T03:00:54.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:53 smithi066 ceph-mon[33436]: pgmap v1042: 129 pgs: 129 active+clean; 317 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 2.4 MiB/s wr, 151 op/s 2024-09-21T03:00:54.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52924 5.54 MB/sec execute 391 sec latency 117.487 ms 2024-09-21T03:00:55.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53051 5.54 MB/sec execute 392 sec latency 84.233 ms 2024-09-21T03:00:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:55 smithi190 ceph-mon[33871]: pgmap v1043: 129 pgs: 129 active+clean; 319 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 3.4 MiB/s wr, 225 op/s 2024-09-21T03:00:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:55 smithi029 ceph-mon[31006]: pgmap v1043: 129 pgs: 129 active+clean; 319 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 3.4 MiB/s wr, 225 op/s 2024-09-21T03:00:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:55 smithi066 ceph-mon[33436]: pgmap v1043: 129 pgs: 129 active+clean; 319 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 3.4 MiB/s wr, 225 op/s 2024-09-21T03:00:56.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53188 5.55 MB/sec execute 393 sec latency 103.519 ms 2024-09-21T03:00:57.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53318 5.55 MB/sec execute 394 sec latency 123.039 ms 2024-09-21T03:00:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:57 smithi190 ceph-mon[33871]: pgmap v1044: 129 pgs: 129 active+clean; 320 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.6 MiB/s wr, 235 op/s 2024-09-21T03:00:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:57 smithi029 ceph-mon[31006]: pgmap v1044: 129 pgs: 129 active+clean; 320 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.6 MiB/s wr, 235 op/s 2024-09-21T03:00:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:57 smithi066 ceph-mon[33436]: pgmap v1044: 129 pgs: 129 active+clean; 320 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.6 MiB/s wr, 235 op/s 2024-09-21T03:00:58.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53432 5.55 MB/sec execute 395 sec latency 168.514 ms 2024-09-21T03:00:59.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53555 5.54 MB/sec execute 396 sec latency 305.938 ms 2024-09-21T03:00:59.812 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:00:59 smithi066 ceph-mon[33436]: pgmap v1045: 129 pgs: 129 active+clean; 325 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 4.1 MiB/s wr, 267 op/s 2024-09-21T03:00:59.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:00:59 smithi190 ceph-mon[33871]: pgmap v1045: 129 pgs: 129 active+clean; 325 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 4.1 MiB/s wr, 267 op/s 2024-09-21T03:00:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:00:59 smithi029 ceph-mon[31006]: pgmap v1045: 129 pgs: 129 active+clean; 325 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 4.1 MiB/s wr, 267 op/s 2024-09-21T03:01:00.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53683 5.54 MB/sec execute 397 sec latency 147.890 ms 2024-09-21T03:01:01.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53804 5.54 MB/sec execute 398 sec latency 108.807 ms 2024-09-21T03:01:01.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:01 smithi029 ceph-mon[31006]: pgmap v1046: 129 pgs: 129 active+clean; 326 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-09-21T03:01:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:01 smithi190 ceph-mon[33871]: pgmap v1046: 129 pgs: 129 active+clean; 326 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-09-21T03:01:02.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:01 smithi066 ceph-mon[33436]: pgmap v1046: 129 pgs: 129 active+clean; 326 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-09-21T03:01:02.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53940 5.54 MB/sec execute 399 sec latency 94.638 ms 2024-09-21T03:01:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/2050857791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1abc101d50 msgr2=0x7f1abc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/2050857791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1abc101d50 0x7f1abc1021c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009860 tx=0x7f1ab002a340 comp rx=0 tx=0).stop 2024-09-21T03:01:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/2050857791 shutdown_connections 2024-09-21T03:01:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/2050857791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1abc103770 0x7f1abc10a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/2050857791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1abc102dd0 0x7f1abc1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/2050857791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1abc101d50 0x7f1abc1021c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/2050857791 >> 172.21.15.29:0/2050857791 conn(0x7f1abc0fd690 msgr2=0x7f1abc0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/2050857791 shutdown_connections 2024-09-21T03:01:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.647+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/2050857791 wait complete. 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.648+0000 7f1ac2bf1640 1 Processor -- start 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.648+0000 7f1ac2bf1640 1 -- start start 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.648+0000 7f1ac2bf1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1abc101d50 0x7f1abc106a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.648+0000 7f1ac2bf1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1abc102dd0 0x7f1abc105310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac2bf1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1abc103770 0x7f1abc105850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac2bf1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc10d820 con 0x7f1abc101d50 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac2bf1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc10d6a0 con 0x7f1abc102dd0 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac2bf1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc10d9a0 con 0x7f1abc103770 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1abc101d50 0x7f1abc106a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1abc101d50 0x7f1abc106a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56442/0 (socket says 172.21.15.29:56442) 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 -- 172.21.15.29:0/1275593631 learned_addr learned my addr 172.21.15.29:0/1275593631 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1abbfff640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1abc102dd0 0x7f1abc105310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac118e640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1abc103770 0x7f1abc105850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 -- 172.21.15.29:0/1275593631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1abc103770 msgr2=0x7f1abc105850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1abc103770 0x7f1abc105850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 -- 172.21.15.29:0/1275593631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1abc102dd0 msgr2=0x7f1abc105310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1abc102dd0 0x7f1abc105310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 -- 172.21.15.29:0/1275593631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1abc105f80 con 0x7f1abc101d50 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.649+0000 7f1ac098d640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1abc101d50 0x7f1abc106a50 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f1ab002a930 tx=0x7f1ab002ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.650+0000 7f1ab9ffb640 1 -- 172.21.15.29:0/1275593631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0040030 con 0x7f1abc101d50 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.650+0000 7f1ab9ffb640 1 -- 172.21.15.29:0/1275593631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ab0032bc0 con 0x7f1abc101d50 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.650+0000 7f1ab9ffb640 1 -- 172.21.15.29:0/1275593631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0037040 con 0x7f1abc101d50 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.650+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/1275593631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1abc1a3f60 con 0x7f1abc101d50 2024-09-21T03:01:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.650+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/1275593631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1abc1a4310 con 0x7f1abc101d50 2024-09-21T03:01:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac9c40740 1 Processor -- start 2024-09-21T03:01:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac9c40740 1 -- start start 2024-09-21T03:01:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb971b0 0x55bb9bf3ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf3f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb764d0 0x55bb9bf40010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bb9beb16f0 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55bb9bea0b20 con 0x55bb9bb971b0 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bb9bea4930 con 0x55bb9bb764d0 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1abbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb764d0 0x55bb9bf40010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ab9ffb640 1 -- 172.21.15.29:0/1275593631 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f1ab0043070 con 0x7f1abc101d50 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1abbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb764d0 0x55bb9bf40010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51934/0 (socket says 172.21.15.29:51934) 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1ac118e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb971b0 0x55bb9bf3ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.654+0000 7f1abbfff640 1 -- 172.21.15.29:0/2592313206 learned_addr learned my addr 172.21.15.29:0/2592313206 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1ac098d640 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf3f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1abbfff640 1 -- 172.21.15.29:0/2592313206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb971b0 msgr2=0x55bb9bf3ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1abbfff640 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb971b0 0x55bb9bf3ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1abbfff640 1 -- 172.21.15.29:0/2592313206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 msgr2=0x55bb9bf3f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1abbfff640 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf3f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1abbfff640 1 -- 172.21.15.29:0/2592313206 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb9be66530 con 0x55bb9bb764d0 2024-09-21T03:01:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1ac098d640 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf3f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:01:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1abbfff640 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb764d0 0x55bb9bf40010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009cc0 tx=0x7f1aa8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f75565c231cae29 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.655+0000 7f1ab9ffb640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1aac0b6590 0x7f1aac0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.656+0000 7f1aa6ffd640 1 -- 172.21.15.29:0/2592313206 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8005220 con 0x55bb9bb764d0 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.656+0000 7f1aa6ffd640 1 -- 172.21.15.29:0/2592313206 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1aa8004c30 con 0x55bb9bb764d0 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.656+0000 7f1ab9ffb640 1 -- 172.21.15.29:0/1275593631 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1ab01072d0 con 0x7f1abc101d50 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.656+0000 7f1ac118e640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1aac0b6590 0x7f1aac0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.656+0000 7f1aa6ffd640 1 -- 172.21.15.29:0/2592313206 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8019650 con 0x55bb9bb764d0 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2592313206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb764d0 msgr2=0x55bb9bf40010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac118e640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1aac0b6590 0x7f1aac0b8aa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4005fd0 tx=0x7f1ab4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb764d0 0x55bb9bf40010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009cc0 tx=0x7f1aa8011810 comp rx=0 tx=0).stop 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2592313206 shutdown_connections 2024-09-21T03:01:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb764d0 0x55bb9bf40010 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf3f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2592313206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb971b0 0x55bb9bf3ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2592313206 >> 172.21.15.29:0/2592313206 conn(0x55bb9bb8b300 msgr2=0x55bb9bf3a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2592313206 shutdown_connections 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2592313206 wait complete. 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 -- start start 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.657+0000 7f1ac9c40740 1 Processor -- start 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf48670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf48d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb971b0 0x55bb9bf498a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bb9bdd8ab0 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55bb9be46730 con 0x55bb9bb764d0 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bb9bf40b30 con 0x55bb9bb971b0 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac9c40740 10 client.0 _ll_register_callbacks cb 0x55bb9be9a6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac118e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb971b0 0x55bb9bf498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac118e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb971b0 0x55bb9bf498a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51950/0 (socket says 172.21.15.29:51950) 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac118e640 1 -- 172.21.15.29:0/593124202 learned_addr learned my addr 172.21.15.29:0/593124202 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac098d640 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf48670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1abbfff640 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf48d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac118e640 1 -- 172.21.15.29:0/593124202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 msgr2=0x55bb9bf48d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac118e640 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf48d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac118e640 1 -- 172.21.15.29:0/593124202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 msgr2=0x55bb9bf48670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac118e640 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf48670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.658+0000 7f1ac118e640 1 -- 172.21.15.29:0/593124202 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb9bc02790 con 0x55bb9bb971b0 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1ac118e640 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb971b0 0x55bb9bf498a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4003fd0 tx=0x7f1ab4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b9b6d13a3c038c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1a9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:01:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1a9bfff640 1 -- 172.21.15.29:0/593124202 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab400f830 con 0x55bb9bb971b0 2024-09-21T03:01:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1a9bfff640 1 -- 172.21.15.29:0/593124202 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1ab400f9d0 con 0x55bb9bb971b0 2024-09-21T03:01:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1ac9c40740 1 -- 172.21.15.29:0/593124202 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aa8004c30 con 0x55bb9bb971b0 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1a9bfff640 1 -- 172.21.15.29:0/593124202 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1ab400fbd0 con 0x55bb9bb971b0 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1ac9c40740 10 client.27793 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1ac9c40740 1 -- 172.21.15.29:0/593124202 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bb9bc7b620 con 0x55bb9bb971b0 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.659+0000 7f1a9bfff640 1 -- 172.21.15.29:0/593124202 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1aa8004c30 con 0x55bb9bb971b0 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1a9bfff640 1 -- 172.21.15.29:0/593124202 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f1ab4026070 con 0x55bb9bb971b0 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1ac9c40740 10 client.27793 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1ac9c40740 10 client.27793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1ac9c40740 20 client.27793 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bb9bb55d80 0x55bb9bf55e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1ac9c40740 4 client.27793 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1ac9c40740 1 -- 172.21.15.29:0/593124202 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb9beb3640 con 0x55bb9bb55d80 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1ac098d640 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bb9bb55d80 0x55bb9bf55e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1a9bfff640 10 client.27793 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.660+0000 7f1ac098d640 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bb9bb55d80 0x55bb9bf55e60 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec83781c247ad0d5 server_cookie=99b993c366442a3d in_seq=0 out_seq=0 2024-09-21T03:01:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.663+0000 7f1a9bfff640 1 -- 172.21.15.29:0/593124202 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bb9beb3640 con 0x55bb9bb55d80 2024-09-21T03:01:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.663+0000 7f1a9bfff640 10 client.27793 handle_command_reply: tid=0 2024-09-21T03:01:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.663+0000 7f1ac9c40740 1 client.27793 shutdown 2024-09-21T03:01:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 -- 172.21.15.29:0/593124202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb971b0 msgr2=0x55bb9bf498a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb971b0 0x55bb9bf498a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4003fd0 tx=0x7f1ab4004ab0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 -- 172.21.15.29:0/593124202 shutdown_connections 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb971b0 0x55bb9bf498a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf48d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bb9bb55d80 0x55bb9bf55e60 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/593124202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf48670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1a9bfff640 0 client.27793 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 -- 172.21.15.29:0/593124202 >> 172.21.15.29:0/593124202 conn(0x55bb9bb8b300 msgr2=0x55bb9bf38ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 -- 172.21.15.29:0/593124202 shutdown_connections 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 1 -- 172.21.15.29:0/593124202 wait complete. 2024-09-21T03:01:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.664+0000 7f1ac9c40740 20 client.27793 trim_cache size 0 max 16384 2024-09-21T03:01:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.684+0000 7f1ac9c40740 1 Processor -- start 2024-09-21T03:01:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.684+0000 7f1ac9c40740 1 -- start start 2024-09-21T03:01:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf3a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf47890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf355c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc1ae760 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc1b02a0 con 0x55bb9bb764d0 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc1aee30 con 0x55bb9bb55d80 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac098d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf3a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac098d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf3a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51952/0 (socket says 172.21.15.29:51952) 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac098d640 1 -- 172.21.15.29:0/3097151235 learned_addr learned my addr 172.21.15.29:0/3097151235 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1abbfff640 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf47890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac118e640 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac118e640 1 -- 172.21.15.29:0/3097151235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 msgr2=0x55bb9bf3a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac118e640 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf3a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac118e640 1 -- 172.21.15.29:0/3097151235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 msgr2=0x55bb9bf47890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac118e640 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf47890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.685+0000 7f1ac118e640 1 -- 172.21.15.29:0/3097151235 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb9bf46800 con 0x55bb9bb764d0 2024-09-21T03:01:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.686+0000 7f1ac118e640 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf355c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1ab401df70 tx=0x7f1ab4009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa65b83b6eb186ae server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.686+0000 7f1aa67fc640 1 -- 172.21.15.29:0/3097151235 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab400fdf0 con 0x55bb9bb764d0 2024-09-21T03:01:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.686+0000 7f1aa67fc640 1 -- 172.21.15.29:0/3097151235 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ab400f830 con 0x55bb9bb764d0 2024-09-21T03:01:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.686+0000 7f1aa67fc640 1 -- 172.21.15.29:0/3097151235 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab401d470 con 0x55bb9bb764d0 2024-09-21T03:01:03.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 -- 172.21.15.29:0/3097151235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 msgr2=0x55bb9bf355c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf355c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1ab401df70 tx=0x7f1ab4009610 comp rx=0 tx=0).stop 2024-09-21T03:01:03.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 -- 172.21.15.29:0/3097151235 shutdown_connections 2024-09-21T03:01:03.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf355c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf47890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/3097151235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf3a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 -- 172.21.15.29:0/3097151235 >> 172.21.15.29:0/3097151235 conn(0x55bb9bb8b300 msgr2=0x55bb9bf49e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 -- 172.21.15.29:0/3097151235 shutdown_connections 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.687+0000 7f1ac9c40740 1 -- 172.21.15.29:0/3097151235 wait complete. 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 1 -- start start 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 1 Processor -- start 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf4d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf4ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf4ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc1afbd0 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc1af500 con 0x55bb9bb764d0 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc1ac740 con 0x55bb9bb55d80 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1ac9c40740 10 client.0 _ll_register_callbacks cb 0x7f1ab4009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.688+0000 7f1abbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf4ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1abbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf4ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56484/0 (socket says 172.21.15.29:56484) 2024-09-21T03:01:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1abbfff640 1 -- 172.21.15.29:0/2690511083 learned_addr learned my addr 172.21.15.29:0/2690511083 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1ac098d640 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf4d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1ac118e640 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf4ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1abbfff640 1 -- 172.21.15.29:0/2690511083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 msgr2=0x55bb9bf4d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1abbfff640 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf4d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1abbfff640 1 -- 172.21.15.29:0/2690511083 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 msgr2=0x55bb9bf4ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1abbfff640 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf4ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1abbfff640 1 -- 172.21.15.29:0/2690511083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb9bf4f1a0 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1ac118e640 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf4ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1abbfff640 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf4ded0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009340 tx=0x7f1aa8009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3bc97c5f65c4d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.689+0000 7f1a9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1a9bfff640 1 -- 172.21.15.29:0/2690511083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8004870 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1a9bfff640 1 -- 172.21.15.29:0/2690511083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f1aa8004a10 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2690511083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ab400f830 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1a9bfff640 1 -- 172.21.15.29:0/2690511083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1aa8004c10 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 10 client.18357 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2690511083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bb9bf504c0 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1a9bfff640 1 -- 172.21.15.29:0/2690511083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ab400f830 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1a9bfff640 1 -- 172.21.15.29:0/2690511083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f1aa802d420 con 0x55bb9bb6e2f0 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 10 client.18357 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 10 client.18357 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 20 client.18357 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bb9bb971b0 0x55bb9bfc28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 4 client.18357 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.690+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2690511083 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb9beaf0f0 con 0x55bb9bb971b0 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.691+0000 7f1ac098d640 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bb9bb971b0 0x55bb9bfc28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.691+0000 7f1a9bfff640 10 client.18357 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.691+0000 7f1ac098d640 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bb9bb971b0 0x55bb9bfc28a0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54784c6c79e2fa1b server_cookie=7500bd68b3253b9d in_seq=0 out_seq=0 2024-09-21T03:01:03.739 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54044 5.53 MB/sec execute 400 sec latency 238.004 ms 2024-09-21T03:01:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.871+0000 7f1a9bfff640 1 -- 172.21.15.29:0/2690511083 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb9beaf0f0 con 0x55bb9bb971b0 2024-09-21T03:01:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.871+0000 7f1a9bfff640 10 client.18357 handle_command_reply: tid=0 2024-09-21T03:01:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.871+0000 7f1ac9c40740 1 client.18357 shutdown 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.871+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2690511083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 msgr2=0x55bb9bf4ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.871+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf4ded0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009340 tx=0x7f1aa8009ed0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.871+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2690511083 shutdown_connections 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.871+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55bb9bb764d0 0x55bb9bf4ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.872+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55bb9bb971b0 0x55bb9bfc28a0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.872+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb9bb6e2f0 0x55bb9bf4ded0 secure :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009340 tx=0x7f1aa8009ed0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.872+0000 7f1ac9c40740 1 --2- 172.21.15.29:0/2690511083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb9bb55d80 0x55bb9bf4d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.872+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2690511083 >> 172.21.15.29:0/2690511083 conn(0x55bb9bb8b300 msgr2=0x55bb9bf49da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.872+0000 7f1a9bfff640 0 client.18357 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:03 smithi190 ceph-mon[33871]: pgmap v1047: 129 pgs: 129 active+clean; 324 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.9 MiB/s wr, 258 op/s 2024-09-21T03:01:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.872+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2690511083 shutdown_connections 2024-09-21T03:01:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.872+0000 7f1ac9c40740 1 -- 172.21.15.29:0/2690511083 wait complete. 2024-09-21T03:01:03.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.872+0000 7f1ac9c40740 20 client.18357 trim_cache size 0 max 16384 2024-09-21T03:01:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:01:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:01:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:01:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:01:03.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.876+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1aac0b6590 msgr2=0x7f1aac0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.876+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1aac0b6590 0x7f1aac0b8aa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4005fd0 tx=0x7f1ab4009500 comp rx=0 tx=0).stop 2024-09-21T03:01:03.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.876+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1abc101d50 msgr2=0x7f1abc106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:03.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.876+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1abc101d50 0x7f1abc106a50 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f1ab002a930 tx=0x7f1ab002ad70 comp rx=0 tx=0).stop 2024-09-21T03:01:03.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.877+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/1275593631 shutdown_connections 2024-09-21T03:01:03.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.877+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f1aac0b6590 0x7f1aac0b8aa0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.877+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1abc103770 0x7f1abc105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.877+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1abc102dd0 0x7f1abc105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:03.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.877+0000 7f1ac2bf1640 1 --2- 172.21.15.29:0/1275593631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1abc101d50 0x7f1abc106a50 secure :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f1ab002a930 tx=0x7f1ab002ad70 comp rx=0 tx=0).stop 2024-09-21T03:01:03.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.877+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/1275593631 >> 172.21.15.29:0/1275593631 conn(0x7f1abc0fd690 msgr2=0x7f1abc06a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:03.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.878+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/1275593631 shutdown_connections 2024-09-21T03:01:03.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:03.878+0000 7f1ac2bf1640 1 -- 172.21.15.29:0/1275593631 wait complete. 2024-09-21T03:01:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:03 smithi029 ceph-mon[31006]: pgmap v1047: 129 pgs: 129 active+clean; 324 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.9 MiB/s wr, 258 op/s 2024-09-21T03:01:04.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:03 smithi066 ceph-mon[33436]: pgmap v1047: 129 pgs: 129 active+clean; 324 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 3.9 MiB/s wr, 258 op/s 2024-09-21T03:01:04.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54172 5.52 MB/sec execute 401 sec latency 227.920 ms 2024-09-21T03:01:05.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54345 5.56 MB/sec execute 402 sec latency 118.420 ms 2024-09-21T03:01:05.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:05 smithi190 ceph-mon[33871]: pgmap v1048: 129 pgs: 129 active+clean; 317 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-09-21T03:01:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:05 smithi029 ceph-mon[31006]: pgmap v1048: 129 pgs: 129 active+clean; 317 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-09-21T03:01:06.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:05 smithi066 ceph-mon[33436]: pgmap v1048: 129 pgs: 129 active+clean; 317 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-09-21T03:01:06.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54480 5.55 MB/sec execute 403 sec latency 338.186 ms 2024-09-21T03:01:07.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54644 5.56 MB/sec execute 404 sec latency 106.945 ms 2024-09-21T03:01:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:07 smithi190 ceph-mon[33871]: pgmap v1049: 129 pgs: 129 active+clean; 314 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-09-21T03:01:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:07 smithi029 ceph-mon[31006]: pgmap v1049: 129 pgs: 129 active+clean; 314 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-09-21T03:01:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:07 smithi066 ceph-mon[33436]: pgmap v1049: 129 pgs: 129 active+clean; 314 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-09-21T03:01:08.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54796 5.57 MB/sec execute 405 sec latency 166.966 ms 2024-09-21T03:01:09.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54944 5.57 MB/sec execute 406 sec latency 135.155 ms 2024-09-21T03:01:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:09 smithi190 ceph-mon[33871]: pgmap v1050: 129 pgs: 129 active+clean; 306 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 3.5 MiB/s wr, 298 op/s 2024-09-21T03:01:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:09 smithi029 ceph-mon[31006]: pgmap v1050: 129 pgs: 129 active+clean; 306 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 3.5 MiB/s wr, 298 op/s 2024-09-21T03:01:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:09 smithi066 ceph-mon[33436]: pgmap v1050: 129 pgs: 129 active+clean; 306 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 3.5 MiB/s wr, 298 op/s 2024-09-21T03:01:10.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55061 5.56 MB/sec execute 407 sec latency 227.623 ms 2024-09-21T03:01:11.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55211 5.57 MB/sec execute 408 sec latency 167.605 ms 2024-09-21T03:01:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:11 smithi190 ceph-mon[33871]: pgmap v1051: 129 pgs: 129 active+clean; 307 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 3.2 MiB/s wr, 279 op/s 2024-09-21T03:01:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:11 smithi029 ceph-mon[31006]: pgmap v1051: 129 pgs: 129 active+clean; 307 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 3.2 MiB/s wr, 279 op/s 2024-09-21T03:01:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:11 smithi066 ceph-mon[33436]: pgmap v1051: 129 pgs: 129 active+clean; 307 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 3.2 MiB/s wr, 279 op/s 2024-09-21T03:01:12.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55351 5.57 MB/sec execute 409 sec latency 104.099 ms 2024-09-21T03:01:13.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55476 5.57 MB/sec execute 410 sec latency 202.937 ms 2024-09-21T03:01:13.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:13 smithi190 ceph-mon[33871]: pgmap v1052: 129 pgs: 129 active+clean; 309 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 3.0 MiB/s wr, 305 op/s 2024-09-21T03:01:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:13 smithi029 ceph-mon[31006]: pgmap v1052: 129 pgs: 129 active+clean; 309 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 3.0 MiB/s wr, 305 op/s 2024-09-21T03:01:14.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:13 smithi066 ceph-mon[33436]: pgmap v1052: 129 pgs: 129 active+clean; 309 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 3.0 MiB/s wr, 305 op/s 2024-09-21T03:01:14.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55619 5.57 MB/sec execute 411 sec latency 132.986 ms 2024-09-21T03:01:15.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55827 5.60 MB/sec execute 412 sec latency 95.313 ms 2024-09-21T03:01:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:15 smithi190 ceph-mon[33871]: pgmap v1053: 129 pgs: 129 active+clean; 312 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.1 MiB/s wr, 296 op/s 2024-09-21T03:01:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:15 smithi029 ceph-mon[31006]: pgmap v1053: 129 pgs: 129 active+clean; 312 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.1 MiB/s wr, 296 op/s 2024-09-21T03:01:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:15 smithi066 ceph-mon[33436]: pgmap v1053: 129 pgs: 129 active+clean; 312 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.1 MiB/s wr, 296 op/s 2024-09-21T03:01:16.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55995 5.61 MB/sec execute 413 sec latency 97.945 ms 2024-09-21T03:01:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 -- 172.21.15.29:0/2535685155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac06c040 msgr2=0x7f47ac10a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 --2- 172.21.15.29:0/2535685155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac06c040 0x7f47ac10a5a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4798009900 tx=0x7f479802a410 comp rx=0 tx=0).stop 2024-09-21T03:01:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 -- 172.21.15.29:0/2535685155 shutdown_connections 2024-09-21T03:01:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 --2- 172.21.15.29:0/2535685155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47ac10ade0 0x7f47ac10b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 --2- 172.21.15.29:0/2535685155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47ac10bf40 0x7f47ac110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 --2- 172.21.15.29:0/2535685155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac06c040 0x7f47ac10a5a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 -- 172.21.15.29:0/2535685155 >> 172.21.15.29:0/2535685155 conn(0x7f47ac070c70 msgr2=0x7f47ac073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 -- 172.21.15.29:0/2535685155 shutdown_connections 2024-09-21T03:01:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.035+0000 7f47b2c25640 1 -- 172.21.15.29:0/2535685155 wait complete. 2024-09-21T03:01:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.036+0000 7f47b2c25640 1 Processor -- start 2024-09-21T03:01:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.036+0000 7f47b2c25640 1 -- start start 2024-09-21T03:01:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b2c25640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47ac06c040 0x7f47ac1a26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b2c25640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47ac10ade0 0x7f47ac1a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b2c25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac10bf40 0x7f47ac1a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b2c25640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1179e0 con 0x7f47ac06c040 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b2c25640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac117860 con 0x7f47ac10ade0 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b2c25640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac117b60 con 0x7f47ac10bf40 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47ac06c040 0x7f47ac1a26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47abfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47ac10ade0 0x7f47ac1a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47ac06c040 0x7f47ac1a26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46810/0 (socket says 172.21.15.29:46810) 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b09c1640 1 -- 172.21.15.29:0/4262517860 learned_addr learned my addr 172.21.15.29:0/4262517860 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b11c2640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac10bf40 0x7f47ac1a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47abfff640 1 -- 172.21.15.29:0/4262517860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac10bf40 msgr2=0x7f47ac1a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47abfff640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac10bf40 0x7f47ac1a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47abfff640 1 -- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47ac06c040 msgr2=0x7f47ac1a26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47abfff640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47ac06c040 0x7f47ac1a26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47abfff640 1 -- 172.21.15.29:0/4262517860 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ac1a4080 con 0x7f47ac10ade0 2024-09-21T03:01:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.037+0000 7f47b09c1640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47ac06c040 0x7f47ac1a26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.038+0000 7f47abfff640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47ac10ade0 0x7f47ac1a2d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f47a000e9c0 tx=0x7f47a000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.038+0000 7f47a9ffb640 1 -- 172.21.15.29:0/4262517860 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a000ccc0 con 0x7f47ac10ade0 2024-09-21T03:01:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.038+0000 7f47a9ffb640 1 -- 172.21.15.29:0/4262517860 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47a00108c0 con 0x7f47ac10ade0 2024-09-21T03:01:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.038+0000 7f47b2c25640 1 -- 172.21.15.29:0/4262517860 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47ac1a4370 con 0x7f47ac10ade0 2024-09-21T03:01:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.038+0000 7f47a9ffb640 1 -- 172.21.15.29:0/4262517860 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0018750 con 0x7f47ac10ade0 2024-09-21T03:01:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.038+0000 7f47b2c25640 1 -- 172.21.15.29:0/4262517860 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47ac110d30 con 0x7f47ac10ade0 2024-09-21T03:01:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.041+0000 7f47a9ffb640 1 -- 172.21.15.29:0/4262517860 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f47a0010430 con 0x7f47ac10ade0 2024-09-21T03:01:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b9c75740 1 Processor -- start 2024-09-21T03:01:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b9c75740 1 -- start start 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b52660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87792f0 0x5614d8b52cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87814d0 0x5614d8b53820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b9c75740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5614d8abc6f0 con 0x5614d87814d0 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b9c75740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5614d8aabb20 con 0x5614d87a21b0 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b9c75740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5614d8aaf930 con 0x5614d87792f0 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87792f0 0x5614d8b52cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87814d0 0x5614d8b53820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87792f0 0x5614d8b52cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49312/0 (socket says 172.21.15.29:49312) 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b11c2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b52660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b09c1640 1 -- 172.21.15.29:0/1716748006 learned_addr learned my addr 172.21.15.29:0/1716748006 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b09c1640 1 -- 172.21.15.29:0/1716748006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 msgr2=0x5614d8b52660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b09c1640 1 --2- 172.21.15.29:0/1716748006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b52660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b09c1640 1 -- 172.21.15.29:0/1716748006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87814d0 msgr2=0x5614d8b53820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.042+0000 7f47b09c1640 1 --2- 172.21.15.29:0/1716748006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87814d0 0x5614d8b53820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.043+0000 7f47b09c1640 1 -- 172.21.15.29:0/1716748006 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614d8a71530 con 0x5614d87792f0 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.043+0000 7f47b09c1640 1 --2- 172.21.15.29:0/1716748006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87792f0 0x5614d8b52cd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4798031a90 tx=0x7f4798005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f30b19066f075ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.043+0000 7f47a9ffb640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f479c0b6610 0x7f479c0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.043+0000 7f4796ffd640 1 -- 172.21.15.29:0/1716748006 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798035070 con 0x5614d87792f0 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.043+0000 7f47a9ffb640 1 -- 172.21.15.29:0/4262517860 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f47a00e7940 con 0x7f47ac10ade0 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.043+0000 7f47b11c2640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f479c0b6610 0x7f479c0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.043+0000 7f4796ffd640 1 -- 172.21.15.29:0/1716748006 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47980044c0 con 0x5614d87792f0 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.043+0000 7f4796ffd640 1 -- 172.21.15.29:0/1716748006 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798044450 con 0x5614d87792f0 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 -- 172.21.15.29:0/1716748006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87792f0 msgr2=0x5614d8b52cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 --2- 172.21.15.29:0/1716748006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87792f0 0x5614d8b52cd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4798031a90 tx=0x7f4798005a20 comp rx=0 tx=0).stop 2024-09-21T03:01:17.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b11c2640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f479c0b6610 0x7f479c0b8b20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f47a4007f80 tx=0x7f47a4007520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 -- 172.21.15.29:0/1716748006 shutdown_connections 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 --2- 172.21.15.29:0/1716748006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87814d0 0x5614d8b53820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 --2- 172.21.15.29:0/1716748006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87792f0 0x5614d8b52cd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 --2- 172.21.15.29:0/1716748006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 -- 172.21.15.29:0/1716748006 >> 172.21.15.29:0/1716748006 conn(0x5614d8796300 msgr2=0x5614d8b458e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 -- 172.21.15.29:0/1716748006 shutdown_connections 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.044+0000 7f47b9c75740 1 -- 172.21.15.29:0/1716748006 wait complete. 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 1 -- start start 2024-09-21T03:01:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 1 Processor -- start 2024-09-21T03:01:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b5bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87814d0 0x5614d8b5c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b5cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5614d89e3ab0 con 0x5614d87792f0 2024-09-21T03:01:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5614d8a51730 con 0x5614d87a21b0 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5614d8b542e0 con 0x5614d87814d0 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b9c75740 10 client.0 _ll_register_callbacks cb 0x5614d8aa56b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b5bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47abfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87814d0 0x5614d8b5c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b5bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46832/0 (socket says 172.21.15.29:46832) 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.045+0000 7f47b09c1640 1 -- 172.21.15.29:0/4009848055 learned_addr learned my addr 172.21.15.29:0/4009848055 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47b11c2640 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b5cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47abfff640 1 -- 172.21.15.29:0/4009848055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 msgr2=0x5614d8b5cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47abfff640 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b5cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47abfff640 1 -- 172.21.15.29:0/4009848055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 msgr2=0x5614d8b5bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47abfff640 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b5bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47abfff640 1 -- 172.21.15.29:0/4009848055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614d880d790 con 0x5614d87814d0 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47b11c2640 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b5cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:01:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47b09c1640 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b5bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47abfff640 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87814d0 0x5614d8b5c420 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f47a00ad4a0 tx=0x7f47a0010ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43669b5e7d2b908d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f478bfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f478bfff640 1 -- 172.21.15.29:0/4009848055 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a00ad570 con 0x5614d87814d0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f478bfff640 1 -- 172.21.15.29:0/4009848055 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f47a00ac470 con 0x5614d87814d0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.046+0000 7f47b9c75740 1 -- 172.21.15.29:0/4009848055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5614d8abe640 con 0x5614d87814d0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f478bfff640 1 -- 172.21.15.29:0/4009848055 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f47a0010720 con 0x5614d87814d0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f47b9c75740 10 client.27811 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f47b9c75740 1 -- 172.21.15.29:0/4009848055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5614d8886620 con 0x5614d87814d0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f478bfff640 1 -- 172.21.15.29:0/4009848055 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5614d8abe640 con 0x5614d87814d0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f478bfff640 1 -- 172.21.15.29:0/4009848055 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f47a00ac670 con 0x5614d87814d0 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f47b9c75740 10 client.27811 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f47b9c75740 10 client.27811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f47b9c75740 20 client.27811 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f47b9c75740 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614d8760d80 0x5614d8b69580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f47b9c75740 4 client.27811 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:01:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.047+0000 7f47b9c75740 1 -- 172.21.15.29:0/4009848055 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614d8aba0f0 con 0x5614d8760d80 2024-09-21T03:01:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.048+0000 7f47b09c1640 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614d8760d80 0x5614d8b69580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.048+0000 7f478bfff640 10 client.27811 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.048+0000 7f47b09c1640 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614d8760d80 0x5614d8b69580 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f7f3ac755dccec8 server_cookie=fd25bd377cbfee58 in_seq=0 out_seq=0 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f478bfff640 1 -- 172.21.15.29:0/4009848055 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5614d8aba0f0 con 0x5614d8760d80 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f478bfff640 10 client.27811 handle_command_reply: tid=0 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 client.27811 shutdown 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 -- 172.21.15.29:0/4009848055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87814d0 msgr2=0x5614d8b5c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87814d0 0x5614d8b5c420 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f47a00ad4a0 tx=0x7f47a0010ca0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 -- 172.21.15.29:0/4009848055 shutdown_connections 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87a21b0 0x5614d8b5cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87814d0 0x5614d8b5c420 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614d8760d80 0x5614d8b69580 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 --2- 172.21.15.29:0/4009848055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b5bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 -- 172.21.15.29:0/4009848055 >> 172.21.15.29:0/4009848055 conn(0x5614d8796300 msgr2=0x5614d8b43ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f478bfff640 0 client.27811 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 -- 172.21.15.29:0/4009848055 shutdown_connections 2024-09-21T03:01:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 1 -- 172.21.15.29:0/4009848055 wait complete. 2024-09-21T03:01:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.049+0000 7f47b9c75740 20 client.27811 trim_cache size 0 max 16384 2024-09-21T03:01:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.071+0000 7f47b9c75740 1 Processor -- start 2024-09-21T03:01:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.071+0000 7f47b9c75740 1 -- start start 2024-09-21T03:01:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.071+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b45220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.071+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b5a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.071+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 0x5614d8b5a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.071+0000 7f47b9c75740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b2b10 con 0x5614d87792f0 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.071+0000 7f47b9c75740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b4650 con 0x5614d87814d0 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b9c75740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b31e0 con 0x5614d87a21b0 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b45220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b45220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46842/0 (socket says 172.21.15.29:46842) 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 -- 172.21.15.29:0/3960969970 learned_addr learned my addr 172.21.15.29:0/3960969970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b11c2640 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 0x5614d8b5a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47abfff640 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b5a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 -- 172.21.15.29:0/3960969970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 msgr2=0x5614d8b5a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 0x5614d8b5a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 -- 172.21.15.29:0/3960969970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 msgr2=0x5614d8b5a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b5a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 -- 172.21.15.29:0/3960969970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614d8b698f0 con 0x5614d87792f0 2024-09-21T03:01:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47abfff640 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b5a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47b09c1640 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b45220 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f4798009a20 tx=0x7f479802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b06c3d2063e12d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.072+0000 7f47967fc640 1 -- 172.21.15.29:0/3960969970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798035040 con 0x5614d87792f0 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.073+0000 7f47967fc640 1 -- 172.21.15.29:0/3960969970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47980041c0 con 0x5614d87792f0 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.073+0000 7f47967fc640 1 -- 172.21.15.29:0/3960969970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798044a30 con 0x5614d87792f0 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.073+0000 7f47b9c75740 1 -- 172.21.15.29:0/3960969970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 msgr2=0x5614d8b45220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.073+0000 7f47b9c75740 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b45220 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f4798009a20 tx=0x7f479802a410 comp rx=0 tx=0).stop 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 -- 172.21.15.29:0/3960969970 shutdown_connections 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 0x5614d8b5a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b5a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 --2- 172.21.15.29:0/3960969970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b45220 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 -- 172.21.15.29:0/3960969970 >> 172.21.15.29:0/3960969970 conn(0x5614d8796300 msgr2=0x5614d8b5af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 -- 172.21.15.29:0/3960969970 shutdown_connections 2024-09-21T03:01:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 -- 172.21.15.29:0/3960969970 wait complete. 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 -- start start 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 Processor -- start 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b61870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 0x5614d8b623e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b3f80 con 0x5614d87792f0 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b38b0 con 0x5614d87814d0 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b0af0 con 0x5614d87a21b0 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b9c75740 10 client.0 _ll_register_callbacks cb 0x7f47a00a9df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47abfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b61870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b09c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b611b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46846/0 (socket says 172.21.15.29:46846) 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b09c1640 1 -- 172.21.15.29:0/161152655 learned_addr learned my addr 172.21.15.29:0/161152655 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.074+0000 7f47b11c2640 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 0x5614d8b623e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47abfff640 1 -- 172.21.15.29:0/161152655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 msgr2=0x5614d8b623e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47abfff640 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 0x5614d8b623e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47abfff640 1 -- 172.21.15.29:0/161152655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 msgr2=0x5614d8b611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47abfff640 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47abfff640 1 -- 172.21.15.29:0/161152655 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614d8b62b40 con 0x5614d87814d0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47b09c1640 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47abfff640 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b61870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f47a00b65b0 tx=0x7f47a00a9530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc3dfd1ca28811f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f478bfff640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f478bfff640 1 -- 172.21.15.29:0/161152655 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a00acf10 con 0x5614d87814d0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47b9c75740 1 -- 172.21.15.29:0/161152655 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47980041c0 con 0x5614d87814d0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f478bfff640 1 -- 172.21.15.29:0/161152655 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f47a00b3360 con 0x5614d87814d0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f478bfff640 1 -- 172.21.15.29:0/161152655 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f47a00ad0f0 con 0x5614d87814d0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47b9c75740 10 client.27653 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.075+0000 7f47b9c75740 1 -- 172.21.15.29:0/161152655 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5614d8b63ea0 con 0x5614d87814d0 2024-09-21T03:01:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f478bfff640 1 -- 172.21.15.29:0/161152655 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47980041c0 con 0x5614d87814d0 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f478bfff640 1 -- 172.21.15.29:0/161152655 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f47a00b3560 con 0x5614d87814d0 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f47b9c75740 10 client.27653 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f47b9c75740 10 client.27653 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f47b9c75740 20 client.27653 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f47b9c75740 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614d87361c0 0x5614d8bd6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f47b9c75740 4 client.27653 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f47b9c75740 1 -- 172.21.15.29:0/161152655 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614d8ab7520 con 0x5614d87361c0 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f47b09c1640 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614d87361c0 0x5614d8bd6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f47b09c1640 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614d87361c0 0x5614d8bd6240 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5102c8f6d7a5d618 server_cookie=bafc13dd8b0ae384 in_seq=0 out_seq=0 2024-09-21T03:01:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.076+0000 7f478bfff640 10 client.27653 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f478bfff640 1 -- 172.21.15.29:0/161152655 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614d8ab7520 con 0x5614d87361c0 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f478bfff640 10 client.27653 handle_command_reply: tid=0 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 client.27653 shutdown 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 -- 172.21.15.29:0/161152655 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 msgr2=0x5614d8b61870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b61870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f47a00b65b0 tx=0x7f47a00a9530 comp rx=0 tx=0).stop 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 -- 172.21.15.29:0/161152655 shutdown_connections 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614d87a21b0 0x5614d8b623e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5614d87814d0 0x5614d8b61870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5614d87361c0 0x5614d8bd6240 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 --2- 172.21.15.29:0/161152655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614d87792f0 0x5614d8b611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f478bfff640 0 client.27653 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 -- 172.21.15.29:0/161152655 >> 172.21.15.29:0/161152655 conn(0x5614d8796300 msgr2=0x5614d8b52600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 -- 172.21.15.29:0/161152655 shutdown_connections 2024-09-21T03:01:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 1 -- 172.21.15.29:0/161152655 wait complete. 2024-09-21T03:01:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.334+0000 7f47b9c75740 20 client.27653 trim_cache size 0 max 16384 2024-09-21T03:01:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:01:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:01:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:01:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:01:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 -- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f479c0b6610 msgr2=0x7f479c0b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f479c0b6610 0x7f479c0b8b20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f47a4007f80 tx=0x7f47a4007520 comp rx=0 tx=0).stop 2024-09-21T03:01:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 -- 172.21.15.29:0/4262517860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47ac10ade0 msgr2=0x7f47ac1a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47ac10ade0 0x7f47ac1a2d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f47a000e9c0 tx=0x7f47a000ee80 comp rx=0 tx=0).stop 2024-09-21T03:01:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 -- 172.21.15.29:0/4262517860 shutdown_connections 2024-09-21T03:01:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f479c0b6610 0x7f479c0b8b20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac10bf40 0x7f47ac1a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47ac10ade0 0x7f47ac1a2d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 --2- 172.21.15.29:0/4262517860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47ac06c040 0x7f47ac1a26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 -- 172.21.15.29:0/4262517860 >> 172.21.15.29:0/4262517860 conn(0x7f47ac070c70 msgr2=0x7f47ac071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 -- 172.21.15.29:0/4262517860 shutdown_connections 2024-09-21T03:01:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:17.335+0000 7f47b2c25640 1 -- 172.21.15.29:0/4262517860 wait complete. 2024-09-21T03:01:17.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56160 5.63 MB/sec execute 414 sec latency 218.344 ms 2024-09-21T03:01:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:17 smithi190 ceph-mon[33871]: pgmap v1054: 129 pgs: 129 active+clean; 314 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.9 MiB/s wr, 256 op/s 2024-09-21T03:01:17.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:17 smithi029 ceph-mon[31006]: pgmap v1054: 129 pgs: 129 active+clean; 314 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.9 MiB/s wr, 256 op/s 2024-09-21T03:01:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:17 smithi066 ceph-mon[33436]: pgmap v1054: 129 pgs: 129 active+clean; 314 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.9 MiB/s wr, 256 op/s 2024-09-21T03:01:18.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56273 5.63 MB/sec execute 415 sec latency 132.984 ms 2024-09-21T03:01:19.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56406 5.63 MB/sec execute 416 sec latency 82.029 ms 2024-09-21T03:01:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:19 smithi190 ceph-mon[33871]: pgmap v1055: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 3.2 MiB/s wr, 271 op/s 2024-09-21T03:01:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:19 smithi029 ceph-mon[31006]: pgmap v1055: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 3.2 MiB/s wr, 271 op/s 2024-09-21T03:01:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:19 smithi066 ceph-mon[33436]: pgmap v1055: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 3.2 MiB/s wr, 271 op/s 2024-09-21T03:01:20.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56540 5.64 MB/sec execute 417 sec latency 96.349 ms 2024-09-21T03:01:21.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56709 5.66 MB/sec execute 418 sec latency 107.846 ms 2024-09-21T03:01:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:21 smithi190 ceph-mon[33871]: pgmap v1056: 129 pgs: 129 active+clean; 323 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-09-21T03:01:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:21 smithi029 ceph-mon[31006]: pgmap v1056: 129 pgs: 129 active+clean; 323 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-09-21T03:01:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:21 smithi066 ceph-mon[33436]: pgmap v1056: 129 pgs: 129 active+clean; 323 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-09-21T03:01:22.740 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56864 5.67 MB/sec execute 419 sec latency 111.916 ms 2024-09-21T03:01:23.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56966 5.67 MB/sec execute 420 sec latency 427.958 ms 2024-09-21T03:01:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:23 smithi190 ceph-mon[33871]: pgmap v1057: 129 pgs: 129 active+clean; 323 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-09-21T03:01:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:23 smithi029 ceph-mon[31006]: pgmap v1057: 129 pgs: 129 active+clean; 323 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-09-21T03:01:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:23 smithi066 ceph-mon[33436]: pgmap v1057: 129 pgs: 129 active+clean; 323 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-09-21T03:01:24.579 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.8 2024-09-21T03:01:24.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:01:24.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57102 5.68 MB/sec execute 421 sec latency 90.549 ms 2024-09-21T03:01:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:01:25.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:25 smithi029 ceph-mon[31006]: pgmap v1058: 129 pgs: 129 active+clean; 328 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 4.5 MiB/s wr, 281 op/s 2024-09-21T03:01:25.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57181 5.67 MB/sec execute 422 sec latency 136.990 ms 2024-09-21T03:01:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:25 smithi190 ceph-mon[33871]: pgmap v1058: 129 pgs: 129 active+clean; 328 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 4.5 MiB/s wr, 281 op/s 2024-09-21T03:01:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:25 smithi066 ceph-mon[33436]: pgmap v1058: 129 pgs: 129 active+clean; 328 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 4.5 MiB/s wr, 281 op/s 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 -- 172.21.15.29:0/4143235681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa934108cf0 msgr2=0x7fa93410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 --2- 172.21.15.29:0/4143235681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa934108cf0 0x7fa93410f520 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009ca0 tx=0x7fa92c02a8c0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 -- 172.21.15.29:0/4143235681 shutdown_connections 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 --2- 172.21.15.29:0/4143235681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa934108cf0 0x7fa93410f520 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 --2- 172.21.15.29:0/4143235681 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9341083a0 0x7fa9341087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 --2- 172.21.15.29:0/4143235681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa934102a60 0x7fa934102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 -- 172.21.15.29:0/4143235681 >> 172.21.15.29:0/4143235681 conn(0x7fa934079730 msgr2=0x7fa934079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 -- 172.21.15.29:0/4143235681 shutdown_connections 2024-09-21T03:01:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.113+0000 7fa93b1da640 1 -- 172.21.15.29:0/4143235681 wait complete. 2024-09-21T03:01:26.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.114+0000 7fa93b1da640 1 Processor -- start 2024-09-21T03:01:26.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.114+0000 7fa93b1da640 1 -- start start 2024-09-21T03:01:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa93b1da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa934102a60 0x7fa93419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa93b1da640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9341083a0 0x7fa93419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa93b1da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa934108cf0 0x7fa93419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa93b1da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa934113960 con 0x7fa934102a60 2024-09-21T03:01:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa93b1da640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341137e0 con 0x7fa9341083a0 2024-09-21T03:01:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa93b1da640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa934113ae0 con 0x7fa934108cf0 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa933fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9341083a0 0x7fa93419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa933fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9341083a0 0x7fa93419a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:39086/0 (socket says 172.21.15.29:39086) 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa933fff640 1 -- 172.21.15.29:0/2921231378 learned_addr learned my addr 172.21.15.29:0/2921231378 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa938f76640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa934102a60 0x7fa93419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.115+0000 7fa939777640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa934108cf0 0x7fa93419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.116+0000 7fa933fff640 1 -- 172.21.15.29:0/2921231378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa934108cf0 msgr2=0x7fa93419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.116+0000 7fa933fff640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa934108cf0 0x7fa93419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.116+0000 7fa933fff640 1 -- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa934102a60 msgr2=0x7fa93419a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.116+0000 7fa933fff640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa934102a60 0x7fa93419a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.116+0000 7fa933fff640 1 -- 172.21.15.29:0/2921231378 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9341a0280 con 0x7fa9341083a0 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.116+0000 7fa933fff640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9341083a0 0x7fa93419a8b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa92000e9c0 tx=0x7fa92000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.116+0000 7fa931ffb640 1 -- 172.21.15.29:0/2921231378 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92000cd80 con 0x7fa9341083a0 2024-09-21T03:01:26.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.117+0000 7fa931ffb640 1 -- 172.21.15.29:0/2921231378 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9200054b0 con 0x7fa9341083a0 2024-09-21T03:01:26.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.117+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9341a0570 con 0x7fa9341083a0 2024-09-21T03:01:26.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.117+0000 7fa931ffb640 1 -- 172.21.15.29:0/2921231378 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920010430 con 0x7fa9341083a0 2024-09-21T03:01:26.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.117+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9341a6cd0 con 0x7fa9341083a0 2024-09-21T03:01:26.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.118+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa934102e70 con 0x7fa9341083a0 2024-09-21T03:01:26.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.121+0000 7fa931ffb640 1 -- 172.21.15.29:0/2921231378 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fa920004f20 con 0x7fa9341083a0 2024-09-21T03:01:26.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.122+0000 7fa931ffb640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9240b6590 0x7fa9240b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:26.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.122+0000 7fa938f76640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9240b6590 0x7fa9240b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:26.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.122+0000 7fa931ffb640 1 -- 172.21.15.29:0/2921231378 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fa9200e7be0 con 0x7fa9341083a0 2024-09-21T03:01:26.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.123+0000 7fa938f76640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9240b6590 0x7fa9240b8aa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa928009bd0 tx=0x7fa928005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.127+0000 7fa931ffb640 1 -- 172.21.15.29:0/2921231378 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa9200ae0f0 con 0x7fa9341083a0 2024-09-21T03:01:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.479+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa93402e640 con 0x7fa9341083a0 2024-09-21T03:01:26.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.480+0000 7fa931ffb640 1 -- 172.21.15.29:0/2921231378 <== mon.2 v2:172.21.15.66:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7fa9200b24d0 con 0x7fa9341083a0 2024-09-21T03:01:26.482 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:01:26.482 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:01:26.483 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T03:01:26.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9240b6590 msgr2=0x7fa9240b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:26.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9240b6590 0x7fa9240b8aa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa928009bd0 tx=0x7fa928005f90 comp rx=0 tx=0).stop 2024-09-21T03:01:26.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9341083a0 msgr2=0x7fa93419a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9341083a0 0x7fa93419a8b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa92000e9c0 tx=0x7fa92000ee80 comp rx=0 tx=0).stop 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 shutdown_connections 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fa9240b6590 0x7fa9240b8aa0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa934108cf0 0x7fa93419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9341083a0 0x7fa93419a8b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 --2- 172.21.15.29:0/2921231378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa934102a60 0x7fa93419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 >> 172.21.15.29:0/2921231378 conn(0x7fa934079730 msgr2=0x7fa9341043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 shutdown_connections 2024-09-21T03:01:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:26.483+0000 7fa93b1da640 1 -- 172.21.15.29:0/2921231378 wait complete. 2024-09-21T03:01:26.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57313 5.68 MB/sec execute 423 sec latency 117.670 ms 2024-09-21T03:01:26.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:26 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2921231378' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:01:26.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:26 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2921231378' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:01:27.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:26 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2921231378' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:01:27.370 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 149.8 --await-for 45.0' 2024-09-21T03:01:27.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 149.8 --await-for 45.0 2024-09-21T03:01:27.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:27 smithi029 ceph-mon[31006]: pgmap v1059: 129 pgs: 129 active+clean; 330 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 4.2 MiB/s wr, 266 op/s 2024-09-21T03:01:27.726 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:01:27.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57448 5.68 MB/sec execute 424 sec latency 140.258 ms 2024-09-21T03:01:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:27 smithi190 ceph-mon[33871]: pgmap v1059: 129 pgs: 129 active+clean; 330 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 4.2 MiB/s wr, 266 op/s 2024-09-21T03:01:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:27 smithi066 ceph-mon[33436]: pgmap v1059: 129 pgs: 129 active+clean; 330 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 4.2 MiB/s wr, 266 op/s 2024-09-21T03:01:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 -- 172.21.15.29:0/1627881572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e4010de30 msgr2=0x7f8e40110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:28.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 --2- 172.21.15.29:0/1627881572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e4010de30 0x7f8e40110320 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009ca0 tx=0x7f8e3802a8c0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 -- 172.21.15.29:0/1627881572 shutdown_connections 2024-09-21T03:01:28.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 --2- 172.21.15.29:0/1627881572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e4010de30 0x7f8e40110320 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 --2- 172.21.15.29:0/1627881572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e40105320 0x7f8e4010d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 --2- 172.21.15.29:0/1627881572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e401049d0 0x7f8e40104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 -- 172.21.15.29:0/1627881572 >> 172.21.15.29:0/1627881572 conn(0x7f8e400797c0 msgr2=0x7f8e40079bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:28.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 -- 172.21.15.29:0/1627881572 shutdown_connections 2024-09-21T03:01:28.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.628+0000 7f8e4863e640 1 -- 172.21.15.29:0/1627881572 wait complete. 2024-09-21T03:01:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.629+0000 7f8e4863e640 1 Processor -- start 2024-09-21T03:01:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.629+0000 7f8e4863e640 1 -- start start 2024-09-21T03:01:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e4863e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e401049d0 0x7f8e4019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e4863e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e40105320 0x7f8e4019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e4863e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e4010de30 0x7f8e4019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e4863e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8e401139b0 con 0x7f8e4010de30 2024-09-21T03:01:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e4863e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8e40113830 con 0x7f8e40105320 2024-09-21T03:01:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e4863e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e40113b30 con 0x7f8e401049d0 2024-09-21T03:01:28.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e463da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e401049d0 0x7f8e4019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e45bd9640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e40105320 0x7f8e4019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e463da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e401049d0 0x7f8e4019a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41872/0 (socket says 172.21.15.29:41872) 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e463da640 1 -- 172.21.15.29:0/799588367 learned_addr learned my addr 172.21.15.29:0/799588367 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e4010de30 0x7f8e4019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e463da640 1 -- 172.21.15.29:0/799588367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e40105320 msgr2=0x7f8e4019a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e463da640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e40105320 0x7f8e4019a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e463da640 1 -- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e4010de30 msgr2=0x7f8e4019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e463da640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e4010de30 0x7f8e4019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e463da640 1 -- 172.21.15.29:0/799588367 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e401a0350 con 0x7f8e401049d0 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.630+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e40105320 0x7f8e4019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.631+0000 7f8e463da640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e401049d0 0x7f8e4019a3e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00d950 tx=0x7f8e3c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.631+0000 7f8e377fe640 1 -- 172.21.15.29:0/799588367 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c017030 con 0x7f8e401049d0 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.631+0000 7f8e4863e640 1 -- 172.21.15.29:0/799588367 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e401a0640 con 0x7f8e401049d0 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.631+0000 7f8e4863e640 1 -- 172.21.15.29:0/799588367 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e401025f0 con 0x7f8e401049d0 2024-09-21T03:01:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.631+0000 7f8e377fe640 1 -- 172.21.15.29:0/799588367 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e3c004020 con 0x7f8e401049d0 2024-09-21T03:01:28.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.631+0000 7f8e377fe640 1 -- 172.21.15.29:0/799588367 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c0055d0 con 0x7f8e401049d0 2024-09-21T03:01:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.634+0000 7f8e4ebb2740 1 Processor -- start 2024-09-21T03:01:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.634+0000 7f8e4ebb2740 1 -- start start 2024-09-21T03:01:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.634+0000 7f8e377fe640 1 -- 172.21.15.29:0/799588367 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f8e3c0047a0 con 0x7f8e401049d0 2024-09-21T03:01:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0e20840 0x55fbd11cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fbd113db90 con 0x55fbd0dffb60 2024-09-21T03:01:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55fbd112cfc0 con 0x55fbd0df7980 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55fbd1130dd0 con 0x55fbd0e20840 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e46bdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0e20840 0x55fbd11cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e45bd9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e46bdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0e20840 0x55fbd11cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41888/0 (socket says 172.21.15.29:41888) 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e46bdb640 1 -- 172.21.15.29:0/1160846495 learned_addr learned my addr 172.21.15.29:0/1160846495 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e463da640 1 --2- 172.21.15.29:0/1160846495 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e46bdb640 1 -- 172.21.15.29:0/1160846495 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 msgr2=0x55fbd11d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/1160846495 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e46bdb640 1 -- 172.21.15.29:0/1160846495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 msgr2=0x55fbd11d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/1160846495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.635+0000 7f8e46bdb640 1 -- 172.21.15.29:0/1160846495 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbd10b4060 con 0x55fbd0e20840 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.636+0000 7f8e377fe640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e2c0b6610 0x7f8e2c0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.636+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/1160846495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0e20840 0x55fbd11cfe10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009ca0 tx=0x7f8e38009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12faa29cbda9ad81 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:28.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.636+0000 7f8e377fe640 1 -- 172.21.15.29:0/799588367 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e3c0e6e10 con 0x7f8e401049d0 2024-09-21T03:01:28.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.636+0000 7f8e34ff9640 1 -- 172.21.15.29:0/1160846495 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e38031980 con 0x55fbd0e20840 2024-09-21T03:01:28.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.636+0000 7f8e34ff9640 1 -- 172.21.15.29:0/1160846495 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e38036030 con 0x55fbd0e20840 2024-09-21T03:01:28.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.636+0000 7f8e34ff9640 1 -- 172.21.15.29:0/1160846495 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e38004460 con 0x55fbd0e20840 2024-09-21T03:01:28.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.636+0000 7f8e463da640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e2c0b6610 0x7f8e2c0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1160846495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0e20840 msgr2=0x55fbd11cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1160846495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0e20840 0x55fbd11cfe10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009ca0 tx=0x7f8e38009a20 comp rx=0 tx=0).stop 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e463da640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e2c0b6610 0x7f8e2c0b8b20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c005fd0 tx=0x7f8e3c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1160846495 shutdown_connections 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1160846495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1160846495 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1160846495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0e20840 0x55fbd11cfe10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1160846495 >> 172.21.15.29:0/1160846495 conn(0x55fbd0e14990 msgr2=0x55fbd11cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1160846495 shutdown_connections 2024-09-21T03:01:28.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.637+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1160846495 wait complete. 2024-09-21T03:01:28.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.638+0000 7f8e4ebb2740 1 -- start start 2024-09-21T03:01:28.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.638+0000 7f8e4ebb2740 1 Processor -- start 2024-09-21T03:01:28.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0df7980 0x55fbd11db990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11dbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0e20840 0x55fbd11e7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fbd0f8e7a0 con 0x55fbd0dffb60 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55fbd10d20a0 con 0x55fbd0e20840 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55fbd11de410 con 0x55fbd0df7980 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0df7980 0x55fbd11db990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e45bd9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11dbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0df7980 0x55fbd11db990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41890/0 (socket says 172.21.15.29:41890) 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 -- 172.21.15.29:0/2877237453 learned_addr learned my addr 172.21.15.29:0/2877237453 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:28.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e4ebb2740 10 client.0 _ll_register_callbacks cb 0x55fbd10af3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0e20840 0x55fbd11e7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 -- 172.21.15.29:0/2877237453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0e20840 msgr2=0x55fbd11e7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0e20840 0x55fbd11e7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 -- 172.21.15.29:0/2877237453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 msgr2=0x55fbd11dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11dbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e46bdb640 1 -- 172.21.15.29:0/2877237453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbd10c0880 con 0x55fbd0df7980 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.639+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11dbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.640+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0df7980 0x55fbd11db990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8e3802ade0 tx=0x7f8e380094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=502201cc10c141c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.640+0000 7f8e29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.640+0000 7f8e29ffb640 1 -- 172.21.15.29:0/2877237453 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e38058060 con 0x55fbd0df7980 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.640+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/2877237453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e38036030 con 0x55fbd0df7980 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.640+0000 7f8e29ffb640 1 -- 172.21.15.29:0/2877237453 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e380041e0 con 0x55fbd0df7980 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.640+0000 7f8e29ffb640 1 -- 172.21.15.29:0/2877237453 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8e3805f050 con 0x55fbd0df7980 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.640+0000 7f8e4ebb2740 10 client.27835 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:28.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.640+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/2877237453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fbd0e9ab90 con 0x55fbd0df7980 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e29ffb640 1 -- 172.21.15.29:0/2877237453 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e38036030 con 0x55fbd0df7980 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e29ffb640 1 -- 172.21.15.29:0/2877237453 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f8e38050070 con 0x55fbd0df7980 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e4ebb2740 10 client.27835 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e4ebb2740 10 client.27835 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e4ebb2740 20 client.27835 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fbd0ddf1c0 0x55fbd11ed8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e4ebb2740 4 client.27835 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/2877237453 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbd113fae0 con 0x55fbd0ddf1c0 2024-09-21T03:01:28.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.641+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fbd0ddf1c0 0x55fbd11ed8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.642+0000 7f8e29ffb640 10 client.27835 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:28.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.642+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fbd0ddf1c0 0x55fbd11ed8c0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e8f0f54ea3444c0 server_cookie=f0843becb0e720b5 in_seq=0 out_seq=0 2024-09-21T03:01:28.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.643+0000 7f8e29ffb640 1 -- 172.21.15.29:0/2877237453 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fbd113fae0 con 0x55fbd0ddf1c0 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.643+0000 7f8e29ffb640 10 client.27835 handle_command_reply: tid=0 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.643+0000 7f8e4ebb2740 1 client.27835 shutdown 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/2877237453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0df7980 msgr2=0x55fbd11db990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0df7980 0x55fbd11db990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8e3802ade0 tx=0x7f8e380094a0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/2877237453 shutdown_connections 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0e20840 0x55fbd11e7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fbd0ddf1c0 0x55fbd11ed8c0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0dffb60 0x55fbd11dbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/2877237453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0df7980 0x55fbd11db990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/2877237453 >> 172.21.15.29:0/2877237453 conn(0x55fbd0e14990 msgr2=0x55fbd11d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e29ffb640 0 client.27835 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/2877237453 shutdown_connections 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/2877237453 wait complete. 2024-09-21T03:01:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.644+0000 7f8e4ebb2740 20 client.27835 trim_cache size 0 max 16384 2024-09-21T03:01:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.676+0000 7f8e4ebb2740 1 Processor -- start 2024-09-21T03:01:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.676+0000 7f8e4ebb2740 1 -- start start 2024-09-21T03:01:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.676+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11c4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.676+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11dd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.676+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 0x55fbd11e7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.676+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2400fb00 con 0x55fbd0e20840 2024-09-21T03:01:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8e24011640 con 0x55fbd0df7980 2024-09-21T03:01:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e240101d0 con 0x55fbd0dffb60 2024-09-21T03:01:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e46bdb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e46bdb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11c4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:39128/0 (socket says 172.21.15.29:39128) 2024-09-21T03:01:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e46bdb640 1 -- 172.21.15.29:0/1467290728 learned_addr learned my addr 172.21.15.29:0/1467290728 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 0x55fbd11e7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11dd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e45bd9640 1 -- 172.21.15.29:0/1467290728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 msgr2=0x55fbd11c4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11c4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e45bd9640 1 -- 172.21.15.29:0/1467290728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 msgr2=0x55fbd11e7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 0x55fbd11e7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.677+0000 7f8e45bd9640 1 -- 172.21.15.29:0/1467290728 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbd11d2330 con 0x55fbd0dffb60 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.678+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11dd3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55fbd11edfc0 tx=0x7f8e30005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a41192496af25995 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.678+0000 7f8e2bfff640 1 -- 172.21.15.29:0/1467290728 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30028030 con 0x55fbd0dffb60 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.678+0000 7f8e2bfff640 1 -- 172.21.15.29:0/1467290728 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e30024450 con 0x55fbd0dffb60 2024-09-21T03:01:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.678+0000 7f8e2bfff640 1 -- 172.21.15.29:0/1467290728 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30023630 con 0x55fbd0dffb60 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.679+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1467290728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 msgr2=0x55fbd11dd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11dd3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55fbd11edfc0 tx=0x7f8e30005f90 comp rx=0 tx=0).stop 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1467290728 shutdown_connections 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 0x55fbd11e7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11dd3c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1467290728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11c4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1467290728 >> 172.21.15.29:0/1467290728 conn(0x55fbd0e14990 msgr2=0x55fbd11dccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1467290728 shutdown_connections 2024-09-21T03:01:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1467290728 wait complete. 2024-09-21T03:01:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 -- start start 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.680+0000 7f8e4ebb2740 1 Processor -- start 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11e3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e4ebb2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 0x55fbd11de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8e24010f70 con 0x55fbd0e20840 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f8e240108a0 con 0x55fbd0df7980 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e4ebb2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2400dae0 con 0x55fbd0dffb60 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e46bdb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11e3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e45bd9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e46bdb640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11e3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:39142/0 (socket says 172.21.15.29:39142) 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e46bdb640 1 -- 172.21.15.29:0/1515587345 learned_addr learned my addr 172.21.15.29:0/1515587345 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 0x55fbd11de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.681+0000 7f8e4ebb2740 10 client.0 _ll_register_callbacks cb 0x7f8e30010ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.682+0000 7f8e45bd9640 1 -- 172.21.15.29:0/1515587345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 msgr2=0x55fbd11e3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.682+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11e3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.682+0000 7f8e45bd9640 1 -- 172.21.15.29:0/1515587345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 msgr2=0x55fbd11de6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.682+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 0x55fbd11de6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.682+0000 7f8e45bd9640 1 -- 172.21.15.29:0/1515587345 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbd11dec70 con 0x55fbd0dffb60 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.682+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11e3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.682+0000 7f8e45bd9640 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11e3e40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55fbd11edfc0 tx=0x7f8e300233b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e20a7f63363c359d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.682+0000 7f8e29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.683+0000 7f8e29ffb640 1 -- 172.21.15.29:0/1515587345 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30031030 con 0x55fbd0dffb60 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.683+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1515587345 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e30024450 con 0x55fbd0dffb60 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.683+0000 7f8e29ffb640 1 -- 172.21.15.29:0/1515587345 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f8e3001dcc0 con 0x55fbd0dffb60 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.683+0000 7f8e29ffb640 1 -- 172.21.15.29:0/1515587345 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8e3003e050 con 0x55fbd0dffb60 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.683+0000 7f8e4ebb2740 10 client.27841 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.683+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1515587345 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fbd1258a90 con 0x55fbd0dffb60 2024-09-21T03:01:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.683+0000 7f8e29ffb640 1 -- 172.21.15.29:0/1515587345 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e30024450 con 0x55fbd0dffb60 2024-09-21T03:01:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.684+0000 7f8e29ffb640 1 -- 172.21.15.29:0/1515587345 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f8e300049e0 con 0x55fbd0dffb60 2024-09-21T03:01:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.684+0000 7f8e4ebb2740 10 client.27841 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.684+0000 7f8e4ebb2740 10 client.27841 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:01:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.684+0000 7f8e4ebb2740 20 client.27841 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.684+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fbd0db4570 0x55fbd125b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.684+0000 7f8e4ebb2740 4 client.27841 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.8, "await_for": 45.0}] 2024-09-21T03:01:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.684+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1515587345 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.8, "await_for": 45.0}) -- 0x55fbd113b590 con 0x55fbd0db4570 2024-09-21T03:01:28.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.685+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fbd0db4570 0x55fbd125b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:28.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.685+0000 7f8e29ffb640 10 client.27841 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:28.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:28.685+0000 7f8e46bdb640 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fbd0db4570 0x55fbd125b1f0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43ecfe13e886b868 server_cookie=6b761a2a7f890642 in_seq=0 out_seq=0 2024-09-21T03:01:28.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57575 5.68 MB/sec execute 425 sec latency 131.544 ms 2024-09-21T03:01:29.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.196+0000 7f8e29ffb640 1 -- 172.21.15.29:0/1515587345 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fbd113b590 con 0x55fbd0db4570 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.196+0000 7f8e29ffb640 10 client.27841 handle_command_reply: tid=0 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.196+0000 7f8e4ebb2740 1 client.27841 shutdown 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1515587345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 msgr2=0x55fbd11e3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11e3e40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55fbd11edfc0 tx=0x7f8e300233b0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1515587345 shutdown_connections 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fbd0db4570 0x55fbd125b1f0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fbd0e20840 0x55fbd11de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbd0dffb60 0x55fbd11e3e40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e29ffb640 0 client.27841 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 --2- 172.21.15.29:0/1515587345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fbd0df7980 0x55fbd11e3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1515587345 >> 172.21.15.29:0/1515587345 conn(0x55fbd0e14990 msgr2=0x55fbd11d38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1515587345 shutdown_connections 2024-09-21T03:01:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 1 -- 172.21.15.29:0/1515587345 wait complete. 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.197+0000 7f8e4ebb2740 20 client.27841 trim_cache size 0 max 16384 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 86, 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "955799b9": { 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 86, 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:01:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 149.8, 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:29.203 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:01:29.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 -- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e2c0b6610 msgr2=0x7f8e2c0b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:29.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e2c0b6610 0x7f8e2c0b8b20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c005fd0 tx=0x7f8e3c000bc0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 -- 172.21.15.29:0/799588367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e401049d0 msgr2=0x7f8e4019a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e401049d0 0x7f8e4019a3e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00d950 tx=0x7f8e3c00de10 comp rx=0 tx=0).stop 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 -- 172.21.15.29:0/799588367 shutdown_connections 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f8e2c0b6610 0x7f8e2c0b8b20 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e4010de30 0x7f8e4019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e40105320 0x7f8e4019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 --2- 172.21.15.29:0/799588367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e401049d0 0x7f8e4019a3e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 -- 172.21.15.29:0/799588367 >> 172.21.15.29:0/799588367 conn(0x7f8e400797c0 msgr2=0x7f8e40106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 -- 172.21.15.29:0/799588367 shutdown_connections 2024-09-21T03:01:29.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:29.198+0000 7f8e4863e640 1 -- 172.21.15.29:0/799588367 wait complete. 2024-09-21T03:01:29.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.66 MB/sec execute 426 sec latency 990.042 ms 2024-09-21T03:01:29.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:29 smithi066 ceph-mon[33436]: pgmap v1060: 129 pgs: 129 active+clean; 331 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 4.3 MiB/s wr, 284 op/s 2024-09-21T03:01:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:29 smithi190 ceph-mon[33871]: pgmap v1060: 129 pgs: 129 active+clean; 331 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 4.3 MiB/s wr, 284 op/s 2024-09-21T03:01:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:29 smithi029 ceph-mon[31006]: pgmap v1060: 129 pgs: 129 active+clean; 331 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 4.3 MiB/s wr, 284 op/s 2024-09-21T03:01:30.003 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '955799b9', quiesce took 5.4 seconds. Will release after: 24.4 2024-09-21T03:01:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.451+0000 7ff3f802e640 1 -- 172.21.15.29:0/3026601609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f01025a0 msgr2=0x7ff3f0102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.451+0000 7ff3f802e640 1 --2- 172.21.15.29:0/3026601609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f01025a0 0x7ff3f0102a10 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009900 tx=0x7ff3ec02a410 comp rx=0 tx=0).stop 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.451+0000 7ff3f802e640 1 -- 172.21.15.29:0/3026601609 shutdown_connections 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.452+0000 7ff3f802e640 1 --2- 172.21.15.29:0/3026601609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f0103fc0 0x7ff3f010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.452+0000 7ff3f802e640 1 --2- 172.21.15.29:0/3026601609 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3f0103620 0x7ff3f0103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.452+0000 7ff3f802e640 1 --2- 172.21.15.29:0/3026601609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f01025a0 0x7ff3f0102a10 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.452+0000 7ff3f802e640 1 -- 172.21.15.29:0/3026601609 >> 172.21.15.29:0/3026601609 conn(0x7ff3f00fdee0 msgr2=0x7ff3f0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.452+0000 7ff3f802e640 1 -- 172.21.15.29:0/3026601609 shutdown_connections 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.452+0000 7ff3f802e640 1 -- 172.21.15.29:0/3026601609 wait complete. 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.452+0000 7ff3f802e640 1 Processor -- start 2024-09-21T03:01:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.452+0000 7ff3f802e640 1 -- start start 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f802e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f01025a0 0x7ff3f006b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f802e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3f0103620 0x7ff3f019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f802e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f0103fc0 0x7ff3f01a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f802e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f010e0e0 con 0x7ff3f0103fc0 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f802e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f010df60 con 0x7ff3f0103620 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f802e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f010e260 con 0x7ff3f01025a0 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f0103fc0 0x7ff3f01a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f0103fc0 0x7ff3f01a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59254/0 (socket says 172.21.15.29:59254) 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 -- 172.21.15.29:0/2954625357 learned_addr learned my addr 172.21.15.29:0/2954625357 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3f0103620 0x7ff3f019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f01025a0 0x7ff3f006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 -- 172.21.15.29:0/2954625357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f01025a0 msgr2=0x7ff3f006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f01025a0 0x7ff3f006b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 -- 172.21.15.29:0/2954625357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3f0103620 msgr2=0x7ff3f019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3f0103620 0x7ff3f019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 -- 172.21.15.29:0/2954625357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f01a1950 con 0x7ff3f0103fc0 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3f65cb640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f0103fc0 0x7ff3f01a1150 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800eb40 tx=0x7ff3e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3e6ffd640 1 -- 172.21.15.29:0/2954625357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8010430 con 0x7ff3f0103fc0 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.453+0000 7ff3e6ffd640 1 -- 172.21.15.29:0/2954625357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3e8010a60 con 0x7ff3f0103fc0 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.454+0000 7ff3f802e640 1 -- 172.21.15.29:0/2954625357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f01a1be0 con 0x7ff3f0103fc0 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.454+0000 7ff3e6ffd640 1 -- 172.21.15.29:0/2954625357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8018ac0 con 0x7ff3f0103fc0 2024-09-21T03:01:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.454+0000 7ff3f802e640 1 -- 172.21.15.29:0/2954625357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f01a2130 con 0x7ff3f0103fc0 2024-09-21T03:01:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.456+0000 7ff3ff076740 1 Processor -- start 2024-09-21T03:01:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.456+0000 7ff3e6ffd640 1 -- 172.21.15.29:0/2954625357 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff3e8010c10 con 0x7ff3f0103fc0 2024-09-21T03:01:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3ff076740 1 -- start start 2024-09-21T03:01:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 0x5640162d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f012f0 0x5640162d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f094d0 0x5640162d3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3ff076740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5640162446f0 con 0x564015f094d0 2024-09-21T03:01:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3ff076740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564016233b20 con 0x564015f2a1b0 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3ff076740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564016237930 con 0x564015f012f0 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f094d0 0x5640162d3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f094d0 0x5640162d3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59268/0 (socket says 172.21.15.29:59268) 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 -- 172.21.15.29:0/1258785618 learned_addr learned my addr 172.21.15.29:0/1258785618 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 0x5640162d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f012f0 0x5640162d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3e6ffd640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3dc0b6510 0x7ff3dc0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 -- 172.21.15.29:0/1258785618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f012f0 msgr2=0x5640162d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f012f0 0x5640162d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 -- 172.21.15.29:0/1258785618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 msgr2=0x5640162d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 0x5640162d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.457+0000 7ff3f55c9640 1 -- 172.21.15.29:0/1258785618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640161f9530 con 0x564015f094d0 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.458+0000 7ff3f65cb640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3dc0b6510 0x7ff3dc0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.458+0000 7ff3e6ffd640 1 -- 172.21.15.29:0/2954625357 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff3e80e7cc0 con 0x7ff3f0103fc0 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.458+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f094d0 0x5640162d3010 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0011850 tx=0x7ff3e00097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b8694d1d317d25 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.458+0000 7ff3dbfff640 1 -- 172.21.15.29:0/1258785618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0004030 con 0x564015f094d0 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.458+0000 7ff3dbfff640 1 -- 172.21.15.29:0/1258785618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3e000f7c0 con 0x564015f094d0 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.458+0000 7ff3dbfff640 1 -- 172.21.15.29:0/1258785618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0003850 con 0x564015f094d0 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 -- 172.21.15.29:0/1258785618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f094d0 msgr2=0x5640162d3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f094d0 0x5640162d3010 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0011850 tx=0x7ff3e00097f0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3f65cb640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3dc0b6510 0x7ff3dc0b8a20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80e57b0 tx=0x7ff3e8071c10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 -- 172.21.15.29:0/1258785618 shutdown_connections 2024-09-21T03:01:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f094d0 0x5640162d3010 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f012f0 0x5640162d24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 --2- 172.21.15.29:0/1258785618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 0x5640162d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 -- 172.21.15.29:0/1258785618 >> 172.21.15.29:0/1258785618 conn(0x564015f1e300 msgr2=0x5640162cd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 -- 172.21.15.29:0/1258785618 shutdown_connections 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 -- 172.21.15.29:0/1258785618 wait complete. 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 -- start start 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.459+0000 7ff3ff076740 1 Processor -- start 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f012f0 0x5640162db610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162dbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 0x5640162dc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3ff076740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56401616bab0 con 0x564015f012f0 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3ff076740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5640161d9730 con 0x564015f2a1b0 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3ff076740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5640162d3b30 con 0x564015f094d0 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3ff076740 10 client.0 _ll_register_callbacks cb 0x56401622d6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f012f0 0x5640162db610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f012f0 0x5640162db610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59274/0 (socket says 172.21.15.29:59274) 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3619234528 learned_addr learned my addr 172.21.15.29:0/3619234528 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162dbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 0x5640162dc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3619234528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 msgr2=0x5640162dbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162dbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3619234528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 msgr2=0x5640162dc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 0x5640162dc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.460+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3619234528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564015f95790 con 0x564015f012f0 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f012f0 0x5640162db610 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000feb0 tx=0x7ff3e000ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f4d63ba0691a32 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3619234528 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e002c060 con 0x564015f012f0 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3619234528 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff3e00210d0 con 0x564015f012f0 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 1 -- 172.21.15.29:0/3619234528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3e000f7c0 con 0x564015f012f0 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3619234528 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff3e0033050 con 0x564015f012f0 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 10 client.18411 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 1 -- 172.21.15.29:0/3619234528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56401600e620 con 0x564015f012f0 2024-09-21T03:01:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3619234528 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3e000f7c0 con 0x564015f012f0 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3619234528 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff3e0004a80 con 0x564015f012f0 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 10 client.18411 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 10 client.18411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 20 client.18411 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564015ee8d80 0x5640162e8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 4 client.18411 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.461+0000 7ff3ff076740 1 -- 172.21.15.29:0/3619234528 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564016246640 con 0x564015ee8d80 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.462+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564015ee8d80 0x5640162e8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.462+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564015ee8d80 0x5640162e8af0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fe08670bbb42905 server_cookie=d81033b6bf290604 in_seq=0 out_seq=0 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.462+0000 7ff3d97fa640 10 client.18411 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3619234528 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564016246640 con 0x564015ee8d80 2024-09-21T03:01:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3d97fa640 10 client.18411 handle_command_reply: tid=0 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 client.18411 shutdown 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 -- 172.21.15.29:0/3619234528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f012f0 msgr2=0x5640162db610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f012f0 0x5640162db610 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000feb0 tx=0x7ff3e000ff60 comp rx=0 tx=0).stop 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 -- 172.21.15.29:0/3619234528 shutdown_connections 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f2a1b0 0x5640162dc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162dbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564015ee8d80 0x5640162e8af0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3619234528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f012f0 0x5640162db610 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3d97fa640 0 client.18411 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 -- 172.21.15.29:0/3619234528 >> 172.21.15.29:0/3619234528 conn(0x564015f1e300 msgr2=0x5640162cbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 -- 172.21.15.29:0/3619234528 shutdown_connections 2024-09-21T03:01:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 1 -- 172.21.15.29:0/3619234528 wait complete. 2024-09-21T03:01:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.463+0000 7ff3ff076740 20 client.18411 trim_cache size 0 max 16384 2024-09-21T03:01:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3ff076740 1 Processor -- start 2024-09-21T03:01:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3ff076740 1 -- start start 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 0x5640162da230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3ff076740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f01ac4f0 con 0x564015f2a1b0 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3ff076740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f01ae030 con 0x564015f012f0 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3ff076740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f01acbc0 con 0x564015f094d0 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3f55c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3f55c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162d2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59290/0 (socket says 172.21.15.29:59290) 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3401584968 learned_addr learned my addr 172.21.15.29:0/3401584968 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 0x5640162da230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.478+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3401584968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 msgr2=0x5640162d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3401584968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 msgr2=0x5640162da230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 0x5640162da230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3401584968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640162d34e0 con 0x564015f2a1b0 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162d2330 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0005eb0 tx=0x7ff3e0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d175ba3c4a583255 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3db7fe640 1 -- 172.21.15.29:0/3401584968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e002c070 con 0x564015f2a1b0 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3db7fe640 1 -- 172.21.15.29:0/3401584968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3e000f790 con 0x564015f2a1b0 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.479+0000 7ff3db7fe640 1 -- 172.21.15.29:0/3401584968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e00048e0 con 0x564015f2a1b0 2024-09-21T03:01:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 -- 172.21.15.29:0/3401584968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 msgr2=0x5640162d2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162d2330 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0005eb0 tx=0x7ff3e0005f90 comp rx=0 tx=0).stop 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 -- 172.21.15.29:0/3401584968 shutdown_connections 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162d2330 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3401584968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 0x5640162da230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 -- 172.21.15.29:0/3401584968 >> 172.21.15.29:0/3401584968 conn(0x564015f1e300 msgr2=0x5640162d9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 -- 172.21.15.29:0/3401584968 shutdown_connections 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 -- 172.21.15.29:0/3401584968 wait complete. 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 -- start start 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 Processor -- start 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 0x5640162e2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162e3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.480+0000 7ff3ff076740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162e1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3ff076740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f01ad960 con 0x564015f2a1b0 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3ff076740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f01ad290 con 0x564015f012f0 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3ff076740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f01aa4d0 con 0x564015f094d0 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3ff076740 10 client.0 _ll_register_callbacks cb 0x7ff3e0009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162e1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162e1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59292/0 (socket says 172.21.15.29:59292) 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3499832776 learned_addr learned my addr 172.21.15.29:0/3499832776 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 0x5640162e2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3499832776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 msgr2=0x5640162e3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162e3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3499832776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 msgr2=0x5640162e2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 0x5640162e2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 -- 172.21.15.29:0/3499832776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640162e2000 con 0x564015f2a1b0 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3f55c9640 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162e1870 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00059f0 tx=0x7ff3e00215e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a77519b95428306 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3499832776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e003c040 con 0x564015f2a1b0 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3499832776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff3e0021cf0 con 0x564015f2a1b0 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3ff076740 1 -- 172.21.15.29:0/3499832776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3e000f790 con 0x564015f2a1b0 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3499832776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff3e002c040 con 0x564015f2a1b0 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.481+0000 7ff3ff076740 10 client.18417 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3ff076740 1 -- 172.21.15.29:0/3499832776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5640162e0e90 con 0x564015f2a1b0 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3499832776 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3e000f790 con 0x564015f2a1b0 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3499832776 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff3e0004a00 con 0x564015f2a1b0 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3ff076740 10 client.18417 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3ff076740 10 client.18417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3ff076740 20 client.18417 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564015ebe1c0 0x564016355880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3ff076740 4 client.18417 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3ff076740 1 -- 172.21.15.29:0/3499832776 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640162420f0 con 0x564015ebe1c0 2024-09-21T03:01:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.482+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564015ebe1c0 0x564016355880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.483+0000 7ff3d97fa640 10 client.18417 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.483+0000 7ff3f5dca640 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564015ebe1c0 0x564016355880 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40652a6361f03244 server_cookie=e7245c10eba30f25 in_seq=0 out_seq=0 2024-09-21T03:01:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3d97fa640 1 -- 172.21.15.29:0/3499832776 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640162420f0 con 0x564015ebe1c0 2024-09-21T03:01:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3d97fa640 10 client.18417 handle_command_reply: tid=0 2024-09-21T03:01:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 client.18417 shutdown 2024-09-21T03:01:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 -- 172.21.15.29:0/3499832776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 msgr2=0x5640162e1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162e1870 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00059f0 tx=0x7ff3e00215e0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 -- 172.21.15.29:0/3499832776 shutdown_connections 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564015ebe1c0 0x564016355880 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564015f2a1b0 0x5640162e1870 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564015f094d0 0x5640162e3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3d97fa640 0 client.18417 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 --2- 172.21.15.29:0/3499832776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564015f012f0 0x5640162e2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 -- 172.21.15.29:0/3499832776 >> 172.21.15.29:0/3499832776 conn(0x564015f1e300 msgr2=0x5640162d99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 -- 172.21.15.29:0/3499832776 shutdown_connections 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 1 -- 172.21.15.29:0/3499832776 wait complete. 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3ff076740 20 client.18417 trim_cache size 0 max 16384 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:01:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:01:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:01:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:01:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 -- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3dc0b6510 msgr2=0x7ff3dc0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3dc0b6510 0x7ff3dc0b8a20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80e57b0 tx=0x7ff3e8071c10 comp rx=0 tx=0).stop 2024-09-21T03:01:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 -- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f0103fc0 msgr2=0x7ff3f01a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f0103fc0 0x7ff3f01a1150 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800eb40 tx=0x7ff3e800c2d0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 -- 172.21.15.29:0/2954625357 shutdown_connections 2024-09-21T03:01:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff3dc0b6510 0x7ff3dc0b8a20 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f0103fc0 0x7ff3f01a1150 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3f0103620 0x7ff3f019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 --2- 172.21.15.29:0/2954625357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f01025a0 0x7ff3f006b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 -- 172.21.15.29:0/2954625357 >> 172.21.15.29:0/2954625357 conn(0x7ff3f00fdee0 msgr2=0x7ff3f0106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 -- 172.21.15.29:0/2954625357 shutdown_connections 2024-09-21T03:01:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:30.643+0000 7ff3f802e640 1 -- 172.21.15.29:0/2954625357 wait complete. 2024-09-21T03:01:30.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.65 MB/sec execute 427 sec latency 1990.149 ms 2024-09-21T03:01:31.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.64 MB/sec execute 428 sec latency 2990.253 ms 2024-09-21T03:01:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:31 smithi190 ceph-mon[33871]: pgmap v1061: 129 pgs: 129 active+clean; 329 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 3.9 MiB/s wr, 222 op/s 2024-09-21T03:01:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:31 smithi029 ceph-mon[31006]: pgmap v1061: 129 pgs: 129 active+clean; 329 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 3.9 MiB/s wr, 222 op/s 2024-09-21T03:01:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:31 smithi066 ceph-mon[33436]: pgmap v1061: 129 pgs: 129 active+clean; 329 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 3.9 MiB/s wr, 222 op/s 2024-09-21T03:01:32.741 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.62 MB/sec execute 429 sec latency 3990.364 ms 2024-09-21T03:01:33.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.61 MB/sec execute 430 sec latency 4990.513 ms 2024-09-21T03:01:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:33 smithi190 ceph-mon[33871]: pgmap v1062: 129 pgs: 129 active+clean; 324 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.6 MiB/s wr, 244 op/s 2024-09-21T03:01:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:33 smithi029 ceph-mon[31006]: pgmap v1062: 129 pgs: 129 active+clean; 324 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.6 MiB/s wr, 244 op/s 2024-09-21T03:01:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:33 smithi066 ceph-mon[33436]: pgmap v1062: 129 pgs: 129 active+clean; 324 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 3.6 MiB/s wr, 244 op/s 2024-09-21T03:01:34.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.60 MB/sec execute 431 sec latency 5990.626 ms 2024-09-21T03:01:35.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.58 MB/sec execute 432 sec latency 6990.723 ms 2024-09-21T03:01:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:35 smithi190 ceph-mon[33871]: pgmap v1063: 129 pgs: 129 active+clean; 320 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 2.7 MiB/s wr, 224 op/s 2024-09-21T03:01:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:35 smithi029 ceph-mon[31006]: pgmap v1063: 129 pgs: 129 active+clean; 320 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 2.7 MiB/s wr, 224 op/s 2024-09-21T03:01:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:35 smithi066 ceph-mon[33436]: pgmap v1063: 129 pgs: 129 active+clean; 320 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 2.7 MiB/s wr, 224 op/s 2024-09-21T03:01:36.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.57 MB/sec execute 433 sec latency 7990.801 ms 2024-09-21T03:01:37.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.56 MB/sec execute 434 sec latency 8990.858 ms 2024-09-21T03:01:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:37 smithi190 ceph-mon[33871]: pgmap v1064: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 1.7 MiB/s wr, 198 op/s 2024-09-21T03:01:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:37 smithi029 ceph-mon[31006]: pgmap v1064: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 1.7 MiB/s wr, 198 op/s 2024-09-21T03:01:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:37 smithi066 ceph-mon[33436]: pgmap v1064: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 1.7 MiB/s wr, 198 op/s 2024-09-21T03:01:38.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.55 MB/sec execute 435 sec latency 9990.967 ms 2024-09-21T03:01:39.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.53 MB/sec execute 436 sec latency 10991.036 ms 2024-09-21T03:01:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:39 smithi190 ceph-mon[33871]: pgmap v1065: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 173 op/s 2024-09-21T03:01:39.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:39 smithi029 ceph-mon[31006]: pgmap v1065: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 173 op/s 2024-09-21T03:01:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:39 smithi066 ceph-mon[33436]: pgmap v1065: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 173 op/s 2024-09-21T03:01:40.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.52 MB/sec execute 437 sec latency 11991.174 ms 2024-09-21T03:01:41.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.51 MB/sec execute 438 sec latency 12991.285 ms 2024-09-21T03:01:41.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:41 smithi190 ceph-mon[33871]: pgmap v1066: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 805 KiB/s wr, 133 op/s 2024-09-21T03:01:41.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:41 smithi029 ceph-mon[31006]: pgmap v1066: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 805 KiB/s wr, 133 op/s 2024-09-21T03:01:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:41 smithi066 ceph-mon[33436]: pgmap v1066: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 805 KiB/s wr, 133 op/s 2024-09-21T03:01:42.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.50 MB/sec execute 439 sec latency 13991.396 ms 2024-09-21T03:01:43.742 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.48 MB/sec execute 440 sec latency 14991.458 ms 2024-09-21T03:01:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 -- 172.21.15.29:0/3412546786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 msgr2=0x7f94e4102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3412546786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 0x7f94e4102a10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009900 tx=0x7f94d402a410 comp rx=0 tx=0).stop 2024-09-21T03:01:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 -- 172.21.15.29:0/3412546786 shutdown_connections 2024-09-21T03:01:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3412546786 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e4103fc0 0x7f94e410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3412546786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e4103620 0x7f94e4103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3412546786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 0x7f94e4102a10 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 -- 172.21.15.29:0/3412546786 >> 172.21.15.29:0/3412546786 conn(0x7f94e40fdee0 msgr2=0x7f94e4100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 -- 172.21.15.29:0/3412546786 shutdown_connections 2024-09-21T03:01:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.759+0000 7f94eaba1640 1 -- 172.21.15.29:0/3412546786 wait complete. 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.760+0000 7f94eaba1640 1 Processor -- start 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.760+0000 7f94eaba1640 1 -- start start 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.760+0000 7f94eaba1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 0x7f94e4106190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.760+0000 7f94eaba1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e4103620 0x7f94e4104a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94eaba1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e4103fc0 0x7f94e4104f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94eaba1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94e410e0a0 con 0x7f94e41025a0 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94eaba1640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f94e410df20 con 0x7f94e4103620 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94eaba1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f94e410e220 con 0x7f94e4103fc0 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 0x7f94e4106190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 0x7f94e4106190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60366/0 (socket says 172.21.15.29:60366) 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 -- 172.21.15.29:0/3124286206 learned_addr learned my addr 172.21.15.29:0/3124286206 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e3fff640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e4103620 0x7f94e4104a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e913e640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e4103fc0 0x7f94e4104f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 -- 172.21.15.29:0/3124286206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e4103fc0 msgr2=0x7f94e4104f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e4103fc0 0x7f94e4104f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 -- 172.21.15.29:0/3124286206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e4103620 msgr2=0x7f94e4104a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e4103620 0x7f94e4104a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 -- 172.21.15.29:0/3124286206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e41056c0 con 0x7f94e41025a0 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e893d640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 0x7f94e4106190 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009230 tx=0x7f94d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e1ffb640 1 -- 172.21.15.29:0/3124286206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d403f030 con 0x7f94e41025a0 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.761+0000 7f94e1ffb640 1 -- 172.21.15.29:0/3124286206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94d4003a80 con 0x7f94e41025a0 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.762+0000 7f94eaba1640 1 -- 172.21.15.29:0/3124286206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e41a3f00 con 0x7f94e41025a0 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.762+0000 7f94e1ffb640 1 -- 172.21.15.29:0/3124286206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4035070 con 0x7f94e41025a0 2024-09-21T03:01:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.762+0000 7f94eaba1640 1 -- 172.21.15.29:0/3124286206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e41a42b0 con 0x7f94e41025a0 2024-09-21T03:01:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.764+0000 7f94f1be5740 1 Processor -- start 2024-09-21T03:01:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.764+0000 7f94f1be5740 1 -- start start 2024-09-21T03:01:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff80e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94f1be5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564ddfef36f0 con 0x564ddfbd91b0 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94f1be5740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564ddfee2b20 con 0x564ddfbb84d0 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94f1be5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564ddfee6930 con 0x564ddfbb02f0 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e1ffb640 1 -- 172.21.15.29:0/3124286206 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f94d4042030 con 0x7f94e41025a0 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e913e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff80e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e913e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff80e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60378/0 (socket says 172.21.15.29:60378) 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e913e640 1 -- 172.21.15.29:0/2551208717 learned_addr learned my addr 172.21.15.29:0/2551208717 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e893d640 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e3fff640 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e913e640 1 -- 172.21.15.29:0/2551208717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 msgr2=0x564ddff814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e913e640 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e913e640 1 -- 172.21.15.29:0/2551208717 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 msgr2=0x564ddff82010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e913e640 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e913e640 1 -- 172.21.15.29:0/2551208717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ddfea8530 con 0x564ddfbd91b0 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.765+0000 7f94e893d640 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.766+0000 7f94e913e640 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff80e50 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009fd0 tx=0x7f94dc010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862cfb3f4d943262 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.766+0000 7f94daffd640 1 -- 172.21.15.29:0/2551208717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc003bc0 con 0x564ddfbd91b0 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.766+0000 7f94daffd640 1 -- 172.21.15.29:0/2551208717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94dc003d60 con 0x564ddfbd91b0 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.766+0000 7f94daffd640 1 -- 172.21.15.29:0/2551208717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0119a0 con 0x564ddfbd91b0 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.766+0000 7f94e1ffb640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6510 0x7f94d00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.766+0000 7f94e893d640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6510 0x7f94d00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.766+0000 7f94e1ffb640 1 -- 172.21.15.29:0/3124286206 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f94d4107c50 con 0x7f94e41025a0 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 -- 172.21.15.29:0/2551208717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 msgr2=0x564ddff80e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff80e50 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009fd0 tx=0x7f94dc010860 comp rx=0 tx=0).stop 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 -- 172.21.15.29:0/2551208717 shutdown_connections 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94e893d640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6510 0x7f94d00b8a20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f94d40cd810 tx=0x7f94d40b5040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2551208717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff80e50 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 -- 172.21.15.29:0/2551208717 >> 172.21.15.29:0/2551208717 conn(0x564ddfbcd300 msgr2=0x564ddff7c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 -- 172.21.15.29:0/2551208717 shutdown_connections 2024-09-21T03:01:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 -- 172.21.15.29:0/2551208717 wait complete. 2024-09-21T03:01:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 -- start start 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.767+0000 7f94f1be5740 1 Processor -- start 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff8a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 0x564ddff8acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 0x564ddff8b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94f1be5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564ddfe1aab0 con 0x564ddfbb02f0 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94f1be5740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564ddfe88730 con 0x564ddfbd91b0 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94f1be5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564ddff82b30 con 0x564ddfbb84d0 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94f1be5740 10 client.0 _ll_register_callbacks cb 0x564ddfedc6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94e913e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff8a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94e913e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff8a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60384/0 (socket says 172.21.15.29:60384) 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94e913e640 1 -- 172.21.15.29:0/609678768 learned_addr learned my addr 172.21.15.29:0/609678768 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94e3fff640 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 0x564ddff8acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.768+0000 7f94e913e640 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 0x564ddff8b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94e913e640 1 -- 172.21.15.29:0/609678768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 msgr2=0x564ddff8acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94e913e640 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 0x564ddff8acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94e913e640 1 -- 172.21.15.29:0/609678768 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 msgr2=0x564ddff8b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94e913e640 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 0x564ddff8b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94e913e640 1 -- 172.21.15.29:0/609678768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ddfc44790 con 0x564ddfbb02f0 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94e913e640 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff8a610 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0048e0 tx=0x7f94dc004810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22b9ba211424ae6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94c3fff640 1 -- 172.21.15.29:0/609678768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc023040 con 0x564ddfbb02f0 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94c3fff640 1 -- 172.21.15.29:0/609678768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f94dc004410 con 0x564ddfbb02f0 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94f1be5740 1 -- 172.21.15.29:0/609678768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94dc003d60 con 0x564ddfbb02f0 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94c3fff640 1 -- 172.21.15.29:0/609678768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f94dc004610 con 0x564ddfbb02f0 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94f1be5740 10 client.18429 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94f1be5740 1 -- 172.21.15.29:0/609678768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564ddfcbd620 con 0x564ddfbb02f0 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.769+0000 7f94c3fff640 1 -- 172.21.15.29:0/609678768 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94dc003d60 con 0x564ddfbb02f0 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94c3fff640 1 -- 172.21.15.29:0/609678768 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f94dc029030 con 0x564ddfbb02f0 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94f1be5740 10 client.18429 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94f1be5740 10 client.18429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94f1be5740 20 client.18429 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94f1be5740 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ddfb97d80 0x564ddff979e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94f1be5740 4 client.18429 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94f1be5740 1 -- 172.21.15.29:0/609678768 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ddfef5640 con 0x564ddfb97d80 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94e3fff640 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ddfb97d80 0x564ddff979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94e3fff640 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ddfb97d80 0x564ddff979e0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9949550955b8da06 server_cookie=4e6da03ec4b1a49c in_seq=0 out_seq=0 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.770+0000 7f94c3fff640 10 client.18429 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.771+0000 7f94c3fff640 1 -- 172.21.15.29:0/609678768 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564ddfef5640 con 0x564ddfb97d80 2024-09-21T03:01:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.771+0000 7f94c3fff640 10 client.18429 handle_command_reply: tid=0 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.771+0000 7f94f1be5740 1 client.18429 shutdown 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.771+0000 7f94f1be5740 1 -- 172.21.15.29:0/609678768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 msgr2=0x564ddff8a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.771+0000 7f94f1be5740 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff8a610 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0048e0 tx=0x7f94dc004810 comp rx=0 tx=0).stop 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 1 -- 172.21.15.29:0/609678768 shutdown_connections 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 0x564ddff8b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 0x564ddff8acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ddfb97d80 0x564ddff979e0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 1 --2- 172.21.15.29:0/609678768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff8a610 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 1 -- 172.21.15.29:0/609678768 >> 172.21.15.29:0/609678768 conn(0x564ddfbcd300 msgr2=0x564ddff81ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94c3fff640 0 client.18429 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 1 -- 172.21.15.29:0/609678768 shutdown_connections 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 1 -- 172.21.15.29:0/609678768 wait complete. 2024-09-21T03:01:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.772+0000 7f94f1be5740 20 client.18429 trim_cache size 0 max 16384 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.787+0000 7f94f1be5740 1 Processor -- start 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.787+0000 7f94f1be5740 1 -- start start 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.787+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff81500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.787+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.787+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff834d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.787+0000 7f94f1be5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41ae780 con 0x564ddfbd91b0 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.787+0000 7f94f1be5740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41b02c0 con 0x564ddfbb84d0 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.787+0000 7f94f1be5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41aee50 con 0x564ddfbb02f0 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff834d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff81500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff81500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50948/0 (socket says 172.21.15.29:50948) 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 -- 172.21.15.29:0/899581483 learned_addr learned my addr 172.21.15.29:0/899581483 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 -- 172.21.15.29:0/899581483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 msgr2=0x564ddff82f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 --2- 172.21.15.29:0/899581483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 -- 172.21.15.29:0/899581483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 msgr2=0x564ddff834d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 --2- 172.21.15.29:0/899581483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff834d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 -- 172.21.15.29:0/899581483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ddff887a0 con 0x564ddfbb02f0 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e3fff640 1 --2- 172.21.15.29:0/899581483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94e913e640 1 --2- 172.21.15.29:0/899581483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff81500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f94dc005390 tx=0x7f94dc009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16435ec9e8fb1ce1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.788+0000 7f94da7fc640 1 -- 172.21.15.29:0/899581483 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc010d60 con 0x564ddfbb02f0 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94da7fc640 1 -- 172.21.15.29:0/899581483 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94dc005040 con 0x564ddfbb02f0 2024-09-21T03:01:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94da7fc640 1 -- 172.21.15.29:0/899581483 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc004210 con 0x564ddfbb02f0 2024-09-21T03:01:43.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 -- 172.21.15.29:0/899581483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 msgr2=0x564ddff81500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 --2- 172.21.15.29:0/899581483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff81500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f94dc005390 tx=0x7f94dc009fd0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 -- 172.21.15.29:0/899581483 shutdown_connections 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 --2- 172.21.15.29:0/899581483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbd91b0 0x564ddff834d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 --2- 172.21.15.29:0/899581483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbb84d0 0x564ddff82f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 --2- 172.21.15.29:0/899581483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb02f0 0x564ddff81500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 -- 172.21.15.29:0/899581483 >> 172.21.15.29:0/899581483 conn(0x564ddfbcd300 msgr2=0x564ddff80df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 -- 172.21.15.29:0/899581483 shutdown_connections 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.789+0000 7f94f1be5740 1 -- 172.21.15.29:0/899581483 wait complete. 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 1 -- start start 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 1 Processor -- start 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff92360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 0x564ddff91020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 0x564ddff91560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41afbf0 con 0x564ddfbb02f0 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41af520 con 0x564ddfbd91b0 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41ac760 con 0x564ddfbb84d0 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94f1be5740 10 client.0 _ll_register_callbacks cb 0x7f94dc009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94e913e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff92360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94e913e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff92360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60404/0 (socket says 172.21.15.29:60404) 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.790+0000 7f94e913e640 1 -- 172.21.15.29:0/2976777839 learned_addr learned my addr 172.21.15.29:0/2976777839 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94e913e640 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 0x564ddff91560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94e3fff640 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 0x564ddff91020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94e913e640 1 -- 172.21.15.29:0/2976777839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 msgr2=0x564ddff91020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94e913e640 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 0x564ddff91020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94e913e640 1 -- 172.21.15.29:0/2976777839 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 msgr2=0x564ddff91560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94e913e640 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 0x564ddff91560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94e913e640 1 -- 172.21.15.29:0/2976777839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ddff91c50 con 0x564ddfbb02f0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94e913e640 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff92360 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0051f0 tx=0x7f94dc003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f96d1ee6f42705 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94c3fff640 1 -- 172.21.15.29:0/2976777839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc005890 con 0x564ddfbb02f0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94c3fff640 1 -- 172.21.15.29:0/2976777839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f94dc004210 con 0x564ddfbb02f0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94f1be5740 1 -- 172.21.15.29:0/2976777839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94dc005040 con 0x564ddfbb02f0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94c3fff640 1 -- 172.21.15.29:0/2976777839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f94dc004410 con 0x564ddfbb02f0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94f1be5740 10 client.18435 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.791+0000 7f94f1be5740 1 -- 172.21.15.29:0/2976777839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564de00022e0 con 0x564ddfbb02f0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94c3fff640 1 -- 172.21.15.29:0/2976777839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94dc005040 con 0x564ddfbb02f0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94c3fff640 1 -- 172.21.15.29:0/2976777839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f94dc029070 con 0x564ddfbb02f0 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94f1be5740 10 client.18435 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94f1be5740 10 client.18435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94f1be5740 20 client.18435 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ddfb6d1c0 0x564de0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94f1be5740 4 client.18435 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:01:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94f1be5740 1 -- 172.21.15.29:0/2976777839 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ddfef10f0 con 0x564ddfb6d1c0 2024-09-21T03:01:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94e3fff640 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ddfb6d1c0 0x564de0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94c3fff640 10 client.18435 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.792+0000 7f94e3fff640 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ddfb6d1c0 0x564de0004820 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f95c42f84b110fe server_cookie=695aafc7dc2bdc20 in_seq=0 out_seq=0 2024-09-21T03:01:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94c3fff640 1 -- 172.21.15.29:0/2976777839 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ddfef10f0 con 0x564ddfb6d1c0 2024-09-21T03:01:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94c3fff640 10 client.18435 handle_command_reply: tid=0 2024-09-21T03:01:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 client.18435 shutdown 2024-09-21T03:01:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 -- 172.21.15.29:0/2976777839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 msgr2=0x564ddff92360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff92360 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0051f0 tx=0x7f94dc003b70 comp rx=0 tx=0).stop 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 -- 172.21.15.29:0/2976777839 shutdown_connections 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ddfbd91b0 0x564ddff91560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ddfbb84d0 0x564ddff91020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ddfb6d1c0 0x564de0004820 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 --2- 172.21.15.29:0/2976777839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ddfbb02f0 0x564ddff92360 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94f1be5740 1 -- 172.21.15.29:0/2976777839 >> 172.21.15.29:0/2976777839 conn(0x564ddfbcd300 msgr2=0x564ddff7aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.838+0000 7f94c3fff640 0 client.18435 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94f1be5740 1 -- 172.21.15.29:0/2976777839 shutdown_connections 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94f1be5740 1 -- 172.21.15.29:0/2976777839 wait complete. 2024-09-21T03:01:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94f1be5740 20 client.18435 trim_cache size 0 max 16384 2024-09-21T03:01:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:01:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:01:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:01:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:01:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 -- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6510 msgr2=0x7f94d00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6510 0x7f94d00b8a20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f94d40cd810 tx=0x7f94d40b5040 comp rx=0 tx=0).stop 2024-09-21T03:01:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 -- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 msgr2=0x7f94e4106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 0x7f94e4106190 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009230 tx=0x7f94d4004270 comp rx=0 tx=0).stop 2024-09-21T03:01:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 -- 172.21.15.29:0/3124286206 shutdown_connections 2024-09-21T03:01:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f94d00b6510 0x7f94d00b8a20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94e4103fc0 0x7f94e4104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94e4103620 0x7f94e4104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 --2- 172.21.15.29:0/3124286206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e41025a0 0x7f94e4106190 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.839+0000 7f94eaba1640 1 -- 172.21.15.29:0/3124286206 >> 172.21.15.29:0/3124286206 conn(0x7f94e40fdee0 msgr2=0x7f94e40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.840+0000 7f94eaba1640 1 -- 172.21.15.29:0/3124286206 shutdown_connections 2024-09-21T03:01:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:43.840+0000 7f94eaba1640 1 -- 172.21.15.29:0/3124286206 wait complete. 2024-09-21T03:01:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:43 smithi190 ceph-mon[33871]: pgmap v1067: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 401 KiB/s wr, 124 op/s 2024-09-21T03:01:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:43 smithi029 ceph-mon[31006]: pgmap v1067: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 401 KiB/s wr, 124 op/s 2024-09-21T03:01:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:43 smithi066 ceph-mon[33436]: pgmap v1067: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 401 KiB/s wr, 124 op/s 2024-09-21T03:01:44.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.47 MB/sec execute 441 sec latency 15991.555 ms 2024-09-21T03:01:45.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.46 MB/sec execute 442 sec latency 16991.663 ms 2024-09-21T03:01:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:45 smithi190 ceph-mon[33871]: pgmap v1068: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 58 KiB/s wr, 71 op/s 2024-09-21T03:01:45.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:45 smithi029 ceph-mon[31006]: pgmap v1068: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 58 KiB/s wr, 71 op/s 2024-09-21T03:01:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:45 smithi066 ceph-mon[33436]: pgmap v1068: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 58 KiB/s wr, 71 op/s 2024-09-21T03:01:46.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.45 MB/sec execute 443 sec latency 17991.809 ms 2024-09-21T03:01:47.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.43 MB/sec execute 444 sec latency 18991.912 ms 2024-09-21T03:01:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:47 smithi190 ceph-mon[33871]: pgmap v1069: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 18 KiB/s wr, 26 op/s 2024-09-21T03:01:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:47 smithi029 ceph-mon[31006]: pgmap v1069: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 18 KiB/s wr, 26 op/s 2024-09-21T03:01:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:47 smithi066 ceph-mon[33436]: pgmap v1069: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 18 KiB/s wr, 26 op/s 2024-09-21T03:01:48.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.42 MB/sec execute 445 sec latency 19991.961 ms 2024-09-21T03:01:48.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:48 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:01:48.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:48 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:01:49.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:48 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:01:49.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.41 MB/sec execute 446 sec latency 20992.064 ms 2024-09-21T03:01:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:49 smithi190 ceph-mon[33871]: pgmap v1070: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 21 op/s 2024-09-21T03:01:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:01:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:01:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:01:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:49 smithi029 ceph-mon[31006]: pgmap v1070: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 21 op/s 2024-09-21T03:01:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:01:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:01:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:01:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:49 smithi066 ceph-mon[33436]: pgmap v1070: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 21 op/s 2024-09-21T03:01:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:01:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:01:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:01:50.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.40 MB/sec execute 447 sec latency 21992.143 ms 2024-09-21T03:01:51.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.38 MB/sec execute 448 sec latency 22992.260 ms 2024-09-21T03:01:51.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:51 smithi190 ceph-mon[33871]: pgmap v1071: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 19 op/s 2024-09-21T03:01:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:51 smithi029 ceph-mon[31006]: pgmap v1071: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 19 op/s 2024-09-21T03:01:52.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:51 smithi066 ceph-mon[33436]: pgmap v1071: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 19 op/s 2024-09-21T03:01:52.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.37 MB/sec execute 449 sec latency 23992.375 ms 2024-09-21T03:01:53.743 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.36 MB/sec execute 450 sec latency 24992.461 ms 2024-09-21T03:01:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:53 smithi190 ceph-mon[33871]: pgmap v1072: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 19 op/s 2024-09-21T03:01:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:53 smithi029 ceph-mon[31006]: pgmap v1072: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 19 op/s 2024-09-21T03:01:54.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:53 smithi066 ceph-mon[33436]: pgmap v1072: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 19 op/s 2024-09-21T03:01:54.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:01:54.688 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:01:54.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.35 MB/sec execute 451 sec latency 25992.574 ms 2024-09-21T03:01:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 -- 172.21.15.29:0/98578005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a041094b0 msgr2=0x7f7a041098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 --2- 172.21.15.29:0/98578005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a041094b0 0x7f7a041098c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009900 tx=0x7f79ec02a3f0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 -- 172.21.15.29:0/98578005 shutdown_connections 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 --2- 172.21.15.29:0/98578005 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7a04109e00 0x7f7a04110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 --2- 172.21.15.29:0/98578005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a041094b0 0x7f7a041098c0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 --2- 172.21.15.29:0/98578005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a04103bc0 0x7f7a04103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 -- 172.21.15.29:0/98578005 >> 172.21.15.29:0/98578005 conn(0x7f7a0407b910 msgr2=0x7f7a0407bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 -- 172.21.15.29:0/98578005 shutdown_connections 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.315+0000 7f7a08db8640 1 -- 172.21.15.29:0/98578005 wait complete. 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a08db8640 1 Processor -- start 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a08db8640 1 -- start start 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a08db8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a04103bc0 0x7f7a0419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a08db8640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7a041094b0 0x7f7a0419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a08db8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a04109e00 0x7f7a0419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a08db8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7a04113970 con 0x7f7a04103bc0 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a08db8640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041137f0 con 0x7f7a041094b0 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a08db8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7a04113af0 con 0x7f7a04109e00 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.316+0000 7f7a0259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a04103bc0 0x7f7a0419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a0259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a04103bc0 0x7f7a0419a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45082/0 (socket says 172.21.15.29:45082) 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a0259c640 1 -- 172.21.15.29:0/3995792261 learned_addr learned my addr 172.21.15.29:0/3995792261 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a01d9b640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7a041094b0 0x7f7a0419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a02d9d640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a04109e00 0x7f7a0419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a0259c640 1 -- 172.21.15.29:0/3995792261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a04109e00 msgr2=0x7f7a0419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a0259c640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a04109e00 0x7f7a0419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a0259c640 1 -- 172.21.15.29:0/3995792261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7a041094b0 msgr2=0x7f7a0419aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a0259c640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7a041094b0 0x7f7a0419aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a0259c640 1 -- 172.21.15.29:0/3995792261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a041a0310 con 0x7f7a04103bc0 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a01d9b640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7a041094b0 0x7f7a0419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a02d9d640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a04109e00 0x7f7a0419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a0259c640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a04103bc0 0x7f7a0419a460 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f79f400c970 tx=0x7f79f400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f79fb7fe640 1 -- 172.21.15.29:0/3995792261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4007bb0 con 0x7f7a04103bc0 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f79fb7fe640 1 -- 172.21.15.29:0/3995792261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79f4007d50 con 0x7f7a04103bc0 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a041a0600 con 0x7f7a04103bc0 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f79fb7fe640 1 -- 172.21.15.29:0/3995792261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4004e40 con 0x7f7a04103bc0 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.317+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a041a0ac0 con 0x7f7a04103bc0 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.318+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a04103fd0 con 0x7f7a04103bc0 2024-09-21T03:01:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.320+0000 7f79fb7fe640 1 -- 172.21.15.29:0/3995792261 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f79f4005000 con 0x7f7a04103bc0 2024-09-21T03:01:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.321+0000 7f79fb7fe640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79f00b6540 0x7f79f00b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.321+0000 7f7a01d9b640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79f00b6540 0x7f79f00b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.321+0000 7f79fb7fe640 1 -- 172.21.15.29:0/3995792261 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f79f40e7800 con 0x7f7a04103bc0 2024-09-21T03:01:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.322+0000 7f7a01d9b640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79f00b6540 0x7f79f00b8a50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0057c0 tx=0x7f79ec009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.324+0000 7f79fb7fe640 1 -- 172.21.15.29:0/3995792261 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f79f4014030 con 0x7f7a04103bc0 2024-09-21T03:01:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.551+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a0419b590 con 0x7f7a04103bc0 2024-09-21T03:01:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.552+0000 7f79fb7fe640 1 -- 172.21.15.29:0/3995792261 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11971 (secure 0 0 0) 0x7f7a0419b590 con 0x7f7a04103bc0 2024-09-21T03:01:55.554 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:01:55.554 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"btime":"2024-09-21T02:54:56:454931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T02:54:56.335127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/4067532455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":4067532455},{"type":"v1","addr":"172.21.15.66:6835","nonce":4067532455}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:01:55.554 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79f00b6540 msgr2=0x7f79f00b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79f00b6540 0x7f79f00b8a50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0057c0 tx=0x7f79ec009680 comp rx=0 tx=0).stop 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a04103bc0 msgr2=0x7f7a0419a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a04103bc0 0x7f7a0419a460 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f79f400c970 tx=0x7f79f400ce30 comp rx=0 tx=0).stop 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 shutdown_connections 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f79f00b6540 0x7f79f00b8a50 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a04109e00 0x7f7a0419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7a041094b0 0x7f7a0419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 --2- 172.21.15.29:0/3995792261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a04103bc0 0x7f7a0419a460 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.554+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 >> 172.21.15.29:0/3995792261 conn(0x7f7a0407b910 msgr2=0x7f7a04105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:55.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.555+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 shutdown_connections 2024-09-21T03:01:55.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:55.555+0000 7f7a08db8640 1 -- 172.21.15.29:0/3995792261 wait complete. 2024-09-21T03:01:55.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.34 MB/sec execute 452 sec latency 26992.660 ms 2024-09-21T03:01:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:55 smithi190 ceph-mon[33871]: pgmap v1073: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 19 op/s 2024-09-21T03:01:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:55 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3995792261' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:01:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:55 smithi029 ceph-mon[31006]: pgmap v1073: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 19 op/s 2024-09-21T03:01:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:55 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3995792261' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:01:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:55 smithi066 ceph-mon[33436]: pgmap v1073: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 19 op/s 2024-09-21T03:01:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:55 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3995792261' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:01:56.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.33 MB/sec execute 453 sec latency 27992.764 ms 2024-09-21T03:01:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 -- 172.21.15.29:0/3081282177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634101d50 msgr2=0x7f36341021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 --2- 172.21.15.29:0/3081282177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634101d50 0x7f36341021c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f362800aa10 tx=0x7f362802b720 comp rx=0 tx=0).stop 2024-09-21T03:01:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 -- 172.21.15.29:0/3081282177 shutdown_connections 2024-09-21T03:01:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 --2- 172.21.15.29:0/3081282177 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3634103770 0x7f363410a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 --2- 172.21.15.29:0/3081282177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3634102dd0 0x7f36341031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 --2- 172.21.15.29:0/3081282177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634101d50 0x7f36341021c0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 -- 172.21.15.29:0/3081282177 >> 172.21.15.29:0/3081282177 conn(0x7f36340fd6d0 msgr2=0x7f36340ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 -- 172.21.15.29:0/3081282177 shutdown_connections 2024-09-21T03:01:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.956+0000 7f363a084640 1 -- 172.21.15.29:0/3081282177 wait complete. 2024-09-21T03:01:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f363a084640 1 Processor -- start 2024-09-21T03:01:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f363a084640 1 -- start start 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f363a084640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3634101d50 0x7f3634199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f363a084640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634102dd0 0x7f363419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f363a084640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3634103770 0x7f363419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f363a084640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f363410cfa0 con 0x7f3634102dd0 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f363a084640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f363410ce20 con 0x7f3634101d50 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f363a084640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f363410d120 con 0x7f3634103770 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f3632ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634102dd0 0x7f363419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f3632ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634102dd0 0x7f363419a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45096/0 (socket says 172.21.15.29:45096) 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f3632ffd640 1 -- 172.21.15.29:0/3896050431 learned_addr learned my addr 172.21.15.29:0/3896050431 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.957+0000 7f36337fe640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3634101d50 0x7f3634199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3633fff640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3634103770 0x7f363419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3632ffd640 1 -- 172.21.15.29:0/3896050431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3634103770 msgr2=0x7f363419b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3632ffd640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3634103770 0x7f363419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3632ffd640 1 -- 172.21.15.29:0/3896050431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3634101d50 msgr2=0x7f3634199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3632ffd640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3634101d50 0x7f3634199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3632ffd640 1 -- 172.21.15.29:0/3896050431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363419b860 con 0x7f3634102dd0 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3632ffd640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634102dd0 0x7f363419a520 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f362400d950 tx=0x7f362400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3630ff9640 1 -- 172.21.15.29:0/3896050431 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3624017030 con 0x7f3634102dd0 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3630ff9640 1 -- 172.21.15.29:0/3896050431 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3624004020 con 0x7f3634102dd0 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f363a084640 1 -- 172.21.15.29:0/3896050431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f363419bb50 con 0x7f3634102dd0 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f3630ff9640 1 -- 172.21.15.29:0/3896050431 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36240055d0 con 0x7f3634102dd0 2024-09-21T03:01:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.958+0000 7f363a084640 1 -- 172.21.15.29:0/3896050431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36341a22d0 con 0x7f3634102dd0 2024-09-21T03:01:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3640ff2740 1 Processor -- start 2024-09-21T03:01:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3640ff2740 1 -- start start 2024-09-21T03:01:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 0x564ba48a9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48ab010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3640ff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564ba481c6f0 con 0x564ba44d92f0 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3640ff2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564ba480bb20 con 0x564ba44e14d0 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3640ff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564ba480f930 con 0x564ba45021b0 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3630ff9640 1 -- 172.21.15.29:0/3896050431 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f3624009a30 con 0x7f3634102dd0 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f36337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f36337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45098/0 (socket says 172.21.15.29:45098) 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f36337fe640 1 -- 172.21.15.29:0/3423491553 learned_addr learned my addr 172.21.15.29:0/3423491553 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3632ffd640 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48ab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.961+0000 7f3633fff640 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 0x564ba48a9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f36337fe640 1 -- 172.21.15.29:0/3423491553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 msgr2=0x564ba48a9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f36337fe640 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 0x564ba48a9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f36337fe640 1 -- 172.21.15.29:0/3423491553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 msgr2=0x564ba48ab010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f36337fe640 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48ab010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f36337fe640 1 -- 172.21.15.29:0/3423491553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ba47d1530 con 0x564ba44d92f0 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f3632ffd640 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48ab010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f36337fe640 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa4c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f362802bc40 tx=0x7f3628005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2e42b5e582cb38 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f3630ff9640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f361c0b6510 0x7f361c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f3619ffb640 1 -- 172.21.15.29:0/3423491553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628036040 con 0x564ba44d92f0 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f3619ffb640 1 -- 172.21.15.29:0/3423491553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36280044c0 con 0x564ba44d92f0 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f3619ffb640 1 -- 172.21.15.29:0/3423491553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628003c20 con 0x564ba44d92f0 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f3633fff640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f361c0b6510 0x7f361c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.962+0000 7f3630ff9640 1 -- 172.21.15.29:0/3896050431 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f36240e70e0 con 0x7f3634102dd0 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 -- 172.21.15.29:0/3423491553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 msgr2=0x564ba48aa4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3633fff640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f361c0b6510 0x7f361c0b8a20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f362c005cc0 tx=0x7f362c00b340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa4c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f362802bc40 tx=0x7f3628005fd0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 -- 172.21.15.29:0/3423491553 shutdown_connections 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48ab010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa4c0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 --2- 172.21.15.29:0/3423491553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 0x564ba48a9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 -- 172.21.15.29:0/3423491553 >> 172.21.15.29:0/3423491553 conn(0x564ba44f6300 msgr2=0x564ba48a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 -- 172.21.15.29:0/3423491553 shutdown_connections 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.963+0000 7f3640ff2740 1 -- 172.21.15.29:0/3423491553 wait complete. 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 1 -- start start 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 1 Processor -- start 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44d92f0 0x564ba48b3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44e14d0 0x564ba48b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48b4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564ba4743ab0 con 0x564ba44e14d0 2024-09-21T03:01:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x564ba47b1730 con 0x564ba45021b0 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564ba48abad0 con 0x564ba44d92f0 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3640ff2740 10 client.0 _ll_register_callbacks cb 0x564ba48056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3632ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44e14d0 0x564ba48b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3632ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44e14d0 0x564ba48b3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45112/0 (socket says 172.21.15.29:45112) 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3632ffd640 1 -- 172.21.15.29:0/4014239239 learned_addr learned my addr 172.21.15.29:0/4014239239 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f36337fe640 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44d92f0 0x564ba48b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3633fff640 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3632ffd640 1 -- 172.21.15.29:0/4014239239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44d92f0 msgr2=0x564ba48b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3632ffd640 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44d92f0 0x564ba48b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3632ffd640 1 -- 172.21.15.29:0/4014239239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 msgr2=0x564ba48b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3632ffd640 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48b4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3632ffd640 1 -- 172.21.15.29:0/4014239239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ba456d790 con 0x564ba44e14d0 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.964+0000 7f3633fff640 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:01:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3632ffd640 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44e14d0 0x564ba48b3c20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f36240a94c0 tx=0x7f36240a8d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3a81690f22e367 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3612ffd640 1 -- 172.21.15.29:0/4014239239 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36240ac660 con 0x564ba44e14d0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3612ffd640 1 -- 172.21.15.29:0/4014239239 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f36240e9110 con 0x564ba44e14d0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 1 -- 172.21.15.29:0/4014239239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564ba481e640 con 0x564ba44e14d0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3612ffd640 1 -- 172.21.15.29:0/4014239239 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f36240aca00 con 0x564ba44e14d0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 10 client.18453 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 1 -- 172.21.15.29:0/4014239239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564ba45e6620 con 0x564ba44e14d0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3612ffd640 1 -- 172.21.15.29:0/4014239239 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x564ba481e640 con 0x564ba44e14d0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3612ffd640 1 -- 172.21.15.29:0/4014239239 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f36240a8840 con 0x564ba44e14d0 2024-09-21T03:01:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 10 client.18453 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 10 client.18453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 20 client.18453 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ba44c0d80 0x564ba48c0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 4 client.18453 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.965+0000 7f3640ff2740 1 -- 172.21.15.29:0/4014239239 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ba481a0f0 con 0x564ba44c0d80 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.966+0000 7f36337fe640 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ba44c0d80 0x564ba48c0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.966+0000 7f36337fe640 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ba44c0d80 0x564ba48c0c60 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb94992dc3599ee6 server_cookie=a557c265b3dad8a1 in_seq=0 out_seq=0 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.966+0000 7f3612ffd640 10 client.18453 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3612ffd640 1 -- 172.21.15.29:0/4014239239 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564ba481a0f0 con 0x564ba44c0d80 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3612ffd640 10 client.18453 handle_command_reply: tid=0 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 client.18453 shutdown 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 -- 172.21.15.29:0/4014239239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44e14d0 msgr2=0x564ba48b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44e14d0 0x564ba48b3c20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f36240a94c0 tx=0x7f36240a8d90 comp rx=0 tx=0).stop 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 -- 172.21.15.29:0/4014239239 shutdown_connections 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ba44c0d80 0x564ba48c0c60 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44e14d0 0x564ba48b3c20 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 --2- 172.21.15.29:0/4014239239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44d92f0 0x564ba48b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3612ffd640 0 client.18453 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 -- 172.21.15.29:0/4014239239 >> 172.21.15.29:0/4014239239 conn(0x564ba44f6300 msgr2=0x564ba48a3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 -- 172.21.15.29:0/4014239239 shutdown_connections 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 1 -- 172.21.15.29:0/4014239239 wait complete. 2024-09-21T03:01:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.967+0000 7f3640ff2740 20 client.18453 trim_cache size 0 max 16384 2024-09-21T03:01:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.982+0000 7f3640ff2740 1 Processor -- start 2024-09-21T03:01:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.982+0000 7f3640ff2740 1 -- start start 2024-09-21T03:01:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 0x564ba48b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f3640ff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36341ac560 con 0x564ba44d92f0 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f3640ff2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f36341ae0a0 con 0x564ba44e14d0 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f3640ff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f36341acc30 con 0x564ba45021b0 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45114/0 (socket says 172.21.15.29:45114) 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 -- 172.21.15.29:0/2616623279 learned_addr learned my addr 172.21.15.29:0/2616623279 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f3633fff640 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 0x564ba48b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f3632ffd640 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 -- 172.21.15.29:0/2616623279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 msgr2=0x564ba48b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 0x564ba48b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 -- 172.21.15.29:0/2616623279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 msgr2=0x564ba48b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 -- 172.21.15.29:0/2616623279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ba48b4b00 con 0x564ba44d92f0 2024-09-21T03:01:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36337fe640 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa500 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f3628030770 tx=0x7f362802b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe43f03fc3c00aa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.983+0000 7f36197fa640 1 -- 172.21.15.29:0/2616623279 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628053070 con 0x564ba44d92f0 2024-09-21T03:01:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f36197fa640 1 -- 172.21.15.29:0/2616623279 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3628030ea0 con 0x564ba44d92f0 2024-09-21T03:01:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f36197fa640 1 -- 172.21.15.29:0/2616623279 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628036040 con 0x564ba44d92f0 2024-09-21T03:01:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 -- 172.21.15.29:0/2616623279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 msgr2=0x564ba48aa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa500 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f3628030770 tx=0x7f362802b720 comp rx=0 tx=0).stop 2024-09-21T03:01:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 -- 172.21.15.29:0/2616623279 shutdown_connections 2024-09-21T03:01:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba45021b0 0x564ba48b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba44e14d0 0x564ba48b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 --2- 172.21.15.29:0/2616623279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48aa500 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 -- 172.21.15.29:0/2616623279 >> 172.21.15.29:0/2616623279 conn(0x564ba44f6300 msgr2=0x564ba48a9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 -- 172.21.15.29:0/2616623279 shutdown_connections 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.984+0000 7f3640ff2740 1 -- 172.21.15.29:0/2616623279 wait complete. 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 1 -- start start 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 1 Processor -- start 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48bb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44e14d0 0x564ba48b9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48ba490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36341ad9d0 con 0x564ba44d92f0 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f36341ad300 con 0x564ba45021b0 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f36341aa540 con 0x564ba44e14d0 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3640ff2740 10 client.0 _ll_register_callbacks cb 0x7f36240044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f36337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48bb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f36337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48bb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45120/0 (socket says 172.21.15.29:45120) 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f36337fe640 1 -- 172.21.15.29:0/458154500 learned_addr learned my addr 172.21.15.29:0/458154500 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3633fff640 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48ba490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.985+0000 7f3632ffd640 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44e14d0 0x564ba48b9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f36337fe640 1 -- 172.21.15.29:0/458154500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44e14d0 msgr2=0x564ba48b9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f36337fe640 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44e14d0 0x564ba48b9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f36337fe640 1 -- 172.21.15.29:0/458154500 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 msgr2=0x564ba48ba490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f36337fe640 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48ba490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f36337fe640 1 -- 172.21.15.29:0/458154500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ba48bab80 con 0x564ba44d92f0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f36337fe640 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48bb290 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f36280057c0 tx=0x7f3628030ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5986c00b553ac841 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f3612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f3612ffd640 1 -- 172.21.15.29:0/458154500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628036070 con 0x564ba44d92f0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f3612ffd640 1 -- 172.21.15.29:0/458154500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7f362803dd80 con 0x564ba44d92f0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f3640ff2740 1 -- 172.21.15.29:0/458154500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3628030ea0 con 0x564ba44d92f0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f3612ffd640 1 -- 172.21.15.29:0/458154500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3628053070 con 0x564ba44d92f0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f3640ff2740 10 client.18459 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f3640ff2740 1 -- 172.21.15.29:0/458154500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564ba492b5e0 con 0x564ba44d92f0 2024-09-21T03:01:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.986+0000 7f3612ffd640 1 -- 172.21.15.29:0/458154500 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3628030ea0 con 0x564ba44d92f0 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3612ffd640 1 -- 172.21.15.29:0/458154500 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f362800b040 con 0x564ba44d92f0 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3640ff2740 10 client.18459 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3640ff2740 10 client.18459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3640ff2740 20 client.18459 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3640ff2740 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ba44961c0 0x564ba492da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3640ff2740 4 client.18459 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3640ff2740 1 -- 172.21.15.29:0/458154500 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ba4817520 con 0x564ba44961c0 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3632ffd640 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ba44961c0 0x564ba492da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3612ffd640 10 client.18459 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:56.987+0000 7f3632ffd640 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ba44961c0 0x564ba492da00 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c55d5713508d3f6d server_cookie=86d27bfb2e4b9d64 in_seq=0 out_seq=0 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3612ffd640 1 -- 172.21.15.29:0/458154500 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ba4817520 con 0x564ba44961c0 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3612ffd640 10 client.18459 handle_command_reply: tid=0 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 client.18459 shutdown 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 -- 172.21.15.29:0/458154500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 msgr2=0x564ba48bb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48bb290 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f36280057c0 tx=0x7f3628030ca0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 -- 172.21.15.29:0/458154500 shutdown_connections 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x564ba45021b0 0x564ba48ba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ba44e14d0 0x564ba48b9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x564ba44961c0 0x564ba492da00 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 --2- 172.21.15.29:0/458154500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ba44d92f0 0x564ba48bb290 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3612ffd640 0 client.18459 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 -- 172.21.15.29:0/458154500 >> 172.21.15.29:0/458154500 conn(0x564ba44f6300 msgr2=0x564ba48adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 -- 172.21.15.29:0/458154500 shutdown_connections 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 1 -- 172.21.15.29:0/458154500 wait complete. 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.015+0000 7f3640ff2740 20 client.18459 trim_cache size 0 max 16384 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:01:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:01:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 -- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f361c0b6510 msgr2=0x7f361c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f361c0b6510 0x7f361c0b8a20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f362c005cc0 tx=0x7f362c00b340 comp rx=0 tx=0).stop 2024-09-21T03:01:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 -- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634102dd0 msgr2=0x7f363419a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634102dd0 0x7f363419a520 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f362400d950 tx=0x7f362400de10 comp rx=0 tx=0).stop 2024-09-21T03:01:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 -- 172.21.15.29:0/3896050431 shutdown_connections 2024-09-21T03:01:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f361c0b6510 0x7f361c0b8a20 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3634103770 0x7f363419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3634102dd0 0x7f363419a520 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 --2- 172.21.15.29:0/3896050431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3634101d50 0x7f3634199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 -- 172.21.15.29:0/3896050431 >> 172.21.15.29:0/3896050431 conn(0x7f36340fd6d0 msgr2=0x7f3634108300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 -- 172.21.15.29:0/3896050431 shutdown_connections 2024-09-21T03:01:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:01:57.016+0000 7f363a084640 1 -- 172.21.15.29:0/3896050431 wait complete. 2024-09-21T03:01:57.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.31 MB/sec execute 454 sec latency 28992.868 ms 2024-09-21T03:01:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:57 smithi029 ceph-mon[31006]: pgmap v1074: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-09-21T03:01:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:57 smithi066 ceph-mon[33436]: pgmap v1074: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-09-21T03:01:58.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:57 smithi190 ceph-mon[33871]: pgmap v1074: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-09-21T03:01:58.460 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id 955799b9 --release --await' 2024-09-21T03:01:58.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id 955799b9 --release --await 2024-09-21T03:01:58.726 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:01:58.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57586 5.30 MB/sec execute 455 sec latency 29992.905 ms 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.272+0000 7fb3ecb30640 1 -- 172.21.15.29:0/2748046272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81020a0 msgr2=0x7fb3e81024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.272+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/2748046272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81020a0 0x7fb3e81024b0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009900 tx=0x7fb3d802a3f0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.272+0000 7fb3ecb30640 1 -- 172.21.15.29:0/2748046272 shutdown_connections 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.272+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/2748046272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3e8106c40 0x7fb3e81106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.272+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/2748046272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3e81062f0 0x7fb3e8106700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.272+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/2748046272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81020a0 0x7fb3e81024b0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.273+0000 7fb3ecb30640 1 -- 172.21.15.29:0/2748046272 >> 172.21.15.29:0/2748046272 conn(0x7fb3e8079f90 msgr2=0x7fb3e807a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.273+0000 7fb3ecb30640 1 -- 172.21.15.29:0/2748046272 shutdown_connections 2024-09-21T03:01:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.273+0000 7fb3ecb30640 1 -- 172.21.15.29:0/2748046272 wait complete. 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.273+0000 7fb3ecb30640 1 Processor -- start 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.273+0000 7fb3ecb30640 1 -- start start 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3ecb30640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3e81020a0 0x7fb3e819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3ecb30640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81062f0 0x7fb3e819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3ecb30640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3e8106c40 0x7fb3e819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3ecb30640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8113a10 con 0x7fb3e81062f0 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3ecb30640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8113890 con 0x7fb3e8106c40 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3ecb30640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8113b90 con 0x7fb3e81020a0 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81062f0 0x7fb3e819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81062f0 0x7fb3e819a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45140/0 (socket says 172.21.15.29:45140) 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 -- 172.21.15.29:0/3113338855 learned_addr learned my addr 172.21.15.29:0/3113338855 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e659c640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3e81020a0 0x7fb3e819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3e8106c40 0x7fb3e819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 -- 172.21.15.29:0/3113338855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3e81020a0 msgr2=0x7fb3e819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3e81020a0 0x7fb3e819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 -- 172.21.15.29:0/3113338855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3e8106c40 msgr2=0x7fb3e819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3e8106c40 0x7fb3e819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 -- 172.21.15.29:0/3113338855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e81a03b0 con 0x7fb3e81062f0 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.274+0000 7fb3e5d9b640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81062f0 0x7fb3e819a9e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00d950 tx=0x7fb3dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.275+0000 7fb3d77fe640 1 -- 172.21.15.29:0/3113338855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc017070 con 0x7fb3e81062f0 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.275+0000 7fb3ecb30640 1 -- 172.21.15.29:0/3113338855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e81a06a0 con 0x7fb3e81062f0 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.275+0000 7fb3d77fe640 1 -- 172.21.15.29:0/3113338855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3dc009c30 con 0x7fb3e81062f0 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.275+0000 7fb3d77fe640 1 -- 172.21.15.29:0/3113338855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc0047a0 con 0x7fb3e81062f0 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.275+0000 7fb3ecb30640 1 -- 172.21.15.29:0/3113338855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e81a6e00 con 0x7fb3e81062f0 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.277+0000 7fb3f30a4740 1 Processor -- start 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3f30a4740 1 -- start start 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 0x561734a9e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3d77fe640 1 -- 172.21.15.29:0/3113338855 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb3dc004940 con 0x7fb3e81062f0 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734a9ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561734989dc0 con 0x5617346cf660 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561734a161f0 con 0x5617346d7840 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561734a05620 con 0x5617346f8520 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45144/0 (socket says 172.21.15.29:45144) 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e659c640 1 -- 172.21.15.29:0/997396698 learned_addr learned my addr 172.21.15.29:0/997396698 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 0x561734a9e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e5d9b640 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734a9ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e659c640 1 -- 172.21.15.29:0/997396698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 msgr2=0x561734a9e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e659c640 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 0x561734a9e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e659c640 1 -- 172.21.15.29:0/997396698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 msgr2=0x561734a9ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e659c640 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734a9ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.278+0000 7fb3e659c640 1 -- 172.21.15.29:0/997396698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617349cdfd0 con 0x5617346cf660 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.279+0000 7fb3e659c640 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9ea50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009900 tx=0x7fb3d8004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee57a2cfbdc1201 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.279+0000 7fb3d4ff9640 1 -- 172.21.15.29:0/997396698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8040070 con 0x5617346cf660 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.279+0000 7fb3d4ff9640 1 -- 172.21.15.29:0/997396698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3d80046c0 con 0x5617346cf660 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.279+0000 7fb3d4ff9640 1 -- 172.21.15.29:0/997396698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8035040 con 0x5617346cf660 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.279+0000 7fb3d77fe640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3d00b6510 0x7fb3d00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.279+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3d00b6510 0x7fb3d00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.279+0000 7fb3d77fe640 1 -- 172.21.15.29:0/3113338855 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb3dc0e7330 con 0x7fb3e81062f0 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 -- 172.21.15.29:0/997396698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 msgr2=0x561734a9ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3d00b6510 0x7fb3d00b8a20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0008dc0 tx=0x7fb3e0008570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9ea50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009900 tx=0x7fb3d8004310 comp rx=0 tx=0).stop 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 -- 172.21.15.29:0/997396698 shutdown_connections 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734a9ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9ea50 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/997396698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 0x561734a9e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 -- 172.21.15.29:0/997396698 >> 172.21.15.29:0/997396698 conn(0x5617346ec910 msgr2=0x561734a9c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 -- 172.21.15.29:0/997396698 shutdown_connections 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 -- 172.21.15.29:0/997396698 wait complete. 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 -- start start 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.280+0000 7fb3f30a4740 1 Processor -- start 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 0x561734aa9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734aaa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5617349a0300 con 0x5617346f8520 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561734a09430 con 0x5617346d7840 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561734866bb0 con 0x5617346cf660 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3f30a4740 10 client.0 _ll_register_callbacks cb 0x5617349ba120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734aaa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734aaa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45156/0 (socket says 172.21.15.29:45156) 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e6d9d640 1 -- 172.21.15.29:0/2226712297 learned_addr learned my addr 172.21.15.29:0/2226712297 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e659c640 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 0x561734aa9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e5d9b640 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e6d9d640 1 -- 172.21.15.29:0/2226712297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 msgr2=0x561734aa9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 0x561734aa9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e6d9d640 1 -- 172.21.15.29:0/2226712297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 msgr2=0x561734aaa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.281+0000 7fb3e6d9d640 1 -- 172.21.15.29:0/2226712297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56173498f780 con 0x5617346f8520 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734aaa5f0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000edb0 tx=0x7fb3e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70379af0e4b02a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3cdffb640 1 -- 172.21.15.29:0/2226712297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e001cd30 con 0x5617346f8520 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3cdffb640 1 -- 172.21.15.29:0/2226712297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb3e0027430 con 0x5617346f8520 2024-09-21T03:01:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2226712297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3d80046c0 con 0x5617346f8520 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3cdffb640 1 -- 172.21.15.29:0/2226712297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb3e0030050 con 0x5617346f8520 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3f30a4740 10 client.18471 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2226712297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561734763b00 con 0x5617346f8520 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3cdffb640 1 -- 172.21.15.29:0/2226712297 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3d80046c0 con 0x5617346f8520 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3cdffb640 1 -- 172.21.15.29:0/2226712297 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fb3e001a390 con 0x5617346f8520 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3f30a4740 10 client.18471 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3f30a4740 10 client.18471 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.282+0000 7fb3f30a4740 20 client.18471 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.283+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617346b6ea0 0x561734abc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.283+0000 7fb3f30a4740 4 client.18471 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.283+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2226712297 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561734a18140 con 0x5617346b6ea0 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.283+0000 7fb3e659c640 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617346b6ea0 0x561734abc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.283+0000 7fb3cdffb640 10 client.18471 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.283+0000 7fb3e659c640 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617346b6ea0 0x561734abc010 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a5c39b970345177 server_cookie=4b109477aec564a3 in_seq=0 out_seq=0 2024-09-21T03:01:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3cdffb640 1 -- 172.21.15.29:0/2226712297 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561734a18140 con 0x5617346b6ea0 2024-09-21T03:01:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3cdffb640 10 client.18471 handle_command_reply: tid=0 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 client.18471 shutdown 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2226712297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 msgr2=0x561734aaa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734aaa5f0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000edb0 tx=0x7fb3e0004a40 comp rx=0 tx=0).stop 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2226712297 shutdown_connections 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5617346b6ea0 0x561734abc010 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734aaa5f0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2226712297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 0x561734aa9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2226712297 >> 172.21.15.29:0/2226712297 conn(0x5617346ec910 msgr2=0x561734aa16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3cdffb640 0 client.18471 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2226712297 shutdown_connections 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.284+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2226712297 wait complete. 2024-09-21T03:01:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.285+0000 7fb3f30a4740 20 client.18471 trim_cache size 0 max 16384 2024-09-21T03:01:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3f30a4740 1 Processor -- start 2024-09-21T03:01:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3f30a4740 1 -- start start 2024-09-21T03:01:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 0x561734aab780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e81b0a00 con 0x5617346cf660 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e81b2540 con 0x5617346d7840 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e81b10d0 con 0x5617346f8520 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45168/0 (socket says 172.21.15.29:45168) 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.302+0000 7fb3e659c640 1 -- 172.21.15.29:0/2215887465 learned_addr learned my addr 172.21.15.29:0/2215887465 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 0x561734aab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e5d9b640 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e659c640 1 -- 172.21.15.29:0/2215887465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 msgr2=0x561734aab780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e659c640 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 0x561734aab780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e659c640 1 -- 172.21.15.29:0/2215887465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 msgr2=0x561734aaa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e659c640 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e659c640 1 -- 172.21.15.29:0/2215887465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617347da960 con 0x5617346cf660 2024-09-21T03:01:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e5d9b640 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:01:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3e659c640 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9fbb0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x561734a9d190 tx=0x7fb3d8004f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e9119a5b6398cf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3cffff640 1 -- 172.21.15.29:0/2215887465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8040070 con 0x5617346cf660 2024-09-21T03:01:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3cffff640 1 -- 172.21.15.29:0/2215887465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3d8004d70 con 0x5617346cf660 2024-09-21T03:01:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.303+0000 7fb3cffff640 1 -- 172.21.15.29:0/2215887465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8035040 con 0x5617346cf660 2024-09-21T03:01:59.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2215887465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 msgr2=0x561734a9fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9fbb0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x561734a9d190 tx=0x7fb3d8004f30 comp rx=0 tx=0).stop 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2215887465 shutdown_connections 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346f8520 0x561734aab780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734aaa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/2215887465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346cf660 0x561734a9fbb0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2215887465 >> 172.21.15.29:0/2215887465 conn(0x5617346ec910 msgr2=0x561734aab070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2215887465 shutdown_connections 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 -- 172.21.15.29:0/2215887465 wait complete. 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.304+0000 7fb3f30a4740 1 -- start start 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3f30a4740 1 Processor -- start 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 0x561734ab06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734ab0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3f30a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734b1fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e81b1e70 con 0x5617346f8520 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e81b17a0 con 0x5617346d7840 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3f30a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e81ae9e0 con 0x5617346cf660 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3f30a4740 10 client.0 _ll_register_callbacks cb 0x7fb3e0008eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734b1fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734b1fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45170/0 (socket says 172.21.15.29:45170) 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e6d9d640 1 -- 172.21.15.29:0/345169366 learned_addr learned my addr 172.21.15.29:0/345169366 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e5d9b640 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734ab0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e659c640 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 0x561734ab06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e6d9d640 1 -- 172.21.15.29:0/345169366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 msgr2=0x561734ab06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 0x561734ab06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e6d9d640 1 -- 172.21.15.29:0/345169366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 msgr2=0x561734ab0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734ab0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.305+0000 7fb3e6d9d640 1 -- 172.21.15.29:0/345169366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561734b200d0 con 0x5617346f8520 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3e6d9d640 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734b1fa60 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0016040 tx=0x7fb3e000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b36b9069ece02 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3cdffb640 1 -- 172.21.15.29:0/345169366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e001ab60 con 0x5617346f8520 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3cdffb640 1 -- 172.21.15.29:0/345169366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7fb3e001cd30 con 0x5617346f8520 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3f30a4740 1 -- 172.21.15.29:0/345169366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561734a13bf0 con 0x5617346f8520 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3cdffb640 1 -- 172.21.15.29:0/345169366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb3e001c770 con 0x5617346f8520 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3f30a4740 10 client.18477 fetch_fsmap learned FSMap version 37 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3f30a4740 1 -- 172.21.15.29:0/345169366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561734b27380 con 0x5617346f8520 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3cdffb640 1 -- 172.21.15.29:0/345169366 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x561734a13bf0 con 0x5617346f8520 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3cdffb640 1 -- 172.21.15.29:0/345169366 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fb3e0033070 con 0x5617346f8520 2024-09-21T03:01:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.306+0000 7fb3f30a4740 10 client.18477 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:01:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.307+0000 7fb3f30a4740 10 client.18477 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:01:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.307+0000 7fb3f30a4740 20 client.18477 populate_metadata read hostname 'smithi029' 2024-09-21T03:01:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.307+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56173468c250 0x561734b29b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:01:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.307+0000 7fb3f30a4740 4 client.18477 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "955799b9", "release": true, "await": true}] 2024-09-21T03:01:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.307+0000 7fb3f30a4740 1 -- 172.21.15.29:0/345169366 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "955799b9", "release": true, "await": true}) -- 0x561734a11020 con 0x56173468c250 2024-09-21T03:01:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.307+0000 7fb3e659c640 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56173468c250 0x561734b29b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:01:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.307+0000 7fb3cdffb640 10 client.18477 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.307+0000 7fb3e659c640 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56173468c250 0x561734b29b90 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8270a955db3a048e server_cookie=f0622363b47dedf8 in_seq=0 out_seq=0 2024-09-21T03:01:59.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.364+0000 7fb3cdffb640 1 -- 172.21.15.29:0/345169366 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561734a11020 con 0x56173468c250 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.364+0000 7fb3cdffb640 10 client.18477 handle_command_reply: tid=0 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 client.18477 shutdown 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 -- 172.21.15.29:0/345169366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 msgr2=0x561734b1fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734b1fa60 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0016040 tx=0x7fb3e000ef20 comp rx=0 tx=0).stop 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 -- 172.21.15.29:0/345169366 shutdown_connections 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56173468c250 0x561734b29b90 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617346f8520 0x561734b1fa60 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5617346d7840 0x561734ab0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 --2- 172.21.15.29:0/345169366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617346cf660 0x561734ab06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 -- 172.21.15.29:0/345169366 >> 172.21.15.29:0/345169366 conn(0x5617346ec910 msgr2=0x561734aaa250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3cdffb640 0 client.18477 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:01:59.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 -- 172.21.15.29:0/345169366 shutdown_connections 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 1 -- 172.21.15.29:0/345169366 wait complete. 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.365+0000 7fb3f30a4740 20 client.18477 trim_cache size 0 max 16384 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 37, 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 88, 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stdout: "955799b9": { 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 88, 2024-09-21T03:01:59.368 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 149.8, 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:01:59.369 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:01:59.370 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:01:59.370 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:01:59.370 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:59.370 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:59.370 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:59.370 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:59.370 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:01:59.370 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:01:59.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 -- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3d00b6510 msgr2=0x7fb3d00b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3d00b6510 0x7fb3d00b8a20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0008dc0 tx=0x7fb3e0008570 comp rx=0 tx=0).stop 2024-09-21T03:01:59.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 -- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81062f0 msgr2=0x7fb3e819a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:01:59.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81062f0 0x7fb3e819a9e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00d950 tx=0x7fb3dc00de10 comp rx=0 tx=0).stop 2024-09-21T03:01:59.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 -- 172.21.15.29:0/3113338855 shutdown_connections 2024-09-21T03:01:59.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb3d00b6510 0x7fb3d00b8a20 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3e8106c40 0x7fb3e819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3e81062f0 0x7fb3e819a9e0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 --2- 172.21.15.29:0/3113338855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3e81020a0 0x7fb3e819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:01:59.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 -- 172.21.15.29:0/3113338855 >> 172.21.15.29:0/3113338855 conn(0x7fb3e8079f90 msgr2=0x7fb3e8107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:01:59.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 -- 172.21.15.29:0/3113338855 shutdown_connections 2024-09-21T03:01:59.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:01:59.366+0000 7fb3ecb30640 1 -- 172.21.15.29:0/3113338855 wait complete. 2024-09-21T03:01:59.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57640 5.29 MB/sec execute 456 sec latency 30605.527 ms 2024-09-21T03:01:59.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:01:59 smithi029 ceph-mon[31006]: pgmap v1075: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-09-21T03:02:00.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:01:59 smithi066 ceph-mon[33436]: pgmap v1075: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-09-21T03:02:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:01:59 smithi190 ceph-mon[33871]: pgmap v1075: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-09-21T03:02:00.348 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '955799b9', total seconds elapsed: 35.8 2024-09-21T03:02:00.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57780 5.29 MB/sec execute 457 sec latency 86.869 ms 2024-09-21T03:02:01.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57918 5.29 MB/sec execute 458 sec latency 89.344 ms 2024-09-21T03:02:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:01 smithi029 ceph-mon[31006]: pgmap v1076: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-21T03:02:02.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:01 smithi066 ceph-mon[33436]: pgmap v1076: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-21T03:02:02.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:01 smithi190 ceph-mon[33871]: pgmap v1076: 129 pgs: 129 active+clean; 319 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-21T03:02:02.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58057 5.29 MB/sec execute 459 sec latency 103.966 ms 2024-09-21T03:02:03.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58181 5.29 MB/sec execute 460 sec latency 241.198 ms 2024-09-21T03:02:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:03 smithi066 ceph-mon[33436]: pgmap v1077: 129 pgs: 129 active+clean; 321 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 219 KiB/s wr, 33 op/s 2024-09-21T03:02:04.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:03 smithi190 ceph-mon[33871]: pgmap v1077: 129 pgs: 129 active+clean; 321 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 219 KiB/s wr, 33 op/s 2024-09-21T03:02:04.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:03 smithi029 ceph-mon[31006]: pgmap v1077: 129 pgs: 129 active+clean; 321 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 219 KiB/s wr, 33 op/s 2024-09-21T03:02:04.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58370 5.32 MB/sec execute 461 sec latency 267.473 ms 2024-09-21T03:02:05.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58491 5.31 MB/sec execute 462 sec latency 197.723 ms 2024-09-21T03:02:05.934 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:05 smithi029 ceph-mon[31006]: pgmap v1078: 129 pgs: 129 active+clean; 320 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 830 KiB/s wr, 62 op/s 2024-09-21T03:02:06.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:05 smithi066 ceph-mon[33436]: pgmap v1078: 129 pgs: 129 active+clean; 320 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 830 KiB/s wr, 62 op/s 2024-09-21T03:02:06.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:05 smithi190 ceph-mon[33871]: pgmap v1078: 129 pgs: 129 active+clean; 320 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 830 KiB/s wr, 62 op/s 2024-09-21T03:02:06.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58627 5.31 MB/sec execute 463 sec latency 132.849 ms 2024-09-21T03:02:07.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58805 5.32 MB/sec execute 464 sec latency 93.732 ms 2024-09-21T03:02:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:07 smithi066 ceph-mon[33436]: pgmap v1079: 129 pgs: 129 active+clean; 321 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-09-21T03:02:08.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:07 smithi190 ceph-mon[33871]: pgmap v1079: 129 pgs: 129 active+clean; 321 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-09-21T03:02:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:07 smithi029 ceph-mon[31006]: pgmap v1079: 129 pgs: 129 active+clean; 321 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-09-21T03:02:08.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58972 5.33 MB/sec execute 465 sec latency 94.480 ms 2024-09-21T03:02:09.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 59138 5.34 MB/sec execute 466 sec latency 198.427 ms 2024-09-21T03:02:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:09 smithi066 ceph-mon[33436]: pgmap v1080: 129 pgs: 129 active+clean; 319 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.8 MiB/s wr, 141 op/s 2024-09-21T03:02:10.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:09 smithi190 ceph-mon[33871]: pgmap v1080: 129 pgs: 129 active+clean; 319 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.8 MiB/s wr, 141 op/s 2024-09-21T03:02:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:09 smithi029 ceph-mon[31006]: pgmap v1080: 129 pgs: 129 active+clean; 319 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.8 MiB/s wr, 141 op/s 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2949496429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8bc103620 msgr2=0x7ff8bc103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2949496429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8bc103620 0x7ff8bc103a30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009900 tx=0x7ff8b002a3f0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2949496429 shutdown_connections 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2949496429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8bc103fc0 0x7ff8bc10a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2949496429 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8bc103620 0x7ff8bc103a30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2949496429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8bc1025a0 0x7ff8bc102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2949496429 >> 172.21.15.29:0/2949496429 conn(0x7ff8bc0fdee0 msgr2=0x7ff8bc100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2949496429 shutdown_connections 2024-09-21T03:02:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.143+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2949496429 wait complete. 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.144+0000 7ff8c1a30640 1 Processor -- start 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.144+0000 7ff8c1a30640 1 -- start start 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.144+0000 7ff8c1a30640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8bc1025a0 0x7ff8bc198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.144+0000 7ff8c1a30640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8bc103620 0x7ff8bc199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.144+0000 7ff8c1a30640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8bc103fc0 0x7ff8bc199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.144+0000 7ff8c1a30640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc10e0b0 con 0x7ff8bc103620 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.144+0000 7ff8c1a30640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc10df30 con 0x7ff8bc103fc0 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.144+0000 7ff8c1a30640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc10e230 con 0x7ff8bc1025a0 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8ba7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8bc103620 0x7ff8bc199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8bc1025a0 0x7ff8bc198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8bc1025a0 0x7ff8bc198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40226/0 (socket says 172.21.15.29:40226) 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 -- 172.21.15.29:0/2750559288 learned_addr learned my addr 172.21.15.29:0/2750559288 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8bc103fc0 0x7ff8bc199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 -- 172.21.15.29:0/2750559288 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8bc103fc0 msgr2=0x7ff8bc199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8bc103fc0 0x7ff8bc199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 -- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8bc103620 msgr2=0x7ff8bc199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8bc103620 0x7ff8bc199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 -- 172.21.15.29:0/2750559288 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc19a7e0 con 0x7ff8bc1025a0 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8bc103620 0x7ff8bc199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8baffd640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8bc1025a0 0x7ff8bc198da0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00e9f0 tx=0x7ff8ac00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.145+0000 7ff8c0a2e640 1 -- 172.21.15.29:0/2750559288 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00cd80 con 0x7ff8bc1025a0 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.146+0000 7ff8c0a2e640 1 -- 172.21.15.29:0/2750559288 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8ac0054b0 con 0x7ff8bc1025a0 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.146+0000 7ff8c0a2e640 1 -- 172.21.15.29:0/2750559288 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac010430 con 0x7ff8bc1025a0 2024-09-21T03:02:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.146+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2750559288 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8bc19aad0 con 0x7ff8bc1025a0 2024-09-21T03:02:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.146+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2750559288 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8bc1a11c0 con 0x7ff8bc1025a0 2024-09-21T03:02:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8c8a7c740 1 Processor -- start 2024-09-21T03:02:10.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8c8a7c740 1 -- start start 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3421b0 0x55987a6ec060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3192f0 0x55987a6ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3214d0 0x55987a6ed220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55987a65c6f0 con 0x55987a3214d0 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55987a64bb20 con 0x55987a3192f0 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55987a64f930 con 0x55987a3421b0 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3421b0 0x55987a6ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8ba7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3214d0 0x55987a6ed220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3421b0 0x55987a6ec060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40232/0 (socket says 172.21.15.29:40232) 2024-09-21T03:02:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.149+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/3198938854 learned_addr learned my addr 172.21.15.29:0/3198938854 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/3198938854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3192f0 msgr2=0x55987a6ec6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/3198938854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3192f0 0x55987a6ec6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/3198938854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3214d0 msgr2=0x55987a6ed220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/3198938854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3214d0 0x55987a6ed220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/3198938854 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55987a611530 con 0x55987a3421b0 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8c0a2e640 1 -- 172.21.15.29:0/2750559288 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ff8ac005020 con 0x7ff8bc1025a0 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/3198938854 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3421b0 0x55987a6ec060 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4011470 tx=0x7ff8b4011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d582d8582c8b3e95 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8a1ffb640 1 -- 172.21.15.29:0/3198938854 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b401e070 con 0x55987a3421b0 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8a1ffb640 1 -- 172.21.15.29:0/3198938854 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8b400f9d0 con 0x55987a3421b0 2024-09-21T03:02:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.150+0000 7ff8a1ffb640 1 -- 172.21.15.29:0/3198938854 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4019b20 con 0x55987a3421b0 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.151+0000 7ff8c0a2e640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8a40b6590 0x7ff8a40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.151+0000 7ff8baffd640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8a40b6590 0x7ff8a40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.151+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/3198938854 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3421b0 msgr2=0x55987a6ec060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/3198938854 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3421b0 0x55987a6ec060 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4011470 tx=0x7ff8b4011930 comp rx=0 tx=0).stop 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c0a2e640 1 -- 172.21.15.29:0/2750559288 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff8ac0e7a50 con 0x7ff8bc1025a0 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/3198938854 shutdown_connections 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/3198938854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3214d0 0x55987a6ed220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/3198938854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3192f0 0x55987a6ec6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/3198938854 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3421b0 0x55987a6ec060 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/3198938854 >> 172.21.15.29:0/3198938854 conn(0x55987a336300 msgr2=0x55987a6e58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8baffd640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8a40b6590 0x7ff8a40b8aa0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00e9c0 tx=0x7ff8ac0de2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/3198938854 shutdown_connections 2024-09-21T03:02:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.152+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/3198938854 wait complete. 2024-09-21T03:02:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.153+0000 7ff8c8a7c740 1 -- start start 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.153+0000 7ff8c8a7c740 1 Processor -- start 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.153+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3192f0 0x55987a6f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.153+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3214d0 0x55987a6fc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.153+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3421b0 0x55987a6fecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.153+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55987a583ab0 con 0x55987a3421b0 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.153+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55987a5f1730 con 0x55987a3214d0 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.153+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55987a6edd40 con 0x55987a3192f0 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8ba7fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3214d0 0x55987a6fc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3192f0 0x55987a6f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8c8a7c740 10 client.0 _ll_register_callbacks cb 0x55987a6456b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3192f0 0x55987a6f3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40248/0 (socket says 172.21.15.29:40248) 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/840990011 learned_addr learned my addr 172.21.15.29:0/840990011 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8ba7fc640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3214d0 0x55987a6fc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:38122/0 (socket says 172.21.15.29:38122) 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3421b0 0x55987a6fecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8ba7fc640 1 -- 172.21.15.29:0/840990011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3192f0 msgr2=0x55987a6f3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3192f0 0x55987a6f3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8ba7fc640 1 -- 172.21.15.29:0/840990011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3421b0 msgr2=0x55987a6fecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3421b0 0x55987a6fecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8ba7fc640 1 -- 172.21.15.29:0/840990011 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55987a3ad790 con 0x55987a3214d0 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3214d0 0x55987a6fc600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0005760 tx=0x7ff8b0031c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11f08f5048dff2d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff89affd640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:02:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.154+0000 7ff89affd640 1 -- 172.21.15.29:0/840990011 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0035040 con 0x55987a3214d0 2024-09-21T03:02:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.155+0000 7ff89affd640 1 -- 172.21.15.29:0/840990011 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff8b0004450 con 0x55987a3214d0 2024-09-21T03:02:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.155+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/840990011 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8b400f9d0 con 0x55987a3214d0 2024-09-21T03:02:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.155+0000 7ff89affd640 1 -- 172.21.15.29:0/840990011 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff8b00035f0 con 0x55987a3214d0 2024-09-21T03:02:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.155+0000 7ff8c8a7c740 10 client.27752 fetch_fsmap learned FSMap version 37 2024-09-21T03:02:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.155+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/840990011 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55987a426620 con 0x55987a3214d0 2024-09-21T03:02:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.155+0000 7ff89affd640 1 -- 172.21.15.29:0/840990011 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8b400f9d0 con 0x55987a3214d0 2024-09-21T03:02:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.155+0000 7ff89affd640 1 -- 172.21.15.29:0/840990011 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff8b003c790 con 0x55987a3214d0 2024-09-21T03:02:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff8c8a7c740 10 client.27752 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:02:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff8c8a7c740 10 client.27752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:02:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff8c8a7c740 20 client.27752 populate_metadata read hostname 'smithi029' 2024-09-21T03:02:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55987a300d80 0x55987a7051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff8c8a7c740 4 client.27752 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:02:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/840990011 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55987a65e640 con 0x55987a300d80 2024-09-21T03:02:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55987a300d80 0x55987a7051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff89affd640 10 client.27752 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.156+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55987a300d80 0x55987a7051c0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6cd3e81de536047 server_cookie=58ac1b7833cffbe3 in_seq=0 out_seq=0 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff89affd640 1 -- 172.21.15.29:0/840990011 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55987a65e640 con 0x55987a300d80 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff89affd640 10 client.27752 handle_command_reply: tid=0 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 client.27752 shutdown 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/840990011 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3214d0 msgr2=0x55987a6fc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3214d0 0x55987a6fc600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0005760 tx=0x7ff8b0031c60 comp rx=0 tx=0).stop 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/840990011 shutdown_connections 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55987a300d80 0x55987a7051c0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3421b0 0x55987a6fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3214d0 0x55987a6fc600 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/840990011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3192f0 0x55987a6f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/840990011 >> 172.21.15.29:0/840990011 conn(0x55987a336300 msgr2=0x55987a6e3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff89affd640 0 client.27752 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/840990011 shutdown_connections 2024-09-21T03:02:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.157+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/840990011 wait complete. 2024-09-21T03:02:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.158+0000 7ff8c8a7c740 20 client.27752 trim_cache size 0 max 16384 2024-09-21T03:02:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.178+0000 7ff8c8a7c740 1 Processor -- start 2024-09-21T03:02:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8c8a7c740 1 -- start start 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a6e6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a6e05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 0x55987a6e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1aa330 con 0x55987a3192f0 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1abe70 con 0x55987a3421b0 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1aaa00 con 0x55987a3214d0 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a6e6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8ba7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a6e05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a6e6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40144/0 (socket says 172.21.15.29:40144) 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8ba7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a6e05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:40250/0 (socket says 172.21.15.29:40250) 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/2056523912 learned_addr learned my addr 172.21.15.29:0/2056523912 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 0x55987a6e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8ba7fc640 1 -- 172.21.15.29:0/2056523912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 msgr2=0x55987a6e46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 0x55987a6e46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8ba7fc640 1 -- 172.21.15.29:0/2056523912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 msgr2=0x55987a6e6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a6e6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8ba7fc640 1 -- 172.21.15.29:0/2056523912 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55987a300fb0 con 0x55987a3214d0 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.179+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a6e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.180+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a6e05c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00057c0 tx=0x7ff8b002a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59ce868f248fb6b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.180+0000 7ff8a17fa640 1 -- 172.21.15.29:0/2056523912 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0035030 con 0x55987a3214d0 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.180+0000 7ff8a17fa640 1 -- 172.21.15.29:0/2056523912 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8b0054070 con 0x55987a3214d0 2024-09-21T03:02:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.180+0000 7ff8a17fa640 1 -- 172.21.15.29:0/2056523912 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0003790 con 0x55987a3214d0 2024-09-21T03:02:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/2056523912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 msgr2=0x55987a6e05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a6e05c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00057c0 tx=0x7ff8b002a3f0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/2056523912 shutdown_connections 2024-09-21T03:02:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 0x55987a6e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a6e05c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/2056523912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a6e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/2056523912 >> 172.21.15.29:0/2056523912 conn(0x55987a336300 msgr2=0x55987a6e3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/2056523912 shutdown_connections 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.181+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/2056523912 wait complete. 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.182+0000 7ff8c8a7c740 1 -- start start 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.182+0000 7ff8c8a7c740 1 Processor -- start 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.182+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a716ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.182+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a715960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.182+0000 7ff8c8a7c740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 0x55987a715ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.182+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1ab7a0 con 0x55987a3192f0 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.182+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1ab0d0 con 0x55987a3421b0 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.182+0000 7ff8c8a7c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1a8310 con 0x55987a3214d0 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8c8a7c740 10 client.0 _ll_register_callbacks cb 0x7ff8b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a716ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a716ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40148/0 (socket says 172.21.15.29:40148) 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/1850560097 learned_addr learned my addr 172.21.15.29:0/1850560097 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a715960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 0x55987a715ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 msgr2=0x55987a715960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a715960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/1850560097 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 msgr2=0x55987a715ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 0x55987a715ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 -- 172.21.15.29:0/1850560097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55987a7163e0 con 0x55987a3192f0 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a715960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff8bb7fe640 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a716ca0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009e20 tx=0x7ff8b4005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd16de529c64305 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.183+0000 7ff89affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff89affd640 1 -- 172.21.15.29:0/1850560097 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b401e040 con 0x55987a3192f0 2024-09-21T03:02:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff89affd640 1 -- 172.21.15.29:0/1850560097 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7ff8b40110f0 con 0x55987a3192f0 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/1850560097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8b0054070 con 0x55987a3192f0 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff89affd640 1 -- 172.21.15.29:0/1850560097 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff8b40112f0 con 0x55987a3192f0 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 10 client.18495 fetch_fsmap learned FSMap version 37 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/1850560097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55987a7152c0 con 0x55987a3192f0 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff89affd640 1 -- 172.21.15.29:0/1850560097 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8b0054070 con 0x55987a3192f0 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff89affd640 1 -- 172.21.15.29:0/1850560097 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff8b4024040 con 0x55987a3192f0 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 10 client.18495 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 10 client.18495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 20 client.18495 populate_metadata read hostname 'smithi029' 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55987a2d61c0 0x55987a7720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 4 client.18495 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.184+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/1850560097 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55987a65a0f0 con 0x55987a2d61c0 2024-09-21T03:02:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.185+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55987a2d61c0 0x55987a7720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.185+0000 7ff89affd640 10 client.18495 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.185+0000 7ff8ba7fc640 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55987a2d61c0 0x55987a7720b0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc396c19cb14566 server_cookie=a3eb6b3ad9eea3bd in_seq=0 out_seq=0 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff89affd640 1 -- 172.21.15.29:0/1850560097 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55987a65a0f0 con 0x55987a2d61c0 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff89affd640 10 client.18495 handle_command_reply: tid=0 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 client.18495 shutdown 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/1850560097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 msgr2=0x55987a716ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a716ca0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009e20 tx=0x7ff8b4005ae0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/1850560097 shutdown_connections 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55987a3421b0 0x55987a715ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55987a3214d0 0x55987a715960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55987a2d61c0 0x55987a7720b0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 --2- 172.21.15.29:0/1850560097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55987a3192f0 0x55987a716ca0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/1850560097 >> 172.21.15.29:0/1850560097 conn(0x55987a336300 msgr2=0x55987a6e05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff89affd640 0 client.18495 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/1850560097 shutdown_connections 2024-09-21T03:02:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 1 -- 172.21.15.29:0/1850560097 wait complete. 2024-09-21T03:02:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.347+0000 7ff8c8a7c740 20 client.18495 trim_cache size 0 max 16384 2024-09-21T03:02:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:02:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:02:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:02:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:02:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8a40b6590 msgr2=0x7ff8a40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8a40b6590 0x7ff8a40b8aa0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00e9c0 tx=0x7ff8ac0de2a0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2750559288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8bc1025a0 msgr2=0x7ff8bc198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8bc1025a0 0x7ff8bc198da0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00e9f0 tx=0x7ff8ac00edf0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2750559288 shutdown_connections 2024-09-21T03:02:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ff8a40b6590 0x7ff8a40b8aa0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8bc103fc0 0x7ff8bc199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8bc103620 0x7ff8bc199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 --2- 172.21.15.29:0/2750559288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8bc1025a0 0x7ff8bc198da0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2750559288 >> 172.21.15.29:0/2750559288 conn(0x7ff8bc0fdee0 msgr2=0x7ff8bc106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2750559288 shutdown_connections 2024-09-21T03:02:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:10.348+0000 7ff8c1a30640 1 -- 172.21.15.29:0/2750559288 wait complete. 2024-09-21T03:02:10.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 59263 5.34 MB/sec execute 467 sec latency 196.377 ms 2024-09-21T03:02:11.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 59367 5.33 MB/sec execute 468 sec latency 107.332 ms 2024-09-21T03:02:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:11 smithi066 ceph-mon[33436]: pgmap v1081: 129 pgs: 129 active+clean; 319 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 2.2 MiB/s wr, 158 op/s 2024-09-21T03:02:12.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:11 smithi190 ceph-mon[33871]: pgmap v1081: 129 pgs: 129 active+clean; 319 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 2.2 MiB/s wr, 158 op/s 2024-09-21T03:02:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:11 smithi029 ceph-mon[31006]: pgmap v1081: 129 pgs: 129 active+clean; 319 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 2.2 MiB/s wr, 158 op/s 2024-09-21T03:02:12.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 59520 5.34 MB/sec execute 469 sec latency 88.669 ms 2024-09-21T03:02:13.744 INFO:tasks.workunit.client.0.smithi029.stdout: 10 59638 5.34 MB/sec execute 470 sec latency 295.042 ms 2024-09-21T03:02:14.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:13 smithi066 ceph-mon[33436]: pgmap v1082: 129 pgs: 129 active+clean; 318 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 2.8 MiB/s wr, 217 op/s 2024-09-21T03:02:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:13 smithi190 ceph-mon[33871]: pgmap v1082: 129 pgs: 129 active+clean; 318 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 2.8 MiB/s wr, 217 op/s 2024-09-21T03:02:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:13 smithi029 ceph-mon[31006]: pgmap v1082: 129 pgs: 129 active+clean; 318 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 2.8 MiB/s wr, 217 op/s 2024-09-21T03:02:14.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 59804 5.36 MB/sec execute 471 sec latency 92.759 ms 2024-09-21T03:02:15.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 59977 5.37 MB/sec execute 472 sec latency 96.591 ms 2024-09-21T03:02:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:15 smithi066 ceph-mon[33436]: pgmap v1083: 129 pgs: 129 active+clean; 323 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 3.5 MiB/s wr, 267 op/s 2024-09-21T03:02:16.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:15 smithi190 ceph-mon[33871]: pgmap v1083: 129 pgs: 129 active+clean; 323 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 3.5 MiB/s wr, 267 op/s 2024-09-21T03:02:16.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:15 smithi029 ceph-mon[31006]: pgmap v1083: 129 pgs: 129 active+clean; 323 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 3.5 MiB/s wr, 267 op/s 2024-09-21T03:02:16.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 60173 5.38 MB/sec execute 473 sec latency 131.308 ms 2024-09-21T03:02:17.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 60280 5.38 MB/sec execute 474 sec latency 99.363 ms 2024-09-21T03:02:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:17 smithi066 ceph-mon[33436]: pgmap v1084: 129 pgs: 129 active+clean; 324 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-21T03:02:18.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:17 smithi190 ceph-mon[33871]: pgmap v1084: 129 pgs: 129 active+clean; 324 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-21T03:02:18.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:17 smithi029 ceph-mon[31006]: pgmap v1084: 129 pgs: 129 active+clean; 324 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-21T03:02:18.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 60401 5.38 MB/sec execute 475 sec latency 91.124 ms 2024-09-21T03:02:19.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 60550 5.39 MB/sec execute 476 sec latency 85.601 ms 2024-09-21T03:02:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:19 smithi066 ceph-mon[33436]: pgmap v1085: 129 pgs: 129 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-09-21T03:02:20.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:19 smithi190 ceph-mon[33871]: pgmap v1085: 129 pgs: 129 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-09-21T03:02:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:19 smithi029 ceph-mon[31006]: pgmap v1085: 129 pgs: 129 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.4 MiB/s wr, 247 op/s 2024-09-21T03:02:20.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 60728 5.41 MB/sec execute 477 sec latency 89.705 ms 2024-09-21T03:02:21.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 60878 5.42 MB/sec execute 478 sec latency 250.629 ms 2024-09-21T03:02:21.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:21 smithi190 ceph-mon[33871]: pgmap v1086: 129 pgs: 129 active+clean; 330 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.5 MiB/s wr, 254 op/s 2024-09-21T03:02:22.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:21 smithi066 ceph-mon[33436]: pgmap v1086: 129 pgs: 129 active+clean; 330 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.5 MiB/s wr, 254 op/s 2024-09-21T03:02:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:21 smithi029 ceph-mon[31006]: pgmap v1086: 129 pgs: 129 active+clean; 330 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.5 MiB/s wr, 254 op/s 2024-09-21T03:02:22.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 61017 5.43 MB/sec execute 479 sec latency 96.425 ms 2024-09-21T03:02:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 -- 172.21.15.29:0/2389563831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feed010e0c0 msgr2=0x7feed010e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 --2- 172.21.15.29:0/2389563831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feed010e0c0 0x7feed010e530 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feec0009900 tx=0x7feec002a3f0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 -- 172.21.15.29:0/2389563831 shutdown_connections 2024-09-21T03:02:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 --2- 172.21.15.29:0/2389563831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feed010fae0 0x7feed0116480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 --2- 172.21.15.29:0/2389563831 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feed010f140 0x7feed010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 --2- 172.21.15.29:0/2389563831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feed010e0c0 0x7feed010e530 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 -- 172.21.15.29:0/2389563831 >> 172.21.15.29:0/2389563831 conn(0x7feed006e320 msgr2=0x7feed010c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 -- 172.21.15.29:0/2389563831 shutdown_connections 2024-09-21T03:02:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.480+0000 7feed560a640 1 -- 172.21.15.29:0/2389563831 wait complete. 2024-09-21T03:02:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feed560a640 1 Processor -- start 2024-09-21T03:02:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feed560a640 1 -- start start 2024-09-21T03:02:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feed560a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feed010e0c0 0x7feed01a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feed560a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feed010f140 0x7feed01a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feed560a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feed010fae0 0x7feed01a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feed560a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feed0119c40 con 0x7feed010fae0 2024-09-21T03:02:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feed560a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feed0119ac0 con 0x7feed010e0c0 2024-09-21T03:02:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feed560a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feed0119dc0 con 0x7feed010f140 2024-09-21T03:02:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feeceffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feed010e0c0 0x7feed01a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feec7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feed010f140 0x7feed01a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feecf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feed010fae0 0x7feed01a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feeceffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feed010e0c0 0x7feed01a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:59102/0 (socket says 172.21.15.29:59102) 2024-09-21T03:02:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.481+0000 7feeceffd640 1 -- 172.21.15.29:0/2349933583 learned_addr learned my addr 172.21.15.29:0/2349933583 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feecf7fe640 1 -- 172.21.15.29:0/2349933583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feed010f140 msgr2=0x7feed01a4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feecf7fe640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feed010f140 0x7feed01a4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feecf7fe640 1 -- 172.21.15.29:0/2349933583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feed010e0c0 msgr2=0x7feed01a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feecf7fe640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feed010e0c0 0x7feed01a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feecf7fe640 1 -- 172.21.15.29:0/2349933583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed01a6300 con 0x7feed010fae0 2024-09-21T03:02:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feeceffd640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feed010e0c0 0x7feed01a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feecf7fe640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feed010fae0 0x7feed01a5ae0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7feec8009fd0 tx=0x7feec800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feeccff9640 1 -- 172.21.15.29:0/2349933583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec800ce20 con 0x7feed010fae0 2024-09-21T03:02:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feed560a640 1 -- 172.21.15.29:0/2349933583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed01a65f0 con 0x7feed010fae0 2024-09-21T03:02:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feed560a640 1 -- 172.21.15.29:0/2349933583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed006ee80 con 0x7feed010fae0 2024-09-21T03:02:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feeccff9640 1 -- 172.21.15.29:0/2349933583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feec8005020 con 0x7feed010fae0 2024-09-21T03:02:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.482+0000 7feeccff9640 1 -- 172.21.15.29:0/2349933583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec8010690 con 0x7feed010fae0 2024-09-21T03:02:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.486+0000 7feeccff9640 1 -- 172.21.15.29:0/2349933583 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7feec8010830 con 0x7feed010fae0 2024-09-21T03:02:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.486+0000 7feedc652740 1 Processor -- start 2024-09-21T03:02:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.486+0000 7feedc652740 1 -- start start 2024-09-21T03:02:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feedc652740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924e51b0 0x56279288ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feedc652740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feec7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924e51b0 0x56279288ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feedc652740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924c44d0 0x56279288e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feedc652740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5627927ff6f0 con 0x5627924e51b0 2024-09-21T03:02:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feedc652740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5627927eeb20 con 0x5627924bc2f0 2024-09-21T03:02:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feedc652740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5627927f2930 con 0x5627924c44d0 2024-09-21T03:02:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feec7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924e51b0 0x56279288ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39888/0 (socket says 172.21.15.29:39888) 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feec7fff640 1 -- 172.21.15.29:0/2466861165 learned_addr learned my addr 172.21.15.29:0/2466861165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.487+0000 7feeceffd640 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feec7fff640 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924c44d0 0x56279288e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feeceffd640 1 -- 172.21.15.29:0/2466861165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924c44d0 msgr2=0x56279288e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feeceffd640 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924c44d0 0x56279288e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feeceffd640 1 -- 172.21.15.29:0/2466861165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924e51b0 msgr2=0x56279288ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feeceffd640 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924e51b0 0x56279288ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feeceffd640 1 -- 172.21.15.29:0/2466861165 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627927b4530 con 0x5627924bc2f0 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feeccff9640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feebc0b6510 0x7feebc0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feec7fff640 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924c44d0 0x56279288e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:02:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feeccff9640 1 -- 172.21.15.29:0/2349933583 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7feec80e7d30 con 0x7feed010fae0 2024-09-21T03:02:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.488+0000 7feecf7fe640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feebc0b6510 0x7feebc0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.489+0000 7feeceffd640 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288d4c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feec0006010 tx=0x7feec0004520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62db5ddc5472633d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.489+0000 7feecf7fe640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feebc0b6510 0x7feebc0b8a20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7feec801a040 tx=0x7feec80d1040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.492+0000 7feec57fa640 1 -- 172.21.15.29:0/2466861165 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0035030 con 0x5627924bc2f0 2024-09-21T03:02:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.492+0000 7feec57fa640 1 -- 172.21.15.29:0/2466861165 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feec0040070 con 0x5627924bc2f0 2024-09-21T03:02:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.492+0000 7feec57fa640 1 -- 172.21.15.29:0/2466861165 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0031c10 con 0x5627924bc2f0 2024-09-21T03:02:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.493+0000 7feedc652740 1 -- 172.21.15.29:0/2466861165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 msgr2=0x56279288d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.493+0000 7feedc652740 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288d4c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feec0006010 tx=0x7feec0004520 comp rx=0 tx=0).stop 2024-09-21T03:02:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.494+0000 7feedc652740 1 -- 172.21.15.29:0/2466861165 shutdown_connections 2024-09-21T03:02:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.494+0000 7feedc652740 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924c44d0 0x56279288e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.494+0000 7feedc652740 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288d4c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.494+0000 7feedc652740 1 --2- 172.21.15.29:0/2466861165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924e51b0 0x56279288ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.494+0000 7feedc652740 1 -- 172.21.15.29:0/2466861165 >> 172.21.15.29:0/2466861165 conn(0x5627924d9300 msgr2=0x5627928888e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.494+0000 7feedc652740 1 -- 172.21.15.29:0/2466861165 shutdown_connections 2024-09-21T03:02:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.494+0000 7feedc652740 1 -- 172.21.15.29:0/2466861165 wait complete. 2024-09-21T03:02:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 1 -- start start 2024-09-21T03:02:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 1 Processor -- start 2024-09-21T03:02:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792896630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792896cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924e51b0 0x562792897860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562792726ab0 con 0x5627924c44d0 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x562792794730 con 0x5627924bc2f0 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56279288ead0 con 0x5627924e51b0 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.495+0000 7feedc652740 10 client.0 _ll_register_callbacks cb 0x5627927e86b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792896630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792896630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:59120/0 (socket says 172.21.15.29:59120) 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 -- 172.21.15.29:0/2364385456 learned_addr learned my addr 172.21.15.29:0/2364385456 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924e51b0 0x562792897860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feeceffd640 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792896cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 -- 172.21.15.29:0/2364385456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924e51b0 msgr2=0x562792897860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924e51b0 0x562792897860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 -- 172.21.15.29:0/2364385456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 msgr2=0x562792896cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792896cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 -- 172.21.15.29:0/2364385456 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562792550790 con 0x5627924bc2f0 2024-09-21T03:02:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feec7fff640 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792896630 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009e10 tx=0x7feeb8015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38d44b598fed51f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.496+0000 7feeb67fc640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.497+0000 7feeb67fc640 1 -- 172.21.15.29:0/2364385456 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8023980 con 0x5627924bc2f0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.497+0000 7feedc652740 1 -- 172.21.15.29:0/2364385456 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feec0040070 con 0x5627924bc2f0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.497+0000 7feeb67fc640 1 -- 172.21.15.29:0/2364385456 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7feeb8023b20 con 0x5627924bc2f0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.497+0000 7feeb67fc640 1 -- 172.21.15.29:0/2364385456 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7feeb8035050 con 0x5627924bc2f0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.497+0000 7feedc652740 10 client.27770 fetch_fsmap learned FSMap version 37 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.497+0000 7feedc652740 1 -- 172.21.15.29:0/2364385456 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5627925c9620 con 0x5627924bc2f0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.497+0000 7feeb67fc640 1 -- 172.21.15.29:0/2364385456 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feec0040070 con 0x5627924bc2f0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.498+0000 7feeb67fc640 1 -- 172.21.15.29:0/2364385456 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7feeb8022040 con 0x5627924bc2f0 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.498+0000 7feedc652740 10 client.27770 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.498+0000 7feedc652740 10 client.27770 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.498+0000 7feedc652740 20 client.27770 populate_metadata read hostname 'smithi029' 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.498+0000 7feedc652740 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627924a3d80 0x5627928a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.498+0000 7feedc652740 4 client.27770 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.498+0000 7feedc652740 1 -- 172.21.15.29:0/2364385456 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562792801640 con 0x5627924a3d80 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.498+0000 7feeceffd640 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627924a3d80 0x5627928a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.499+0000 7feeb67fc640 10 client.27770 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.499+0000 7feeceffd640 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627924a3d80 0x5627928a3d20 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=282d169c2f9b099a server_cookie=34252e569976614a in_seq=0 out_seq=0 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.499+0000 7feeb67fc640 1 -- 172.21.15.29:0/2364385456 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562792801640 con 0x5627924a3d80 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.499+0000 7feeb67fc640 10 client.27770 handle_command_reply: tid=0 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.499+0000 7feedc652740 1 client.27770 shutdown 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 -- 172.21.15.29:0/2364385456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 msgr2=0x562792896630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792896630 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009e10 tx=0x7feeb8015040 comp rx=0 tx=0).stop 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 -- 172.21.15.29:0/2364385456 shutdown_connections 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924e51b0 0x562792897860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627924a3d80 0x5627928a3d20 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792896cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 --2- 172.21.15.29:0/2364385456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792896630 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 -- 172.21.15.29:0/2364385456 >> 172.21.15.29:0/2364385456 conn(0x5627924d9300 msgr2=0x562792886ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feeb67fc640 0 client.27770 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 -- 172.21.15.29:0/2364385456 shutdown_connections 2024-09-21T03:02:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 1 -- 172.21.15.29:0/2364385456 wait complete. 2024-09-21T03:02:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.500+0000 7feedc652740 20 client.27770 trim_cache size 0 max 16384 2024-09-21T03:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.531+0000 7feedc652740 1 Processor -- start 2024-09-21T03:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.531+0000 7feedc652740 1 -- start start 2024-09-21T03:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.531+0000 7feedc652740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x562792888290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.531+0000 7feedc652740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.531+0000 7feedc652740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792895830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.531+0000 7feedc652740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feed01b2bd0 con 0x5627924c44d0 2024-09-21T03:02:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.531+0000 7feedc652740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feed01b4710 con 0x5627924bc2f0 2024-09-21T03:02:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.531+0000 7feedc652740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feed01b32a0 con 0x5627924d9300 2024-09-21T03:02:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x562792888290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792895830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792895830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39916/0 (socket says 172.21.15.29:39916) 2024-09-21T03:02:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 -- 172.21.15.29:0/4166227221 learned_addr learned my addr 172.21.15.29:0/4166227221 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feeceffd640 1 --2- 172.21.15.29:0/4166227221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 -- 172.21.15.29:0/4166227221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 msgr2=0x562792888290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 --2- 172.21.15.29:0/4166227221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x562792888290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 -- 172.21.15.29:0/4166227221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 msgr2=0x56279288dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 --2- 172.21.15.29:0/4166227221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 -- 172.21.15.29:0/4166227221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562792897550 con 0x5627924c44d0 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.532+0000 7feec7fff640 1 --2- 172.21.15.29:0/4166227221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792895830 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7feeb8025810 tx=0x7feeb8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de770d033a2c51a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.533+0000 7feec4ff9640 1 -- 172.21.15.29:0/4166227221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8005850 con 0x5627924c44d0 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.533+0000 7feec4ff9640 1 -- 172.21.15.29:0/4166227221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb80230e0 con 0x5627924c44d0 2024-09-21T03:02:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.533+0000 7feec4ff9640 1 -- 172.21.15.29:0/4166227221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8023be0 con 0x5627924c44d0 2024-09-21T03:02:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 -- 172.21.15.29:0/4166227221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 msgr2=0x562792895830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 --2- 172.21.15.29:0/4166227221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792895830 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7feeb8025810 tx=0x7feeb8000bc0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 -- 172.21.15.29:0/4166227221 shutdown_connections 2024-09-21T03:02:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 --2- 172.21.15.29:0/4166227221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792895830 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 --2- 172.21.15.29:0/4166227221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x56279288dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 --2- 172.21.15.29:0/4166227221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x562792888290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 -- 172.21.15.29:0/4166227221 >> 172.21.15.29:0/4166227221 conn(0x5627928a4050 msgr2=0x562792886ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 -- 172.21.15.29:0/4166227221 shutdown_connections 2024-09-21T03:02:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 -- 172.21.15.29:0/4166227221 wait complete. 2024-09-21T03:02:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.534+0000 7feedc652740 1 -- start start 2024-09-21T03:02:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feedc652740 1 Processor -- start 2024-09-21T03:02:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feedc652740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792899710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feedc652740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792899dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feedc652740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x56279289a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feedc652740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feed01b4040 con 0x5627924c44d0 2024-09-21T03:02:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feedc652740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7feed01b3970 con 0x5627924bc2f0 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feedc652740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feed01b0bd0 con 0x5627924d9300 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feedc652740 10 client.0 _ll_register_callbacks cb 0x7feeb8009730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feeceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792899dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feeceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792899dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39924/0 (socket says 172.21.15.29:39924) 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feeceffd640 1 -- 172.21.15.29:0/2850447538 learned_addr learned my addr 172.21.15.29:0/2850447538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feec7fff640 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x56279289a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feec7fff640 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792899710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feec7fff640 1 -- 172.21.15.29:0/2850447538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 msgr2=0x562792899710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.535+0000 7feec7fff640 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792899710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feec7fff640 1 -- 172.21.15.29:0/2850447538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 msgr2=0x562792899dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feec7fff640 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792899dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feec7fff640 1 -- 172.21.15.29:0/2850447538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56279289b0a0 con 0x5627924d9300 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feeceffd640 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792899dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feec7fff640 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x56279289a940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feeb8025810 tx=0x7feeb8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a14c939ccfae7361 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feeb67fc640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feeb67fc640 1 -- 172.21.15.29:0/2850447538 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8023d90 con 0x5627924d9300 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feedc652740 1 -- 172.21.15.29:0/2850447538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb80230e0 con 0x5627924d9300 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feeb67fc640 1 -- 172.21.15.29:0/2850447538 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9121+0+0 (secure 0 0 0) 0x7feeb80213e0 con 0x5627924d9300 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feeb67fc640 1 -- 172.21.15.29:0/2850447538 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7feeb80097e0 con 0x5627924d9300 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feedc652740 10 client.27940 fetch_fsmap learned FSMap version 37 2024-09-21T03:02:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.536+0000 7feedc652740 1 -- 172.21.15.29:0/2850447538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56279289e3d0 con 0x5627924d9300 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feeb67fc640 1 -- 172.21.15.29:0/2850447538 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb80230e0 con 0x5627924d9300 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feeb67fc640 1 -- 172.21.15.29:0/2850447538 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7feeb8021670 con 0x5627924d9300 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feedc652740 10 client.27940 fetch_fsmap finished waiting for FSMap version 37 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feedc652740 10 client.27940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feedc652740 20 client.27940 populate_metadata read hostname 'smithi029' 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feedc652740 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627924e51b0 0x5627929107d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feedc652740 4 client.27940 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feedc652740 1 -- 172.21.15.29:0/2850447538 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627927fd0f0 con 0x5627924e51b0 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.537+0000 7feeceffd640 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627924e51b0 0x5627929107d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.538+0000 7feeceffd640 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627924e51b0 0x5627929107d0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee18ff7e1dfb1ac9 server_cookie=963d24a0376d1d8c in_seq=0 out_seq=0 2024-09-21T03:02:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.538+0000 7feeb67fc640 10 client.27940 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:23.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 61134 5.43 MB/sec execute 480 sec latency 213.807 ms 2024-09-21T03:02:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.780+0000 7feeb67fc640 1 -- 172.21.15.29:0/2850447538 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627927fd0f0 con 0x5627924e51b0 2024-09-21T03:02:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.780+0000 7feeb67fc640 10 client.27940 handle_command_reply: tid=0 2024-09-21T03:02:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.781+0000 7feedc652740 1 client.27940 shutdown 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.781+0000 7feedc652740 1 -- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 msgr2=0x56279289a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.782+0000 7feedc652740 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x56279289a940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feeb8025810 tx=0x7feeb8000bc0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.782+0000 7feedc652740 1 -- 172.21.15.29:0/2850447538 shutdown_connections 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.782+0000 7feedc652740 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627924d9300 0x56279289a940 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.782+0000 7feedc652740 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627924e51b0 0x5627929107d0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.782+0000 7feedc652740 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627924c44d0 0x562792899dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.782+0000 7feeb67fc640 0 client.27940 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.782+0000 7feedc652740 1 --2- 172.21.15.29:0/2850447538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627924bc2f0 0x562792899710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.783+0000 7feedc652740 1 -- 172.21.15.29:0/2850447538 >> 172.21.15.29:0/2850447538 conn(0x5627928a4050 msgr2=0x562792886ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.783+0000 7feedc652740 1 -- 172.21.15.29:0/2850447538 shutdown_connections 2024-09-21T03:02:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.784+0000 7feedc652740 1 -- 172.21.15.29:0/2850447538 wait complete. 2024-09-21T03:02:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.784+0000 7feedc652740 20 client.27940 trim_cache size 0 max 16384 2024-09-21T03:02:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:02:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:02:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:02:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:02:23.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.787+0000 7feed560a640 1 -- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feebc0b6510 msgr2=0x7feebc0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.787+0000 7feed560a640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feebc0b6510 0x7feebc0b8a20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7feec801a040 tx=0x7feec80d1040 comp rx=0 tx=0).stop 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.787+0000 7feed560a640 1 -- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feed010fae0 msgr2=0x7feed01a5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.787+0000 7feed560a640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feed010fae0 0x7feed01a5ae0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7feec8009fd0 tx=0x7feec800ef10 comp rx=0 tx=0).stop 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.788+0000 7feed560a640 1 -- 172.21.15.29:0/2349933583 shutdown_connections 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.788+0000 7feed560a640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7feebc0b6510 0x7feebc0b8a20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.788+0000 7feed560a640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feed010fae0 0x7feed01a5ae0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.788+0000 7feed560a640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feed010f140 0x7feed01a4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.788+0000 7feed560a640 1 --2- 172.21.15.29:0/2349933583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feed010e0c0 0x7feed01a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.788+0000 7feed560a640 1 -- 172.21.15.29:0/2349933583 >> 172.21.15.29:0/2349933583 conn(0x7feed006e320 msgr2=0x7feed0112460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.788+0000 7feed560a640 1 -- 172.21.15.29:0/2349933583 shutdown_connections 2024-09-21T03:02:23.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:23.788+0000 7feed560a640 1 -- 172.21.15.29:0/2349933583 wait complete. 2024-09-21T03:02:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:23 smithi066 ceph-mon[33436]: pgmap v1087: 129 pgs: 129 active+clean; 333 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-09-21T03:02:24.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:23 smithi190 ceph-mon[33871]: pgmap v1087: 129 pgs: 129 active+clean; 333 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-09-21T03:02:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:23 smithi029 ceph-mon[31006]: pgmap v1087: 129 pgs: 129 active+clean; 333 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-09-21T03:02:24.510 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:02:24 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: ignoring --setuser ceph since I am not root 2024-09-21T03:02:24.510 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:02:24 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: ignoring --setgroup ceph since I am not root 2024-09-21T03:02:24.510 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:02:24 smithi066 ceph-mds[60984]: -- 172.21.15.66:0/3653807731 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c66f73860 con 0x555c6b11b600 2024-09-21T03:02:24.510 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:02:24 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: starting mds.e at 2024-09-21T03:02:24.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 61249 5.43 MB/sec execute 481 sec latency 245.488 ms 2024-09-21T03:02:25.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:24 smithi066 ceph-mon[33436]: daemon mds.e restarted 2024-09-21T03:02:25.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:24 smithi190 ceph-mon[33871]: daemon mds.e restarted 2024-09-21T03:02:25.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:24 smithi029 ceph-mon[31006]: daemon mds.e restarted 2024-09-21T03:02:25.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 61359 5.44 MB/sec execute 482 sec latency 127.800 ms 2024-09-21T03:02:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:25 smithi066 ceph-mon[33436]: pgmap v1088: 129 pgs: 129 active+clean; 339 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-09-21T03:02:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:25 smithi066 ceph-mon[33436]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T03:02:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:25 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6834/3193047193,v1:172.21.15.66:6835/3193047193] up:boot 2024-09-21T03:02:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:25 smithi066 ceph-mon[33436]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-09-21T03:02:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:25 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T03:02:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:25 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:02:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:25 smithi190 ceph-mon[33871]: pgmap v1088: 129 pgs: 129 active+clean; 339 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-09-21T03:02:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:25 smithi190 ceph-mon[33871]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T03:02:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:25 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6834/3193047193,v1:172.21.15.66:6835/3193047193] up:boot 2024-09-21T03:02:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:25 smithi190 ceph-mon[33871]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-09-21T03:02:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:25 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T03:02:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:25 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:02:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:25 smithi029 ceph-mon[31006]: pgmap v1088: 129 pgs: 129 active+clean; 339 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-09-21T03:02:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:25 smithi029 ceph-mon[31006]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T03:02:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:25 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6834/3193047193,v1:172.21.15.66:6835/3193047193] up:boot 2024-09-21T03:02:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:25 smithi029 ceph-mon[31006]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-09-21T03:02:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:25 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T03:02:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:25 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:02:26.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 61507 5.45 MB/sec execute 483 sec latency 170.754 ms 2024-09-21T03:02:27.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 61633 5.45 MB/sec execute 484 sec latency 104.794 ms 2024-09-21T03:02:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:27 smithi066 ceph-mon[33436]: pgmap v1090: 129 pgs: 129 active+clean; 336 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 4.1 MiB/s wr, 278 op/s 2024-09-21T03:02:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:27 smithi190 ceph-mon[33871]: pgmap v1090: 129 pgs: 129 active+clean; 336 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 4.1 MiB/s wr, 278 op/s 2024-09-21T03:02:28.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:27 smithi029 ceph-mon[31006]: pgmap v1090: 129 pgs: 129 active+clean; 336 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 4.1 MiB/s wr, 278 op/s 2024-09-21T03:02:28.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 61754 5.44 MB/sec execute 485 sec latency 195.517 ms 2024-09-21T03:02:29.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 61860 5.43 MB/sec execute 486 sec latency 169.918 ms 2024-09-21T03:02:30.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:29 smithi066 ceph-mon[33436]: pgmap v1091: 129 pgs: 129 active+clean; 338 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 4.0 MiB/s wr, 301 op/s 2024-09-21T03:02:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:29 smithi190 ceph-mon[33871]: pgmap v1091: 129 pgs: 129 active+clean; 338 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 4.0 MiB/s wr, 301 op/s 2024-09-21T03:02:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:29 smithi029 ceph-mon[31006]: pgmap v1091: 129 pgs: 129 active+clean; 338 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 4.0 MiB/s wr, 301 op/s 2024-09-21T03:02:30.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 62001 5.43 MB/sec execute 487 sec latency 89.951 ms 2024-09-21T03:02:31.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 62108 5.43 MB/sec execute 488 sec latency 219.387 ms 2024-09-21T03:02:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:31 smithi066 ceph-mon[33436]: pgmap v1092: 129 pgs: 129 active+clean; 336 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 4.4 MiB/s wr, 333 op/s 2024-09-21T03:02:32.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:31 smithi190 ceph-mon[33871]: pgmap v1092: 129 pgs: 129 active+clean; 336 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 4.4 MiB/s wr, 333 op/s 2024-09-21T03:02:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:31 smithi029 ceph-mon[31006]: pgmap v1092: 129 pgs: 129 active+clean; 336 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 4.4 MiB/s wr, 333 op/s 2024-09-21T03:02:32.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 62250 5.43 MB/sec execute 489 sec latency 117.461 ms 2024-09-21T03:02:33.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 62379 5.44 MB/sec execute 490 sec latency 267.602 ms 2024-09-21T03:02:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:33 smithi066 ceph-mon[33436]: pgmap v1093: 129 pgs: 129 active+clean; 334 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 4.0 MiB/s wr, 325 op/s 2024-09-21T03:02:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:33 smithi190 ceph-mon[33871]: pgmap v1093: 129 pgs: 129 active+clean; 334 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 4.0 MiB/s wr, 325 op/s 2024-09-21T03:02:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:33 smithi029 ceph-mon[31006]: pgmap v1093: 129 pgs: 129 active+clean; 334 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 4.0 MiB/s wr, 325 op/s 2024-09-21T03:02:34.746 INFO:tasks.workunit.client.0.smithi029.stdout: 10 62523 5.44 MB/sec execute 491 sec latency 91.050 ms 2024-09-21T03:02:35.745 INFO:tasks.workunit.client.0.smithi029.stdout: 10 62661 5.44 MB/sec execute 492 sec latency 91.675 ms 2024-09-21T03:02:35.934 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:35 smithi029 ceph-mon[31006]: pgmap v1094: 129 pgs: 129 active+clean; 328 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-09-21T03:02:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:35 smithi066 ceph-mon[33436]: pgmap v1094: 129 pgs: 129 active+clean; 328 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-09-21T03:02:36.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:35 smithi190 ceph-mon[33871]: pgmap v1094: 129 pgs: 129 active+clean; 328 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-09-21T03:02:36.746 INFO:tasks.workunit.client.0.smithi029.stdout: 10 62832 5.44 MB/sec execute 493 sec latency 102.173 ms 2024-09-21T03:02:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 -- 172.21.15.29:0/3948701296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1788103630 msgr2=0x7f1788103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 --2- 172.21.15.29:0/3948701296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1788103630 0x7f1788103a40 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f177c009860 tx=0x7f177c02a340 comp rx=0 tx=0).stop 2024-09-21T03:02:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 -- 172.21.15.29:0/3948701296 shutdown_connections 2024-09-21T03:02:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 --2- 172.21.15.29:0/3948701296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1788103fd0 0x7f178810a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 --2- 172.21.15.29:0/3948701296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1788103630 0x7f1788103a40 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 --2- 172.21.15.29:0/3948701296 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f17881025b0 0x7f1788102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 -- 172.21.15.29:0/3948701296 >> 172.21.15.29:0/3948701296 conn(0x7f17880fdf30 msgr2=0x7f17881003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 -- 172.21.15.29:0/3948701296 shutdown_connections 2024-09-21T03:02:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.927+0000 7f178e28c640 1 -- 172.21.15.29:0/3948701296 wait complete. 2024-09-21T03:02:36.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.928+0000 7f178e28c640 1 Processor -- start 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.928+0000 7f178e28c640 1 -- start start 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f178e28c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17881025b0 0x7f178806bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f178e28c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1788103630 0x7f178819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f178e28c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1788103fd0 0x7f17881a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f178e28c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f178810e0a0 con 0x7f17881025b0 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f178e28c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f178810df20 con 0x7f1788103630 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f178e28c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f178810e220 con 0x7f1788103fd0 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f1787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17881025b0 0x7f178806bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f17877fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1788103630 0x7f178819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f1787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17881025b0 0x7f178806bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43600/0 (socket says 172.21.15.29:43600) 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f1787fff640 1 -- 172.21.15.29:0/2085598951 learned_addr learned my addr 172.21.15.29:0/2085598951 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f178c829640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1788103fd0 0x7f17881a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f17877fe640 1 -- 172.21.15.29:0/2085598951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1788103fd0 msgr2=0x7f17881a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:36.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f17877fe640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1788103fd0 0x7f17881a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f17877fe640 1 -- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17881025b0 msgr2=0x7f178806bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f17877fe640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17881025b0 0x7f178806bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f17877fe640 1 -- 172.21.15.29:0/2085598951 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17881a19a0 con 0x7f1788103630 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.929+0000 7f178c829640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1788103fd0 0x7f17881a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.930+0000 7f17877fe640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1788103630 0x7f178819ec50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f177c02a860 tx=0x7f177c02ad70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.930+0000 7f17857fa640 1 -- 172.21.15.29:0/2085598951 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c040030 con 0x7f1788103630 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.930+0000 7f178e28c640 1 -- 172.21.15.29:0/2085598951 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17881a1c30 con 0x7f1788103630 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.930+0000 7f178e28c640 1 -- 172.21.15.29:0/2085598951 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17881a2140 con 0x7f1788103630 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.930+0000 7f17857fa640 1 -- 172.21.15.29:0/2085598951 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f177c032bc0 con 0x7f1788103630 2024-09-21T03:02:36.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.930+0000 7f17857fa640 1 -- 172.21.15.29:0/2085598951 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c037040 con 0x7f1788103630 2024-09-21T03:02:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f17951f3740 1 Processor -- start 2024-09-21T03:02:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f17951f3740 1 -- start start 2024-09-21T03:02:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f17951f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153e31b0 0x55771578f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f17951f3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771578f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f17951f3740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x557715790420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f17951f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5577156fd6f0 con 0x5577153e31b0 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f17951f3740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5577156ecb20 con 0x5577153c24d0 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f17951f3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5577156f0930 con 0x5577153ba2f0 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f178c829640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153e31b0 0x55771578f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f178c829640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153e31b0 0x55771578f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43602/0 (socket says 172.21.15.29:43602) 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f178c829640 1 -- 172.21.15.29:0/797326197 learned_addr learned my addr 172.21.15.29:0/797326197 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.933+0000 7f1787fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771578f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f178c829640 1 -- 172.21.15.29:0/797326197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 msgr2=0x55771578f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f178c829640 1 --2- 172.21.15.29:0/797326197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771578f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f178c829640 1 -- 172.21.15.29:0/797326197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 msgr2=0x557715790420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f178c829640 1 --2- 172.21.15.29:0/797326197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x557715790420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f178c829640 1 -- 172.21.15.29:0/797326197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577156b2530 con 0x5577153e31b0 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f17857fa640 1 -- 172.21.15.29:0/2085598951 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f177c043070 con 0x7f1788103630 2024-09-21T03:02:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f178c829640 1 --2- 172.21.15.29:0/797326197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153e31b0 0x55771578f260 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f1780009fd0 tx=0x7f178000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4b005a3ade2f59 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f176e7fc640 1 -- 172.21.15.29:0/797326197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178001d070 con 0x5577153e31b0 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f176e7fc640 1 -- 172.21.15.29:0/797326197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1780003e00 con 0x5577153e31b0 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.934+0000 7f176e7fc640 1 -- 172.21.15.29:0/797326197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1780004710 con 0x5577153e31b0 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.935+0000 7f17857fa640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17700b6590 0x7f17700b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 -- 172.21.15.29:0/797326197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153e31b0 msgr2=0x55771578f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 --2- 172.21.15.29:0/797326197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153e31b0 0x55771578f260 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f1780009fd0 tx=0x7f178000ec80 comp rx=0 tx=0).stop 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17857fa640 1 -- 172.21.15.29:0/2085598951 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f177c1071f0 con 0x7f1788103630 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 -- 172.21.15.29:0/797326197 shutdown_connections 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 --2- 172.21.15.29:0/797326197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x557715790420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 --2- 172.21.15.29:0/797326197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771578f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 --2- 172.21.15.29:0/797326197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153e31b0 0x55771578f260 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 -- 172.21.15.29:0/797326197 >> 172.21.15.29:0/797326197 conn(0x5577153d7300 msgr2=0x55771578acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f1787fff640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17700b6590 0x7f17700b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 -- 172.21.15.29:0/797326197 shutdown_connections 2024-09-21T03:02:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 -- 172.21.15.29:0/797326197 wait complete. 2024-09-21T03:02:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f1787fff640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17700b6590 0x7f17700b8aa0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1778009620 tx=0x7f1778005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.936+0000 7f17951f3740 1 -- start start 2024-09-21T03:02:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f17951f3740 1 Processor -- start 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f17951f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715798a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f17951f3740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x557715799130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f17951f3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153e31b0 0x557715799ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f17951f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557715624ab0 con 0x5577153ba2f0 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f17951f3740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x557715692730 con 0x5577153c24d0 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f1787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715798a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f17951f3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x557715790f40 con 0x5577153e31b0 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f1787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715798a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43610/0 (socket says 172.21.15.29:43610) 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f1787fff640 1 -- 172.21.15.29:0/3103314893 learned_addr learned my addr 172.21.15.29:0/3103314893 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f178c829640 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153e31b0 0x557715799ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.937+0000 7f17951f3740 10 client.0 _ll_register_callbacks cb 0x5577156e66b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f17877fe640 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x557715799130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f178c829640 1 -- 172.21.15.29:0/3103314893 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 msgr2=0x557715799130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f178c829640 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x557715799130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f178c829640 1 -- 172.21.15.29:0/3103314893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 msgr2=0x557715798a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f178c829640 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715798a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f178c829640 1 -- 172.21.15.29:0/3103314893 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55771544e790 con 0x5577153e31b0 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f1787fff640 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715798a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f178c829640 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153e31b0 0x557715799ca0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1780006010 tx=0x7f1780009d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd72fe7ddb7b7e10 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f17677fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:02:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f17677fe640 1 -- 172.21.15.29:0/3103314893 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178001d070 con 0x5577153e31b0 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f17951f3740 1 -- 172.21.15.29:0/3103314893 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1780003e00 con 0x5577153e31b0 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.938+0000 7f17677fe640 1 -- 172.21.15.29:0/3103314893 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f1780004650 con 0x5577153e31b0 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17677fe640 1 -- 172.21.15.29:0/3103314893 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1780030050 con 0x5577153e31b0 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17951f3740 10 client.27958 fetch_fsmap learned FSMap version 39 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17951f3740 1 -- 172.21.15.29:0/3103314893 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5577154c7620 con 0x5577153e31b0 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17677fe640 1 -- 172.21.15.29:0/3103314893 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1780003e00 con 0x5577153e31b0 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17677fe640 1 -- 172.21.15.29:0/3103314893 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f178000b040 con 0x5577153e31b0 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17951f3740 10 client.27958 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17951f3740 10 client.27958 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17951f3740 20 client.27958 populate_metadata read hostname 'smithi029' 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17951f3740 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5577153a1d80 0x5577157a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17951f3740 4 client.27958 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f17951f3740 1 -- 172.21.15.29:0/3103314893 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577156ff640 con 0x5577153a1d80 2024-09-21T03:02:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.939+0000 7f1787fff640 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5577153a1d80 0x5577157a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.940+0000 7f17677fe640 10 client.27958 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.940+0000 7f1787fff640 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5577153a1d80 0x5577157a5e70 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39aa81b6ebaae259 server_cookie=63acad88acb8b171 in_seq=0 out_seq=0 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17677fe640 1 -- 172.21.15.29:0/3103314893 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5577156ff640 con 0x5577153a1d80 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17677fe640 10 client.27958 handle_command_reply: tid=0 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 client.27958 shutdown 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 -- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153e31b0 msgr2=0x557715799ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153e31b0 0x557715799ca0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1780006010 tx=0x7f1780009d70 comp rx=0 tx=0).stop 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 -- 172.21.15.29:0/3103314893 shutdown_connections 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153e31b0 0x557715799ca0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x557715799130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5577153a1d80 0x5577157a5e70 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17677fe640 0 client.27958 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 --2- 172.21.15.29:0/3103314893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715798a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 -- 172.21.15.29:0/3103314893 >> 172.21.15.29:0/3103314893 conn(0x5577153d7300 msgr2=0x557715784ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:36.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 -- 172.21.15.29:0/3103314893 shutdown_connections 2024-09-21T03:02:36.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.941+0000 7f17951f3740 1 -- 172.21.15.29:0/3103314893 wait complete. 2024-09-21T03:02:36.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.942+0000 7f17951f3740 20 client.27958 trim_cache size 0 max 16384 2024-09-21T03:02:36.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.968+0000 7f17951f3740 1 Processor -- start 2024-09-21T03:02:36.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.969+0000 7f17951f3740 1 -- start start 2024-09-21T03:02:36.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.969+0000 7f17951f3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153a1d80 0x5577157983a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.969+0000 7f17951f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715799be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.969+0000 7f17951f3740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.969+0000 7f1787fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153a1d80 0x5577157983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.969+0000 7f1787fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153a1d80 0x5577157983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42820/0 (socket says 172.21.15.29:42820) 2024-09-21T03:02:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17951f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17881aa370 con 0x5577153ba2f0 2024-09-21T03:02:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17951f3740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f17881abeb0 con 0x5577153c24d0 2024-09-21T03:02:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715799be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f178c829640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f178c829640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:37532/0 (socket says 172.21.15.29:37532) 2024-09-21T03:02:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f178c829640 1 -- 172.21.15.29:0/3168095548 learned_addr learned my addr 172.21.15.29:0/3168095548 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17951f3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17881aaa40 con 0x5577153a1d80 2024-09-21T03:02:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715799be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43612/0 (socket says 172.21.15.29:43612) 2024-09-21T03:02:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17877fe640 1 -- 172.21.15.29:0/3168095548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153a1d80 msgr2=0x5577157983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17877fe640 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153a1d80 0x5577157983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17877fe640 1 -- 172.21.15.29:0/3168095548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 msgr2=0x55771579a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17877fe640 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f17877fe640 1 -- 172.21.15.29:0/3168095548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557715790e20 con 0x5577153ba2f0 2024-09-21T03:02:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f178c829640 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.970+0000 7f1787fff640 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153a1d80 0x5577157983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.971+0000 7f17877fe640 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715799be0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f177c031000 tx=0x7f177c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b78038cbab0624 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.971+0000 7f176dffb640 1 -- 172.21.15.29:0/3168095548 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c0cc7a0 con 0x5577153ba2f0 2024-09-21T03:02:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.971+0000 7f176dffb640 1 -- 172.21.15.29:0/3168095548 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f177c0b2e50 con 0x5577153ba2f0 2024-09-21T03:02:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.971+0000 7f176dffb640 1 -- 172.21.15.29:0/3168095548 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c0d4070 con 0x5577153ba2f0 2024-09-21T03:02:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 -- 172.21.15.29:0/3168095548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 msgr2=0x557715799be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715799be0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f177c031000 tx=0x7f177c009580 comp rx=0 tx=0).stop 2024-09-21T03:02:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 -- 172.21.15.29:0/3168095548 shutdown_connections 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153ba2f0 0x557715799be0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 --2- 172.21.15.29:0/3168095548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153a1d80 0x5577157983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 -- 172.21.15.29:0/3168095548 >> 172.21.15.29:0/3168095548 conn(0x5577153d7300 msgr2=0x557715797c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 -- 172.21.15.29:0/3168095548 shutdown_connections 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.972+0000 7f17951f3740 1 -- 172.21.15.29:0/3168095548 wait complete. 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.973+0000 7f17951f3740 1 -- start start 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.973+0000 7f17951f3740 1 Processor -- start 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.973+0000 7f17951f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153a1d80 0x5577157a0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.973+0000 7f17951f3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771579f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17951f3740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17951f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17881ab7e0 con 0x5577153a1d80 2024-09-21T03:02:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17951f3740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f17881ab110 con 0x5577153c24d0 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17951f3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17881a8350 con 0x5577153ba2f0 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17951f3740 10 client.0 _ll_register_callbacks cb 0x7f178001b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f1787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153a1d80 0x5577157a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17877fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771579f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f1787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153a1d80 0x5577157a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43622/0 (socket says 172.21.15.29:43622) 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f1787fff640 1 -- 172.21.15.29:0/2724660832 learned_addr learned my addr 172.21.15.29:0/2724660832 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17877fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771579f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42824/0 (socket says 172.21.15.29:42824) 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f178c829640 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17877fe640 1 -- 172.21.15.29:0/2724660832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 msgr2=0x55771579fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17877fe640 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17877fe640 1 -- 172.21.15.29:0/2724660832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153a1d80 msgr2=0x5577157a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17877fe640 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153a1d80 0x5577157a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17877fe640 1 -- 172.21.15.29:0/2724660832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55771579e0e0 con 0x5577153ba2f0 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f1787fff640 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153a1d80 0x5577157a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17877fe640 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771579f520 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f177c0ccfc0 tx=0x7f177c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8702a0395eea97a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.974+0000 7f17677fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.975+0000 7f17677fe640 1 -- 172.21.15.29:0/2724660832 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c0ccb40 con 0x5577153ba2f0 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.975+0000 7f17677fe640 1 -- 172.21.15.29:0/2724660832 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f177c005490 con 0x5577153ba2f0 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.975+0000 7f17951f3740 1 -- 172.21.15.29:0/2724660832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f177c0b2e50 con 0x5577153ba2f0 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.975+0000 7f17677fe640 1 -- 172.21.15.29:0/2724660832 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f177c0cd730 con 0x5577153ba2f0 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.975+0000 7f17951f3740 10 client.27964 fetch_fsmap learned FSMap version 39 2024-09-21T03:02:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.975+0000 7f17951f3740 1 -- 172.21.15.29:0/2724660832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5577158108f0 con 0x5577153ba2f0 2024-09-21T03:02:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f17677fe640 1 -- 172.21.15.29:0/2724660832 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f177c0b2e50 con 0x5577153ba2f0 2024-09-21T03:02:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f17677fe640 1 -- 172.21.15.29:0/2724660832 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f177c04d060 con 0x5577153ba2f0 2024-09-21T03:02:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f17951f3740 10 client.27964 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:02:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f17951f3740 10 client.27964 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:02:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f17951f3740 20 client.27964 populate_metadata read hostname 'smithi029' 2024-09-21T03:02:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f17951f3740 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5577153e31b0 0x557715813240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f17951f3740 4 client.27964 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:02:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f17951f3740 1 -- 172.21.15.29:0/2724660832 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5577156fb0f0 con 0x5577153e31b0 2024-09-21T03:02:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.976+0000 7f1787fff640 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5577153e31b0 0x557715813240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.977+0000 7f17677fe640 10 client.27964 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:36.977+0000 7f1787fff640 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5577153e31b0 0x557715813240 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae2fb8f0c57146e1 server_cookie=3ba6087334524c5f in_seq=0 out_seq=0 2024-09-21T03:02:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.180+0000 7f17677fe640 1 -- 172.21.15.29:0/2724660832 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5577156fb0f0 con 0x5577153e31b0 2024-09-21T03:02:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.180+0000 7f17677fe640 10 client.27964 handle_command_reply: tid=0 2024-09-21T03:02:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.180+0000 7f17951f3740 1 client.27964 shutdown 2024-09-21T03:02:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.180+0000 7f17951f3740 1 -- 172.21.15.29:0/2724660832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 msgr2=0x55771579f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.180+0000 7f17951f3740 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771579f520 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f177c0ccfc0 tx=0x7f177c004270 comp rx=0 tx=0).stop 2024-09-21T03:02:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.181+0000 7f17951f3740 1 -- 172.21.15.29:0/2724660832 shutdown_connections 2024-09-21T03:02:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.181+0000 7f17951f3740 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5577153c24d0 0x55771579fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.181+0000 7f17951f3740 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577153ba2f0 0x55771579f520 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.181+0000 7f17951f3740 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5577153e31b0 0x557715813240 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.181+0000 7f17677fe640 0 client.27964 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.181+0000 7f17951f3740 1 --2- 172.21.15.29:0/2724660832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577153a1d80 0x5577157a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.181+0000 7f17951f3740 1 -- 172.21.15.29:0/2724660832 >> 172.21.15.29:0/2724660832 conn(0x5577153d7300 msgr2=0x557715796b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.182+0000 7f17951f3740 1 -- 172.21.15.29:0/2724660832 shutdown_connections 2024-09-21T03:02:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.182+0000 7f17951f3740 1 -- 172.21.15.29:0/2724660832 wait complete. 2024-09-21T03:02:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.182+0000 7f17951f3740 20 client.27964 trim_cache size 0 max 16384 2024-09-21T03:02:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:02:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:02:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:02:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.183+0000 7f178e28c640 1 -- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17700b6590 msgr2=0x7f17700b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.183+0000 7f178e28c640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17700b6590 0x7f17700b8aa0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1778009620 tx=0x7f1778005f90 comp rx=0 tx=0).stop 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.183+0000 7f178e28c640 1 -- 172.21.15.29:0/2085598951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1788103630 msgr2=0x7f178819ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.183+0000 7f178e28c640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1788103630 0x7f178819ec50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f177c02a860 tx=0x7f177c02ad70 comp rx=0 tx=0).stop 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.184+0000 7f178e28c640 1 -- 172.21.15.29:0/2085598951 shutdown_connections 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.184+0000 7f178e28c640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f17700b6590 0x7f17700b8aa0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.184+0000 7f178e28c640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1788103fd0 0x7f17881a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.184+0000 7f178e28c640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1788103630 0x7f178819ec50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.184+0000 7f178e28c640 1 --2- 172.21.15.29:0/2085598951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17881025b0 0x7f178806bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.184+0000 7f178e28c640 1 -- 172.21.15.29:0/2085598951 >> 172.21.15.29:0/2085598951 conn(0x7f17880fdf30 msgr2=0x7f1788106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.185+0000 7f178e28c640 1 -- 172.21.15.29:0/2085598951 shutdown_connections 2024-09-21T03:02:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:37.185+0000 7f178e28c640 1 -- 172.21.15.29:0/2085598951 wait complete. 2024-09-21T03:02:37.746 INFO:tasks.workunit.client.0.smithi029.stdout: 10 62975 5.44 MB/sec execute 494 sec latency 219.073 ms 2024-09-21T03:02:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:37 smithi066 ceph-mon[33436]: pgmap v1095: 129 pgs: 129 active+clean; 324 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 3.1 MiB/s wr, 291 op/s 2024-09-21T03:02:38.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:37 smithi190 ceph-mon[33871]: pgmap v1095: 129 pgs: 129 active+clean; 324 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 3.1 MiB/s wr, 291 op/s 2024-09-21T03:02:38.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:37 smithi029 ceph-mon[31006]: pgmap v1095: 129 pgs: 129 active+clean; 324 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 3.1 MiB/s wr, 291 op/s 2024-09-21T03:02:38.746 INFO:tasks.workunit.client.0.smithi029.stdout: 10 63111 5.45 MB/sec execute 495 sec latency 259.928 ms 2024-09-21T03:02:39.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 63269 5.45 MB/sec execute 496 sec latency 92.573 ms 2024-09-21T03:02:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:39 smithi066 ceph-mon[33436]: pgmap v1096: 129 pgs: 129 active+clean; 320 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 3.0 MiB/s wr, 288 op/s 2024-09-21T03:02:40.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:39 smithi190 ceph-mon[33871]: pgmap v1096: 129 pgs: 129 active+clean; 320 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 3.0 MiB/s wr, 288 op/s 2024-09-21T03:02:40.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:39 smithi029 ceph-mon[31006]: pgmap v1096: 129 pgs: 129 active+clean; 320 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 3.0 MiB/s wr, 288 op/s 2024-09-21T03:02:40.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 63420 5.46 MB/sec execute 497 sec latency 98.049 ms 2024-09-21T03:02:41.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 63522 5.46 MB/sec execute 498 sec latency 360.618 ms 2024-09-21T03:02:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:41 smithi066 ceph-mon[33436]: pgmap v1097: 129 pgs: 129 active+clean; 320 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-09-21T03:02:42.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:41 smithi190 ceph-mon[33871]: pgmap v1097: 129 pgs: 129 active+clean; 320 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-09-21T03:02:42.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:41 smithi029 ceph-mon[31006]: pgmap v1097: 129 pgs: 129 active+clean; 320 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-09-21T03:02:42.746 INFO:tasks.workunit.client.0.smithi029.stdout: 10 63676 5.46 MB/sec execute 499 sec latency 91.021 ms 2024-09-21T03:02:43.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 63870 5.49 MB/sec execute 500 sec latency 87.396 ms 2024-09-21T03:02:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:43 smithi066 ceph-mon[33436]: pgmap v1098: 129 pgs: 129 active+clean; 323 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 2.6 MiB/s wr, 272 op/s 2024-09-21T03:02:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:43 smithi190 ceph-mon[33871]: pgmap v1098: 129 pgs: 129 active+clean; 323 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 2.6 MiB/s wr, 272 op/s 2024-09-21T03:02:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:43 smithi029 ceph-mon[31006]: pgmap v1098: 129 pgs: 129 active+clean; 323 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 2.6 MiB/s wr, 272 op/s 2024-09-21T03:02:44.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 64049 5.50 MB/sec execute 501 sec latency 109.634 ms 2024-09-21T03:02:45.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 64233 5.51 MB/sec execute 502 sec latency 91.455 ms 2024-09-21T03:02:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:45 smithi066 ceph-mon[33436]: pgmap v1099: 129 pgs: 129 active+clean; 329 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 3.3 MiB/s wr, 306 op/s 2024-09-21T03:02:46.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:45 smithi190 ceph-mon[33871]: pgmap v1099: 129 pgs: 129 active+clean; 329 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 3.3 MiB/s wr, 306 op/s 2024-09-21T03:02:46.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:45 smithi029 ceph-mon[31006]: pgmap v1099: 129 pgs: 129 active+clean; 329 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 3.3 MiB/s wr, 306 op/s 2024-09-21T03:02:46.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 64328 5.51 MB/sec execute 503 sec latency 85.032 ms 2024-09-21T03:02:47.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 64444 5.51 MB/sec execute 504 sec latency 84.972 ms 2024-09-21T03:02:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:47 smithi066 ceph-mon[33436]: pgmap v1100: 129 pgs: 129 active+clean; 332 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 2.9 MiB/s wr, 248 op/s 2024-09-21T03:02:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:47 smithi190 ceph-mon[33871]: pgmap v1100: 129 pgs: 129 active+clean; 332 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 2.9 MiB/s wr, 248 op/s 2024-09-21T03:02:48.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:47 smithi029 ceph-mon[31006]: pgmap v1100: 129 pgs: 129 active+clean; 332 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 2.9 MiB/s wr, 248 op/s 2024-09-21T03:02:48.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 64582 5.52 MB/sec execute 505 sec latency 98.844 ms 2024-09-21T03:02:49.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 64759 5.54 MB/sec execute 506 sec latency 127.860 ms 2024-09-21T03:02:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:49 smithi066 ceph-mon[33436]: pgmap v1101: 129 pgs: 129 active+clean; 337 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 3.5 MiB/s wr, 246 op/s 2024-09-21T03:02:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:02:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:02:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:02:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:49 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:02:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:49 smithi190 ceph-mon[33871]: pgmap v1101: 129 pgs: 129 active+clean; 337 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 3.5 MiB/s wr, 246 op/s 2024-09-21T03:02:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:02:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:02:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:02:50.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:49 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:02:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:49 smithi029 ceph-mon[31006]: pgmap v1101: 129 pgs: 129 active+clean; 337 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 3.5 MiB/s wr, 246 op/s 2024-09-21T03:02:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:02:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:02:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:02:50.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:49 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:02:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 -- 172.21.15.29:0/598269591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 msgr2=0x7fea38108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/598269591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 0x7fea38108fe0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fea28009900 tx=0x7fea2802a410 comp rx=0 tx=0).stop 2024-09-21T03:02:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 -- 172.21.15.29:0/598269591 shutdown_connections 2024-09-21T03:02:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/598269591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea38109520 0x7fea3810b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/598269591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 0x7fea38108fe0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/598269591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea38101a70 0x7fea38101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 -- 172.21.15.29:0/598269591 >> 172.21.15.29:0/598269591 conn(0x7fea380fd690 msgr2=0x7fea380ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 -- 172.21.15.29:0/598269591 shutdown_connections 2024-09-21T03:02:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.332+0000 7fea3dc4d640 1 -- 172.21.15.29:0/598269591 wait complete. 2024-09-21T03:02:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.333+0000 7fea3dc4d640 1 Processor -- start 2024-09-21T03:02:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.333+0000 7fea3dc4d640 1 -- start start 2024-09-21T03:02:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.333+0000 7fea3dc4d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea38101a70 0x7fea38199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.333+0000 7fea3dc4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 0x7fea3819a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.333+0000 7fea3dc4d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea38109520 0x7fea3819b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.333+0000 7fea3dc4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea3810f1b0 con 0x7fea38106bc0 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.333+0000 7fea3dc4d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fea3810f030 con 0x7fea38101a70 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.333+0000 7fea3dc4d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea3810f330 con 0x7fea38109520 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea36ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 0x7fea3819a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea377fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea38101a70 0x7fea38199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea377fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea38101a70 0x7fea38199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46846/0 (socket says 172.21.15.29:46846) 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea377fe640 1 -- 172.21.15.29:0/3769665337 learned_addr learned my addr 172.21.15.29:0/3769665337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea37fff640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea38109520 0x7fea3819b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea36ffd640 1 -- 172.21.15.29:0/3769665337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea38109520 msgr2=0x7fea3819b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea36ffd640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea38109520 0x7fea3819b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea36ffd640 1 -- 172.21.15.29:0/3769665337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea38101a70 msgr2=0x7fea38199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea36ffd640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea38101a70 0x7fea38199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea36ffd640 1 -- 172.21.15.29:0/3769665337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3819b900 con 0x7fea38106bc0 2024-09-21T03:02:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.334+0000 7fea36ffd640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 0x7fea3819a530 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fea28005760 tx=0x7fea28005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.335+0000 7fea34ff9640 1 -- 172.21.15.29:0/3769665337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28035050 con 0x7fea38106bc0 2024-09-21T03:02:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.335+0000 7fea34ff9640 1 -- 172.21.15.29:0/3769665337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea28004350 con 0x7fea38106bc0 2024-09-21T03:02:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.335+0000 7fea3dc4d640 1 -- 172.21.15.29:0/3769665337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea3819bb90 con 0x7fea38106bc0 2024-09-21T03:02:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.335+0000 7fea34ff9640 1 -- 172.21.15.29:0/3769665337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28003a50 con 0x7fea38106bc0 2024-09-21T03:02:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.335+0000 7fea3dc4d640 1 -- 172.21.15.29:0/3769665337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea38102b80 con 0x7fea38106bc0 2024-09-21T03:02:50.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea44bbb740 1 Processor -- start 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea44bbb740 1 -- start start 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea34ff9640 1 -- 172.21.15.29:0/3769665337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fea2803ac20 con 0x7fea38106bc0 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e38b1b0 0x55712e732e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e3622f0 0x55712e7334c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e734010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea44bbb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55712e6a56f0 con 0x55712e36a4d0 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea44bbb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55712e694b20 con 0x55712e38b1b0 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea44bbb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55712e698930 con 0x55712e3622f0 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea36ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e734010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea377fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e3622f0 0x55712e7334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea37fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e38b1b0 0x55712e732e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea37fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e38b1b0 0x55712e732e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46852/0 (socket says 172.21.15.29:46852) 2024-09-21T03:02:50.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea37fff640 1 -- 172.21.15.29:0/1930545304 learned_addr learned my addr 172.21.15.29:0/1930545304 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:50.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.339+0000 7fea377fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e3622f0 0x55712e7334c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:33044/0 (socket says 172.21.15.29:33044) 2024-09-21T03:02:50.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea377fe640 1 -- 172.21.15.29:0/1930545304 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e38b1b0 msgr2=0x55712e732e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:50.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea377fe640 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e38b1b0 0x55712e732e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea377fe640 1 -- 172.21.15.29:0/1930545304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 msgr2=0x55712e734010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea377fe640 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e734010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea377fe640 1 -- 172.21.15.29:0/1930545304 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55712e65a530 con 0x55712e3622f0 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea37fff640 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e38b1b0 0x55712e732e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea36ffd640 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e734010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea377fe640 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e3622f0 0x55712e7334c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009c80 tx=0x7fea2c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=835bc941f08703b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea34ff9640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fea200b6590 0x7fea200b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea1dffb640 1 -- 172.21.15.29:0/1930545304 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c004230 con 0x55712e3622f0 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea1dffb640 1 -- 172.21.15.29:0/1930545304 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea2c0043d0 con 0x55712e3622f0 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea1dffb640 1 -- 172.21.15.29:0/1930545304 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c004c00 con 0x55712e3622f0 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.340+0000 7fea37fff640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fea200b6590 0x7fea200b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.341+0000 7fea34ff9640 1 -- 172.21.15.29:0/3769665337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fea2810b290 con 0x7fea38106bc0 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 -- 172.21.15.29:0/1930545304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e3622f0 msgr2=0x55712e7334c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea37fff640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fea200b6590 0x7fea200b8aa0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fea30009fd0 tx=0x7fea30009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e3622f0 0x55712e7334c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009c80 tx=0x7fea2c011810 comp rx=0 tx=0).stop 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 -- 172.21.15.29:0/1930545304 shutdown_connections 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e734010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e3622f0 0x55712e7334c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 --2- 172.21.15.29:0/1930545304 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e38b1b0 0x55712e732e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 -- 172.21.15.29:0/1930545304 >> 172.21.15.29:0/1930545304 conn(0x55712e37f300 msgr2=0x55712e72e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 -- 172.21.15.29:0/1930545304 shutdown_connections 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 -- 172.21.15.29:0/1930545304 wait complete. 2024-09-21T03:02:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.342+0000 7fea44bbb740 1 -- start start 2024-09-21T03:02:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea44bbb740 1 Processor -- start 2024-09-21T03:02:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e73c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e73cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e38b1b0 0x55712e73d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea44bbb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55712e5ccab0 con 0x55712e36a4d0 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea44bbb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55712e63a730 con 0x55712e3622f0 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea44bbb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55712e734b30 con 0x55712e38b1b0 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea377fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e73c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea36ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e73cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea37fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e38b1b0 0x55712e73d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea377fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e73c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46856/0 (socket says 172.21.15.29:46856) 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea37fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e38b1b0 0x55712e73d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:33048/0 (socket says 172.21.15.29:33048) 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea377fe640 1 -- 172.21.15.29:0/527383181 learned_addr learned my addr 172.21.15.29:0/527383181 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea36ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e73cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60356/0 (socket says 172.21.15.29:60356) 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.343+0000 7fea44bbb740 10 client.0 _ll_register_callbacks cb 0x55712e68e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea377fe640 1 -- 172.21.15.29:0/527383181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e38b1b0 msgr2=0x55712e73d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea377fe640 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e38b1b0 0x55712e73d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea377fe640 1 -- 172.21.15.29:0/527383181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 msgr2=0x55712e73cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea377fe640 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e73cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea377fe640 1 -- 172.21.15.29:0/527383181 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55712e3f6790 con 0x55712e3622f0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea377fe640 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e73c5b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009620 tx=0x7fea2c019ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e97b1cbafd4dc0af server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea16ffd640 1 -- 172.21.15.29:0/527383181 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c028020 con 0x55712e3622f0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea44bbb740 1 -- 172.21.15.29:0/527383181 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea2c0043d0 con 0x55712e3622f0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea16ffd640 1 -- 172.21.15.29:0/527383181 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fea2c024620 con 0x55712e3622f0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.344+0000 7fea16ffd640 1 -- 172.21.15.29:0/527383181 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fea2c033050 con 0x55712e3622f0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea44bbb740 10 client.27809 fetch_fsmap learned FSMap version 39 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea44bbb740 1 -- 172.21.15.29:0/527383181 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55712e46f620 con 0x55712e3622f0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea16ffd640 1 -- 172.21.15.29:0/527383181 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea2c01f040 con 0x55712e3622f0 2024-09-21T03:02:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea16ffd640 1 -- 172.21.15.29:0/527383181 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fea2c018070 con 0x55712e3622f0 2024-09-21T03:02:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea44bbb740 10 client.27809 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:02:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea44bbb740 10 client.27809 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:02:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea44bbb740 20 client.27809 populate_metadata read hostname 'smithi029' 2024-09-21T03:02:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea44bbb740 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55712e349d80 0x55712e749b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea44bbb740 4 client.27809 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:02:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea44bbb740 1 -- 172.21.15.29:0/527383181 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fea2c0043d0 con 0x55712e349d80 2024-09-21T03:02:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.345+0000 7fea36ffd640 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55712e349d80 0x55712e749b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.346+0000 7fea16ffd640 10 client.27809 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.347+0000 7fea36ffd640 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55712e349d80 0x55712e749b50 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4061296c544985da server_cookie=c5c50efe01a5aac6 in_seq=0 out_seq=0 2024-09-21T03:02:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea16ffd640 1 -- 172.21.15.29:0/527383181 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fea2c0043d0 con 0x55712e349d80 2024-09-21T03:02:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea16ffd640 10 client.27809 handle_command_reply: tid=0 2024-09-21T03:02:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 client.27809 shutdown 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 -- 172.21.15.29:0/527383181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 msgr2=0x55712e73c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e73c5b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009620 tx=0x7fea2c019ad0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 -- 172.21.15.29:0/527383181 shutdown_connections 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e38b1b0 0x55712e73d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55712e349d80 0x55712e749b50 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e36a4d0 0x55712e73cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea16ffd640 0 client.27809 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 --2- 172.21.15.29:0/527383181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e73c5b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.348+0000 7fea44bbb740 1 -- 172.21.15.29:0/527383181 >> 172.21.15.29:0/527383181 conn(0x55712e37f300 msgr2=0x55712e72cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.349+0000 7fea44bbb740 1 -- 172.21.15.29:0/527383181 shutdown_connections 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.349+0000 7fea44bbb740 1 -- 172.21.15.29:0/527383181 wait complete. 2024-09-21T03:02:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.349+0000 7fea44bbb740 20 client.27809 trim_cache size 0 max 16384 2024-09-21T03:02:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.371+0000 7fea44bbb740 1 Processor -- start 2024-09-21T03:02:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.371+0000 7fea44bbb740 1 -- start start 2024-09-21T03:02:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.371+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e349d80 0x55712e732df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.371+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e3622f0 0x55712e733330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.371+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e7446c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.371+0000 7fea44bbb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea381aa330 con 0x55712e3622f0 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.371+0000 7fea44bbb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fea381abe70 con 0x55712e349d80 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.371+0000 7fea44bbb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea381aaa00 con 0x55712e36a4d0 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e349d80 0x55712e732df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e349d80 0x55712e732df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:46870/0 (socket says 172.21.15.29:46870) 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 -- 172.21.15.29:0/3959301224 learned_addr learned my addr 172.21.15.29:0/3959301224 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea37fff640 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e7446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 -- 172.21.15.29:0/3959301224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 msgr2=0x55712e7446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e7446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 -- 172.21.15.29:0/3959301224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e3622f0 msgr2=0x55712e733330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e3622f0 0x55712e733330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 -- 172.21.15.29:0/3959301224 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55712e73db50 con 0x55712e349d80 2024-09-21T03:02:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea37fff640 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e7446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:02:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea377fe640 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e349d80 0x55712e732df0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fea2c011d10 tx=0x7fea2c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4cd27f1def9a3b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.372+0000 7fea1d7fa640 1 -- 172.21.15.29:0/3959301224 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c03c070 con 0x55712e349d80 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.373+0000 7fea1d7fa640 1 -- 172.21.15.29:0/3959301224 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea2c019b40 con 0x55712e349d80 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.373+0000 7fea1d7fa640 1 -- 172.21.15.29:0/3959301224 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c0378c0 con 0x55712e349d80 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 -- 172.21.15.29:0/3959301224 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e349d80 msgr2=0x55712e732df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e349d80 0x55712e732df0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fea2c011d10 tx=0x7fea2c009b80 comp rx=0 tx=0).stop 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 -- 172.21.15.29:0/3959301224 shutdown_connections 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e7446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e3622f0 0x55712e733330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 --2- 172.21.15.29:0/3959301224 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e349d80 0x55712e732df0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 -- 172.21.15.29:0/3959301224 >> 172.21.15.29:0/3959301224 conn(0x55712e37f300 msgr2=0x55712e72d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 -- 172.21.15.29:0/3959301224 shutdown_connections 2024-09-21T03:02:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 -- 172.21.15.29:0/3959301224 wait complete. 2024-09-21T03:02:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.374+0000 7fea44bbb740 1 -- start start 2024-09-21T03:02:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea44bbb740 1 Processor -- start 2024-09-21T03:02:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e349d80 0x55712e73f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e740010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea44bbb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e740b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea44bbb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea381ab7a0 con 0x55712e349d80 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea44bbb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fea381ab0d0 con 0x55712e3622f0 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea44bbb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea381a8310 con 0x55712e36a4d0 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea44bbb740 10 client.0 _ll_register_callbacks cb 0x7fea2810ec40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e349d80 0x55712e73f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea37fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e740b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e349d80 0x55712e73f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60386/0 (socket says 172.21.15.29:60386) 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea37fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e740b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:33064/0 (socket says 172.21.15.29:33064) 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea377fe640 1 -- 172.21.15.29:0/11356183 learned_addr learned my addr 172.21.15.29:0/11356183 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.375+0000 7fea36ffd640 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e740010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea36ffd640 1 -- 172.21.15.29:0/11356183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 msgr2=0x55712e740b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea36ffd640 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e740b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea36ffd640 1 -- 172.21.15.29:0/11356183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e349d80 msgr2=0x55712e73f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea36ffd640 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e349d80 0x55712e73f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea36ffd640 1 -- 172.21.15.29:0/11356183 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55712e741290 con 0x55712e3622f0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea377fe640 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e349d80 0x55712e73f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea36ffd640 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e740010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fea28108a60 tx=0x7fea2803af10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7dcbd915ad3ae78 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea16ffd640 1 -- 172.21.15.29:0/11356183 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea280d0e70 con 0x55712e3622f0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea16ffd640 1 -- 172.21.15.29:0/11356183 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fea28003bf0 con 0x55712e3622f0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.376+0000 7fea44bbb740 1 -- 172.21.15.29:0/11356183 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea2c019b40 con 0x55712e3622f0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea16ffd640 1 -- 172.21.15.29:0/11356183 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fea280d0810 con 0x55712e3622f0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea44bbb740 10 client.27815 fetch_fsmap learned FSMap version 39 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea44bbb740 1 -- 172.21.15.29:0/11356183 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55712e744660 con 0x55712e3622f0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea16ffd640 1 -- 172.21.15.29:0/11356183 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea280d0bb0 con 0x55712e3622f0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea16ffd640 1 -- 172.21.15.29:0/11356183 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fea280da6c0 con 0x55712e3622f0 2024-09-21T03:02:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea44bbb740 10 client.27815 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:02:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea44bbb740 10 client.27815 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:02:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea44bbb740 20 client.27815 populate_metadata read hostname 'smithi029' 2024-09-21T03:02:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea44bbb740 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55712e38b1b0 0x55712e7b6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:02:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea44bbb740 4 client.27815 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:02:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea44bbb740 1 -- 172.21.15.29:0/11356183 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fea2c019b40 con 0x55712e38b1b0 2024-09-21T03:02:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.377+0000 7fea377fe640 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55712e38b1b0 0x55712e7b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:02:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.378+0000 7fea16ffd640 10 client.27815 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.378+0000 7fea377fe640 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55712e38b1b0 0x55712e7b6950 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b90794b65d51040 server_cookie=fab453923ee7f325 in_seq=0 out_seq=0 2024-09-21T03:02:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.599+0000 7fea16ffd640 1 -- 172.21.15.29:0/11356183 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fea2c019b40 con 0x55712e38b1b0 2024-09-21T03:02:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.599+0000 7fea16ffd640 10 client.27815 handle_command_reply: tid=0 2024-09-21T03:02:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.599+0000 7fea44bbb740 1 client.27815 shutdown 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 -- 172.21.15.29:0/11356183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 msgr2=0x55712e740010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e740010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fea28108a60 tx=0x7fea2803af10 comp rx=0 tx=0).stop 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 -- 172.21.15.29:0/11356183 shutdown_connections 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55712e36a4d0 0x55712e740b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55712e3622f0 0x55712e740010 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55712e38b1b0 0x55712e7b6950 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 --2- 172.21.15.29:0/11356183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55712e349d80 0x55712e73f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 -- 172.21.15.29:0/11356183 >> 172.21.15.29:0/11356183 conn(0x55712e37f300 msgr2=0x55712e72d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea16ffd640 0 client.27815 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 -- 172.21.15.29:0/11356183 shutdown_connections 2024-09-21T03:02:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 1 -- 172.21.15.29:0/11356183 wait complete. 2024-09-21T03:02:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.600+0000 7fea44bbb740 20 client.27815 trim_cache size 0 max 16384 2024-09-21T03:02:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:02:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:02:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:02:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.601+0000 7fea3dc4d640 1 -- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fea200b6590 msgr2=0x7fea200b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.601+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fea200b6590 0x7fea200b8aa0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fea30009fd0 tx=0x7fea30009500 comp rx=0 tx=0).stop 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.601+0000 7fea3dc4d640 1 -- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 msgr2=0x7fea3819a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.601+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 0x7fea3819a530 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fea28005760 tx=0x7fea28005e20 comp rx=0 tx=0).stop 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.602+0000 7fea3dc4d640 1 -- 172.21.15.29:0/3769665337 shutdown_connections 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.602+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fea200b6590 0x7fea200b8aa0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.602+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea38109520 0x7fea3819b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.602+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea38106bc0 0x7fea3819a530 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.602+0000 7fea3dc4d640 1 --2- 172.21.15.29:0/3769665337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea38101a70 0x7fea38199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:02:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.602+0000 7fea3dc4d640 1 -- 172.21.15.29:0/3769665337 >> 172.21.15.29:0/3769665337 conn(0x7fea380fd690 msgr2=0x7fea38107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:02:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.602+0000 7fea3dc4d640 1 -- 172.21.15.29:0/3769665337 shutdown_connections 2024-09-21T03:02:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:02:50.602+0000 7fea3dc4d640 1 -- 172.21.15.29:0/3769665337 wait complete. 2024-09-21T03:02:50.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 64899 5.54 MB/sec execute 507 sec latency 272.063 ms 2024-09-21T03:02:51.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65023 5.54 MB/sec execute 508 sec latency 213.946 ms 2024-09-21T03:02:52.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:51 smithi066 ceph-mon[33436]: pgmap v1102: 129 pgs: 129 active+clean; 343 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 4.3 MiB/s wr, 269 op/s 2024-09-21T03:02:52.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:51 smithi190 ceph-mon[33871]: pgmap v1102: 129 pgs: 129 active+clean; 343 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 4.3 MiB/s wr, 269 op/s 2024-09-21T03:02:52.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:51 smithi029 ceph-mon[31006]: pgmap v1102: 129 pgs: 129 active+clean; 343 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 4.3 MiB/s wr, 269 op/s 2024-09-21T03:02:52.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65131 5.54 MB/sec execute 509 sec latency 109.090 ms 2024-09-21T03:02:53.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65249 5.54 MB/sec execute 510 sec latency 92.325 ms 2024-09-21T03:02:54.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:53 smithi066 ceph-mon[33436]: pgmap v1103: 129 pgs: 129 active+clean; 343 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-09-21T03:02:54.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:53 smithi190 ceph-mon[33871]: pgmap v1103: 129 pgs: 129 active+clean; 343 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-09-21T03:02:54.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:53 smithi029 ceph-mon[31006]: pgmap v1103: 129 pgs: 129 active+clean; 343 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-09-21T03:02:54.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65392 5.54 MB/sec execute 511 sec latency 100.213 ms 2024-09-21T03:02:55.692 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.9 2024-09-21T03:02:55.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:02:55.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65530 5.55 MB/sec execute 512 sec latency 153.196 ms 2024-09-21T03:02:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:55 smithi066 ceph-mon[33436]: pgmap v1104: 129 pgs: 129 active+clean; 339 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 4.3 MiB/s wr, 264 op/s 2024-09-21T03:02:56.031 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:02:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:55 smithi190 ceph-mon[33871]: pgmap v1104: 129 pgs: 129 active+clean; 339 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 4.3 MiB/s wr, 264 op/s 2024-09-21T03:02:56.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:55 smithi029 ceph-mon[31006]: pgmap v1104: 129 pgs: 129 active+clean; 339 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 4.3 MiB/s wr, 264 op/s 2024-09-21T03:02:56.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65651 5.55 MB/sec execute 513 sec latency 205.692 ms 2024-09-21T03:02:57.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65772 5.55 MB/sec execute 514 sec latency 249.134 ms 2024-09-21T03:02:58.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:57 smithi066 ceph-mon[33436]: pgmap v1105: 129 pgs: 129 active+clean; 341 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.8 MiB/s wr, 263 op/s 2024-09-21T03:02:58.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:57 smithi190 ceph-mon[33871]: pgmap v1105: 129 pgs: 129 active+clean; 341 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.8 MiB/s wr, 263 op/s 2024-09-21T03:02:58.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:57 smithi029 ceph-mon[31006]: pgmap v1105: 129 pgs: 129 active+clean; 341 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 3.8 MiB/s wr, 263 op/s 2024-09-21T03:02:58.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65875 5.55 MB/sec execute 515 sec latency 168.268 ms 2024-09-21T03:02:59.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 65984 5.55 MB/sec execute 516 sec latency 290.066 ms 2024-09-21T03:02:59.761 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:02:59 smithi029 ceph-mon[31006]: pgmap v1106: 129 pgs: 129 active+clean; 341 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 3.9 MiB/s wr, 286 op/s 2024-09-21T03:03:00.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:02:59 smithi066 ceph-mon[33436]: pgmap v1106: 129 pgs: 129 active+clean; 341 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 3.9 MiB/s wr, 286 op/s 2024-09-21T03:03:00.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:02:59 smithi190 ceph-mon[33871]: pgmap v1106: 129 pgs: 129 active+clean; 341 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 3.9 MiB/s wr, 286 op/s 2024-09-21T03:03:00.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 -- 172.21.15.29:0/843095032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ac010a190 msgr2=0x7f2ac010a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:00.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/843095032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ac010a190 0x7f2ac010a5a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009860 tx=0x7f2ab002a340 comp rx=0 tx=0).stop 2024-09-21T03:03:00.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 -- 172.21.15.29:0/843095032 shutdown_connections 2024-09-21T03:03:00.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/843095032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac0110420 0x7f2ac0116c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/843095032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac010fad0 0x7f2ac010fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/843095032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ac010a190 0x7f2ac010a5a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 -- 172.21.15.29:0/843095032 >> 172.21.15.29:0/843095032 conn(0x7f2ac00780f0 msgr2=0x7f2ac01081e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:00.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 -- 172.21.15.29:0/843095032 shutdown_connections 2024-09-21T03:03:00.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.553+0000 7f2ac4c08640 1 -- 172.21.15.29:0/843095032 wait complete. 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.554+0000 7f2ac4c08640 1 Processor -- start 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.554+0000 7f2ac4c08640 1 -- start start 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ac4c08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac010a190 0x7f2ac01a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ac4c08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac010fad0 0x7f2ac01a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ac4c08640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ac0110420 0x7f2ac01a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ac4c08640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac0119fc0 con 0x7f2ac010fad0 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ac4c08640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac0119e40 con 0x7f2ac0110420 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ac4c08640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac011a140 con 0x7f2ac010a190 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ab7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac010fad0 0x7f2ac01a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ab7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac010fad0 0x7f2ac01a0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53172/0 (socket says 172.21.15.29:53172) 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ab7fff640 1 -- 172.21.15.29:0/3337523042 learned_addr learned my addr 172.21.15.29:0/3337523042 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2abe59c640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac010a190 0x7f2ac01a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2abed9d640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ac0110420 0x7f2ac01a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ab7fff640 1 -- 172.21.15.29:0/3337523042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac010a190 msgr2=0x7f2ac01a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac010a190 0x7f2ac01a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ab7fff640 1 -- 172.21.15.29:0/3337523042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ac0110420 msgr2=0x7f2ac01a6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ac0110420 0x7f2ac01a6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.555+0000 7f2ab7fff640 1 -- 172.21.15.29:0/3337523042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac01a6940 con 0x7f2ac010fad0 2024-09-21T03:03:00.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.556+0000 7f2abe59c640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac010a190 0x7f2ac01a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.556+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac010fad0 0x7f2ac01a0f70 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800e9c0 tx=0x7f2aa800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.556+0000 7f2ab77fe640 1 -- 172.21.15.29:0/3337523042 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800ccc0 con 0x7f2ac010fad0 2024-09-21T03:03:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.556+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac01a6c30 con 0x7f2ac010fad0 2024-09-21T03:03:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.556+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac01ad390 con 0x7f2ac010fad0 2024-09-21T03:03:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.556+0000 7f2ab77fe640 1 -- 172.21.15.29:0/3337523042 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2aa80054b0 con 0x7f2ac010fad0 2024-09-21T03:03:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.556+0000 7f2ab77fe640 1 -- 172.21.15.29:0/3337523042 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8004e60 con 0x7f2ac010fad0 2024-09-21T03:03:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.558+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac010b410 con 0x7f2ac010fad0 2024-09-21T03:03:00.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/3337523042 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f2aa8005020 con 0x7f2ac010fad0 2024-09-21T03:03:00.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.561+0000 7f2ab77fe640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2aac0b6510 0x7f2aac0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:00.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.561+0000 7f2abe59c640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2aac0b6510 0x7f2aac0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:00.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.561+0000 7f2ab77fe640 1 -- 172.21.15.29:0/3337523042 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f2aa80e7a80 con 0x7f2ac010fad0 2024-09-21T03:03:00.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.562+0000 7f2abe59c640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2aac0b6510 0x7f2aac0b8a20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00057c0 tx=0x7f2ab00058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:00.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.566+0000 7f2ab77fe640 1 -- 172.21.15.29:0/3337523042 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2aa80adfc0 con 0x7f2ac010fad0 2024-09-21T03:03:00.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66121 5.55 MB/sec execute 517 sec latency 106.816 ms 2024-09-21T03:03:00.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.946+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ac01121a0 con 0x7f2ac010fad0 2024-09-21T03:03:00.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.948+0000 7f2ab77fe640 1 -- 172.21.15.29:0/3337523042 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11971 (secure 0 0 0) 0x7f2aa80b23a0 con 0x7f2ac010fad0 2024-09-21T03:03:00.949 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:03:00.950 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"btime":"2024-09-21T03:02:24:667358+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:02:24.667336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:03:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.950+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2aac0b6510 msgr2=0x7f2aac0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.950+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2aac0b6510 0x7f2aac0b8a20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00057c0 tx=0x7f2ab00058b0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.950+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac010fad0 msgr2=0x7f2ac01a0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.950+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac010fad0 0x7f2ac01a0f70 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800e9c0 tx=0x7f2aa800ee80 comp rx=0 tx=0).stop 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.950+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 shutdown_connections 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.950+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2aac0b6510 0x7f2aac0b8a20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.950+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ac0110420 0x7f2ac01a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.951+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac010fad0 0x7f2ac01a0f70 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.951+0000 7f2ac4c08640 1 --2- 172.21.15.29:0/3337523042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac010a190 0x7f2ac01a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.951+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 >> 172.21.15.29:0/3337523042 conn(0x7f2ac00780f0 msgr2=0x7f2ac010dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.951+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 shutdown_connections 2024-09-21T03:03:00.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:00.951+0000 7f2ac4c08640 1 -- 172.21.15.29:0/3337523042 wait complete. 2024-09-21T03:03:01.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66239 5.55 MB/sec execute 518 sec latency 119.838 ms 2024-09-21T03:03:01.999 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:01 smithi029 ceph-mon[31006]: pgmap v1107: 129 pgs: 129 active+clean; 340 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 3.9 MiB/s wr, 304 op/s 2024-09-21T03:03:01.999 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:01 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3337523042' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:03:02.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:01 smithi066 ceph-mon[33436]: pgmap v1107: 129 pgs: 129 active+clean; 340 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 3.9 MiB/s wr, 304 op/s 2024-09-21T03:03:02.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:01 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3337523042' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:03:02.044 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 139.9 --await-for 45.0' 2024-09-21T03:03:02.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 139.9 --await-for 45.0 2024-09-21T03:03:02.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:01 smithi190 ceph-mon[33871]: pgmap v1107: 129 pgs: 129 active+clean; 340 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 3.9 MiB/s wr, 304 op/s 2024-09-21T03:03:02.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:01 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3337523042' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:03:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:03:02.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66367 5.56 MB/sec execute 519 sec latency 125.990 ms 2024-09-21T03:03:03.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 -- 172.21.15.29:0/892284848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d0810de50 msgr2=0x7f4d08110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/892284848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d0810de50 0x7f4d08110320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009be0 tx=0x7f4d0002a6d0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 -- 172.21.15.29:0/892284848 shutdown_connections 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/892284848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d0810de50 0x7f4d08110320 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/892284848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d081052d0 0x7f4d0810d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/892284848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d081048a0 0x7f4d08104cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 -- 172.21.15.29:0/892284848 >> 172.21.15.29:0/892284848 conn(0x7f4d08079760 msgr2=0x7f4d08079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 -- 172.21.15.29:0/892284848 shutdown_connections 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.265+0000 7f4d0ffba640 1 -- 172.21.15.29:0/892284848 wait complete. 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.266+0000 7f4d0ffba640 1 Processor -- start 2024-09-21T03:03:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.266+0000 7f4d0ffba640 1 -- start start 2024-09-21T03:03:03.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0ffba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d081048a0 0x7f4d0819c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0ffba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d081052d0 0x7f4d0819c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0ffba640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d0810de50 0x7f4d081a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0ffba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d08113940 con 0x7f4d081048a0 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0ffba640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4d081137c0 con 0x7f4d0810de50 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0ffba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4d08113ac0 con 0x7f4d081052d0 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0d555640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d081052d0 0x7f4d0819c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0e557640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d0810de50 0x7f4d081a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0d555640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d081052d0 0x7f4d0819c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45544/0 (socket says 172.21.15.29:45544) 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0d555640 1 -- 172.21.15.29:0/1040051549 learned_addr learned my addr 172.21.15.29:0/1040051549 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.267+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d081048a0 0x7f4d0819c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4d0d555640 1 -- 172.21.15.29:0/1040051549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d0810de50 msgr2=0x7f4d081a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4d0d555640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d0810de50 0x7f4d081a1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4d0d555640 1 -- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d081048a0 msgr2=0x7f4d0819c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4d0d555640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d081048a0 0x7f4d0819c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4d0d555640 1 -- 172.21.15.29:0/1040051549 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d081a2340 con 0x7f4d081052d0 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d081048a0 0x7f4d0819c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4d0e557640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d0810de50 0x7f4d081a1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4d0d555640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d081052d0 0x7f4d0819c970 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800e9c0 tx=0x7f4cf800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.268+0000 7f4cfeffd640 1 -- 172.21.15.29:0/1040051549 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf800ccc0 con 0x7f4d081052d0 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.269+0000 7f4d0ffba640 1 -- 172.21.15.29:0/1040051549 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d081a2630 con 0x7f4d081052d0 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.269+0000 7f4cfeffd640 1 -- 172.21.15.29:0/1040051549 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4cf80054b0 con 0x7f4d081052d0 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.269+0000 7f4cfeffd640 1 -- 172.21.15.29:0/1040051549 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8010430 con 0x7f4d081052d0 2024-09-21T03:03:03.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.269+0000 7f4d0ffba640 1 -- 172.21.15.29:0/1040051549 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d081a8d90 con 0x7f4d081052d0 2024-09-21T03:03:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.272+0000 7f4d1652e740 1 Processor -- start 2024-09-21T03:03:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.272+0000 7f4d1652e740 1 -- start start 2024-09-21T03:03:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.272+0000 7f4cfeffd640 1 -- 172.21.15.29:0/1040051549 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4cf8004eb0 con 0x7f4d081052d0 2024-09-21T03:03:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd93c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd93cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd93d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d1652e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cedd8b2b90 con 0x55cedd595840 2024-09-21T03:03:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d1652e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55cedd8a1fc0 con 0x55cedd56c980 2024-09-21T03:03:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d1652e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cedd8a5dd0 con 0x55cedd574b60 2024-09-21T03:03:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d0dd56640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd93cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d0dd56640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd93cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:32990/0 (socket says 172.21.15.29:32990) 2024-09-21T03:03:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d0dd56640 1 -- 172.21.15.29:0/2346702096 learned_addr learned my addr 172.21.15.29:0/2346702096 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.273+0000 7f4d0e557640 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd93c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.274+0000 7f4d0d555640 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd93d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.274+0000 7f4d0dd56640 1 -- 172.21.15.29:0/2346702096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 msgr2=0x55cedd93d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.274+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd93d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.274+0000 7f4d0dd56640 1 -- 172.21.15.29:0/2346702096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 msgr2=0x55cedd93c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.274+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd93c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.274+0000 7f4d0dd56640 1 -- 172.21.15.29:0/2346702096 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cedd829060 con 0x55cedd56c980 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.274+0000 7f4d0d555640 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd93d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:03:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.274+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd93cb40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4d040114f0 tx=0x7f4d040119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab5578cf438995c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.275+0000 7f4cfeffd640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cf40b6650 0x7f4cf40b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.275+0000 7f4cf3fff640 1 -- 172.21.15.29:0/2346702096 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d04005020 con 0x55cedd56c980 2024-09-21T03:03:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.275+0000 7f4cf3fff640 1 -- 172.21.15.29:0/2346702096 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d040047a0 con 0x55cedd56c980 2024-09-21T03:03:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.275+0000 7f4cf3fff640 1 -- 172.21.15.29:0/2346702096 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0400f8e0 con 0x55cedd56c980 2024-09-21T03:03:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.275+0000 7f4d0e557640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cf40b6650 0x7f4cf40b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.275+0000 7f4cfeffd640 1 -- 172.21.15.29:0/1040051549 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f4cf80e7b80 con 0x7f4d081052d0 2024-09-21T03:03:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.276+0000 7f4d1652e740 1 -- 172.21.15.29:0/2346702096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 msgr2=0x55cedd93cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.276+0000 7f4d1652e740 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd93cb40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4d040114f0 tx=0x7f4d040119b0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.276+0000 7f4d0e557640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cf40b6650 0x7f4cf40b8b60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009b30 tx=0x7f4d0000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.276+0000 7f4d1652e740 1 -- 172.21.15.29:0/2346702096 shutdown_connections 2024-09-21T03:03:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.276+0000 7f4d1652e740 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd93d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.276+0000 7f4d1652e740 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd93cb40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.276+0000 7f4d1652e740 1 --2- 172.21.15.29:0/2346702096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd93c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.276+0000 7f4d1652e740 1 -- 172.21.15.29:0/2346702096 >> 172.21.15.29:0/2346702096 conn(0x55cedd589990 msgr2=0x55cedd93a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.277+0000 7f4d1652e740 1 -- 172.21.15.29:0/2346702096 shutdown_connections 2024-09-21T03:03:03.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.277+0000 7f4d1652e740 1 -- 172.21.15.29:0/2346702096 wait complete. 2024-09-21T03:03:03.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.280+0000 7f4d1652e740 1 -- start start 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.280+0000 7f4d1652e740 1 Processor -- start 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd56c980 0x55cedd9481c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd948700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd595840 0x55cedd95c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d1652e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cedd7037a0 con 0x55cedd56c980 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d0dd56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd56c980 0x55cedd9481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d0dd56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd56c980 0x55cedd9481c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53210/0 (socket says 172.21.15.29:53210) 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d0dd56640 1 -- 172.21.15.29:0/1739908136 learned_addr learned my addr 172.21.15.29:0/1739908136 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55cedd8470a0 con 0x55cedd595840 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cedd94ade0 con 0x55cedd574b60 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d0e557640 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd595840 0x55cedd95c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d0d555640 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd948700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.281+0000 7f4d1652e740 10 client.0 _ll_register_callbacks cb 0x55cedd8243c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:03.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.282+0000 7f4d0e557640 1 -- 172.21.15.29:0/1739908136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 msgr2=0x55cedd948700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.282+0000 7f4d0e557640 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd948700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.282+0000 7f4d0e557640 1 -- 172.21.15.29:0/1739908136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd56c980 msgr2=0x55cedd9481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.282+0000 7f4d0e557640 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd56c980 0x55cedd9481c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.282+0000 7f4d0e557640 1 -- 172.21.15.29:0/1739908136 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cedd835880 con 0x55cedd595840 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.282+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d040047a0 con 0x55cedd595840 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.282+0000 7f4d0e557640 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd595840 0x55cedd95c8e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009560 tx=0x7f4d0002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=768036e95d2b1dcb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.282+0000 7f4cf17fa640 10 client.27833 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.283+0000 7f4cf17fa640 1 -- 172.21.15.29:0/1739908136 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0004f020 con 0x55cedd595840 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.283+0000 7f4cf17fa640 1 -- 172.21.15.29:0/1739908136 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f4d00003d80 con 0x55cedd595840 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.283+0000 7f4cf17fa640 1 -- 172.21.15.29:0/1739908136 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d00036050 con 0x55cedd595840 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.283+0000 7f4d1652e740 10 client.27833 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.283+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({fsmap=39}) -- 0x55cedd60fb90 con 0x55cedd595840 2024-09-21T03:03:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.284+0000 7f4cf17fa640 1 -- 172.21.15.29:0/1739908136 <== mon.2 v2:172.21.15.66:3300/0 4 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f4d00004040 con 0x55cedd595840 2024-09-21T03:03:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.284+0000 7f4d1652e740 10 client.27833 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.284+0000 7f4d1652e740 10 client.27833 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:03:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.284+0000 7f4d1652e740 20 client.27833 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.284+0000 7f4d1652e740 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cedd5541c0 0x55cedd962b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.284+0000 7f4d1652e740 4 client.27833 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:03:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.284+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cedd8b4ae0 con 0x55cedd5541c0 2024-09-21T03:03:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.284+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cedd5541c0 0x55cedd962b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.285+0000 7f4cf17fa640 10 client.27833 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.285+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cedd5541c0 0x55cedd962b30 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b21ff21bd62313ff server_cookie=aa1bfae0fd7a63b8 in_seq=0 out_seq=0 2024-09-21T03:03:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4cf17fa640 1 -- 172.21.15.29:0/1739908136 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cedd8b4ae0 con 0x55cedd5541c0 2024-09-21T03:03:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4cf17fa640 10 client.27833 handle_command_reply: tid=0 2024-09-21T03:03:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 client.27833 shutdown 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd595840 msgr2=0x55cedd95c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd595840 0x55cedd95c8e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009560 tx=0x7f4d0002a6d0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 shutdown_connections 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd595840 0x55cedd95c8e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd948700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cedd5541c0 0x55cedd962b30 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 --2- 172.21.15.29:0/1739908136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd56c980 0x55cedd9481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 >> 172.21.15.29:0/1739908136 conn(0x55cedd589990 msgr2=0x55cedd93f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4cf17fa640 0 client.27833 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 shutdown_connections 2024-09-21T03:03:03.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 1 -- 172.21.15.29:0/1739908136 wait complete. 2024-09-21T03:03:03.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.287+0000 7f4d1652e740 20 client.27833 trim_cache size 0 max 16384 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.322+0000 7f4d1652e740 1 Processor -- start 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.322+0000 7f4d1652e740 1 -- start start 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.322+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd95c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.322+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd941880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.322+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd941dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.322+0000 7f4d1652e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d081ae920 con 0x55cedd595840 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.322+0000 7f4d1652e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4d081b0460 con 0x55cedd56c980 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.322+0000 7f4d1652e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4d081aeff0 con 0x55cedd574b60 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd941880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd941880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45556/0 (socket says 172.21.15.29:45556) 2024-09-21T03:03:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 -- 172.21.15.29:0/2535811077 learned_addr learned my addr 172.21.15.29:0/2535811077 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0e557640 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd941dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd95c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 -- 172.21.15.29:0/2535811077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 msgr2=0x55cedd95c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd95c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 -- 172.21.15.29:0/2535811077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 msgr2=0x55cedd941dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd941dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 -- 172.21.15.29:0/2535811077 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cedd9488a0 con 0x55cedd574b60 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.323+0000 7f4d0d555640 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd941880 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009740 tx=0x7f4cf80aa040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31aae3ae4de8d230 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.324+0000 7f4cf37fe640 1 -- 172.21.15.29:0/2535811077 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80ad0a0 con 0x55cedd574b60 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.324+0000 7f4cf37fe640 1 -- 172.21.15.29:0/2535811077 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4cf80046c0 con 0x55cedd574b60 2024-09-21T03:03:03.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.324+0000 7f4cf37fe640 1 -- 172.21.15.29:0/2535811077 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf800ce60 con 0x55cedd574b60 2024-09-21T03:03:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 -- 172.21.15.29:0/2535811077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 msgr2=0x55cedd941880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd941880 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009740 tx=0x7f4cf80aa040 comp rx=0 tx=0).stop 2024-09-21T03:03:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 -- 172.21.15.29:0/2535811077 shutdown_connections 2024-09-21T03:03:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd941dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd941880 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 --2- 172.21.15.29:0/2535811077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd95c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 -- 172.21.15.29:0/2535811077 >> 172.21.15.29:0/2535811077 conn(0x55cedd589990 msgr2=0x55cedd946840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 -- 172.21.15.29:0/2535811077 shutdown_connections 2024-09-21T03:03:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 -- 172.21.15.29:0/2535811077 wait complete. 2024-09-21T03:03:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.325+0000 7f4d1652e740 1 -- start start 2024-09-21T03:03:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.326+0000 7f4d1652e740 1 Processor -- start 2024-09-21T03:03:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.326+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd958f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd953880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d1652e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd953dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d0d555640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd953880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d0d555640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd953880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45566/0 (socket says 172.21.15.29:45566) 2024-09-21T03:03:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d0dd56640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd958f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d0dd56640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd958f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:33022/0 (socket says 172.21.15.29:33022) 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d0e557640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd953dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d0e557640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd953dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53222/0 (socket says 172.21.15.29:53222) 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d0d555640 1 -- 172.21.15.29:0/657701208 learned_addr learned my addr 172.21.15.29:0/657701208 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d081afd90 con 0x55cedd595840 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4d081af6c0 con 0x55cedd56c980 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.327+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4d081ac970 con 0x55cedd574b60 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d1652e740 10 client.0 _ll_register_callbacks cb 0x7f4d00042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d0d555640 1 -- 172.21.15.29:0/657701208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 msgr2=0x55cedd958f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d0d555640 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd958f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d0d555640 1 -- 172.21.15.29:0/657701208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 msgr2=0x55cedd953dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d0d555640 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd953dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d0d555640 1 -- 172.21.15.29:0/657701208 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cedd93ee90 con 0x55cedd574b60 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd958f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cedd8b0590 con 0x55cedd574b60 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4d0d555640 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd953880 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009830 tx=0x7f4cf80a9dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6625eca6ef4c47d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4cf17fa640 10 client.28003 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4cf17fa640 1 -- 172.21.15.29:0/657701208 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80eb070 con 0x55cedd574b60 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.328+0000 7f4cf17fa640 1 -- 172.21.15.29:0/657701208 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f4cf80a9280 con 0x55cedd574b60 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4cf17fa640 1 -- 172.21.15.29:0/657701208 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4cf80a9480 con 0x55cedd574b60 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4d1652e740 10 client.28003 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=39}) -- 0x55cedd9cdfa0 con 0x55cedd574b60 2024-09-21T03:03:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4cf17fa640 1 -- 172.21.15.29:0/657701208 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f4cf80ac600 con 0x55cedd574b60 2024-09-21T03:03:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4d1652e740 10 client.28003 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4d1652e740 10 client.28003 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:03:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4d1652e740 20 client.28003 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4d1652e740 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cedd529570 0x55cedd9d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4d1652e740 4 client.28003 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.9, "await_for": 45.0}] 2024-09-21T03:03:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.329+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.9, "await_for": 45.0}) -- 0x55cedd8ad9c0 con 0x55cedd529570 2024-09-21T03:03:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.330+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cedd529570 0x55cedd9d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.330+0000 7f4d0dd56640 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cedd529570 0x55cedd9d0530 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30e4efe9c8f2cd18 server_cookie=b8859dfab679e605 in_seq=0 out_seq=0 2024-09-21T03:03:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.330+0000 7f4cf17fa640 10 client.28003 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:03.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66453 5.55 MB/sec execute 520 sec latency 420.875 ms 2024-09-21T03:03:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.762+0000 7fd781f9b640 1 -- 172.21.15.29:0/1249831907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c103fc0 msgr2=0x7fd77c10a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.762+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1249831907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c103fc0 0x7fd77c10a960 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd774009c90 tx=0x7fd77402a860 comp rx=0 tx=0).stop 2024-09-21T03:03:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.762+0000 7fd781f9b640 1 -- 172.21.15.29:0/1249831907 shutdown_connections 2024-09-21T03:03:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.762+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1249831907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c103fc0 0x7fd77c10a960 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.762+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1249831907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c103620 0x7fd77c103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.763+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1249831907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c1025a0 0x7fd77c102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.763+0000 7fd781f9b640 1 -- 172.21.15.29:0/1249831907 >> 172.21.15.29:0/1249831907 conn(0x7fd77c0fdee0 msgr2=0x7fd77c100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.763+0000 7fd781f9b640 1 -- 172.21.15.29:0/1249831907 shutdown_connections 2024-09-21T03:03:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.763+0000 7fd781f9b640 1 -- 172.21.15.29:0/1249831907 wait complete. 2024-09-21T03:03:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd781f9b640 1 Processor -- start 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd781f9b640 1 -- start start 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd781f9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1025a0 0x7fd77c19c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd781f9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c103620 0x7fd77c19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd781f9b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103fc0 0x7fd77c19d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd781f9b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c10e090 con 0x7fd77c1025a0 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd781f9b640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c10df10 con 0x7fd77c103fc0 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd781f9b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c10e210 con 0x7fd77c103620 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1025a0 0x7fd77c19c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103fc0 0x7fd77c19d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103fc0 0x7fd77c19d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:33052/0 (socket says 172.21.15.29:33052) 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1025a0 0x7fd77c19c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53234/0 (socket says 172.21.15.29:53234) 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd77bfff640 1 -- 172.21.15.29:0/1853776353 learned_addr learned my addr 172.21.15.29:0/1853776353 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.764+0000 7fd77affd640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c103620 0x7fd77c19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.765+0000 7fd77b7fe640 1 -- 172.21.15.29:0/1853776353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c103620 msgr2=0x7fd77c19c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.765+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c103620 0x7fd77c19c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.765+0000 7fd77b7fe640 1 -- 172.21.15.29:0/1853776353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103fc0 msgr2=0x7fd77c19d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.765+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103fc0 0x7fd77c19d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.765+0000 7fd77b7fe640 1 -- 172.21.15.29:0/1853776353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c19daa0 con 0x7fd77c1025a0 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.765+0000 7fd77bfff640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103fc0 0x7fd77c19d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.765+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1025a0 0x7fd77c19c060 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00e9f0 tx=0x7fd76c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.766+0000 7fd778ff9640 1 -- 172.21.15.29:0/1853776353 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c00cd80 con 0x7fd77c1025a0 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.766+0000 7fd778ff9640 1 -- 172.21.15.29:0/1853776353 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd76c0035f0 con 0x7fd77c1025a0 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.766+0000 7fd781f9b640 1 -- 172.21.15.29:0/1853776353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c19dd90 con 0x7fd77c1025a0 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.766+0000 7fd778ff9640 1 -- 172.21.15.29:0/1853776353 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c010670 con 0x7fd77c1025a0 2024-09-21T03:03:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.766+0000 7fd781f9b640 1 -- 172.21.15.29:0/1853776353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c1a4480 con 0x7fd77c1025a0 2024-09-21T03:03:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd788fe4740 1 Processor -- start 2024-09-21T03:03:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd788fe4740 1 -- start start 2024-09-21T03:03:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f14660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f14cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f15820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd788fe4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b385e7e6f0 con 0x55b385b641b0 2024-09-21T03:03:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd788fe4740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b385e6db20 con 0x55b385b3b2f0 2024-09-21T03:03:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd788fe4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b385e71930 con 0x55b385b434d0 2024-09-21T03:03:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f14660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.769+0000 7fd77bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f14660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53242/0 (socket says 172.21.15.29:53242) 2024-09-21T03:03:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd77bfff640 1 -- 172.21.15.29:0/1220412200 learned_addr learned my addr 172.21.15.29:0/1220412200 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd77affd640 1 --2- 172.21.15.29:0/1220412200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f15820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd77affd640 1 -- 172.21.15.29:0/1220412200 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 msgr2=0x55b385f14cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd77affd640 1 --2- 172.21.15.29:0/1220412200 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f14cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd77affd640 1 -- 172.21.15.29:0/1220412200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 msgr2=0x55b385f14660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd77affd640 1 --2- 172.21.15.29:0/1220412200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f14660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd77affd640 1 -- 172.21.15.29:0/1220412200 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b385e33530 con 0x55b385b434d0 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd77affd640 1 --2- 172.21.15.29:0/1220412200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f15820 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd770009c20 tx=0x7fd770011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e8f577b8f4c6b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd761ffb640 1 -- 172.21.15.29:0/1220412200 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd770009dd0 con 0x55b385b434d0 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.770+0000 7fd761ffb640 1 -- 172.21.15.29:0/1220412200 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd770005020 con 0x55b385b434d0 2024-09-21T03:03:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.771+0000 7fd761ffb640 1 -- 172.21.15.29:0/1220412200 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd770004a00 con 0x55b385b434d0 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.771+0000 7fd778ff9640 1 -- 172.21.15.29:0/1853776353 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd76c004000 con 0x7fd77c1025a0 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.771+0000 7fd788fe4740 1 -- 172.21.15.29:0/1220412200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 msgr2=0x55b385f15820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.771+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1220412200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f15820 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd770009c20 tx=0x7fd770011810 comp rx=0 tx=0).stop 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd788fe4740 1 -- 172.21.15.29:0/1220412200 shutdown_connections 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1220412200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f15820 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1220412200 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f14cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1220412200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f14660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd788fe4740 1 -- 172.21.15.29:0/1220412200 >> 172.21.15.29:0/1220412200 conn(0x55b385b58300 msgr2=0x55b385f100f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd788fe4740 1 -- 172.21.15.29:0/1220412200 shutdown_connections 2024-09-21T03:03:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd788fe4740 1 -- 172.21.15.29:0/1220412200 wait complete. 2024-09-21T03:03:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd778ff9640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7640b6420 0x7fd7640b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd788fe4740 1 -- start start 2024-09-21T03:03:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.772+0000 7fd778ff9640 1 -- 172.21.15.29:0/1853776353 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fd76c0e7960 con 0x7fd77c1025a0 2024-09-21T03:03:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.773+0000 7fd77bfff640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7640b6420 0x7fd7640b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.773+0000 7fd788fe4740 1 Processor -- start 2024-09-21T03:03:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.773+0000 7fd77bfff640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7640b6420 0x7fd7640b8930 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd774009c60 tx=0x7fd774031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.774+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f1de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.774+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f1e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.774+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f1f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.774+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f1de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.774+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f1de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:33066/0 (socket says 172.21.15.29:33066) 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.774+0000 7fd77b7fe640 1 -- 172.21.15.29:0/649639743 learned_addr learned my addr 172.21.15.29:0/649639743 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.774+0000 7fd77affd640 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f1e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.774+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b385da5ab0 con 0x55b385b641b0 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd77bfff640 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f1f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55b385e13730 con 0x55b385b3b2f0 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b385f163e0 con 0x55b385b434d0 2024-09-21T03:03:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd788fe4740 10 client.0 _ll_register_callbacks cb 0x55b385e676b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:03.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd77affd640 1 -- 172.21.15.29:0/649639743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 msgr2=0x55b385f1de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd77affd640 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f1de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd77affd640 1 -- 172.21.15.29:0/649639743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 msgr2=0x55b385f1f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd77affd640 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f1f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd77affd640 1 -- 172.21.15.29:0/649639743 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b385bcf790 con 0x55b385b434d0 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.775+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f1de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.776+0000 7fd77affd640 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f1e510 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd77001b040 tx=0x7fd7700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=451dd1903cdc911e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.776+0000 7fd75affd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.776+0000 7fd75affd640 1 -- 172.21.15.29:0/649639743 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd770029020 con 0x55b385b434d0 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.776+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd770005020 con 0x55b385b434d0 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.776+0000 7fd75affd640 1 -- 172.21.15.29:0/649639743 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fd770004a40 con 0x55b385b434d0 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.776+0000 7fd75affd640 1 -- 172.21.15.29:0/649639743 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd770034050 con 0x55b385b434d0 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.776+0000 7fd788fe4740 10 client.28015 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b385c48620 con 0x55b385b434d0 2024-09-21T03:03:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd75affd640 1 -- 172.21.15.29:0/649639743 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd770005020 con 0x55b385b434d0 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd75affd640 1 -- 172.21.15.29:0/649639743 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fd770023420 con 0x55b385b434d0 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd788fe4740 10 client.28015 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd788fe4740 10 client.28015 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd788fe4740 20 client.28015 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd788fe4740 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b385b22d80 0x55b385f2b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd788fe4740 4 client.28015 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.777+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b385e80640 con 0x55b385b22d80 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.778+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b385b22d80 0x55b385f2b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.778+0000 7fd75affd640 10 client.28015 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.778+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b385b22d80 0x55b385f2b290 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7439858607f5ea32 server_cookie=56f215ec8333bba5 in_seq=0 out_seq=0 2024-09-21T03:03:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.781+0000 7fd75affd640 1 -- 172.21.15.29:0/649639743 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b385e80640 con 0x55b385b22d80 2024-09-21T03:03:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.781+0000 7fd75affd640 10 client.28015 handle_command_reply: tid=0 2024-09-21T03:03:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.781+0000 7fd788fe4740 1 client.28015 shutdown 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 msgr2=0x55b385f1e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f1e510 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd77001b040 tx=0x7fd7700046c0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 shutdown_connections 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b385b22d80 0x55b385f2b290 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd75affd640 0 client.28015 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b641b0 0x55b385f1f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b434d0 0x55b385f1e510 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 --2- 172.21.15.29:0/649639743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f1de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 >> 172.21.15.29:0/649639743 conn(0x55b385b58300 msgr2=0x55b385f05ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 shutdown_connections 2024-09-21T03:03:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.782+0000 7fd788fe4740 1 -- 172.21.15.29:0/649639743 wait complete. 2024-09-21T03:03:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.783+0000 7fd788fe4740 20 client.28015 trim_cache size 0 max 16384 2024-09-21T03:03:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.813+0000 7fd788fe4740 1 Processor -- start 2024-09-21T03:03:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.813+0000 7fd788fe4740 1 -- start start 2024-09-21T03:03:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.814+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b22d80 0x55b385f19240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.814+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f0fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.814+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b22d80 0x55b385f19240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.814+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b22d80 0x55b385f19240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45606/0 (socket says 172.21.15.29:45606) 2024-09-21T03:03:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.814+0000 7fd77b7fe640 1 -- 172.21.15.29:0/1641021887 learned_addr learned my addr 172.21.15.29:0/1641021887 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.814+0000 7fd77affd640 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f0fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.814+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b434d0 0x55b385f10530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.815+0000 7fd788fe4740 1 -- 172.21.15.29:0/1641021887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1aa490 con 0x55b385b434d0 2024-09-21T03:03:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.815+0000 7fd788fe4740 1 -- 172.21.15.29:0/1641021887 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1abfd0 con 0x55b385b3b2f0 2024-09-21T03:03:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.815+0000 7fd788fe4740 1 -- 172.21.15.29:0/1641021887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1aab60 con 0x55b385b22d80 2024-09-21T03:03:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.815+0000 7fd77b7fe640 1 -- 172.21.15.29:0/1641021887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 msgr2=0x55b385f0fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.815+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f0fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.815+0000 7fd77b7fe640 1 -- 172.21.15.29:0/1641021887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b434d0 msgr2=0x55b385f10530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.815+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b434d0 0x55b385f10530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.815+0000 7fd77b7fe640 1 -- 172.21.15.29:0/1641021887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b385f1bf20 con 0x55b385b22d80 2024-09-21T03:03:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.816+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b22d80 0x55b385f19240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009230 tx=0x7fd76c003d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f14c5e2c5447c803 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.817+0000 7fd7617fa640 1 -- 172.21.15.29:0/1641021887 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0ade10 con 0x55b385b22d80 2024-09-21T03:03:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.817+0000 7fd7617fa640 1 -- 172.21.15.29:0/1641021887 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd76c0b21f0 con 0x55b385b22d80 2024-09-21T03:03:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.817+0000 7fd7617fa640 1 -- 172.21.15.29:0/1641021887 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0ac6d0 con 0x55b385b22d80 2024-09-21T03:03:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.818+0000 7fd788fe4740 1 -- 172.21.15.29:0/1641021887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b22d80 msgr2=0x55b385f19240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.818+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b22d80 0x55b385f19240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009230 tx=0x7fd76c003d90 comp rx=0 tx=0).stop 2024-09-21T03:03:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.819+0000 7fd788fe4740 1 -- 172.21.15.29:0/1641021887 shutdown_connections 2024-09-21T03:03:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.819+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b434d0 0x55b385f10530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.819+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b3b2f0 0x55b385f0fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.819+0000 7fd788fe4740 1 --2- 172.21.15.29:0/1641021887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b22d80 0x55b385f19240 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.819+0000 7fd788fe4740 1 -- 172.21.15.29:0/1641021887 >> 172.21.15.29:0/1641021887 conn(0x55b385b58300 msgr2=0x55b385f02700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.819+0000 7fd788fe4740 1 -- 172.21.15.29:0/1641021887 shutdown_connections 2024-09-21T03:03:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.819+0000 7fd788fe4740 1 -- 172.21.15.29:0/1641021887 wait complete. 2024-09-21T03:03:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.820+0000 7fd788fe4740 1 -- start start 2024-09-21T03:03:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.820+0000 7fd788fe4740 1 Processor -- start 2024-09-21T03:03:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.820+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b22d80 0x55b385f25ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.820+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b3b2f0 0x55b385f24790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.820+0000 7fd788fe4740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b434d0 0x55b385f24cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.820+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b22d80 0x55b385f25ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b22d80 0x55b385f25ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53276/0 (socket says 172.21.15.29:53276) 2024-09-21T03:03:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77b7fe640 1 -- 172.21.15.29:0/2886385246 learned_addr learned my addr 172.21.15.29:0/2886385246 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1ab900 con 0x55b385b22d80 2024-09-21T03:03:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1ab230 con 0x55b385b434d0 2024-09-21T03:03:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1a84b0 con 0x55b385b3b2f0 2024-09-21T03:03:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77bfff640 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b434d0 0x55b385f24cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77affd640 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b3b2f0 0x55b385f24790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd788fe4740 10 client.0 _ll_register_callbacks cb 0x7fd76c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77bfff640 1 -- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b3b2f0 msgr2=0x55b385f24790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77bfff640 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b3b2f0 0x55b385f24790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77bfff640 1 -- 172.21.15.29:0/2886385246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b22d80 msgr2=0x55b385f25ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77bfff640 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b22d80 0x55b385f25ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77bfff640 1 -- 172.21.15.29:0/2886385246 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b385f25310 con 0x55b385b434d0 2024-09-21T03:03:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd76c0b21f0 con 0x55b385b434d0 2024-09-21T03:03:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77affd640 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b3b2f0 0x55b385f24790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd77bfff640 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b434d0 0x55b385f24cd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd774009a20 tx=0x7fd7740092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3492d2ee55bec2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.821+0000 7fd75affd640 10 client.27854 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:03:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.822+0000 7fd75affd640 1 -- 172.21.15.29:0/2886385246 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774030830 con 0x55b385b434d0 2024-09-21T03:03:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.822+0000 7fd75affd640 1 -- 172.21.15.29:0/2886385246 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fd7740309d0 con 0x55b385b434d0 2024-09-21T03:03:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.822+0000 7fd75affd640 1 -- 172.21.15.29:0/2886385246 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd774030c10 con 0x55b385b434d0 2024-09-21T03:03:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.822+0000 7fd788fe4740 10 client.27854 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.822+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({fsmap=39}) -- 0x55b385f240b0 con 0x55b385b434d0 2024-09-21T03:03:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.823+0000 7fd75affd640 1 -- 172.21.15.29:0/2886385246 <== mon.2 v2:172.21.15.66:3300/0 4 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fd774044050 con 0x55b385b434d0 2024-09-21T03:03:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.823+0000 7fd788fe4740 10 client.27854 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.823+0000 7fd788fe4740 10 client.27854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.823+0000 7fd788fe4740 20 client.27854 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.824+0000 7fd788fe4740 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b385b641b0 0x55b385f98360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.824+0000 7fd788fe4740 4 client.27854 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:03:03.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.824+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b385e7c0f0 con 0x55b385b641b0 2024-09-21T03:03:03.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.824+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b385b641b0 0x55b385f98360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:03.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.825+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b385b641b0 0x55b385f98360 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9e4c5cdcd5f774 server_cookie=6f9581934287a9d7 in_seq=0 out_seq=0 2024-09-21T03:03:03.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:03.825+0000 7fd75affd640 10 client.27854 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:03.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.890+0000 7f4cf17fa640 1 -- 172.21.15.29:0/657701208 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cedd8ad9c0 con 0x55cedd529570 2024-09-21T03:03:03.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.890+0000 7f4cf17fa640 10 client.28003 handle_command_reply: tid=0 2024-09-21T03:03:03.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.890+0000 7f4d1652e740 1 client.28003 shutdown 2024-09-21T03:03:03.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.890+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 msgr2=0x55cedd953880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.890+0000 7f4d1652e740 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd953880 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009830 tx=0x7f4cf80a9dd0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.890+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 shutdown_connections 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4d1652e740 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cedd529570 0x55cedd9d0530 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4cf17fa640 0 client.28003 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4d1652e740 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cedd595840 0x55cedd953dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4d1652e740 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cedd574b60 0x55cedd953880 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4d1652e740 1 --2- 172.21.15.29:0/657701208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cedd56c980 0x55cedd958f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 >> 172.21.15.29:0/657701208 conn(0x55cedd589990 msgr2=0x55cedd93f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 shutdown_connections 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4d1652e740 1 -- 172.21.15.29:0/657701208 wait complete. 2024-09-21T03:03:03.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.891+0000 7f4d1652e740 20 client.28003 trim_cache size 0 max 16384 2024-09-21T03:03:03.894 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:03:03.894 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 39, 2024-09-21T03:03:03.894 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:03:03.894 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 90, 2024-09-21T03:03:03.894 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:03:03.894 INFO:teuthology.orchestra.run.smithi029.stdout: "a177e407": { 2024-09-21T03:03:03.894 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 90, 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.9, 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:03:03.895 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:03.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:03.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:03.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:03.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:03.896 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:03:03.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 -- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cf40b6650 msgr2=0x7f4cf40b8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cf40b6650 0x7f4cf40b8b60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009b30 tx=0x7f4d0000b040 comp rx=0 tx=0).stop 2024-09-21T03:03:03.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 -- 172.21.15.29:0/1040051549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d081052d0 msgr2=0x7f4d0819c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d081052d0 0x7f4d0819c970 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800e9c0 tx=0x7f4cf800edc0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 -- 172.21.15.29:0/1040051549 shutdown_connections 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4cf40b6650 0x7f4cf40b8b60 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d0810de50 0x7f4d081a1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d081052d0 0x7f4d0819c970 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 --2- 172.21.15.29:0/1040051549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d081048a0 0x7f4d0819c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.893+0000 7f4d0ffba640 1 -- 172.21.15.29:0/1040051549 >> 172.21.15.29:0/1040051549 conn(0x7f4d08079760 msgr2=0x7f4d0810cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.894+0000 7f4d0ffba640 1 -- 172.21.15.29:0/1040051549 shutdown_connections 2024-09-21T03:03:03.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:03.894+0000 7f4d0ffba640 1 -- 172.21.15.29:0/1040051549 wait complete. 2024-09-21T03:03:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:03 smithi066 ceph-mon[33436]: pgmap v1108: 129 pgs: 129 active+clean; 340 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 3.3 MiB/s wr, 263 op/s 2024-09-21T03:03:04.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:03 smithi190 ceph-mon[33871]: pgmap v1108: 129 pgs: 129 active+clean; 340 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 3.3 MiB/s wr, 263 op/s 2024-09-21T03:03:04.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:03 smithi029 ceph-mon[31006]: pgmap v1108: 129 pgs: 129 active+clean; 340 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 3.3 MiB/s wr, 263 op/s 2024-09-21T03:03:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd75affd640 1 -- 172.21.15.29:0/2886385246 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b385e7c0f0 con 0x55b385b641b0 2024-09-21T03:03:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd75affd640 10 client.27854 handle_command_reply: tid=0 2024-09-21T03:03:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 client.27854 shutdown 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b434d0 msgr2=0x55b385f24cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b434d0 0x55b385f24cd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd774009a20 tx=0x7fd7740092a0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 shutdown_connections 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55b385b434d0 0x55b385f24cd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b385b3b2f0 0x55b385f24790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55b385b641b0 0x55b385f98360 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 --2- 172.21.15.29:0/2886385246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b385b22d80 0x55b385f25ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd75affd640 0 client.27854 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 >> 172.21.15.29:0/2886385246 conn(0x55b385b58300 msgr2=0x55b385f025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.502+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 shutdown_connections 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd788fe4740 1 -- 172.21.15.29:0/2886385246 wait complete. 2024-09-21T03:03:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd788fe4740 20 client.27854 trim_cache size 0 max 16384 2024-09-21T03:03:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:03:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:03:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:03:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 -- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7640b6420 msgr2=0x7fd7640b8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7640b6420 0x7fd7640b8930 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd774009c60 tx=0x7fd774031000 comp rx=0 tx=0).stop 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 -- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1025a0 msgr2=0x7fd77c19c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1025a0 0x7fd77c19c060 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00e9f0 tx=0x7fd76c00eeb0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 -- 172.21.15.29:0/1853776353 shutdown_connections 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd7640b6420 0x7fd7640b8930 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd77c103fc0 0x7fd77c19d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd77c103620 0x7fd77c19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 --2- 172.21.15.29:0/1853776353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1025a0 0x7fd77c19c060 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.503+0000 7fd781f9b640 1 -- 172.21.15.29:0/1853776353 >> 172.21.15.29:0/1853776353 conn(0x7fd77c0fdee0 msgr2=0x7fd77c0ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.504+0000 7fd781f9b640 1 -- 172.21.15.29:0/1853776353 shutdown_connections 2024-09-21T03:03:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:04.504+0000 7fd781f9b640 1 -- 172.21.15.29:0/1853776353 wait complete. 2024-09-21T03:03:04.747 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.54 MB/sec execute 521 sec latency 1420.944 ms 2024-09-21T03:03:05.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.53 MB/sec execute 522 sec latency 2420.995 ms 2024-09-21T03:03:06.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:05 smithi066 ceph-mon[33436]: pgmap v1109: 129 pgs: 129 active+clean; 341 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.6 MiB/s wr, 312 op/s 2024-09-21T03:03:06.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:05 smithi029 ceph-mon[31006]: pgmap v1109: 129 pgs: 129 active+clean; 341 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.6 MiB/s wr, 312 op/s 2024-09-21T03:03:06.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:05 smithi190 ceph-mon[33871]: pgmap v1109: 129 pgs: 129 active+clean; 341 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.6 MiB/s wr, 312 op/s 2024-09-21T03:03:06.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.52 MB/sec execute 523 sec latency 3421.105 ms 2024-09-21T03:03:07.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.51 MB/sec execute 524 sec latency 4421.214 ms 2024-09-21T03:03:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:07 smithi066 ceph-mon[33436]: pgmap v1110: 129 pgs: 129 active+clean; 334 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 3.0 MiB/s wr, 266 op/s 2024-09-21T03:03:08.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:07 smithi190 ceph-mon[33871]: pgmap v1110: 129 pgs: 129 active+clean; 334 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 3.0 MiB/s wr, 266 op/s 2024-09-21T03:03:08.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:07 smithi029 ceph-mon[31006]: pgmap v1110: 129 pgs: 129 active+clean; 334 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 3.0 MiB/s wr, 266 op/s 2024-09-21T03:03:08.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.50 MB/sec execute 525 sec latency 5421.262 ms 2024-09-21T03:03:09.643 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a177e407', quiesce took 14.0 seconds. Will release after: 5.899999999999999 2024-09-21T03:03:09.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.49 MB/sec execute 526 sec latency 6421.384 ms 2024-09-21T03:03:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:09 smithi029 ceph-mon[31006]: pgmap v1111: 129 pgs: 129 active+clean; 326 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 2.5 MiB/s wr, 255 op/s 2024-09-21T03:03:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:09 smithi066 ceph-mon[33436]: pgmap v1111: 129 pgs: 129 active+clean; 326 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 2.5 MiB/s wr, 255 op/s 2024-09-21T03:03:10.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:09 smithi190 ceph-mon[33871]: pgmap v1111: 129 pgs: 129 active+clean; 326 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 2.5 MiB/s wr, 255 op/s 2024-09-21T03:03:10.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.48 MB/sec execute 527 sec latency 7421.489 ms 2024-09-21T03:03:11.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.47 MB/sec execute 528 sec latency 8421.529 ms 2024-09-21T03:03:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:11 smithi066 ceph-mon[33436]: pgmap v1112: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.0 MiB/s wr, 227 op/s 2024-09-21T03:03:12.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:11 smithi190 ceph-mon[33871]: pgmap v1112: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.0 MiB/s wr, 227 op/s 2024-09-21T03:03:12.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:11 smithi029 ceph-mon[31006]: pgmap v1112: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.0 MiB/s wr, 227 op/s 2024-09-21T03:03:12.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.46 MB/sec execute 529 sec latency 9421.669 ms 2024-09-21T03:03:13.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.45 MB/sec execute 530 sec latency 10421.781 ms 2024-09-21T03:03:14.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:13 smithi066 ceph-mon[33436]: pgmap v1113: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 192 op/s 2024-09-21T03:03:14.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:13 smithi190 ceph-mon[33871]: pgmap v1113: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 192 op/s 2024-09-21T03:03:14.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:13 smithi029 ceph-mon[31006]: pgmap v1113: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.3 MiB/s wr, 192 op/s 2024-09-21T03:03:14.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.44 MB/sec execute 531 sec latency 11421.887 ms 2024-09-21T03:03:15.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:03:15.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.43 MB/sec execute 532 sec latency 12421.944 ms 2024-09-21T03:03:15.821 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:03:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:15 smithi029 ceph-mon[31006]: pgmap v1114: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 912 KiB/s wr, 169 op/s 2024-09-21T03:03:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:15 smithi066 ceph-mon[33436]: pgmap v1114: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 912 KiB/s wr, 169 op/s 2024-09-21T03:03:16.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:15 smithi190 ceph-mon[33871]: pgmap v1114: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 912 KiB/s wr, 169 op/s 2024-09-21T03:03:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.359+0000 7f82ef1e2640 1 -- 172.21.15.29:0/3742239666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e8109df0 msgr2=0x7f82e8110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.359+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/3742239666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e8109df0 0x7f82e8110620 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009ca0 tx=0x7f82e002a8c0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 -- 172.21.15.29:0/3742239666 shutdown_connections 2024-09-21T03:03:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/3742239666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e8109df0 0x7f82e8110620 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/3742239666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82e81094a0 0x7f82e81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/3742239666 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8103b60 0x7f82e8103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 -- 172.21.15.29:0/3742239666 >> 172.21.15.29:0/3742239666 conn(0x7f82e807b850 msgr2=0x7f82e807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 -- 172.21.15.29:0/3742239666 shutdown_connections 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 -- 172.21.15.29:0/3742239666 wait complete. 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 Processor -- start 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.360+0000 7f82ef1e2640 1 -- start start 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82ef1e2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82e8103b60 0x7f82e819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82ef1e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e81094a0 0x7f82e819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82ef1e2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8109df0 0x7f82e819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82ef1e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81139b0 con 0x7f82e81094a0 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82ef1e2640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8113830 con 0x7f82e8109df0 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82ef1e2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8113b30 con 0x7f82e8103b60 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e81094a0 0x7f82e819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e81094a0 0x7f82e819a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36100/0 (socket says 172.21.15.29:36100) 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 -- 172.21.15.29:0/2678567064 learned_addr learned my addr 172.21.15.29:0/2678567064 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82ed77f640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8109df0 0x7f82e819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82ecf7e640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82e8103b60 0x7f82e819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 -- 172.21.15.29:0/2678567064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82e8103b60 msgr2=0x7f82e819a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82e8103b60 0x7f82e819a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 -- 172.21.15.29:0/2678567064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8109df0 msgr2=0x7f82e819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8109df0 0x7f82e819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 -- 172.21.15.29:0/2678567064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e81a0310 con 0x7f82e81094a0 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e7fff640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e81094a0 0x7f82e819a940 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f82d400c970 tx=0x7f82d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.361+0000 7f82e5ffb640 1 -- 172.21.15.29:0/2678567064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4007bb0 con 0x7f82e81094a0 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.362+0000 7f82e5ffb640 1 -- 172.21.15.29:0/2678567064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82d4007d50 con 0x7f82e81094a0 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.362+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e81a0600 con 0x7f82e81094a0 2024-09-21T03:03:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.362+0000 7f82e5ffb640 1 -- 172.21.15.29:0/2678567064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4004da0 con 0x7f82e81094a0 2024-09-21T03:03:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.362+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e81a6d60 con 0x7f82e81094a0 2024-09-21T03:03:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.363+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e8103f70 con 0x7f82e81094a0 2024-09-21T03:03:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.364+0000 7f82e5ffb640 1 -- 172.21.15.29:0/2678567064 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f82d4004f80 con 0x7f82e81094a0 2024-09-21T03:03:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.365+0000 7f82e5ffb640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82d80b6510 0x7f82d80b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.365+0000 7f82ecf7e640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82d80b6510 0x7f82d80b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.365+0000 7f82e5ffb640 1 -- 172.21.15.29:0/2678567064 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f82d40e7360 con 0x7f82e81094a0 2024-09-21T03:03:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.366+0000 7f82ecf7e640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82d80b6510 0x7f82d80b8a20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009b60 tx=0x7f82dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.369+0000 7f82e5ffb640 1 -- 172.21.15.29:0/2678567064 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f82d4014030 con 0x7f82e81094a0 2024-09-21T03:03:16.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.594+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82e810bc40 con 0x7f82e81094a0 2024-09-21T03:03:16.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.595+0000 7f82e5ffb640 1 -- 172.21.15.29:0/2678567064 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11971 (secure 0 0 0) 0x7f82d40ad870 con 0x7f82e81094a0 2024-09-21T03:03:16.597 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:03:16.597 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"btime":"2024-09-21T03:02:24:667358+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:02:24.667336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:03:16.597 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-09-21T03:03:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82d80b6510 msgr2=0x7f82d80b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82d80b6510 0x7f82d80b8a20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009b60 tx=0x7f82dc005f90 comp rx=0 tx=0).stop 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e81094a0 msgr2=0x7f82e819a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e81094a0 0x7f82e819a940 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f82d400c970 tx=0x7f82d400ce30 comp rx=0 tx=0).stop 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 shutdown_connections 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f82d80b6510 0x7f82d80b8a20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8109df0 0x7f82e819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e81094a0 0x7f82e819a940 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 --2- 172.21.15.29:0/2678567064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82e8103b60 0x7f82e819a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 >> 172.21.15.29:0/2678567064 conn(0x7f82e807b850 msgr2=0x7f82e81054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 shutdown_connections 2024-09-21T03:03:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:16.598+0000 7f82ef1e2640 1 -- 172.21.15.29:0/2678567064 wait complete. 2024-09-21T03:03:16.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.42 MB/sec execute 533 sec latency 13422.041 ms 2024-09-21T03:03:16.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:16 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2678567064' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:03:17.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:16 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2678567064' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:03:17.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:16 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2678567064' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:03:17.357 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id a177e407 --release --await' 2024-09-21T03:03:17.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id a177e407 --release --await 2024-09-21T03:03:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 -- 172.21.15.29:0/2705499617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341093d0 msgr2=0x7f643410b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 --2- 172.21.15.29:0/2705499617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341093d0 0x7f643410b8e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f642c009c90 tx=0x7f642c02a860 comp rx=0 tx=0).stop 2024-09-21T03:03:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 -- 172.21.15.29:0/2705499617 shutdown_connections 2024-09-21T03:03:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 --2- 172.21.15.29:0/2705499617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341093d0 0x7f643410b8e0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 --2- 172.21.15.29:0/2705499617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64341068c0 0x7f6434108ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 --2- 172.21.15.29:0/2705499617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f643406b670 0x7f6434106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 -- 172.21.15.29:0/2705499617 >> 172.21.15.29:0/2705499617 conn(0x7f64340fdee0 msgr2=0x7f6434100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 -- 172.21.15.29:0/2705499617 shutdown_connections 2024-09-21T03:03:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.612+0000 7f6438b5f640 1 -- 172.21.15.29:0/2705499617 wait complete. 2024-09-21T03:03:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6438b5f640 1 Processor -- start 2024-09-21T03:03:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6438b5f640 1 -- start start 2024-09-21T03:03:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6438b5f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f643406b670 0x7f64341039c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6438b5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341068c0 0x7f6434102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6438b5f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64341093d0 0x7f64341027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6438b5f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f643410f190 con 0x7f64341068c0 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6438b5f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f643410f010 con 0x7f64341093d0 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6438b5f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f643410f310 con 0x7f643406b670 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6431d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341068c0 0x7f6434102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6431d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341068c0 0x7f6434102280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36110/0 (socket says 172.21.15.29:36110) 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6431d9b640 1 -- 172.21.15.29:0/4098112393 learned_addr learned my addr 172.21.15.29:0/4098112393 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.613+0000 7f6432d9d640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64341093d0 0x7f64341027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f643259c640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f643406b670 0x7f64341039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6431d9b640 1 -- 172.21.15.29:0/4098112393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f643406b670 msgr2=0x7f64341039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6431d9b640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f643406b670 0x7f64341039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6431d9b640 1 -- 172.21.15.29:0/4098112393 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64341093d0 msgr2=0x7f64341027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6431d9b640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64341093d0 0x7f64341027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6431d9b640 1 -- 172.21.15.29:0/4098112393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6434102ef0 con 0x7f64341068c0 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f643259c640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f643406b670 0x7f64341039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6432d9d640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64341093d0 0x7f64341027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6431d9b640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341068c0 0x7f6434102280 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f642800e9f0 tx=0x7f642800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f64237fe640 1 -- 172.21.15.29:0/4098112393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642800cd80 con 0x7f64341068c0 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f64237fe640 1 -- 172.21.15.29:0/4098112393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64280054b0 con 0x7f64341068c0 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6438b5f640 1 -- 172.21.15.29:0/4098112393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64341a3ed0 con 0x7f64341068c0 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f64237fe640 1 -- 172.21.15.29:0/4098112393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428010430 con 0x7f64341068c0 2024-09-21T03:03:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.614+0000 7f6438b5f640 1 -- 172.21.15.29:0/4098112393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64341a42e0 con 0x7f64341068c0 2024-09-21T03:03:17.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.617+0000 7f64237fe640 1 -- 172.21.15.29:0/4098112393 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f64280046c0 con 0x7f64341068c0 2024-09-21T03:03:17.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643fac9740 1 Processor -- start 2024-09-21T03:03:17.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643fac9740 1 -- start start 2024-09-21T03:03:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643fac9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae23e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643fac9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae244c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643fac9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 0x56050ae25010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643fac9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56050ad966f0 con 0x56050aa7c1b0 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643fac9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56050ad85b20 con 0x56050aa532f0 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643fac9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56050ad89930 con 0x56050aa5b4d0 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f6432d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae23e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f6432d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae23e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36114/0 (socket says 172.21.15.29:36114) 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f6432d9d640 1 -- 172.21.15.29:0/3363822463 learned_addr learned my addr 172.21.15.29:0/3363822463 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f643259c640 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.618+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 0x56050ae25010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6432d9d640 1 -- 172.21.15.29:0/3363822463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 msgr2=0x56050ae25010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6432d9d640 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 0x56050ae25010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6432d9d640 1 -- 172.21.15.29:0/3363822463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 msgr2=0x56050ae244c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6432d9d640 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae244c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f64237fe640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f641c0b6590 0x7f641c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6432d9d640 1 -- 172.21.15.29:0/3363822463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050ad4b530 con 0x56050aa7c1b0 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6432d9d640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f641c0b6590 0x7f641c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f64237fe640 1 -- 172.21.15.29:0/4098112393 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f64280e7a20 con 0x7f64341068c0 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6432d9d640 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae23e50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f642c009c90 tx=0x7f642c031830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1964df83f58983db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6420ff9640 1 -- 172.21.15.29:0/3363822463 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c04d070 con 0x56050aa7c1b0 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6420ff9640 1 -- 172.21.15.29:0/3363822463 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f642c036070 con 0x56050aa7c1b0 2024-09-21T03:03:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.619+0000 7f6420ff9640 1 -- 172.21.15.29:0/3363822463 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c003f50 con 0x56050aa7c1b0 2024-09-21T03:03:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 -- 172.21.15.29:0/3363822463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 msgr2=0x56050ae23e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae23e50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f642c009c90 tx=0x7f642c031830 comp rx=0 tx=0).stop 2024-09-21T03:03:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f6432d9d640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f641c0b6590 0x7f641c0b8aa0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f642c04a000 tx=0x7f642c04b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 -- 172.21.15.29:0/3363822463 shutdown_connections 2024-09-21T03:03:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 0x56050ae25010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 --2- 172.21.15.29:0/3363822463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae23e50 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 -- 172.21.15.29:0/3363822463 >> 172.21.15.29:0/3363822463 conn(0x56050aa70300 msgr2=0x56050ae1f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 -- 172.21.15.29:0/3363822463 shutdown_connections 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.620+0000 7f643fac9740 1 -- 172.21.15.29:0/3363822463 wait complete. 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 1 -- start start 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 1 Processor -- start 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae2d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 0x56050ae2dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae2e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56050acbdab0 con 0x56050aa7c1b0 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56050ad2b730 con 0x56050aa532f0 2024-09-21T03:03:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56050ae25b30 con 0x56050aa5b4d0 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643fac9740 10 client.0 _ll_register_callbacks cb 0x56050ad7f6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f6432d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae2e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f6432d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae2e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36126/0 (socket says 172.21.15.29:36126) 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f6432d9d640 1 -- 172.21.15.29:0/3495943194 learned_addr learned my addr 172.21.15.29:0/3495943194 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.621+0000 7f643259c640 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae2d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 0x56050ae2dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f643259c640 1 -- 172.21.15.29:0/3495943194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 msgr2=0x56050ae2dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f643259c640 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 0x56050ae2dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f643259c640 1 -- 172.21.15.29:0/3495943194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 msgr2=0x56050ae2e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f643259c640 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae2e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f643259c640 1 -- 172.21.15.29:0/3495943194 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56050aae7790 con 0x56050aa532f0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f6432d9d640 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae2e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f643259c640 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae2d5c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6424009c00 tx=0x7f64240039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfbdf9fa23d843c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f6419ffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f6419ffb640 1 -- 172.21.15.29:0/3495943194 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424012860 con 0x56050aa532f0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f643fac9740 1 -- 172.21.15.29:0/3495943194 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f642c036070 con 0x56050aa532f0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.622+0000 7f6419ffb640 1 -- 172.21.15.29:0/3495943194 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f6424012a00 con 0x56050aa532f0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f6419ffb640 1 -- 172.21.15.29:0/3495943194 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6424024050 con 0x56050aa532f0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f643fac9740 10 client.27872 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f643fac9740 1 -- 172.21.15.29:0/3495943194 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56050ab60620 con 0x56050aa532f0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f6419ffb640 1 -- 172.21.15.29:0/3495943194 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f642c036070 con 0x56050aa532f0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f6419ffb640 1 -- 172.21.15.29:0/3495943194 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f642401a030 con 0x56050aa532f0 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f643fac9740 10 client.27872 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f643fac9740 10 client.27872 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f643fac9740 20 client.27872 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f643fac9740 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56050aa3ad80 0x56050ae3add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f643fac9740 4 client.27872 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.623+0000 7f643fac9740 1 -- 172.21.15.29:0/3495943194 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56050ad98640 con 0x56050aa3ad80 2024-09-21T03:03:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.624+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56050aa3ad80 0x56050ae3add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.624+0000 7f6419ffb640 10 client.27872 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.624+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56050aa3ad80 0x56050ae3add0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=631439eb7d7dee73 server_cookie=4c203fd901ffd469 in_seq=0 out_seq=0 2024-09-21T03:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f6419ffb640 1 -- 172.21.15.29:0/3495943194 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56050ad98640 con 0x56050aa3ad80 2024-09-21T03:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f6419ffb640 10 client.27872 handle_command_reply: tid=0 2024-09-21T03:03:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f643fac9740 1 client.27872 shutdown 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f643fac9740 1 -- 172.21.15.29:0/3495943194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 msgr2=0x56050ae2d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f643fac9740 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae2d5c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6424009c00 tx=0x7f64240039c0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f643fac9740 1 -- 172.21.15.29:0/3495943194 shutdown_connections 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f643fac9740 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56050aa3ad80 0x56050ae3add0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f643fac9740 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa7c1b0 0x56050ae2e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f6419ffb640 0 client.27872 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.625+0000 7f643fac9740 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa5b4d0 0x56050ae2dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.626+0000 7f643fac9740 1 --2- 172.21.15.29:0/3495943194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa532f0 0x56050ae2d5c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.626+0000 7f643fac9740 1 -- 172.21.15.29:0/3495943194 >> 172.21.15.29:0/3495943194 conn(0x56050aa70300 msgr2=0x56050ae1dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.626+0000 7f643fac9740 1 -- 172.21.15.29:0/3495943194 shutdown_connections 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.626+0000 7f643fac9740 1 -- 172.21.15.29:0/3495943194 wait complete. 2024-09-21T03:03:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.626+0000 7f643fac9740 20 client.27872 trim_cache size 0 max 16384 2024-09-21T03:03:17.632 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643fac9740 1 Processor -- start 2024-09-21T03:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643fac9740 1 -- start start 2024-09-21T03:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643fac9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa532f0 0x56050ae2ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643fac9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa5b4d0 0x56050ae1f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643fac9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa7c1b0 0x56050ae1e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643fac9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f64341ae650 con 0x56050aa532f0 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643fac9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f64341b0190 con 0x56050aa5b4d0 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643fac9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64341aed20 con 0x56050aa7c1b0 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa532f0 0x56050ae2ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa532f0 0x56050ae2ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36142/0 (socket says 172.21.15.29:36142) 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643259c640 1 -- 172.21.15.29:0/3772732675 learned_addr learned my addr 172.21.15.29:0/3772732675 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa5b4d0 0x56050ae1f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f6432d9d640 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa7c1b0 0x56050ae1e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643259c640 1 -- 172.21.15.29:0/3772732675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa7c1b0 msgr2=0x56050ae1e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643259c640 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa7c1b0 0x56050ae1e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643259c640 1 -- 172.21.15.29:0/3772732675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa5b4d0 msgr2=0x56050ae1f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643259c640 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa5b4d0 0x56050ae1f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f643259c640 1 -- 172.21.15.29:0/3772732675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050ae2b790 con 0x56050aa532f0 2024-09-21T03:03:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.642+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa5b4d0 0x56050ae1f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.643+0000 7f643259c640 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa532f0 0x56050ae2ed50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6424022040 tx=0x7f6424005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f40db2558e43d1c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.643+0000 7f641bfff640 1 -- 172.21.15.29:0/3772732675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642401b790 con 0x56050aa532f0 2024-09-21T03:03:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.643+0000 7f641bfff640 1 -- 172.21.15.29:0/3772732675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f642401bdc0 con 0x56050aa532f0 2024-09-21T03:03:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.643+0000 7f641bfff640 1 -- 172.21.15.29:0/3772732675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424012660 con 0x56050aa532f0 2024-09-21T03:03:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 -- 172.21.15.29:0/3772732675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa532f0 msgr2=0x56050ae2ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa532f0 0x56050ae2ed50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6424022040 tx=0x7f6424005e30 comp rx=0 tx=0).stop 2024-09-21T03:03:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 -- 172.21.15.29:0/3772732675 shutdown_connections 2024-09-21T03:03:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa7c1b0 0x56050ae1e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa5b4d0 0x56050ae1f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 --2- 172.21.15.29:0/3772732675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa532f0 0x56050ae2ed50 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 -- 172.21.15.29:0/3772732675 >> 172.21.15.29:0/3772732675 conn(0x56050aa70300 msgr2=0x56050ae1a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 -- 172.21.15.29:0/3772732675 shutdown_connections 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 -- 172.21.15.29:0/3772732675 wait complete. 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 -- start start 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.644+0000 7f643fac9740 1 Processor -- start 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643fac9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa532f0 0x56050ae35130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643fac9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa5b4d0 0x56050ae33df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643fac9740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa7c1b0 0x56050ae34330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643fac9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f64341afac0 con 0x56050aa5b4d0 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643fac9740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f64341af3f0 con 0x56050aa7c1b0 2024-09-21T03:03:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643fac9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64341ac630 con 0x56050aa532f0 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643fac9740 10 client.0 _ll_register_callbacks cb 0x7f6424004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6431d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa5b4d0 0x56050ae33df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6431d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa5b4d0 0x56050ae33df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36158/0 (socket says 172.21.15.29:36158) 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6431d9b640 1 -- 172.21.15.29:0/3785723123 learned_addr learned my addr 172.21.15.29:0/3785723123 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643259c640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa532f0 0x56050ae35130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6432d9d640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa7c1b0 0x56050ae34330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6431d9b640 1 -- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa532f0 msgr2=0x56050ae35130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa532f0 0x56050ae35130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6431d9b640 1 -- 172.21.15.29:0/3785723123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa7c1b0 msgr2=0x56050ae34330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa7c1b0 0x56050ae34330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6431d9b640 1 -- 172.21.15.29:0/3785723123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56050ae34ac0 con 0x56050aa5b4d0 2024-09-21T03:03:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f6432d9d640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa7c1b0 0x56050ae34330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.645+0000 7f643259c640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa532f0 0x56050ae35130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f6431d9b640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa5b4d0 0x56050ae33df0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f64280e51a0 tx=0x7f64280ad6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43b8fb7d3c7a75 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f6419ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f6419ffb640 1 -- 172.21.15.29:0/3785723123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64280aed60 con 0x56050aa5b4d0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f6419ffb640 1 -- 172.21.15.29:0/3785723123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f6428005000 con 0x56050aa5b4d0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 1 -- 172.21.15.29:0/3785723123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56050ae25490 con 0x56050aa5b4d0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f6419ffb640 1 -- 172.21.15.29:0/3785723123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f64280adf30 con 0x56050aa5b4d0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 10 client.18615 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 1 -- 172.21.15.29:0/3785723123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56050aea57f0 con 0x56050aa5b4d0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f6419ffb640 1 -- 172.21.15.29:0/3785723123 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56050ae25490 con 0x56050aa5b4d0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f6419ffb640 1 -- 172.21.15.29:0/3785723123 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f64280a9180 con 0x56050aa5b4d0 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 10 client.18615 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 10 client.18615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 20 client.18615 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56050aa101c0 0x56050aea81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 4 client.18615 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:03:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.646+0000 7f643fac9740 1 -- 172.21.15.29:0/3785723123 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56050ae1e440 con 0x56050aa101c0 2024-09-21T03:03:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.647+0000 7f643259c640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56050aa101c0 0x56050aea81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.647+0000 7f643259c640 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56050aa101c0 0x56050aea81e0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d71948521d6a58a server_cookie=714fb16c08d549b1 in_seq=0 out_seq=0 2024-09-21T03:03:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.647+0000 7f6419ffb640 10 client.18615 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.694+0000 7f6419ffb640 1 -- 172.21.15.29:0/3785723123 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56050ae1e440 con 0x56050aa101c0 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.694+0000 7f6419ffb640 10 client.18615 handle_command_reply: tid=0 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 client.18615 shutdown 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 -- 172.21.15.29:0/3785723123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa5b4d0 msgr2=0x56050ae33df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa5b4d0 0x56050ae33df0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f64280e51a0 tx=0x7f64280ad6c0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 -- 172.21.15.29:0/3785723123 shutdown_connections 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56050aa7c1b0 0x56050ae34330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56050aa101c0 0x56050aea81e0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050aa5b4d0 0x56050ae33df0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 --2- 172.21.15.29:0/3785723123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56050aa532f0 0x56050ae35130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f6419ffb640 0 client.18615 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 -- 172.21.15.29:0/3785723123 >> 172.21.15.29:0/3785723123 conn(0x56050aa70300 msgr2=0x56050ae1a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 -- 172.21.15.29:0/3785723123 shutdown_connections 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 1 -- 172.21.15.29:0/3785723123 wait complete. 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f643fac9740 20 client.18615 trim_cache size 0 max 16384 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:03:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:03:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.695+0000 7f6438b5f640 1 -- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f641c0b6590 msgr2=0x7f641c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f641c0b6590 0x7f641c0b8aa0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f642c04a000 tx=0x7f642c04b000 comp rx=0 tx=0).stop 2024-09-21T03:03:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 -- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341068c0 msgr2=0x7f6434102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341068c0 0x7f6434102280 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f642800e9f0 tx=0x7f642800edf0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 -- 172.21.15.29:0/4098112393 shutdown_connections 2024-09-21T03:03:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f641c0b6590 0x7f641c0b8aa0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64341093d0 0x7f64341027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64341068c0 0x7f6434102280 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 --2- 172.21.15.29:0/4098112393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f643406b670 0x7f64341039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 -- 172.21.15.29:0/4098112393 >> 172.21.15.29:0/4098112393 conn(0x7f64340fdee0 msgr2=0x7f6434104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 -- 172.21.15.29:0/4098112393 shutdown_connections 2024-09-21T03:03:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:17.696+0000 7f6438b5f640 1 -- 172.21.15.29:0/4098112393 wait complete. 2024-09-21T03:03:17.748 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66463 5.41 MB/sec execute 534 sec latency 14422.140 ms 2024-09-21T03:03:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:17 smithi066 ceph-mon[33436]: pgmap v1115: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 218 KiB/s wr, 99 op/s 2024-09-21T03:03:18.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:17 smithi029 ceph-mon[31006]: pgmap v1115: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 218 KiB/s wr, 99 op/s 2024-09-21T03:03:18.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:17 smithi190 ceph-mon[33871]: pgmap v1115: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 218 KiB/s wr, 99 op/s 2024-09-21T03:03:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 -- 172.21.15.29:0/1574198665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe4103b80 msgr2=0x7f7fe4103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/1574198665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe4103b80 0x7f7fe4103f90 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009ca0 tx=0x7f7fe002a8c0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 -- 172.21.15.29:0/1574198665 shutdown_connections 2024-09-21T03:03:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/1574198665 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fe4109e10 0x7f7fe4110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/1574198665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe41094c0 0x7f7fe41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/1574198665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe4103b80 0x7f7fe4103f90 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 -- 172.21.15.29:0/1574198665 >> 172.21.15.29:0/1574198665 conn(0x7f7fe40ff610 msgr2=0x7f7fe4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 -- 172.21.15.29:0/1574198665 shutdown_connections 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.155+0000 7f7feb8ca640 1 -- 172.21.15.29:0/1574198665 wait complete. 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7feb8ca640 1 Processor -- start 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7feb8ca640 1 -- start start 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7feb8ca640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fe4103b80 0x7f7fe419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7feb8ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe41094c0 0x7f7fe419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7feb8ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe4109e10 0x7f7fe419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7feb8ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41139d0 con 0x7f7fe41094c0 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7feb8ca640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe4113850 con 0x7f7fe4103b80 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7feb8ca640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe4113b50 con 0x7f7fe4109e10 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7fe8e65640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe41094c0 0x7f7fe419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7fe8e65640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe41094c0 0x7f7fe419a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36180/0 (socket says 172.21.15.29:36180) 2024-09-21T03:03:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fe8e65640 1 -- 172.21.15.29:0/4102258466 learned_addr learned my addr 172.21.15.29:0/4102258466 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.156+0000 7f7fe9666640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fe4103b80 0x7f7fe419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fe8e65640 1 -- 172.21.15.29:0/4102258466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe4109e10 msgr2=0x7f7fe419fc60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe4109e10 0x7f7fe419fc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fe8e65640 1 -- 172.21.15.29:0/4102258466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fe4103b80 msgr2=0x7f7fe419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fe4103b80 0x7f7fe419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fe8e65640 1 -- 172.21.15.29:0/4102258466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe41a02d0 con 0x7f7fe41094c0 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe41094c0 0x7f7fe419a9c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7fd400c8c0 tx=0x7f7fd400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fda7fc640 1 -- 172.21.15.29:0/4102258466 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4007bb0 con 0x7f7fe41094c0 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fda7fc640 1 -- 172.21.15.29:0/4102258466 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd4007d50 con 0x7f7fe41094c0 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7fda7fc640 1 -- 172.21.15.29:0/4102258466 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40186a0 con 0x7f7fe41094c0 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7feb8ca640 1 -- 172.21.15.29:0/4102258466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe41a05c0 con 0x7f7fe41094c0 2024-09-21T03:03:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.157+0000 7f7feb8ca640 1 -- 172.21.15.29:0/4102258466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe41a6de0 con 0x7f7fe41094c0 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7ff1e3e740 1 Processor -- start 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7ff1e3e740 1 -- start start 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7fda7fc640 1 -- 172.21.15.29:0/4102258466 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f7fd400fa00 con 0x7f7fe41094c0 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e4118520 0x5594e44be510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44bea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44bef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5594e43a9dc0 con 0x5594e40f7840 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5594e44361f0 con 0x5594e40ef660 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5594e4425620 con 0x5594e4118520 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.160+0000 7f7fe8e65640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44bef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe8e65640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44bef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36190/0 (socket says 172.21.15.29:36190) 2024-09-21T03:03:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe9666640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44bea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe8e65640 1 -- 172.21.15.29:0/4288791703 learned_addr learned my addr 172.21.15.29:0/4288791703 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e4118520 0x5594e44be510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe8e65640 1 -- 172.21.15.29:0/4288791703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e4118520 msgr2=0x5594e44be510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e4118520 0x5594e44be510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe8e65640 1 -- 172.21.15.29:0/4288791703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 msgr2=0x5594e44bea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44bea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe8e65640 1 -- 172.21.15.29:0/4288791703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594e43edfd0 con 0x5594e40f7840 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe9666640 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44bea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e4118520 0x5594e44be510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fda7fc640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fd00b6540 0x7f7fd00b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44bef90 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40edc10 tx=0x7f7fd40edd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f914d328281694d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fd00b6540 0x7f7fd00b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fda7fc640 1 -- 172.21.15.29:0/4102258466 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f7fd40e76f0 con 0x7f7fe41094c0 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.161+0000 7f7fcf7fe640 1 -- 172.21.15.29:0/4288791703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40a9b50 con 0x5594e40f7840 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7fcf7fe640 1 -- 172.21.15.29:0/4288791703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd40ad010 con 0x5594e40f7840 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7fcf7fe640 1 -- 172.21.15.29:0/4288791703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40acc70 con 0x5594e40f7840 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fd00b6540 0x7f7fd00b8a50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009a40 tx=0x7f7fdc005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/4288791703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 msgr2=0x5594e44bef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44bef90 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40edc10 tx=0x7f7fd40edd00 comp rx=0 tx=0).stop 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/4288791703 shutdown_connections 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44bef90 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44bea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/4288791703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e4118520 0x5594e44be510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.162+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/4288791703 >> 172.21.15.29:0/4288791703 conn(0x5594e410c910 msgr2=0x5594e44bc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.163+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/4288791703 shutdown_connections 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.163+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/4288791703 wait complete. 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.163+0000 7f7ff1e3e740 1 -- start start 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.163+0000 7f7ff1e3e740 1 Processor -- start 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ca100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e4118520 0x5594e44d5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5594e43c0300 con 0x5594e4118520 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5594e4429430 con 0x5594e40ef660 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5594e4286bb0 con 0x5594e40f7840 2024-09-21T03:03:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7ff1e3e740 10 client.0 _ll_register_callbacks cb 0x5594e43da120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e4118520 0x5594e44d5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e4118520 0x5594e44d5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36202/0 (socket says 172.21.15.29:36202) 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 -- 172.21.15.29:0/1974110314 learned_addr learned my addr 172.21.15.29:0/1974110314 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9666640 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 -- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 msgr2=0x5594e44ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 -- 172.21.15.29:0/1974110314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 msgr2=0x5594e44c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 -- 172.21.15.29:0/1974110314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594e43af780 con 0x5594e4118520 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e4118520 0x5594e44d5e60 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0116d0 tx=0x7f7fdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8e0198ebe02d5b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.164+0000 7f7fccff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7fccff9640 1 -- 172.21.15.29:0/1974110314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc022020 con 0x5594e4118520 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7fccff9640 1 -- 172.21.15.29:0/1974110314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f7fdc0108d0 con 0x5594e4118520 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/1974110314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd40ad010 con 0x5594e4118520 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7fccff9640 1 -- 172.21.15.29:0/1974110314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7fdc02d050 con 0x5594e4118520 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 10 client.18627 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/1974110314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5594e4183b00 con 0x5594e4118520 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7fccff9640 1 -- 172.21.15.29:0/1974110314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd40ad010 con 0x5594e4118520 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7fccff9640 1 -- 172.21.15.29:0/1974110314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f7fdc00f040 con 0x5594e4118520 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 10 client.18627 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 10 client.18627 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:03:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 20 client.18627 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5594e40d6ea0 0x5594e44dc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 4 client.18627 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.165+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/1974110314 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594e4438140 con 0x5594e40d6ea0 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.166+0000 7f7fe9666640 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5594e40d6ea0 0x5594e44dc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.166+0000 7f7fccff9640 10 client.18627 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.166+0000 7f7fe9666640 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5594e40d6ea0 0x5594e44dc000 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ffdcd217f56c8d5 server_cookie=335f82af397c4d3a in_seq=0 out_seq=0 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7fccff9640 1 -- 172.21.15.29:0/1974110314 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5594e4438140 con 0x5594e40d6ea0 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7fccff9640 10 client.18627 handle_command_reply: tid=0 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 client.18627 shutdown 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/1974110314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e4118520 msgr2=0x5594e44d5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e4118520 0x5594e44d5e60 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0116d0 tx=0x7f7fdc0046c0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/1974110314 shutdown_connections 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5594e40d6ea0 0x5594e44dc000 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e4118520 0x5594e44d5e60 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7fccff9640 0 client.18627 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/1974110314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/1974110314 >> 172.21.15.29:0/1974110314 conn(0x5594e410c910 msgr2=0x5594e44c16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/1974110314 shutdown_connections 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.167+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/1974110314 wait complete. 2024-09-21T03:03:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.168+0000 7f7ff1e3e740 20 client.18627 trim_cache size 0 max 16384 2024-09-21T03:03:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7ff1e3e740 1 Processor -- start 2024-09-21T03:03:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7ff1e3e740 1 -- start start 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40d6ea0 0x5594e44d5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40ef660 0x5594e44cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41ae7b0 con 0x5594e40f7840 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41b02f0 con 0x5594e40d6ea0 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41aee80 con 0x5594e40ef660 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7fe9e67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7fe9e67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44c6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36212/0 (socket says 172.21.15.29:36212) 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7fe9e67640 1 -- 172.21.15.29:0/19279159 learned_addr learned my addr 172.21.15.29:0/19279159 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.185+0000 7f7fe9666640 1 --2- 172.21.15.29:0/19279159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40d6ea0 0x5594e44d5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fe9e67640 1 -- 172.21.15.29:0/19279159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40ef660 msgr2=0x5594e44cb7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T03:03:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/19279159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40ef660 0x5594e44cb7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fe9e67640 1 -- 172.21.15.29:0/19279159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40d6ea0 msgr2=0x5594e44d5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/19279159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40d6ea0 0x5594e44d5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fe9e67640 1 -- 172.21.15.29:0/19279159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594e41fa960 con 0x5594e40f7840 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/19279159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44c6db0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc01ed30 tx=0x7f7fdc009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413a25221bd06b55 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fceffd640 1 -- 172.21.15.29:0/19279159 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc035020 con 0x5594e40f7840 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fceffd640 1 -- 172.21.15.29:0/19279159 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fdc017030 con 0x5594e40f7840 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.186+0000 7f7fceffd640 1 -- 172.21.15.29:0/19279159 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc022070 con 0x5594e40f7840 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/19279159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 msgr2=0x5594e44c6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/19279159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44c6db0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc01ed30 tx=0x7f7fdc009700 comp rx=0 tx=0).stop 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/19279159 shutdown_connections 2024-09-21T03:03:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/19279159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40f7840 0x5594e44c6db0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/19279159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40ef660 0x5594e44cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/19279159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40d6ea0 0x5594e44d5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/19279159 >> 172.21.15.29:0/19279159 conn(0x5594e410c910 msgr2=0x5594e44cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/19279159 shutdown_connections 2024-09-21T03:03:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/19279159 wait complete. 2024-09-21T03:03:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 -- start start 2024-09-21T03:03:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.187+0000 7f7ff1e3e740 1 Processor -- start 2024-09-21T03:03:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40d6ea0 0x5594e44d22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7ff1e3e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41afc20 con 0x5594e40d6ea0 2024-09-21T03:03:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41af550 con 0x5594e40ef660 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9666640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40d6ea0 0x5594e44d22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7ff1e3e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41ac790 con 0x5594e40f7840 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7ff1e3e740 10 client.0 _ll_register_callbacks cb 0x7f7fdc009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9666640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40d6ea0 0x5594e44d22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36222/0 (socket says 172.21.15.29:36222) 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9666640 1 -- 172.21.15.29:0/2045236779 learned_addr learned my addr 172.21.15.29:0/2045236779 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9e67640 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9666640 1 -- 172.21.15.29:0/2045236779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 msgr2=0x5594e44ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9666640 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9666640 1 -- 172.21.15.29:0/2045236779 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 msgr2=0x5594e44ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:18.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9666640 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.188+0000 7f7fe9666640 1 -- 172.21.15.29:0/2045236779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594e44cf250 con 0x5594e40d6ea0 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7fe9666640 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40d6ea0 0x5594e44d22c0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009620 tx=0x7f7fe0031b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41f33f4689e9ca9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7fccff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7fccff9640 1 -- 172.21.15.29:0/2045236779 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0003b70 con 0x5594e40d6ea0 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7fccff9640 1 -- 172.21.15.29:0/2045236779 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f7fe0003d10 con 0x5594e40d6ea0 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/2045236779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fdc017030 con 0x5594e40d6ea0 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7fccff9640 1 -- 172.21.15.29:0/2045236779 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7fe0060050 con 0x5594e40d6ea0 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7ff1e3e740 10 client.18633 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/2045236779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5594e4547690 con 0x5594e40d6ea0 2024-09-21T03:03:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.189+0000 7f7fccff9640 1 -- 172.21.15.29:0/2045236779 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fdc017030 con 0x5594e40d6ea0 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.190+0000 7f7fccff9640 1 -- 172.21.15.29:0/2045236779 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f7fe0049040 con 0x5594e40d6ea0 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.190+0000 7f7ff1e3e740 10 client.18633 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.190+0000 7f7ff1e3e740 10 client.18633 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.190+0000 7f7ff1e3e740 20 client.18633 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.190+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5594e4118520 0x5594e4549e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.190+0000 7f7ff1e3e740 4 client.18633 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a177e407", "release": true, "await": true}] 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.190+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/2045236779 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a177e407", "release": true, "await": true}) -- 0x5594e4433bf0 con 0x5594e4118520 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.191+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5594e4118520 0x5594e4549e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.191+0000 7f7fe8e65640 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5594e4118520 0x5594e4549e80 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52d8c436df12387a server_cookie=7ac13789db47720d in_seq=0 out_seq=0 2024-09-21T03:03:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.191+0000 7f7fccff9640 10 client.18633 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:18.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7fccff9640 1 -- 172.21.15.29:0/2045236779 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5594e4433bf0 con 0x5594e4118520 2024-09-21T03:03:18.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7fccff9640 10 client.18633 handle_command_reply: tid=0 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 client.18633 shutdown 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/2045236779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40d6ea0 msgr2=0x5594e44d22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40d6ea0 0x5594e44d22c0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009620 tx=0x7f7fe0031b90 comp rx=0 tx=0).stop 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/2045236779 shutdown_connections 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594e40f7840 0x5594e44ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5594e40ef660 0x5594e44ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5594e4118520 0x5594e4549e80 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 --2- 172.21.15.29:0/2045236779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594e40d6ea0 0x5594e44d22c0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/2045236779 >> 172.21.15.29:0/2045236779 conn(0x5594e410c910 msgr2=0x5594e44cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.250+0000 7f7fccff9640 0 client.18633 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.251+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/2045236779 shutdown_connections 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.251+0000 7f7ff1e3e740 1 -- 172.21.15.29:0/2045236779 wait complete. 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.251+0000 7f7ff1e3e740 20 client.18633 trim_cache size 0 max 16384 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 39, 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 92, 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "a177e407": { 2024-09-21T03:03:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 92, 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.9, 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:03:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:03:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:03:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:03:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:03:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:03:18.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 -- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fd00b6540 msgr2=0x7f7fd00b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:18.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fd00b6540 0x7f7fd00b8a50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009a40 tx=0x7f7fdc005e50 comp rx=0 tx=0).stop 2024-09-21T03:03:18.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 -- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe41094c0 msgr2=0x7f7fe419a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:18.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe41094c0 0x7f7fe419a9c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7fd400c8c0 tx=0x7f7fd400cd80 comp rx=0 tx=0).stop 2024-09-21T03:03:18.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 -- 172.21.15.29:0/4102258466 shutdown_connections 2024-09-21T03:03:18.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7fd00b6540 0x7f7fd00b8a50 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe4109e10 0x7f7fe419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe41094c0 0x7f7fe419a9c0 secure :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7fd400c8c0 tx=0x7f7fd400cd80 comp rx=0 tx=0).stop 2024-09-21T03:03:18.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 --2- 172.21.15.29:0/4102258466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fe4103b80 0x7f7fe419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:18.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 -- 172.21.15.29:0/4102258466 >> 172.21.15.29:0/4102258466 conn(0x7f7fe40ff610 msgr2=0x7f7fe4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:18.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 -- 172.21.15.29:0/4102258466 shutdown_connections 2024-09-21T03:03:18.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:03:18.252+0000 7f7feb8ca640 1 -- 172.21.15.29:0/4102258466 wait complete. 2024-09-21T03:03:18.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66541 5.41 MB/sec execute 535 sec latency 14871.079 ms 2024-09-21T03:03:19.122 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a177e407', total seconds elapsed: 23.4 2024-09-21T03:03:19.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66681 5.41 MB/sec execute 536 sec latency 97.499 ms 2024-09-21T03:03:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:19 smithi066 ceph-mon[33436]: pgmap v1116: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 78 KiB/s wr, 68 op/s 2024-09-21T03:03:20.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:19 smithi190 ceph-mon[33871]: pgmap v1116: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 78 KiB/s wr, 68 op/s 2024-09-21T03:03:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:19 smithi029 ceph-mon[31006]: pgmap v1116: 129 pgs: 129 active+clean; 323 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 78 KiB/s wr, 68 op/s 2024-09-21T03:03:20.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66827 5.40 MB/sec execute 537 sec latency 83.848 ms 2024-09-21T03:03:21.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 66979 5.40 MB/sec execute 538 sec latency 91.122 ms 2024-09-21T03:03:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:21 smithi066 ceph-mon[33436]: pgmap v1117: 129 pgs: 129 active+clean; 324 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 286 KiB/s wr, 51 op/s 2024-09-21T03:03:22.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:21 smithi190 ceph-mon[33871]: pgmap v1117: 129 pgs: 129 active+clean; 324 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 286 KiB/s wr, 51 op/s 2024-09-21T03:03:22.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:21 smithi029 ceph-mon[31006]: pgmap v1117: 129 pgs: 129 active+clean; 324 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 286 KiB/s wr, 51 op/s 2024-09-21T03:03:22.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 67145 5.41 MB/sec execute 539 sec latency 85.451 ms 2024-09-21T03:03:23.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 67312 5.41 MB/sec execute 540 sec latency 122.711 ms 2024-09-21T03:03:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:23 smithi066 ceph-mon[33436]: pgmap v1118: 129 pgs: 129 active+clean; 325 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 391 KiB/s wr, 40 op/s 2024-09-21T03:03:24.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:23 smithi190 ceph-mon[33871]: pgmap v1118: 129 pgs: 129 active+clean; 325 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 391 KiB/s wr, 40 op/s 2024-09-21T03:03:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:23 smithi029 ceph-mon[31006]: pgmap v1118: 129 pgs: 129 active+clean; 325 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 391 KiB/s wr, 40 op/s 2024-09-21T03:03:24.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 67453 5.41 MB/sec execute 541 sec latency 102.283 ms 2024-09-21T03:03:25.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 67599 5.42 MB/sec execute 542 sec latency 89.445 ms 2024-09-21T03:03:26.011 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:25 smithi066 ceph-mon[33436]: pgmap v1119: 129 pgs: 129 active+clean; 327 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 973 KiB/s wr, 93 op/s 2024-09-21T03:03:26.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:25 smithi190 ceph-mon[33871]: pgmap v1119: 129 pgs: 129 active+clean; 327 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 973 KiB/s wr, 93 op/s 2024-09-21T03:03:26.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:25 smithi029 ceph-mon[31006]: pgmap v1119: 129 pgs: 129 active+clean; 327 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 973 KiB/s wr, 93 op/s 2024-09-21T03:03:26.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 67761 5.43 MB/sec execute 543 sec latency 98.383 ms 2024-09-21T03:03:27.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 67940 5.44 MB/sec execute 544 sec latency 94.384 ms 2024-09-21T03:03:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:27 smithi066 ceph-mon[33436]: pgmap v1120: 129 pgs: 129 active+clean; 329 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-09-21T03:03:28.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:27 smithi190 ceph-mon[33871]: pgmap v1120: 129 pgs: 129 active+clean; 329 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-09-21T03:03:28.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:27 smithi029 ceph-mon[31006]: pgmap v1120: 129 pgs: 129 active+clean; 329 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-09-21T03:03:28.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 68135 5.45 MB/sec execute 545 sec latency 121.866 ms 2024-09-21T03:03:29.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 68306 5.46 MB/sec execute 546 sec latency 126.338 ms 2024-09-21T03:03:30.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:29 smithi066 ceph-mon[33436]: pgmap v1121: 129 pgs: 129 active+clean; 332 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 2.1 MiB/s wr, 158 op/s 2024-09-21T03:03:30.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:29 smithi190 ceph-mon[33871]: pgmap v1121: 129 pgs: 129 active+clean; 332 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 2.1 MiB/s wr, 158 op/s 2024-09-21T03:03:30.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:29 smithi029 ceph-mon[31006]: pgmap v1121: 129 pgs: 129 active+clean; 332 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 2.1 MiB/s wr, 158 op/s 2024-09-21T03:03:30.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 68402 5.46 MB/sec execute 547 sec latency 80.670 ms 2024-09-21T03:03:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.829+0000 7fec8f806640 1 -- 172.21.15.29:0/1378169326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec881036c0 msgr2=0x7fec88103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.829+0000 7fec8f806640 1 --2- 172.21.15.29:0/1378169326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec881036c0 0x7fec88103b30 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fec84009900 tx=0x7fec8402a410 comp rx=0 tx=0).stop 2024-09-21T03:03:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.829+0000 7fec8f806640 1 -- 172.21.15.29:0/1378169326 shutdown_connections 2024-09-21T03:03:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.829+0000 7fec8f806640 1 --2- 172.21.15.29:0/1378169326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec881050e0 0x7fec8810ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.829+0000 7fec8f806640 1 --2- 172.21.15.29:0/1378169326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec88104740 0x7fec88104b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.829+0000 7fec8f806640 1 --2- 172.21.15.29:0/1378169326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec881036c0 0x7fec88103b30 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.829+0000 7fec8f806640 1 -- 172.21.15.29:0/1378169326 >> 172.21.15.29:0/1378169326 conn(0x7fec880ff020 msgr2=0x7fec881014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.830+0000 7fec8f806640 1 -- 172.21.15.29:0/1378169326 shutdown_connections 2024-09-21T03:03:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.830+0000 7fec8f806640 1 -- 172.21.15.29:0/1378169326 wait complete. 2024-09-21T03:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.830+0000 7fec8f806640 1 Processor -- start 2024-09-21T03:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.830+0000 7fec8f806640 1 -- start start 2024-09-21T03:03:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8f806640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec881036c0 0x7fec88076e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8f806640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88104740 0x7fec88075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8f806640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec881050e0 0x7fec88075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8f806640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec8810f200 con 0x7fec88104740 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8f806640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fec8810f080 con 0x7fec881050e0 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8f806640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec8810f380 con 0x7fec881036c0 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8cda1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88104740 0x7fec88075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8cda1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88104740 0x7fec88075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53474/0 (socket says 172.21.15.29:53474) 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8cda1640 1 -- 172.21.15.29:0/2791478007 learned_addr learned my addr 172.21.15.29:0/2791478007 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec881036c0 0x7fec88076e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8dda3640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec881050e0 0x7fec88075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8cda1640 1 -- 172.21.15.29:0/2791478007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec881036c0 msgr2=0x7fec88076e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8cda1640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec881036c0 0x7fec88076e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8cda1640 1 -- 172.21.15.29:0/2791478007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec881050e0 msgr2=0x7fec88075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8cda1640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec881050e0 0x7fec88075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.831+0000 7fec8cda1640 1 -- 172.21.15.29:0/2791478007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec88076380 con 0x7fec88104740 2024-09-21T03:03:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.832+0000 7fec8cda1640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88104740 0x7fec88075710 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fec7800d950 tx=0x7fec7800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.832+0000 7fec7e7fc640 1 -- 172.21.15.29:0/2791478007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78017070 con 0x7fec88104740 2024-09-21T03:03:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.832+0000 7fec7e7fc640 1 -- 172.21.15.29:0/2791478007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec78009c30 con 0x7fec88104740 2024-09-21T03:03:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.832+0000 7fec7e7fc640 1 -- 172.21.15.29:0/2791478007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7800f620 con 0x7fec88104740 2024-09-21T03:03:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.832+0000 7fec8f806640 1 -- 172.21.15.29:0/2791478007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec881a3f00 con 0x7fec88104740 2024-09-21T03:03:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.832+0000 7fec8f806640 1 -- 172.21.15.29:0/2791478007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec881a4310 con 0x7fec88104740 2024-09-21T03:03:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.835+0000 7fec9684a740 1 Processor -- start 2024-09-21T03:03:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec9684a740 1 -- start start 2024-09-21T03:03:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec9684a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec9684a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f02f0 0x5627bb6c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec9684a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f84d0 0x5627bb6c2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec9684a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5627bb6336f0 con 0x5627bb2f84d0 2024-09-21T03:03:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec9684a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5627bb622b20 con 0x5627bb2f02f0 2024-09-21T03:03:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec9684a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5627bb626930 con 0x5627bb3191b0 2024-09-21T03:03:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec8dda3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec8dda3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43548/0 (socket says 172.21.15.29:43548) 2024-09-21T03:03:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec8d5a2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f02f0 0x5627bb6c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec8dda3640 1 -- 172.21.15.29:0/2006914683 learned_addr learned my addr 172.21.15.29:0/2006914683 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec7e7fc640 1 -- 172.21.15.29:0/2791478007 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fec7800f7c0 con 0x7fec88104740 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec8dda3640 1 -- 172.21.15.29:0/2006914683 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f02f0 msgr2=0x5627bb6c14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec8dda3640 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f02f0 0x5627bb6c14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.836+0000 7fec8dda3640 1 -- 172.21.15.29:0/2006914683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f84d0 msgr2=0x5627bb6c2010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec8cda1640 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f84d0 0x5627bb6c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec8dda3640 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f84d0 0x5627bb6c2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec8dda3640 1 -- 172.21.15.29:0/2006914683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627bb5e8530 con 0x5627bb3191b0 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f02f0 0x5627bb6c14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec8cda1640 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f84d0 0x5627bb6c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec8dda3640 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c0e50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec80009fd0 tx=0x7fec800118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91296faeaa22f462 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec737fe640 1 -- 172.21.15.29:0/2006914683 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80004d20 con 0x5627bb3191b0 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec737fe640 1 -- 172.21.15.29:0/2006914683 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec80004050 con 0x5627bb3191b0 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec737fe640 1 -- 172.21.15.29:0/2006914683 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80005260 con 0x5627bb3191b0 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.837+0000 7fec7e7fc640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fec740b6510 0x7fec740b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fec740b6510 0x7fec740b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec7e7fc640 1 -- 172.21.15.29:0/2791478007 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fec780e7200 con 0x7fec88104740 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 -- 172.21.15.29:0/2006914683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 msgr2=0x5627bb6c0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c0e50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec80009fd0 tx=0x7fec800118d0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fec740b6510 0x7fec740b8a20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fec84005760 tx=0x7fec84031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 -- 172.21.15.29:0/2006914683 shutdown_connections 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f84d0 0x5627bb6c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f02f0 0x5627bb6c14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 --2- 172.21.15.29:0/2006914683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c0e50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 -- 172.21.15.29:0/2006914683 >> 172.21.15.29:0/2006914683 conn(0x5627bb30d300 msgr2=0x5627bb6bc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 -- 172.21.15.29:0/2006914683 shutdown_connections 2024-09-21T03:03:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.838+0000 7fec9684a740 1 -- 172.21.15.29:0/2006914683 wait complete. 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.839+0000 7fec9684a740 1 -- start start 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.839+0000 7fec9684a740 1 Processor -- start 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec9684a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f02f0 0x5627bb6ca640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec9684a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec9684a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec9684a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5627bb55aab0 con 0x5627bb2f02f0 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec9684a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5627bb5c8730 con 0x5627bb2f84d0 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec9684a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5627bb6c2b30 con 0x5627bb3191b0 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec8dda3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec8cda1640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec8d5a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f02f0 0x5627bb6ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec8dda3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43564/0 (socket says 172.21.15.29:43564) 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec8d5a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f02f0 0x5627bb6ca640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53496/0 (socket says 172.21.15.29:53496) 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec8dda3640 1 -- 172.21.15.29:0/2908765385 learned_addr learned my addr 172.21.15.29:0/2908765385 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.840+0000 7fec9684a740 10 client.0 _ll_register_callbacks cb 0x5627bb61c6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec8d5a2640 1 -- 172.21.15.29:0/2908765385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 msgr2=0x5627bb6cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec8d5a2640 1 -- 172.21.15.29:0/2908765385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 msgr2=0x5627bb6cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec8d5a2640 1 -- 172.21.15.29:0/2908765385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627bb384790 con 0x5627bb2f02f0 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec8dda3640 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f02f0 0x5627bb6ca640 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fec8402a930 tx=0x7fec84009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8c5fad218beb23 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:03:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec70ff9640 1 -- 172.21.15.29:0/2908765385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84035040 con 0x5627bb2f02f0 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec9684a740 1 -- 172.21.15.29:0/2908765385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec80004050 con 0x5627bb2f02f0 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.841+0000 7fec70ff9640 1 -- 172.21.15.29:0/2908765385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fec84031500 con 0x5627bb2f02f0 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec70ff9640 1 -- 172.21.15.29:0/2908765385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fec84031700 con 0x5627bb2f02f0 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec9684a740 10 client.18642 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec9684a740 1 -- 172.21.15.29:0/2908765385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5627bb3fd620 con 0x5627bb2f02f0 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec70ff9640 1 -- 172.21.15.29:0/2908765385 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec80004050 con 0x5627bb2f02f0 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec70ff9640 1 -- 172.21.15.29:0/2908765385 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fec84044030 con 0x5627bb2f02f0 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec9684a740 10 client.18642 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec9684a740 10 client.18642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec9684a740 20 client.18642 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec9684a740 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627bb2d7d80 0x5627bb6d7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec9684a740 4 client.18642 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec9684a740 1 -- 172.21.15.29:0/2908765385 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627bb635640 con 0x5627bb2d7d80 2024-09-21T03:03:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.842+0000 7fec8cda1640 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627bb2d7d80 0x5627bb6d7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.843+0000 7fec70ff9640 10 client.18642 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.843+0000 7fec8cda1640 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627bb2d7d80 0x5627bb6d7c90 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d8f91b2a30f2721 server_cookie=a14b0bbe86b69334 in_seq=0 out_seq=0 2024-09-21T03:03:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.846+0000 7fec70ff9640 1 -- 172.21.15.29:0/2908765385 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5627bb635640 con 0x5627bb2d7d80 2024-09-21T03:03:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.846+0000 7fec70ff9640 10 client.18642 handle_command_reply: tid=0 2024-09-21T03:03:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.846+0000 7fec9684a740 1 client.18642 shutdown 2024-09-21T03:03:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 -- 172.21.15.29:0/2908765385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f02f0 msgr2=0x5627bb6ca640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f02f0 0x5627bb6ca640 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fec8402a930 tx=0x7fec84009260 comp rx=0 tx=0).stop 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 -- 172.21.15.29:0/2908765385 shutdown_connections 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627bb2d7d80 0x5627bb6d7c90 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 --2- 172.21.15.29:0/2908765385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2f02f0 0x5627bb6ca640 secure :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fec8402a930 tx=0x7fec84009260 comp rx=0 tx=0).stop 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec70ff9640 0 client.18642 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 -- 172.21.15.29:0/2908765385 >> 172.21.15.29:0/2908765385 conn(0x5627bb30d300 msgr2=0x5627bb6baea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 -- 172.21.15.29:0/2908765385 shutdown_connections 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 1 -- 172.21.15.29:0/2908765385 wait complete. 2024-09-21T03:03:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.847+0000 7fec9684a740 20 client.18642 trim_cache size 0 max 16384 2024-09-21T03:03:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec9684a740 1 Processor -- start 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec9684a740 1 -- start start 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec9684a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec9684a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6bbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec9684a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec9684a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec881ae800 con 0x5627bb2d7d80 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec9684a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fec881b0340 con 0x5627bb2f84d0 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec9684a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec881aeed0 con 0x5627bb3191b0 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec8d5a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec8d5a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6bb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53500/0 (socket says 172.21.15.29:53500) 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec8d5a2640 1 -- 172.21.15.29:0/3406242354 learned_addr learned my addr 172.21.15.29:0/3406242354 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec8cda1640 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6bbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.870+0000 7fec8dda3640 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec8d5a2640 1 -- 172.21.15.29:0/3406242354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 msgr2=0x5627bb6c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec8d5a2640 1 -- 172.21.15.29:0/3406242354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 msgr2=0x5627bb6bbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6bbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec8d5a2640 1 -- 172.21.15.29:0/3406242354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627bb6cbbe0 con 0x5627bb2d7d80 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec8cda1640 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6bbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6bb650 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fec84051a80 tx=0x7fec84003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a55348856a821d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec72ffd640 1 -- 172.21.15.29:0/3406242354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84035040 con 0x5627bb2d7d80 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec72ffd640 1 -- 172.21.15.29:0/3406242354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec8404c070 con 0x5627bb2d7d80 2024-09-21T03:03:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.871+0000 7fec72ffd640 1 -- 172.21.15.29:0/3406242354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84031d30 con 0x5627bb2d7d80 2024-09-21T03:03:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 -- 172.21.15.29:0/3406242354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 msgr2=0x5627bb6bb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6bb650 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fec84051a80 tx=0x7fec84003a70 comp rx=0 tx=0).stop 2024-09-21T03:03:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 -- 172.21.15.29:0/3406242354 shutdown_connections 2024-09-21T03:03:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb3191b0 0x5627bb6c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb2f84d0 0x5627bb6bbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 --2- 172.21.15.29:0/3406242354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6bb650 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 -- 172.21.15.29:0/3406242354 >> 172.21.15.29:0/3406242354 conn(0x5627bb30d300 msgr2=0x5627bb6bc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 -- 172.21.15.29:0/3406242354 shutdown_connections 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 -- 172.21.15.29:0/3406242354 wait complete. 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.872+0000 7fec9684a740 1 -- start start 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec9684a740 1 Processor -- start 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec9684a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6d17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec9684a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb2f84d0 0x5627bb6d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec9684a740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb3191b0 0x5627bb6d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec9684a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec881afc70 con 0x5627bb2d7d80 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec9684a740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fec881af5a0 con 0x5627bb3191b0 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec9684a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec881ac7e0 con 0x5627bb2f84d0 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec8cda1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb2f84d0 0x5627bb6d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec8d5a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6d17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec9684a740 10 client.0 _ll_register_callbacks cb 0x7fec84004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec8cda1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb2f84d0 0x5627bb6d1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43574/0 (socket says 172.21.15.29:43574) 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec8d5a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6d17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53516/0 (socket says 172.21.15.29:53516) 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.873+0000 7fec8cda1640 1 -- 172.21.15.29:0/122148623 learned_addr learned my addr 172.21.15.29:0/122148623 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec8dda3640 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb3191b0 0x5627bb6d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec8d5a2640 1 -- 172.21.15.29:0/122148623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb2f84d0 msgr2=0x5627bb6d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb2f84d0 0x5627bb6d1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec8d5a2640 1 -- 172.21.15.29:0/122148623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb3191b0 msgr2=0x5627bb6d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb3191b0 0x5627bb6d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec8d5a2640 1 -- 172.21.15.29:0/122148623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627bb6d25f0 con 0x5627bb2d7d80 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec8dda3640 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb3191b0 0x5627bb6d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec8d5a2640 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6d17c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fec8402a930 tx=0x7fec84009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39179a5f94d614c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.874+0000 7fec70ff9640 1 -- 172.21.15.29:0/122148623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84062070 con 0x5627bb2d7d80 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.875+0000 7fec70ff9640 1 -- 172.21.15.29:0/122148623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fec84031cc0 con 0x5627bb2d7d80 2024-09-21T03:03:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.875+0000 7fec9684a740 1 -- 172.21.15.29:0/122148623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec8404c070 con 0x5627bb2d7d80 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.875+0000 7fec70ff9640 1 -- 172.21.15.29:0/122148623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fec8406c050 con 0x5627bb2d7d80 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.875+0000 7fec9684a740 10 client.18648 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.875+0000 7fec9684a740 1 -- 172.21.15.29:0/122148623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5627bb742c80 con 0x5627bb2d7d80 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.875+0000 7fec70ff9640 1 -- 172.21.15.29:0/122148623 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec8404c070 con 0x5627bb2d7d80 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec70ff9640 1 -- 172.21.15.29:0/122148623 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fec84044040 con 0x5627bb2d7d80 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec9684a740 10 client.18648 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec9684a740 10 client.18648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec9684a740 20 client.18648 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec9684a740 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627bb2ad1c0 0x5627bb7456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec9684a740 4 client.18648 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec9684a740 1 -- 172.21.15.29:0/122148623 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627bb6310f0 con 0x5627bb2ad1c0 2024-09-21T03:03:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec8cda1640 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627bb2ad1c0 0x5627bb7456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.876+0000 7fec70ff9640 10 client.18648 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:30.877+0000 7fec8cda1640 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627bb2ad1c0 0x5627bb7456f0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=553b9430a00f058b server_cookie=f66ce1c2f303da46 in_seq=0 out_seq=0 2024-09-21T03:03:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.100+0000 7fec70ff9640 1 -- 172.21.15.29:0/122148623 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627bb6310f0 con 0x5627bb2ad1c0 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.100+0000 7fec70ff9640 10 client.18648 handle_command_reply: tid=0 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.100+0000 7fec9684a740 1 client.18648 shutdown 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 -- 172.21.15.29:0/122148623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 msgr2=0x5627bb6d17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6d17c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fec8402a930 tx=0x7fec84009670 comp rx=0 tx=0).stop 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 -- 172.21.15.29:0/122148623 shutdown_connections 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5627bb3191b0 0x5627bb6d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627bb2f84d0 0x5627bb6d1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5627bb2ad1c0 0x5627bb7456f0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 --2- 172.21.15.29:0/122148623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bb2d7d80 0x5627bb6d17c0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec70ff9640 0 client.18648 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 -- 172.21.15.29:0/122148623 >> 172.21.15.29:0/122148623 conn(0x5627bb30d300 msgr2=0x5627bb6bc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 -- 172.21.15.29:0/122148623 shutdown_connections 2024-09-21T03:03:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 1 -- 172.21.15.29:0/122148623 wait complete. 2024-09-21T03:03:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.101+0000 7fec9684a740 20 client.18648 trim_cache size 0 max 16384 2024-09-21T03:03:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:03:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:03:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:03:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:03:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 -- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fec740b6510 msgr2=0x7fec740b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fec740b6510 0x7fec740b8a20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fec84005760 tx=0x7fec84031400 comp rx=0 tx=0).stop 2024-09-21T03:03:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 -- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88104740 msgr2=0x7fec88075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88104740 0x7fec88075710 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fec7800d950 tx=0x7fec7800de10 comp rx=0 tx=0).stop 2024-09-21T03:03:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 -- 172.21.15.29:0/2791478007 shutdown_connections 2024-09-21T03:03:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fec740b6510 0x7fec740b8a20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec881050e0 0x7fec88075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec88104740 0x7fec88075710 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 --2- 172.21.15.29:0/2791478007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec881036c0 0x7fec88076e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.102+0000 7fec8f806640 1 -- 172.21.15.29:0/2791478007 >> 172.21.15.29:0/2791478007 conn(0x7fec880ff020 msgr2=0x7fec88107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.103+0000 7fec8f806640 1 -- 172.21.15.29:0/2791478007 shutdown_connections 2024-09-21T03:03:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:31.103+0000 7fec8f806640 1 -- 172.21.15.29:0/2791478007 wait complete. 2024-09-21T03:03:31.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 68501 5.46 MB/sec execute 548 sec latency 237.711 ms 2024-09-21T03:03:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:31 smithi066 ceph-mon[33436]: pgmap v1122: 129 pgs: 129 active+clean; 339 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-09-21T03:03:32.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:31 smithi190 ceph-mon[33871]: pgmap v1122: 129 pgs: 129 active+clean; 339 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-09-21T03:03:32.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:31 smithi029 ceph-mon[31006]: pgmap v1122: 129 pgs: 129 active+clean; 339 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-09-21T03:03:32.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 68662 5.48 MB/sec execute 549 sec latency 102.124 ms 2024-09-21T03:03:33.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 68855 5.50 MB/sec execute 550 sec latency 99.038 ms 2024-09-21T03:03:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:33 smithi066 ceph-mon[33436]: pgmap v1123: 129 pgs: 129 active+clean; 340 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-09-21T03:03:34.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:33 smithi190 ceph-mon[33871]: pgmap v1123: 129 pgs: 129 active+clean; 340 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-09-21T03:03:34.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:33 smithi029 ceph-mon[31006]: pgmap v1123: 129 pgs: 129 active+clean; 340 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-09-21T03:03:34.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 68957 5.50 MB/sec execute 551 sec latency 227.397 ms 2024-09-21T03:03:35.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 69114 5.51 MB/sec execute 552 sec latency 149.486 ms 2024-09-21T03:03:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:35 smithi066 ceph-mon[33436]: pgmap v1124: 129 pgs: 129 active+clean; 344 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 4.2 MiB/s wr, 269 op/s 2024-09-21T03:03:36.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:35 smithi190 ceph-mon[33871]: pgmap v1124: 129 pgs: 129 active+clean; 344 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 4.2 MiB/s wr, 269 op/s 2024-09-21T03:03:36.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:35 smithi029 ceph-mon[31006]: pgmap v1124: 129 pgs: 129 active+clean; 344 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 4.2 MiB/s wr, 269 op/s 2024-09-21T03:03:36.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 69225 5.50 MB/sec execute 553 sec latency 142.719 ms 2024-09-21T03:03:37.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 69340 5.50 MB/sec execute 554 sec latency 128.681 ms 2024-09-21T03:03:38.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:37 smithi190 ceph-mon[33871]: pgmap v1125: 129 pgs: 129 active+clean; 346 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 4.2 MiB/s wr, 247 op/s 2024-09-21T03:03:38.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:37 smithi029 ceph-mon[31006]: pgmap v1125: 129 pgs: 129 active+clean; 346 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 4.2 MiB/s wr, 247 op/s 2024-09-21T03:03:38.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:37 smithi066 ceph-mon[33436]: pgmap v1125: 129 pgs: 129 active+clean; 346 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 4.2 MiB/s wr, 247 op/s 2024-09-21T03:03:38.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 69437 5.50 MB/sec execute 555 sec latency 141.312 ms 2024-09-21T03:03:39.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 69570 5.50 MB/sec execute 556 sec latency 190.591 ms 2024-09-21T03:03:40.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:39 smithi190 ceph-mon[33871]: pgmap v1126: 129 pgs: 129 active+clean; 349 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 4.4 MiB/s wr, 277 op/s 2024-09-21T03:03:40.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:39 smithi029 ceph-mon[31006]: pgmap v1126: 129 pgs: 129 active+clean; 349 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 4.4 MiB/s wr, 277 op/s 2024-09-21T03:03:40.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:39 smithi066 ceph-mon[33436]: pgmap v1126: 129 pgs: 129 active+clean; 349 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 4.4 MiB/s wr, 277 op/s 2024-09-21T03:03:40.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 69736 5.51 MB/sec execute 557 sec latency 128.767 ms 2024-09-21T03:03:41.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 69878 5.51 MB/sec execute 558 sec latency 132.490 ms 2024-09-21T03:03:42.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:41 smithi190 ceph-mon[33871]: pgmap v1127: 129 pgs: 129 active+clean; 350 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 4.3 MiB/s wr, 258 op/s 2024-09-21T03:03:42.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:41 smithi029 ceph-mon[31006]: pgmap v1127: 129 pgs: 129 active+clean; 350 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 4.3 MiB/s wr, 258 op/s 2024-09-21T03:03:42.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:41 smithi066 ceph-mon[33436]: pgmap v1127: 129 pgs: 129 active+clean; 350 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 4.3 MiB/s wr, 258 op/s 2024-09-21T03:03:42.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 69986 5.51 MB/sec execute 559 sec latency 90.262 ms 2024-09-21T03:03:43.749 INFO:tasks.workunit.client.0.smithi029.stdout: 10 70119 5.51 MB/sec execute 560 sec latency 150.362 ms 2024-09-21T03:03:44.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:43 smithi190 ceph-mon[33871]: pgmap v1128: 129 pgs: 129 active+clean; 349 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 3.6 MiB/s wr, 241 op/s 2024-09-21T03:03:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:43 smithi029 ceph-mon[31006]: pgmap v1128: 129 pgs: 129 active+clean; 349 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 3.6 MiB/s wr, 241 op/s 2024-09-21T03:03:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 -- 172.21.15.29:0/4021600423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6401036a0 msgr2=0x7fe640103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 --2- 172.21.15.29:0/4021600423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6401036a0 0x7fe640103b10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe630009900 tx=0x7fe63002a410 comp rx=0 tx=0).stop 2024-09-21T03:03:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 -- 172.21.15.29:0/4021600423 shutdown_connections 2024-09-21T03:03:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 --2- 172.21.15.29:0/4021600423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6401050c0 0x7fe64010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 --2- 172.21.15.29:0/4021600423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe640104720 0x7fe640104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 --2- 172.21.15.29:0/4021600423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6401036a0 0x7fe640103b10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 -- 172.21.15.29:0/4021600423 >> 172.21.15.29:0/4021600423 conn(0x7fe6400fefe0 msgr2=0x7fe640101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 -- 172.21.15.29:0/4021600423 shutdown_connections 2024-09-21T03:03:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.240+0000 7fe64647e640 1 -- 172.21.15.29:0/4021600423 wait complete. 2024-09-21T03:03:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe64647e640 1 Processor -- start 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe64647e640 1 -- start start 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe64647e640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe6401036a0 0x7fe640199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe64647e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe640104720 0x7fe64019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe64647e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6401050c0 0x7fe64019b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe64647e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe64010f1c0 con 0x7fe6401050c0 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe64647e640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe64010f040 con 0x7fe6401036a0 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe64647e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe64010f340 con 0x7fe640104720 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe63f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe640104720 0x7fe64019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe644a1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6401050c0 0x7fe64019b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe63ffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe6401036a0 0x7fe640199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe63f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe640104720 0x7fe64019a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55012/0 (socket says 172.21.15.29:55012) 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.241+0000 7fe63f7fe640 1 -- 172.21.15.29:0/2348876702 learned_addr learned my addr 172.21.15.29:0/2348876702 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63f7fe640 1 -- 172.21.15.29:0/2348876702 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe6401036a0 msgr2=0x7fe640199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe6401036a0 0x7fe640199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63f7fe640 1 -- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6401050c0 msgr2=0x7fe64019b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6401050c0 0x7fe64019b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63f7fe640 1 -- 172.21.15.29:0/2348876702 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64019b8b0 con 0x7fe640104720 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe640104720 0x7fe64019a4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e9f0 tx=0x7fe63400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63d7fa640 1 -- 172.21.15.29:0/2348876702 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63400cd80 con 0x7fe640104720 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63d7fa640 1 -- 172.21.15.29:0/2348876702 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6340108c0 con 0x7fe640104720 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe64647e640 1 -- 172.21.15.29:0/2348876702 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64019bba0 con 0x7fe640104720 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe63d7fa640 1 -- 172.21.15.29:0/2348876702 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6340187c0 con 0x7fe640104720 2024-09-21T03:03:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.242+0000 7fe64647e640 1 -- 172.21.15.29:0/2348876702 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6401a2290 con 0x7fe640104720 2024-09-21T03:03:44.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.245+0000 7fe63d7fa640 1 -- 172.21.15.29:0/2348876702 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fe634018a60 con 0x7fe640104720 2024-09-21T03:03:44.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.245+0000 7fe64d3eb740 1 Processor -- start 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe64d3eb740 1 -- start start 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd7be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd7c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd7d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55977fcee6f0 con 0x55977f9d41b0 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55977fcddb20 con 0x55977f9ab2f0 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55977fce1930 con 0x55977f9b34d0 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe644a1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd7be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe644a1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd7be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34406/0 (socket says 172.21.15.29:34406) 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe644a1b640 1 -- 172.21.15.29:0/3584440267 learned_addr learned my addr 172.21.15.29:0/3584440267 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe63ffff640 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd7c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.246+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd7d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe644a1b640 1 -- 172.21.15.29:0/3584440267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 msgr2=0x55977fd7d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe644a1b640 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd7d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe644a1b640 1 -- 172.21.15.29:0/3584440267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 msgr2=0x55977fd7c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe644a1b640 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd7c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe644a1b640 1 -- 172.21.15.29:0/3584440267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55977fca3530 con 0x55977f9d41b0 2024-09-21T03:03:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe63d7fa640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe6280b6590 0x7fe6280b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe644a1b640 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd7be50 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fe638009fd0 tx=0x7fe6380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a51876233daa1c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe63d7fa640 1 -- 172.21.15.29:0/2348876702 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fe6340e7a20 con 0x7fe640104720 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe63ffff640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe6280b6590 0x7fe6280b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe6267fc640 1 -- 172.21.15.29:0/3584440267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638004d20 con 0x55977f9d41b0 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe6267fc640 1 -- 172.21.15.29:0/3584440267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe638004050 con 0x55977f9d41b0 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.247+0000 7fe6267fc640 1 -- 172.21.15.29:0/3584440267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638005260 con 0x55977f9d41b0 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.248+0000 7fe63ffff640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe6280b6590 0x7fe6280b8aa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe630005760 tx=0x7fe630031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.248+0000 7fe64d3eb740 1 -- 172.21.15.29:0/3584440267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 msgr2=0x55977fd7be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.248+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd7be50 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fe638009fd0 tx=0x7fe6380118d0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.248+0000 7fe64d3eb740 1 -- 172.21.15.29:0/3584440267 shutdown_connections 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.249+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd7d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.249+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd7c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.249+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/3584440267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd7be50 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.249+0000 7fe64d3eb740 1 -- 172.21.15.29:0/3584440267 >> 172.21.15.29:0/3584440267 conn(0x55977f9c8300 msgr2=0x55977fd778e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.249+0000 7fe64d3eb740 1 -- 172.21.15.29:0/3584440267 shutdown_connections 2024-09-21T03:03:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.249+0000 7fe64d3eb740 1 -- 172.21.15.29:0/3584440267 wait complete. 2024-09-21T03:03:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.249+0000 7fe64d3eb740 1 -- start start 2024-09-21T03:03:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.249+0000 7fe64d3eb740 1 Processor -- start 2024-09-21T03:03:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd85690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd85d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd868c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55977fc15ab0 con 0x55977f9d41b0 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55977fc83730 con 0x55977f9ab2f0 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55977fd7db30 con 0x55977f9b34d0 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe64d3eb740 10 client.0 _ll_register_callbacks cb 0x55977fcd76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd85d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe644a1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd868c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63ffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd85690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63ffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd85690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:47826/0 (socket says 172.21.15.29:47826) 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63ffff640 1 -- 172.21.15.29:0/1124696001 learned_addr learned my addr 172.21.15.29:0/1124696001 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd85d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55026/0 (socket says 172.21.15.29:55026) 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe644a1b640 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd868c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34408/0 (socket says 172.21.15.29:34408) 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63ffff640 1 -- 172.21.15.29:0/1124696001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 msgr2=0x55977fd85d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63ffff640 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd85d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63ffff640 1 -- 172.21.15.29:0/1124696001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 msgr2=0x55977fd868c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63ffff640 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd868c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.250+0000 7fe63ffff640 1 -- 172.21.15.29:0/1124696001 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55977fa3f790 con 0x55977f9ab2f0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe63ffff640 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd85690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe630000c00 tx=0x7fe630009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c64cce849fe4dbb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe61f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1124696001 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe630035040 con 0x55977f9ab2f0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1124696001 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fe630003d80 con 0x55977f9ab2f0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1124696001 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe638004050 con 0x55977f9ab2f0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1124696001 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe630031500 con 0x55977f9ab2f0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe64d3eb740 10 client.27923 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1124696001 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55977fab8620 con 0x55977f9ab2f0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.251+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1124696001 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe638004050 con 0x55977f9ab2f0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1124696001 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fe630044030 con 0x55977f9ab2f0 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe64d3eb740 10 client.27923 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe64d3eb740 10 client.27923 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe64d3eb740 20 client.27923 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55977f992d80 0x55977fd97210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe64d3eb740 4 client.27923 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:03:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1124696001 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55977fcf0640 con 0x55977f992d80 2024-09-21T03:03:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55977f992d80 0x55977fd97210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe61f7fe640 10 client.27923 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.252+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55977f992d80 0x55977fd97210 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d58bb8ec3a178a0 server_cookie=5a84deb828622e48 in_seq=0 out_seq=0 2024-09-21T03:03:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.254+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1124696001 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55977fcf0640 con 0x55977f992d80 2024-09-21T03:03:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.254+0000 7fe61f7fe640 10 client.27923 handle_command_reply: tid=0 2024-09-21T03:03:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.254+0000 7fe64d3eb740 1 client.27923 shutdown 2024-09-21T03:03:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1124696001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 msgr2=0x55977fd85690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd85690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe630000c00 tx=0x7fe630009260 comp rx=0 tx=0).stop 2024-09-21T03:03:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1124696001 shutdown_connections 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55977f992d80 0x55977fd97210 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd868c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9b34d0 0x55977fd85d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1124696001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd85690 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1124696001 >> 172.21.15.29:0/1124696001 conn(0x55977f9c8300 msgr2=0x55977fd75ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe61f7fe640 0 client.27923 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1124696001 shutdown_connections 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1124696001 wait complete. 2024-09-21T03:03:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.255+0000 7fe64d3eb740 20 client.27923 trim_cache size 0 max 16384 2024-09-21T03:03:44.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:43 smithi066 ceph-mon[33436]: pgmap v1128: 129 pgs: 129 active+clean; 349 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 3.6 MiB/s wr, 241 op/s 2024-09-21T03:03:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.282+0000 7fe64d3eb740 1 Processor -- start 2024-09-21T03:03:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.282+0000 7fe64d3eb740 1 -- start start 2024-09-21T03:03:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd76650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9b34d0 0x55977fd76b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9d41b0 0x55977fd7bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401aa380 con 0x55977f9b34d0 2024-09-21T03:03:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401abec0 con 0x55977f9ab2f0 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401aaa50 con 0x55977f9d41b0 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe63f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9b34d0 0x55977fd76b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe63f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9b34d0 0x55977fd76b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34424/0 (socket says 172.21.15.29:34424) 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe63f7fe640 1 -- 172.21.15.29:0/4070365999 learned_addr learned my addr 172.21.15.29:0/4070365999 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe644a1b640 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9d41b0 0x55977fd7bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe63ffff640 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd76650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe63f7fe640 1 -- 172.21.15.29:0/4070365999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9d41b0 msgr2=0x55977fd7bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9d41b0 0x55977fd7bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe63f7fe640 1 -- 172.21.15.29:0/4070365999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 msgr2=0x55977fd76650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.283+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd76650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.284+0000 7fe63f7fe640 1 -- 172.21.15.29:0/4070365999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55977fd87490 con 0x55977f9b34d0 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.284+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9b34d0 0x55977fd76b90 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe6340ad580 tx=0x7fe6340eaf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15e234211444dea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.284+0000 7fe625ffb640 1 -- 172.21.15.29:0/4070365999 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6340b22b0 con 0x55977f9b34d0 2024-09-21T03:03:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.284+0000 7fe625ffb640 1 -- 172.21.15.29:0/4070365999 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6340b71f0 con 0x55977f9b34d0 2024-09-21T03:03:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.284+0000 7fe625ffb640 1 -- 172.21.15.29:0/4070365999 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6340a9430 con 0x55977f9b34d0 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 -- 172.21.15.29:0/4070365999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9b34d0 msgr2=0x55977fd76b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9b34d0 0x55977fd76b90 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe6340ad580 tx=0x7fe6340eaf90 comp rx=0 tx=0).stop 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 -- 172.21.15.29:0/4070365999 shutdown_connections 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9d41b0 0x55977fd7bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9b34d0 0x55977fd76b90 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/4070365999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9ab2f0 0x55977fd76650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 -- 172.21.15.29:0/4070365999 >> 172.21.15.29:0/4070365999 conn(0x55977f9c8300 msgr2=0x55977fd725c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 -- 172.21.15.29:0/4070365999 shutdown_connections 2024-09-21T03:03:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.285+0000 7fe64d3eb740 1 -- 172.21.15.29:0/4070365999 wait complete. 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.286+0000 7fe64d3eb740 1 -- start start 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.286+0000 7fe64d3eb740 1 Processor -- start 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.286+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9ab2f0 0x55977fd8a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.286+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9b34d0 0x55977fd8a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.286+0000 7fe64d3eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd8b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9ab2f0 0x55977fd8a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe644a1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd8b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe644a1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd8b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34426/0 (socket says 172.21.15.29:34426) 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe644a1b640 1 -- 172.21.15.29:0/1748660906 learned_addr learned my addr 172.21.15.29:0/1748660906 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9ab2f0 0x55977fd8a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55042/0 (socket says 172.21.15.29:55042) 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe64d3eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401ab7f0 con 0x55977f9d41b0 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401ab120 con 0x55977f9b34d0 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401a8360 con 0x55977f9ab2f0 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe64d3eb740 10 client.0 _ll_register_callbacks cb 0x7fe630009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9b34d0 0x55977fd8a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63ffff640 1 -- 172.21.15.29:0/1748660906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9b34d0 msgr2=0x55977fd8a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63ffff640 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9b34d0 0x55977fd8a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63ffff640 1 -- 172.21.15.29:0/1748660906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 msgr2=0x55977fd8b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63ffff640 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd8b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63ffff640 1 -- 172.21.15.29:0/1748660906 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55977fd8bb50 con 0x55977f9ab2f0 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe63ffff640 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9ab2f0 0x55977fd8a1c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe630051870 tx=0x7fe630003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6086de21eaa90570 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.287+0000 7fe61f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.288+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1748660906 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe630035040 con 0x55977f9ab2f0 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.288+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1748660906 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fe6300318a0 con 0x55977f9ab2f0 2024-09-21T03:03:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.288+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6340b71f0 con 0x55977f9ab2f0 2024-09-21T03:03:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.288+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1748660906 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe630064050 con 0x55977f9ab2f0 2024-09-21T03:03:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.288+0000 7fe64d3eb740 10 client.28084 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.288+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55977fd8ce70 con 0x55977f9ab2f0 2024-09-21T03:03:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.288+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1748660906 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6340b71f0 con 0x55977f9ab2f0 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.288+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1748660906 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fe630044040 con 0x55977f9ab2f0 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe64d3eb740 10 client.28084 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe64d3eb740 10 client.28084 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe64d3eb740 20 client.28084 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55977f9681c0 0x55977fe03c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe64d3eb740 4 client.28084 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55977fcec0f0 con 0x55977f9681c0 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55977f9681c0 0x55977fe03c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe61f7fe640 10 client.28084 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.289+0000 7fe63f7fe640 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55977f9681c0 0x55977fe03c70 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d96f81a0e9a09a3 server_cookie=508e08fef7aa6ff0 in_seq=0 out_seq=0 2024-09-21T03:03:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe61f7fe640 1 -- 172.21.15.29:0/1748660906 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55977fcec0f0 con 0x55977f9681c0 2024-09-21T03:03:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe61f7fe640 10 client.28084 handle_command_reply: tid=0 2024-09-21T03:03:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe64d3eb740 1 client.28084 shutdown 2024-09-21T03:03:44.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9ab2f0 msgr2=0x55977fd8a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9ab2f0 0x55977fd8a1c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe630051870 tx=0x7fe630003eb0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 shutdown_connections 2024-09-21T03:03:44.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55977f9681c0 0x55977fe03c70 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55977f9d41b0 0x55977fd8b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.496+0000 7fe61f7fe640 0 client.28084 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.495+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55977f9b34d0 0x55977fd8a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.496+0000 7fe64d3eb740 1 --2- 172.21.15.29:0/1748660906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55977f9ab2f0 0x55977fd8a1c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.496+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 >> 172.21.15.29:0/1748660906 conn(0x55977f9c8300 msgr2=0x55977fd7bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.496+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 shutdown_connections 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.496+0000 7fe64d3eb740 1 -- 172.21.15.29:0/1748660906 wait complete. 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.496+0000 7fe64d3eb740 20 client.28084 trim_cache size 0 max 16384 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:03:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:03:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:03:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 -- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe6280b6590 msgr2=0x7fe6280b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe6280b6590 0x7fe6280b8aa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe630005760 tx=0x7fe630031400 comp rx=0 tx=0).stop 2024-09-21T03:03:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 -- 172.21.15.29:0/2348876702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe640104720 msgr2=0x7fe64019a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe640104720 0x7fe64019a4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e9f0 tx=0x7fe63400eeb0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 -- 172.21.15.29:0/2348876702 shutdown_connections 2024-09-21T03:03:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fe6280b6590 0x7fe6280b8aa0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6401050c0 0x7fe64019b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe640104720 0x7fe64019a4e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 --2- 172.21.15.29:0/2348876702 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe6401036a0 0x7fe640199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.498+0000 7fe64647e640 1 -- 172.21.15.29:0/2348876702 >> 172.21.15.29:0/2348876702 conn(0x7fe6400fefe0 msgr2=0x7fe640107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.499+0000 7fe64647e640 1 -- 172.21.15.29:0/2348876702 shutdown_connections 2024-09-21T03:03:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:44.499+0000 7fe64647e640 1 -- 172.21.15.29:0/2348876702 wait complete. 2024-09-21T03:03:44.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 70241 5.51 MB/sec execute 561 sec latency 248.379 ms 2024-09-21T03:03:45.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 70390 5.52 MB/sec execute 562 sec latency 104.044 ms 2024-09-21T03:03:46.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:45 smithi190 ceph-mon[33871]: pgmap v1129: 129 pgs: 129 active+clean; 346 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 4.4 MiB/s wr, 297 op/s 2024-09-21T03:03:46.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:45 smithi029 ceph-mon[31006]: pgmap v1129: 129 pgs: 129 active+clean; 346 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 4.4 MiB/s wr, 297 op/s 2024-09-21T03:03:46.259 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:45 smithi066 ceph-mon[33436]: pgmap v1129: 129 pgs: 129 active+clean; 346 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 4.4 MiB/s wr, 297 op/s 2024-09-21T03:03:46.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 70519 5.51 MB/sec execute 563 sec latency 99.440 ms 2024-09-21T03:03:47.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 70660 5.51 MB/sec execute 564 sec latency 103.803 ms 2024-09-21T03:03:47.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:47 smithi029 ceph-mon[31006]: pgmap v1130: 129 pgs: 129 active+clean; 343 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 3.5 MiB/s wr, 244 op/s 2024-09-21T03:03:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:47 smithi190 ceph-mon[33871]: pgmap v1130: 129 pgs: 129 active+clean; 343 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 3.5 MiB/s wr, 244 op/s 2024-09-21T03:03:48.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:47 smithi066 ceph-mon[33436]: pgmap v1130: 129 pgs: 129 active+clean; 343 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 3.5 MiB/s wr, 244 op/s 2024-09-21T03:03:48.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 70805 5.51 MB/sec execute 565 sec latency 124.030 ms 2024-09-21T03:03:49.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 70946 5.51 MB/sec execute 566 sec latency 120.887 ms 2024-09-21T03:03:49.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:49 smithi066 ceph-mon[33436]: pgmap v1131: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 3.2 MiB/s wr, 274 op/s 2024-09-21T03:03:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:49 smithi190 ceph-mon[33871]: pgmap v1131: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 3.2 MiB/s wr, 274 op/s 2024-09-21T03:03:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:49 smithi029 ceph-mon[31006]: pgmap v1131: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 3.2 MiB/s wr, 274 op/s 2024-09-21T03:03:50.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 71054 5.51 MB/sec execute 567 sec latency 359.089 ms 2024-09-21T03:03:50.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:03:50.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:03:50.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:03:50.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:03:50.868 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:03:50.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:03:50.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:03:50.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:03:50.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:03:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:03:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:03:51.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:03:51.750 INFO:tasks.workunit.client.0.smithi029.stdout: 10 71177 5.51 MB/sec execute 568 sec latency 520.015 ms 2024-09-21T03:03:51.762 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:51 smithi066 ceph-mon[33436]: pgmap v1132: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 3.2 MiB/s wr, 311 op/s 2024-09-21T03:03:51.774 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:51 smithi190 ceph-mon[33871]: pgmap v1132: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 3.2 MiB/s wr, 311 op/s 2024-09-21T03:03:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:51 smithi029 ceph-mon[31006]: pgmap v1132: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 3.2 MiB/s wr, 311 op/s 2024-09-21T03:03:52.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 71362 5.52 MB/sec execute 569 sec latency 95.959 ms 2024-09-21T03:03:53.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 71485 5.52 MB/sec execute 570 sec latency 120.938 ms 2024-09-21T03:03:53.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:53 smithi190 ceph-mon[33871]: pgmap v1133: 129 pgs: 129 active+clean; 332 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.8 MiB/s wr, 292 op/s 2024-09-21T03:03:53.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:53 smithi029 ceph-mon[31006]: pgmap v1133: 129 pgs: 129 active+clean; 332 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.8 MiB/s wr, 292 op/s 2024-09-21T03:03:54.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:53 smithi066 ceph-mon[33436]: pgmap v1133: 129 pgs: 129 active+clean; 332 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.8 MiB/s wr, 292 op/s 2024-09-21T03:03:54.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 71652 5.53 MB/sec execute 571 sec latency 117.154 ms 2024-09-21T03:03:55.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 71802 5.54 MB/sec execute 572 sec latency 98.801 ms 2024-09-21T03:03:55.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:55 smithi190 ceph-mon[33871]: pgmap v1134: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 3.3 MiB/s wr, 335 op/s 2024-09-21T03:03:55.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:55 smithi029 ceph-mon[31006]: pgmap v1134: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 3.3 MiB/s wr, 335 op/s 2024-09-21T03:03:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:55 smithi066 ceph-mon[33436]: pgmap v1134: 129 pgs: 129 active+clean; 334 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 3.3 MiB/s wr, 335 op/s 2024-09-21T03:03:56.079 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T03:03:56.080 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.8 2024-09-21T03:03:56.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:03:56.450 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:03:56.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 71964 5.55 MB/sec execute 573 sec latency 229.802 ms 2024-09-21T03:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 -- 172.21.15.29:0/271271643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7741034c0 msgr2=0x7fb774103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 --2- 172.21.15.29:0/271271643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7741034c0 0x7fb774103990 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009900 tx=0x7fb75c02a410 comp rx=0 tx=0).stop 2024-09-21T03:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 -- 172.21.15.29:0/271271643 shutdown_connections 2024-09-21T03:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 --2- 172.21.15.29:0/271271643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7741040d0 0x7fb77410ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 --2- 172.21.15.29:0/271271643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7741034c0 0x7fb774103990 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 --2- 172.21.15.29:0/271271643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb774102ae0 0x7fb774102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 -- 172.21.15.29:0/271271643 >> 172.21.15.29:0/271271643 conn(0x7fb7740fd690 msgr2=0x7fb7740ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 -- 172.21.15.29:0/271271643 shutdown_connections 2024-09-21T03:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.614+0000 7fb779ace640 1 -- 172.21.15.29:0/271271643 wait complete. 2024-09-21T03:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.615+0000 7fb779ace640 1 Processor -- start 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.615+0000 7fb779ace640 1 -- start start 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.615+0000 7fb779ace640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb774102ae0 0x7fb774102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.615+0000 7fb779ace640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7741034c0 0x7fb77419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.615+0000 7fb779ace640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7741040d0 0x7fb7741a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.615+0000 7fb779ace640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410e0c0 con 0x7fb774102ae0 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.615+0000 7fb779ace640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410df40 con 0x7fb7741040d0 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.615+0000 7fb779ace640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410e240 con 0x7fb7741034c0 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb774102ae0 0x7fb774102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb774102ae0 0x7fb774102040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47404/0 (socket says 172.21.15.29:47404) 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 -- 172.21.15.29:0/2028459505 learned_addr learned my addr 172.21.15.29:0/2028459505 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb773fff640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7741040d0 0x7fb7741a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7741034c0 0x7fb77419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 -- 172.21.15.29:0/2028459505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7741034c0 msgr2=0x7fb77419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7741034c0 0x7fb77419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 -- 172.21.15.29:0/2028459505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7741040d0 msgr2=0x7fb7741a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7741040d0 0x7fb7741a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 -- 172.21.15.29:0/2028459505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7741a1910 con 0x7fb774102ae0 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7741034c0 0x7fb77419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb7737fe640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb774102ae0 0x7fb774102040 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb76400e9f0 tx=0x7fb76400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb770ff9640 1 -- 172.21.15.29:0/2028459505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76400cd80 con 0x7fb774102ae0 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb770ff9640 1 -- 172.21.15.29:0/2028459505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb7640108c0 con 0x7fb774102ae0 2024-09-21T03:03:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb779ace640 1 -- 172.21.15.29:0/2028459505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7741a1c00 con 0x7fb774102ae0 2024-09-21T03:03:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb770ff9640 1 -- 172.21.15.29:0/2028459505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7640187c0 con 0x7fb774102ae0 2024-09-21T03:03:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.616+0000 7fb779ace640 1 -- 172.21.15.29:0/2028459505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7741a21a0 con 0x7fb774102ae0 2024-09-21T03:03:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb780a37740 1 Processor -- start 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb780a37740 1 -- start start 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb770ff9640 1 -- 172.21.15.29:0/2028459505 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb764010430 con 0x7fb774102ae0 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb780a37740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fcce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb780a37740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fcd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb780a37740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb780a37740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x565407f3f6f0 con 0x565407c044d0 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb780a37740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x565407f2eb20 con 0x565407bfc2f0 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb780a37740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x565407f32930 con 0x565407c251b0 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb772ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb772ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47410/0 (socket says 172.21.15.29:47410) 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb772ffd640 1 -- 172.21.15.29:0/1477560837 learned_addr learned my addr 172.21.15.29:0/1477560837 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb7737fe640 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fcd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.619+0000 7fb773fff640 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fcce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb773fff640 1 -- 172.21.15.29:0/1477560837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 msgr2=0x565407fcd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb773fff640 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fcd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb773fff640 1 -- 172.21.15.29:0/1477560837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 msgr2=0x565407fce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb773fff640 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb773fff640 1 -- 172.21.15.29:0/1477560837 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565407ef4530 con 0x565407c251b0 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb773fff640 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fcce50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009f00 tx=0x7fb76c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79368ee874e80d49 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb769ffb640 1 -- 172.21.15.29:0/1477560837 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c004d20 con 0x565407c251b0 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb769ffb640 1 -- 172.21.15.29:0/1477560837 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb76c004030 con 0x565407c251b0 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb769ffb640 1 -- 172.21.15.29:0/1477560837 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c005240 con 0x565407c251b0 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb770ff9640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7600b6590 0x7fb7600b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.620+0000 7fb7737fe640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7600b6590 0x7fb7600b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb770ff9640 1 -- 172.21.15.29:0/2028459505 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fb7640e7a00 con 0x7fb774102ae0 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 -- 172.21.15.29:0/1477560837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 msgr2=0x565407fcce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fcce50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009f00 tx=0x7fb76c011810 comp rx=0 tx=0).stop 2024-09-21T03:03:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 -- 172.21.15.29:0/1477560837 shutdown_connections 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fcd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 --2- 172.21.15.29:0/1477560837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fcce50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 -- 172.21.15.29:0/1477560837 >> 172.21.15.29:0/1477560837 conn(0x565407c19300 msgr2=0x565407fc88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 -- 172.21.15.29:0/1477560837 shutdown_connections 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb7737fe640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7600b6590 0x7fb7600b8aa0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb7640aaa80 tx=0x7fb7640eb430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 -- 172.21.15.29:0/1477560837 wait complete. 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.621+0000 7fb780a37740 1 -- start start 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb780a37740 1 Processor -- start 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb780a37740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fd6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb780a37740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c044d0 0x565407fd6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb780a37740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c251b0 0x565407fd77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb780a37740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x565407e66ab0 con 0x565407c251b0 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb780a37740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x565407ed4730 con 0x565407bfc2f0 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb780a37740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x565407fceb30 con 0x565407c044d0 2024-09-21T03:03:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb780a37740 10 client.0 _ll_register_callbacks cb 0x565407f286b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c251b0 0x565407fd77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c251b0 0x565407fd77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47424/0 (socket says 172.21.15.29:47424) 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 -- 172.21.15.29:0/2179504171 learned_addr learned my addr 172.21.15.29:0/2179504171 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c044d0 0x565407fd6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fd6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 -- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c044d0 msgr2=0x565407fd6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c044d0 0x565407fd6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 -- 172.21.15.29:0/2179504171 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 msgr2=0x565407fd6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fd6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 -- 172.21.15.29:0/2179504171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565407c90790 con 0x565407c251b0 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c044d0 0x565407fd6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb773fff640 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c251b0 0x565407fd77b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb76c004760 tx=0x7fb76c0041e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50603295f5b2ba67 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.622+0000 7fb752ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:03:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb752ffd640 1 -- 172.21.15.29:0/2179504171 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c02b060 con 0x565407c251b0 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb752ffd640 1 -- 172.21.15.29:0/2179504171 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fb76c004a40 con 0x565407c251b0 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 1 -- 172.21.15.29:0/2179504171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb76c004030 con 0x565407c251b0 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb752ffd640 1 -- 172.21.15.29:0/2179504171 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb76c01e040 con 0x565407c251b0 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 10 client.18678 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 1 -- 172.21.15.29:0/2179504171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x565407d09620 con 0x565407c251b0 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb752ffd640 1 -- 172.21.15.29:0/2179504171 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb76c004030 con 0x565407c251b0 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb752ffd640 1 -- 172.21.15.29:0/2179504171 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fb76c024070 con 0x565407c251b0 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 10 client.18678 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 10 client.18678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 20 client.18678 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565407be3d80 0x565407fe3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 4 client.18678 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.623+0000 7fb780a37740 1 -- 172.21.15.29:0/2179504171 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565407f41640 con 0x565407be3d80 2024-09-21T03:03:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.624+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565407be3d80 0x565407fe3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.624+0000 7fb752ffd640 10 client.18678 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.624+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565407be3d80 0x565407fe3a30 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59472fa9bbf92751 server_cookie=83076c6e2ca3f3be in_seq=0 out_seq=0 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb752ffd640 1 -- 172.21.15.29:0/2179504171 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x565407f41640 con 0x565407be3d80 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb752ffd640 10 client.18678 handle_command_reply: tid=0 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 client.18678 shutdown 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 -- 172.21.15.29:0/2179504171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c251b0 msgr2=0x565407fd77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c251b0 0x565407fd77b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb76c004760 tx=0x7fb76c0041e0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 -- 172.21.15.29:0/2179504171 shutdown_connections 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565407be3d80 0x565407fe3a30 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c251b0 0x565407fd77b0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c044d0 0x565407fd6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 --2- 172.21.15.29:0/2179504171 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fd6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 -- 172.21.15.29:0/2179504171 >> 172.21.15.29:0/2179504171 conn(0x565407c19300 msgr2=0x565407fc6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb752ffd640 0 client.18678 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 -- 172.21.15.29:0/2179504171 shutdown_connections 2024-09-21T03:03:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 1 -- 172.21.15.29:0/2179504171 wait complete. 2024-09-21T03:03:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.625+0000 7fb780a37740 20 client.18678 trim_cache size 0 max 16384 2024-09-21T03:03:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb780a37740 1 Processor -- start 2024-09-21T03:03:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb780a37740 1 -- start start 2024-09-21T03:03:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb780a37740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407bfc2f0 0x565407fd51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb780a37740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407c044d0 0x565407fc7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb780a37740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fc83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb780a37740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741ac560 con 0x565407bfc2f0 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb780a37740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741ae0a0 con 0x565407c044d0 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb780a37740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741acc30 con 0x565407c251b0 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407bfc2f0 0x565407fd51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407bfc2f0 0x565407fd51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47434/0 (socket says 172.21.15.29:47434) 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 -- 172.21.15.29:0/186955275 learned_addr learned my addr 172.21.15.29:0/186955275 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb772ffd640 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407c044d0 0x565407fc7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fc83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 -- 172.21.15.29:0/186955275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 msgr2=0x565407fc83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fc83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 -- 172.21.15.29:0/186955275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407c044d0 msgr2=0x565407fc7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407c044d0 0x565407fc7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb773fff640 1 -- 172.21.15.29:0/186955275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565407fd6310 con 0x565407bfc2f0 2024-09-21T03:03:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.640+0000 7fb772ffd640 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407c044d0 0x565407fc7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:03:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.641+0000 7fb773fff640 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407bfc2f0 0x565407fd51a0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb76c011ef0 tx=0x7fb76c0021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57664fb29e5cc158 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.641+0000 7fb7697fa640 1 -- 172.21.15.29:0/186955275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c02b070 con 0x565407bfc2f0 2024-09-21T03:03:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.641+0000 7fb7697fa640 1 -- 172.21.15.29:0/186955275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb76c005830 con 0x565407bfc2f0 2024-09-21T03:03:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.641+0000 7fb7697fa640 1 -- 172.21.15.29:0/186955275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0388d0 con 0x565407bfc2f0 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 -- 172.21.15.29:0/186955275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407bfc2f0 msgr2=0x565407fd51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407bfc2f0 0x565407fd51a0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb76c011ef0 tx=0x7fb76c0021d0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 -- 172.21.15.29:0/186955275 shutdown_connections 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fc83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407c044d0 0x565407fc7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 --2- 172.21.15.29:0/186955275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407bfc2f0 0x565407fd51a0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 -- 172.21.15.29:0/186955275 >> 172.21.15.29:0/186955275 conn(0x565407c19300 msgr2=0x565407fd4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 -- 172.21.15.29:0/186955275 shutdown_connections 2024-09-21T03:03:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 -- 172.21.15.29:0/186955275 wait complete. 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 -- start start 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.642+0000 7fb780a37740 1 Processor -- start 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb780a37740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fdb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb780a37740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fdb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb780a37740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fdc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb780a37740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741ad9d0 con 0x565407c044d0 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb780a37740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741ad300 con 0x565407bfc2f0 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb780a37740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741aa540 con 0x565407c251b0 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb780a37740 10 client.0 _ll_register_callbacks cb 0x7fb764005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb772ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fdb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb773fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fdb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb773fff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fdb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:40668/0 (socket says 172.21.15.29:40668) 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb773fff640 1 -- 172.21.15.29:0/2628437381 learned_addr learned my addr 172.21.15.29:0/2628437381 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:03:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb772ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fdb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47438/0 (socket says 172.21.15.29:47438) 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb773fff640 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fdc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb772ffd640 1 -- 172.21.15.29:0/2628437381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 msgr2=0x565407fdc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fdc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb772ffd640 1 -- 172.21.15.29:0/2628437381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 msgr2=0x565407fdb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fdb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb772ffd640 1 -- 172.21.15.29:0/2628437381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565407fdcac0 con 0x565407c044d0 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb773fff640 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fdb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb772ffd640 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fdb860 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0057c0 tx=0x7fb75c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec5ed01ed7f4c44 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb752ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb752ffd640 1 -- 172.21.15.29:0/2628437381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c035040 con 0x565407c044d0 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb752ffd640 1 -- 172.21.15.29:0/2628437381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fb75c031880 con 0x565407c044d0 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.643+0000 7fb780a37740 1 -- 172.21.15.29:0/2628437381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb76c005830 con 0x565407c044d0 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb752ffd640 1 -- 172.21.15.29:0/2628437381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb75c053050 con 0x565407c044d0 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb780a37740 10 client.18684 fetch_fsmap learned FSMap version 39 2024-09-21T03:03:57.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb780a37740 1 -- 172.21.15.29:0/2628437381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x565407fdde80 con 0x565407c044d0 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb752ffd640 1 -- 172.21.15.29:0/2628437381 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb76c005830 con 0x565407c044d0 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb752ffd640 1 -- 172.21.15.29:0/2628437381 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fb75c003c20 con 0x565407c044d0 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb780a37740 10 client.18684 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb780a37740 10 client.18684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb780a37740 20 client.18684 populate_metadata read hostname 'smithi029' 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb780a37740 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565407bb91c0 0x565408050610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb780a37740 4 client.18684 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb780a37740 1 -- 172.21.15.29:0/2628437381 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565407f3d0f0 con 0x565407bb91c0 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.644+0000 7fb773fff640 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565407bb91c0 0x565408050610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.645+0000 7fb752ffd640 10 client.18684 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.645+0000 7fb773fff640 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565407bb91c0 0x565408050610 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb9a176ef48c56e4 server_cookie=464b4516561d890f in_seq=0 out_seq=0 2024-09-21T03:03:57.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72084 5.55 MB/sec execute 574 sec latency 245.266 ms 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.845+0000 7fb752ffd640 1 -- 172.21.15.29:0/2628437381 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565407f3d0f0 con 0x565407bb91c0 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.845+0000 7fb752ffd640 10 client.18684 handle_command_reply: tid=0 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 client.18684 shutdown 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 -- 172.21.15.29:0/2628437381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 msgr2=0x565407fdb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fdb860 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0057c0 tx=0x7fb75c031630 comp rx=0 tx=0).stop 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 -- 172.21.15.29:0/2628437381 shutdown_connections 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565407c251b0 0x565407fdc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565407bb91c0 0x565408050610 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565407c044d0 0x565407fdb860 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 --2- 172.21.15.29:0/2628437381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565407bfc2f0 0x565407fdb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 -- 172.21.15.29:0/2628437381 >> 172.21.15.29:0/2628437381 conn(0x565407c19300 msgr2=0x565407fd4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb752ffd640 0 client.18684 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:03:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 -- 172.21.15.29:0/2628437381 shutdown_connections 2024-09-21T03:03:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 1 -- 172.21.15.29:0/2628437381 wait complete. 2024-09-21T03:03:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.846+0000 7fb780a37740 20 client.18684 trim_cache size 0 max 16384 2024-09-21T03:03:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:03:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:03:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:03:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 -- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7600b6590 msgr2=0x7fb7600b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7600b6590 0x7fb7600b8aa0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb7640aaa80 tx=0x7fb7640eb430 comp rx=0 tx=0).stop 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 -- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb774102ae0 msgr2=0x7fb774102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb774102ae0 0x7fb774102040 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb76400e9f0 tx=0x7fb76400eeb0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 -- 172.21.15.29:0/2028459505 shutdown_connections 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb7600b6590 0x7fb7600b8aa0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7741040d0 0x7fb7741a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7741034c0 0x7fb77419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 --2- 172.21.15.29:0/2028459505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb774102ae0 0x7fb774102040 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:03:57.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 -- 172.21.15.29:0/2028459505 >> 172.21.15.29:0/2028459505 conn(0x7fb7740fd690 msgr2=0x7fb774106b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:03:57.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 -- 172.21.15.29:0/2028459505 shutdown_connections 2024-09-21T03:03:57.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:03:57.847+0000 7fb779ace640 1 -- 172.21.15.29:0/2028459505 wait complete. 2024-09-21T03:03:57.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:57 smithi190 ceph-mon[33871]: pgmap v1135: 129 pgs: 129 active+clean; 336 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-09-21T03:03:57.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:57 smithi029 ceph-mon[31006]: pgmap v1135: 129 pgs: 129 active+clean; 336 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-09-21T03:03:58.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:57 smithi066 ceph-mon[33436]: pgmap v1135: 129 pgs: 129 active+clean; 336 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-09-21T03:03:58.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72230 5.55 MB/sec execute 575 sec latency 321.977 ms 2024-09-21T03:03:59.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:03:59 smithi029 ceph-mon[31006]: pgmap v1136: 129 pgs: 129 active+clean; 341 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-09-21T03:03:59.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72302 5.55 MB/sec execute 576 sec latency 356.743 ms 2024-09-21T03:03:59.812 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:03:59 smithi066 ceph-mon[33436]: pgmap v1136: 129 pgs: 129 active+clean; 341 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-09-21T03:03:59.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:03:59 smithi190 ceph-mon[33871]: pgmap v1136: 129 pgs: 129 active+clean; 341 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-09-21T03:04:00.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72402 5.56 MB/sec execute 577 sec latency 278.987 ms 2024-09-21T03:04:01.568 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:01 smithi029 ceph-mon[31006]: pgmap v1137: 129 pgs: 129 active+clean; 343 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 3.4 MiB/s wr, 283 op/s 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 -- 172.21.15.29:0/2233509891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 msgr2=0x7fce281061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 --2- 172.21.15.29:0/2233509891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 0x7fce281061c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fce18009900 tx=0x7fce1802a410 comp rx=0 tx=0).stop 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 -- 172.21.15.29:0/2233509891 shutdown_connections 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 --2- 172.21.15.29:0/2233509891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2810c040 0x7fce28112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 --2- 172.21.15.29:0/2233509891 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2810b6f0 0x7fce2810bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 --2- 172.21.15.29:0/2233509891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 0x7fce281061c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 -- 172.21.15.29:0/2233509891 >> 172.21.15.29:0/2233509891 conn(0x7fce281017c0 msgr2=0x7fce28103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 -- 172.21.15.29:0/2233509891 shutdown_connections 2024-09-21T03:04:01.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.580+0000 7fce2e355640 1 -- 172.21.15.29:0/2233509891 wait complete. 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.581+0000 7fce2e355640 1 Processor -- start 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.581+0000 7fce2e355640 1 -- start start 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.581+0000 7fce2e355640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 0x7fce2819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.581+0000 7fce2e355640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2810b6f0 0x7fce2819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.581+0000 7fce2e355640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2810c040 0x7fce281a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2e355640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce28115c20 con 0x7fce2810b6f0 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2e355640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fce28115aa0 con 0x7fce2810c040 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2e355640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fce28115da0 con 0x7fce28105db0 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce27fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 0x7fce2819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce27fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 0x7fce2819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45304/0 (socket says 172.21.15.29:45304) 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce27fff640 1 -- 172.21.15.29:0/2751391727 learned_addr learned my addr 172.21.15.29:0/2751391727 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2c8f2640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2810c040 0x7fce281a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce277fe640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2810b6f0 0x7fce2819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2c8f2640 1 -- 172.21.15.29:0/2751391727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 msgr2=0x7fce2819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2c8f2640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 0x7fce2819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2c8f2640 1 -- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2810b6f0 msgr2=0x7fce2819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2c8f2640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2810b6f0 0x7fce2819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.582+0000 7fce2c8f2640 1 -- 172.21.15.29:0/2751391727 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce281a25a0 con 0x7fce2810c040 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.583+0000 7fce2c8f2640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2810c040 0x7fce281a1e70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fce20009fd0 tx=0x7fce2000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.583+0000 7fce257fa640 1 -- 172.21.15.29:0/2751391727 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2000ce20 con 0x7fce2810c040 2024-09-21T03:04:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.583+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce281a2890 con 0x7fce2810c040 2024-09-21T03:04:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.583+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce281a8ff0 con 0x7fce2810c040 2024-09-21T03:04:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.583+0000 7fce257fa640 1 -- 172.21.15.29:0/2751391727 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce20005020 con 0x7fce2810c040 2024-09-21T03:04:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.584+0000 7fce257fa640 1 -- 172.21.15.29:0/2751391727 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce20010690 con 0x7fce2810c040 2024-09-21T03:04:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.585+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce140052c0 con 0x7fce2810c040 2024-09-21T03:04:01.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.586+0000 7fce257fa640 1 -- 172.21.15.29:0/2751391727 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fce20004b20 con 0x7fce2810c040 2024-09-21T03:04:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.587+0000 7fce257fa640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce100b6590 0x7fce100b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.587+0000 7fce257fa640 1 -- 172.21.15.29:0/2751391727 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fce200e7aa0 con 0x7fce2810c040 2024-09-21T03:04:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.588+0000 7fce27fff640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce100b6590 0x7fce100b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.588+0000 7fce27fff640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce100b6590 0x7fce100b8aa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fce180096c0 tx=0x7fce18005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.592+0000 7fce257fa640 1 -- 172.21.15.29:0/2751391727 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fce200adfb0 con 0x7fce2810c040 2024-09-21T03:04:01.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72516 5.56 MB/sec execute 578 sec latency 86.754 ms 2024-09-21T03:04:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:01 smithi190 ceph-mon[33871]: pgmap v1137: 129 pgs: 129 active+clean; 343 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 3.4 MiB/s wr, 283 op/s 2024-09-21T03:04:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.935+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce140055b0 con 0x7fce2810c040 2024-09-21T03:04:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.936+0000 7fce257fa640 1 -- 172.21.15.29:0/2751391727 <== mon.2 v2:172.21.15.66:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11971 (secure 0 0 0) 0x7fce200b2390 con 0x7fce2810c040 2024-09-21T03:04:01.939 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:04:01.940 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"btime":"2024-09-21T03:02:24:667358+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:02:24.667336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:04:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-09-21T03:04:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce100b6590 msgr2=0x7fce100b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce100b6590 0x7fce100b8aa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fce180096c0 tx=0x7fce18005e80 comp rx=0 tx=0).stop 2024-09-21T03:04:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2810c040 msgr2=0x7fce281a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2810c040 0x7fce281a1e70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fce20009fd0 tx=0x7fce2000ef10 comp rx=0 tx=0).stop 2024-09-21T03:04:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 shutdown_connections 2024-09-21T03:04:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fce100b6590 0x7fce100b8aa0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce2810c040 0x7fce281a1e70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce2810b6f0 0x7fce2819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 --2- 172.21.15.29:0/2751391727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce28105db0 0x7fce2819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 >> 172.21.15.29:0/2751391727 conn(0x7fce281017c0 msgr2=0x7fce28109ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 shutdown_connections 2024-09-21T03:04:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:01.942+0000 7fce2e355640 1 -- 172.21.15.29:0/2751391727 wait complete. 2024-09-21T03:04:02.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:01 smithi066 ceph-mon[33436]: pgmap v1137: 129 pgs: 129 active+clean; 343 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 3.4 MiB/s wr, 283 op/s 2024-09-21T03:04:02.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:02 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2751391727' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:02.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72620 5.56 MB/sec execute 579 sec latency 109.980 ms 2024-09-21T03:04:02.786 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 139.8 --await-for 45.0' 2024-09-21T03:04:02.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 139.8 --await-for 45.0 2024-09-21T03:04:02.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:02 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2751391727' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:03.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:02 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2751391727' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:03.156 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:03.542 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:03 smithi029 ceph-mon[31006]: pgmap v1138: 129 pgs: 129 active+clean; 345 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-09-21T03:04:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 -- 172.21.15.29:0/97181320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74201083c0 msgr2=0x7f74201087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 --2- 172.21.15.29:0/97181320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74201083c0 0x7f74201087d0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f741000d910 tx=0x7f7410032980 comp rx=0 tx=0).stop 2024-09-21T03:04:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 -- 172.21.15.29:0/97181320 shutdown_connections 2024-09-21T03:04:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 --2- 172.21.15.29:0/97181320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7420108d10 0x7f742010f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 --2- 172.21.15.29:0/97181320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74201083c0 0x7f74201087d0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 --2- 172.21.15.29:0/97181320 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7420102a80 0x7f7420102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 -- 172.21.15.29:0/97181320 >> 172.21.15.29:0/97181320 conn(0x7f7420079760 msgr2=0x7f7420079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 -- 172.21.15.29:0/97181320 shutdown_connections 2024-09-21T03:04:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.743+0000 7f7424aef640 1 -- 172.21.15.29:0/97181320 wait complete. 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f7424aef640 1 Processor -- start 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f7424aef640 1 -- start start 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f7424aef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7420102a80 0x7f7420074dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f7424aef640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74201083c0 0x7f742006f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f7424aef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7420108d10 0x7f742006f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f7424aef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f742006d500 con 0x7f7420108d10 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f7424aef640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f742006d380 con 0x7f74201083c0 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f7424aef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f742006d680 con 0x7f7420102a80 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f741e59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7420102a80 0x7f7420074dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f741e59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7420102a80 0x7f7420074dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45314/0 (socket says 172.21.15.29:45314) 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f741e59c640 1 -- 172.21.15.29:0/2359500390 learned_addr learned my addr 172.21.15.29:0/2359500390 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.745+0000 7f741dd9b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74201083c0 0x7f742006f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f741ed9d640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7420108d10 0x7f742006f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f741e59c640 1 -- 172.21.15.29:0/2359500390 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74201083c0 msgr2=0x7f742006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f741e59c640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74201083c0 0x7f742006f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f741e59c640 1 -- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7420108d10 msgr2=0x7f742006f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f741e59c640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7420108d10 0x7f742006f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f741e59c640 1 -- 172.21.15.29:0/2359500390 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742006fda0 con 0x7f7420102a80 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f741e59c640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7420102a80 0x7f7420074dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f74140099f0 tx=0x7f7414009eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f740f7fe640 1 -- 172.21.15.29:0/2359500390 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741401c070 con 0x7f7420102a80 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f7424aef640 1 -- 172.21.15.29:0/2359500390 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74201acda0 con 0x7f7420102a80 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f740f7fe640 1 -- 172.21.15.29:0/2359500390 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f741400cc60 con 0x7f7420102a80 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f740f7fe640 1 -- 172.21.15.29:0/2359500390 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140165c0 con 0x7f7420102a80 2024-09-21T03:04:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.746+0000 7f7424aef640 1 -- 172.21.15.29:0/2359500390 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74201ad200 con 0x7f7420102a80 2024-09-21T03:04:03.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72782 5.58 MB/sec execute 580 sec latency 114.676 ms 2024-09-21T03:04:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.750+0000 7f742b063740 1 Processor -- start 2024-09-21T03:04:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.750+0000 7f740f7fe640 1 -- 172.21.15.29:0/2359500390 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f741401f070 con 0x7f7420102a80 2024-09-21T03:04:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.750+0000 7f742b063740 1 -- start start 2024-09-21T03:04:03.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.750+0000 7f742b063740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x5560642f5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f742b063740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560642f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f742b063740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f742b063740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55606426bb90 con 0x556063f25980 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f742b063740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55606425afc0 con 0x556063f4e840 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f742b063740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55606425edd0 con 0x556063f2db60 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f741dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f741ed9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x5560642f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f741dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45318/0 (socket says 172.21.15.29:45318) 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f741dd9b640 1 -- 172.21.15.29:0/2144256923 learned_addr learned my addr 172.21.15.29:0/2144256923 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f741e59c640 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560642f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.751+0000 7f740f7fe640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f74080b6590 0x7f74080b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f740f7fe640 1 -- 172.21.15.29:0/2359500390 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f74140e7310 con 0x7f7420102a80 2024-09-21T03:04:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f741dd9b640 1 -- 172.21.15.29:0/2144256923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 msgr2=0x5560642f5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f741dd9b640 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x5560642f5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f741dd9b640 1 -- 172.21.15.29:0/2144256923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 msgr2=0x5560642f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f741dd9b640 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560642f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f741dd9b640 1 -- 172.21.15.29:0/2144256923 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560641e2060 con 0x556063f2db60 2024-09-21T03:04:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f741e59c640 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560642f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f741ed9d640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f74080b6590 0x7f74080b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f741dd9b640 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f6080 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f742006a890 tx=0x7f7410005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3a1def4a1e54bce server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.752+0000 7f740cff9640 1 -- 172.21.15.29:0/2144256923 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741004f070 con 0x556063f2db60 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.753+0000 7f740cff9640 1 -- 172.21.15.29:0/2144256923 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f741000d530 con 0x556063f2db60 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.753+0000 7f740cff9640 1 -- 172.21.15.29:0/2144256923 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74100418e0 con 0x556063f2db60 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 -- 172.21.15.29:0/2144256923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 msgr2=0x5560642f6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f6080 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f742006a890 tx=0x7f7410005d50 comp rx=0 tx=0).stop 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f741ed9d640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f74080b6590 0x7f74080b8aa0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7418005fd0 tx=0x7f7418014040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 -- 172.21.15.29:0/2144256923 shutdown_connections 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f6080 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560642f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 --2- 172.21.15.29:0/2144256923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x5560642f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 -- 172.21.15.29:0/2144256923 >> 172.21.15.29:0/2144256923 conn(0x556063f42990 msgr2=0x5560642f34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 -- 172.21.15.29:0/2144256923 shutdown_connections 2024-09-21T03:04:03.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.754+0000 7f742b063740 1 -- 172.21.15.29:0/2144256923 wait complete. 2024-09-21T03:04:03.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.755+0000 7f742b063740 1 -- start start 2024-09-21T03:04:03.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.755+0000 7f742b063740 1 Processor -- start 2024-09-21T03:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.755+0000 7f742b063740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560643011c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.755+0000 7f742b063740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f2db60 0x556064301700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f742b063740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f4e840 0x55606430d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f742b063740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5560640bc7a0 con 0x556063f25980 2024-09-21T03:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f742b063740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5560642000a0 con 0x556063f2db60 2024-09-21T03:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f742b063740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556064303d80 con 0x556063f4e840 2024-09-21T03:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560643011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560643011c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47486/0 (socket says 172.21.15.29:47486) 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741e59c640 1 -- 172.21.15.29:0/1800161928 learned_addr learned my addr 172.21.15.29:0/1800161928 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f742b063740 10 client.0 _ll_register_callbacks cb 0x5560641dd3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741ed9d640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f4e840 0x55606430d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741dd9b640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f2db60 0x556064301700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741ed9d640 1 -- 172.21.15.29:0/1800161928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f2db60 msgr2=0x556064301700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741ed9d640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f2db60 0x556064301700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741ed9d640 1 -- 172.21.15.29:0/1800161928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 msgr2=0x5560643011c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741ed9d640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560643011c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741ed9d640 1 -- 172.21.15.29:0/1800161928 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560641ee880 con 0x556063f4e840 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741dd9b640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f2db60 0x556064301700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741e59c640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560643011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f741ed9d640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f4e840 0x55606430d070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7418010800 tx=0x7f7418009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b2682f74e28489c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.756+0000 7f7405ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.757+0000 7f7405ffb640 1 -- 172.21.15.29:0/1800161928 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741801c890 con 0x556063f4e840 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.757+0000 7f7405ffb640 1 -- 172.21.15.29:0/1800161928 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f741801ca30 con 0x556063f4e840 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.757+0000 7f742b063740 1 -- 172.21.15.29:0/1800161928 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f741000d530 con 0x556063f4e840 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.757+0000 7f7405ffb640 1 -- 172.21.15.29:0/1800161928 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f741801cc30 con 0x556063f4e840 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.757+0000 7f742b063740 10 client.28120 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.757+0000 7f742b063740 1 -- 172.21.15.29:0/1800161928 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x556063fc8b90 con 0x556063f4e840 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.757+0000 7f7405ffb640 1 -- 172.21.15.29:0/1800161928 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f741000d530 con 0x556063f4e840 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f7405ffb640 1 -- 172.21.15.29:0/1800161928 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f7418025020 con 0x556063f4e840 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f742b063740 10 client.28120 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f742b063740 10 client.28120 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f742b063740 20 client.28120 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f742b063740 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556063f0d1c0 0x556064313200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f742b063740 4 client.28120 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f742b063740 1 -- 172.21.15.29:0/1800161928 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55606426dae0 con 0x556063f0d1c0 2024-09-21T03:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f741e59c640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556063f0d1c0 0x556064313200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f7405ffb640 10 client.28120 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.758+0000 7f741e59c640 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556063f0d1c0 0x556064313200 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a46e5ec182671e4a server_cookie=17f03b36d9beded in_seq=0 out_seq=0 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f7405ffb640 1 -- 172.21.15.29:0/1800161928 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55606426dae0 con 0x556063f0d1c0 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f7405ffb640 10 client.28120 handle_command_reply: tid=0 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 client.28120 shutdown 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 -- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f4e840 msgr2=0x55606430d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f4e840 0x55606430d070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7418010800 tx=0x7f7418009500 comp rx=0 tx=0).stop 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 -- 172.21.15.29:0/1800161928 shutdown_connections 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f4e840 0x55606430d070 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f2db60 0x556064301700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556063f0d1c0 0x556064313200 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 --2- 172.21.15.29:0/1800161928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x5560643011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 -- 172.21.15.29:0/1800161928 >> 172.21.15.29:0/1800161928 conn(0x556063f42990 msgr2=0x5560642f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f7405ffb640 0 client.28120 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 -- 172.21.15.29:0/1800161928 shutdown_connections 2024-09-21T03:04:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.760+0000 7f742b063740 1 -- 172.21.15.29:0/1800161928 wait complete. 2024-09-21T03:04:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.761+0000 7f742b063740 20 client.28120 trim_cache size 0 max 16384 2024-09-21T03:04:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.784+0000 7f742b063740 1 Processor -- start 2024-09-21T03:04:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.784+0000 7f742b063740 1 -- start start 2024-09-21T03:04:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f742b063740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f25980 0x5560642f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f742b063740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f742b063740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f4e840 0x5560642fe1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f742b063740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b6f40 con 0x556063f4e840 2024-09-21T03:04:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f742b063740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b8a80 con 0x556063f25980 2024-09-21T03:04:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f742b063740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b7610 con 0x556063f2db60 2024-09-21T03:04:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741e59c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f25980 0x5560642f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f4e840 0x5560642fe1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741e59c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f25980 0x5560642f2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:40726/0 (socket says 172.21.15.29:40726) 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f4e840 0x5560642fe1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47494/0 (socket says 172.21.15.29:47494) 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741e59c640 1 -- 172.21.15.29:0/3981578330 learned_addr learned my addr 172.21.15.29:0/3981578330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741dd9b640 1 --2- 172.21.15.29:0/3981578330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741e59c640 1 -- 172.21.15.29:0/3981578330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 msgr2=0x5560642f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741e59c640 1 --2- 172.21.15.29:0/3981578330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741e59c640 1 -- 172.21.15.29:0/3981578330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f4e840 msgr2=0x5560642fe1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741e59c640 1 --2- 172.21.15.29:0/3981578330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f4e840 0x5560642fe1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.785+0000 7f741e59c640 1 -- 172.21.15.29:0/3981578330 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560642f90c0 con 0x556063f25980 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.786+0000 7f741e59c640 1 --2- 172.21.15.29:0/3981578330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f25980 0x5560642f2aa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74140099c0 tx=0x7f74140e4900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d64e67381180747 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.786+0000 7f7407fff640 1 -- 172.21.15.29:0/3981578330 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741400ce10 con 0x556063f25980 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.786+0000 7f7407fff640 1 -- 172.21.15.29:0/3981578330 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f74140b1ba0 con 0x556063f25980 2024-09-21T03:04:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.786+0000 7f7407fff640 1 -- 172.21.15.29:0/3981578330 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140a8d20 con 0x556063f25980 2024-09-21T03:04:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 -- 172.21.15.29:0/3981578330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f25980 msgr2=0x5560642f2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 --2- 172.21.15.29:0/3981578330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f25980 0x5560642f2aa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74140099c0 tx=0x7f74140e4900 comp rx=0 tx=0).stop 2024-09-21T03:04:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 -- 172.21.15.29:0/3981578330 shutdown_connections 2024-09-21T03:04:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 --2- 172.21.15.29:0/3981578330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f4e840 0x5560642fe1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 --2- 172.21.15.29:0/3981578330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x5560642f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 --2- 172.21.15.29:0/3981578330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f25980 0x5560642f2aa0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 -- 172.21.15.29:0/3981578330 >> 172.21.15.29:0/3981578330 conn(0x556063f42990 msgr2=0x5560643024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 -- 172.21.15.29:0/3981578330 shutdown_connections 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.787+0000 7f742b063740 1 -- 172.21.15.29:0/3981578330 wait complete. 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 1 -- start start 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 1 Processor -- start 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x556064309960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x556064305e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x556064306510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b83b0 con 0x556063f25980 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b7ce0 con 0x556063f4e840 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b4f20 con 0x556063f2db60 2024-09-21T03:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f742b063740 10 client.0 _ll_register_callbacks cb 0x7f7418009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f741dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x556064305e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f741ed9d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x556064306510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f741dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x556064305e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45334/0 (socket says 172.21.15.29:45334) 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.788+0000 7f741dd9b640 1 -- 172.21.15.29:0/3854147268 learned_addr learned my addr 172.21.15.29:0/3854147268 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f741e59c640 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x556064309960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f741ed9d640 1 -- 172.21.15.29:0/3854147268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 msgr2=0x556064305e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f741ed9d640 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x556064305e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f741ed9d640 1 -- 172.21.15.29:0/3854147268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 msgr2=0x556064309960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f741ed9d640 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x556064309960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f741ed9d640 1 -- 172.21.15.29:0/3854147268 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55606437daa0 con 0x556063f4e840 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f741ed9d640 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x556064306510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7418027f60 tx=0x7f7418004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30160602ab71b8d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f7405ffb640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:04:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f7405ffb640 1 -- 172.21.15.29:0/3854147268 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418039060 con 0x556063f4e840 2024-09-21T03:04:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f7405ffb640 1 -- 172.21.15.29:0/3854147268 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f741801d5d0 con 0x556063f4e840 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.789+0000 7f742b063740 1 -- 172.21.15.29:0/3854147268 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74140b1ba0 con 0x556063f4e840 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f7405ffb640 1 -- 172.21.15.29:0/3854147268 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7418010cc0 con 0x556063f4e840 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f742b063740 10 client.27968 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f742b063740 1 -- 172.21.15.29:0/3854147268 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55606437ebd0 con 0x556063f4e840 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f7405ffb640 1 -- 172.21.15.29:0/3854147268 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7418004e10 con 0x556063f4e840 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f7405ffb640 1 -- 172.21.15.29:0/3854147268 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f7418025040 con 0x556063f4e840 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f742b063740 10 client.27968 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f742b063740 10 client.27968 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f742b063740 20 client.27968 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f742b063740 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556063ee2570 0x556064381400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f742b063740 4 client.27968 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.8, "await_for": 45.0}] 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.790+0000 7f742b063740 1 -- 172.21.15.29:0/3854147268 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.8, "await_for": 45.0}) -- 0x7f74140b1ba0 con 0x556063ee2570 2024-09-21T03:04:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.791+0000 7f741e59c640 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556063ee2570 0x556064381400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:03.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.791+0000 7f7405ffb640 10 client.27968 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:03.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:03.791+0000 7f741e59c640 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556063ee2570 0x556064381400 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcb99ca388d1c089 server_cookie=7ecc353e264829e0 in_seq=0 out_seq=0 2024-09-21T03:04:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:03 smithi190 ceph-mon[33871]: pgmap v1138: 129 pgs: 129 active+clean; 345 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-09-21T03:04:04.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:03 smithi066 ceph-mon[33436]: pgmap v1138: 129 pgs: 129 active+clean; 345 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-09-21T03:04:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f7405ffb640 1 -- 172.21.15.29:0/3854147268 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f74140b1ba0 con 0x556063ee2570 2024-09-21T03:04:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f7405ffb640 10 client.27968 handle_command_reply: tid=0 2024-09-21T03:04:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f742b063740 1 client.27968 shutdown 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f742b063740 1 -- 172.21.15.29:0/3854147268 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 msgr2=0x556064306510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f742b063740 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x556064306510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7418027f60 tx=0x7f7418004530 comp rx=0 tx=0).stop 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f742b063740 1 -- 172.21.15.29:0/3854147268 shutdown_connections 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f742b063740 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x556063f4e840 0x556064306510 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f742b063740 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556063f2db60 0x556064305e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.339+0000 7f742b063740 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x556063ee2570 0x556064381400 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f742b063740 1 --2- 172.21.15.29:0/3854147268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556063f25980 0x556064309960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f7405ffb640 0 client.27968 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f742b063740 1 -- 172.21.15.29:0/3854147268 >> 172.21.15.29:0/3854147268 conn(0x556063f42990 msgr2=0x5560642f6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f742b063740 1 -- 172.21.15.29:0/3854147268 shutdown_connections 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f742b063740 1 -- 172.21.15.29:0/3854147268 wait complete. 2024-09-21T03:04:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f742b063740 20 client.27968 trim_cache size 0 max 16384 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 39, 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 94, 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "fba12f0e": { 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 94, 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:04:04.343 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.8, 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:04.344 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:04.345 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:04.345 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f7424aef640 1 -- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f74080b6590 msgr2=0x7f74080b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f7424aef640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f74080b6590 0x7f74080b8aa0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7418005fd0 tx=0x7f7418014040 comp rx=0 tx=0).stop 2024-09-21T03:04:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f7424aef640 1 -- 172.21.15.29:0/2359500390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7420102a80 msgr2=0x7f7420074dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.340+0000 7f7424aef640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7420102a80 0x7f7420074dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f74140099f0 tx=0x7f7414009eb0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.341+0000 7f7424aef640 1 -- 172.21.15.29:0/2359500390 shutdown_connections 2024-09-21T03:04:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.341+0000 7f7424aef640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f74080b6590 0x7f74080b8aa0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.341+0000 7f7424aef640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7420108d10 0x7f742006f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.341+0000 7f7424aef640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74201083c0 0x7f742006f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.341+0000 7f7424aef640 1 --2- 172.21.15.29:0/2359500390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7420102a80 0x7f7420074dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.341+0000 7f7424aef640 1 -- 172.21.15.29:0/2359500390 >> 172.21.15.29:0/2359500390 conn(0x7f7420079760 msgr2=0x7f74201044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.341+0000 7f7424aef640 1 -- 172.21.15.29:0/2359500390 shutdown_connections 2024-09-21T03:04:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:04.341+0000 7f7424aef640 1 -- 172.21.15.29:0/2359500390 wait complete. 2024-09-21T03:04:04.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.58 MB/sec execute 581 sec latency 967.578 ms 2024-09-21T03:04:05.179 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'fba12f0e', quiesce took 9.1 seconds. Will release after: 10.700000000000001 2024-09-21T03:04:05.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.57 MB/sec execute 582 sec latency 1967.648 ms 2024-09-21T03:04:05.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:05 smithi190 ceph-mon[33871]: pgmap v1139: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.7 MiB/s wr, 247 op/s 2024-09-21T03:04:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:05 smithi029 ceph-mon[31006]: pgmap v1139: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.7 MiB/s wr, 247 op/s 2024-09-21T03:04:06.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:05 smithi066 ceph-mon[33436]: pgmap v1139: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.7 MiB/s wr, 247 op/s 2024-09-21T03:04:06.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.56 MB/sec execute 583 sec latency 2967.743 ms 2024-09-21T03:04:07.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.55 MB/sec execute 584 sec latency 3967.853 ms 2024-09-21T03:04:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:07 smithi190 ceph-mon[33871]: pgmap v1140: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.5 MiB/s wr, 235 op/s 2024-09-21T03:04:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:07 smithi029 ceph-mon[31006]: pgmap v1140: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.5 MiB/s wr, 235 op/s 2024-09-21T03:04:08.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:07 smithi066 ceph-mon[33436]: pgmap v1140: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.5 MiB/s wr, 235 op/s 2024-09-21T03:04:08.751 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.54 MB/sec execute 585 sec latency 4967.967 ms 2024-09-21T03:04:09.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.53 MB/sec execute 586 sec latency 5968.097 ms 2024-09-21T03:04:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:09 smithi190 ceph-mon[33871]: pgmap v1141: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-21T03:04:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:09 smithi029 ceph-mon[31006]: pgmap v1141: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-21T03:04:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:09 smithi066 ceph-mon[33436]: pgmap v1141: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-21T03:04:10.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.52 MB/sec execute 587 sec latency 6968.199 ms 2024-09-21T03:04:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 -- 172.21.15.29:0/4088797320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 msgr2=0x7f7e3c10baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 --2- 172.21.15.29:0/4088797320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 0x7f7e3c10baa0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009ca0 tx=0x7f7e2c02a8c0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 -- 172.21.15.29:0/4088797320 shutdown_connections 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 --2- 172.21.15.29:0/4088797320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 0x7f7e3c10baa0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 --2- 172.21.15.29:0/4088797320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c104760 0x7f7e3c104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 --2- 172.21.15.29:0/4088797320 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e3c1036e0 0x7f7e3c103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 -- 172.21.15.29:0/4088797320 >> 172.21.15.29:0/4088797320 conn(0x7f7e3c0ff020 msgr2=0x7f7e3c1014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 -- 172.21.15.29:0/4088797320 shutdown_connections 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.965+0000 7f7e41815640 1 -- 172.21.15.29:0/4088797320 wait complete. 2024-09-21T03:04:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e41815640 1 Processor -- start 2024-09-21T03:04:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e41815640 1 -- start start 2024-09-21T03:04:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e41815640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c1036e0 0x7f7e3c076e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e41815640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e3c104760 0x7f7e3c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e41815640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 0x7f7e3c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e41815640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c10f170 con 0x7f7e3c105100 2024-09-21T03:04:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e41815640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c10eff0 con 0x7f7e3c104760 2024-09-21T03:04:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e41815640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c10f2f0 con 0x7f7e3c1036e0 2024-09-21T03:04:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 0x7f7e3c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 0x7f7e3c075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60392/0 (socket says 172.21.15.29:60392) 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3b7fe640 1 -- 172.21.15.29:0/743817658 learned_addr learned my addr 172.21.15.29:0/743817658 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3affd640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c1036e0 0x7f7e3c076e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e3c104760 0x7f7e3c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3b7fe640 1 -- 172.21.15.29:0/743817658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c1036e0 msgr2=0x7f7e3c076e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3b7fe640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c1036e0 0x7f7e3c076e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3b7fe640 1 -- 172.21.15.29:0/743817658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e3c104760 msgr2=0x7f7e3c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.966+0000 7f7e3b7fe640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e3c104760 0x7f7e3c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.967+0000 7f7e3b7fe640 1 -- 172.21.15.29:0/743817658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c0762f0 con 0x7f7e3c105100 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.967+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e3c104760 0x7f7e3c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.967+0000 7f7e3b7fe640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 0x7f7e3c075c50 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009dc0 tx=0x7f7e2c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.967+0000 7f7e40813640 1 -- 172.21.15.29:0/743817658 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c004240 con 0x7f7e3c105100 2024-09-21T03:04:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.967+0000 7f7e40813640 1 -- 172.21.15.29:0/743817658 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e2c040030 con 0x7f7e3c105100 2024-09-21T03:04:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.967+0000 7f7e40813640 1 -- 172.21.15.29:0/743817658 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c03b430 con 0x7f7e3c105100 2024-09-21T03:04:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.967+0000 7f7e41815640 1 -- 172.21.15.29:0/743817658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3c1a3f10 con 0x7f7e3c105100 2024-09-21T03:04:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.967+0000 7f7e41815640 1 -- 172.21.15.29:0/743817658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3c1a42c0 con 0x7f7e3c105100 2024-09-21T03:04:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.970+0000 7f7e48862740 1 Processor -- start 2024-09-21T03:04:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e40813640 1 -- 172.21.15.29:0/743817658 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f7e2c0043e0 con 0x7f7e3c105100 2024-09-21T03:04:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e48862740 1 -- start start 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e48862740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd455ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e48862740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd455f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e48862740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 0x55fcd4560010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e48862740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fcd44d16f0 con 0x55fcd418e2f0 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e48862740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55fcd44c0b20 con 0x55fcd41b71b0 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e48862740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55fcd44c4930 con 0x55fcd41964d0 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd455f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd455f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60394/0 (socket says 172.21.15.29:60394) 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3affd640 1 -- 172.21.15.29:0/2215750374 learned_addr learned my addr 172.21.15.29:0/2215750374 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 0x55fcd4560010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd455ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3affd640 1 -- 172.21.15.29:0/2215750374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 msgr2=0x55fcd4560010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3affd640 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 0x55fcd4560010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3affd640 1 -- 172.21.15.29:0/2215750374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 msgr2=0x55fcd455ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3affd640 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd455ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.971+0000 7f7e3affd640 1 -- 172.21.15.29:0/2215750374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd4486530 con 0x55fcd418e2f0 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.972+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd455ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.972+0000 7f7e3affd640 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd455f4c0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400cc60 tx=0x7f7e3400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2a27d2969295aa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.972+0000 7f7e31ffb640 1 -- 172.21.15.29:0/2215750374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34014070 con 0x55fcd418e2f0 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.972+0000 7f7e40813640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e280b6510 0x7f7e280b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.972+0000 7f7e31ffb640 1 -- 172.21.15.29:0/2215750374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e340047a0 con 0x55fcd418e2f0 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.972+0000 7f7e31ffb640 1 -- 172.21.15.29:0/2215750374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34010690 con 0x55fcd418e2f0 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.972+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e280b6510 0x7f7e280b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.972+0000 7f7e40813640 1 -- 172.21.15.29:0/743817658 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f7e2c1085b0 con 0x7f7e3c105100 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 -- 172.21.15.29:0/2215750374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 msgr2=0x55fcd455f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e280b6510 0x7f7e280b8a20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7e240058c0 tx=0x7f7e24005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd455f4c0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400cc60 tx=0x7f7e3400e720 comp rx=0 tx=0).stop 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 -- 172.21.15.29:0/2215750374 shutdown_connections 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 0x55fcd4560010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd455f4c0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 --2- 172.21.15.29:0/2215750374 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd455ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 -- 172.21.15.29:0/2215750374 >> 172.21.15.29:0/2215750374 conn(0x55fcd41ab300 msgr2=0x55fcd455a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 -- 172.21.15.29:0/2215750374 shutdown_connections 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 -- 172.21.15.29:0/2215750374 wait complete. 2024-09-21T03:04:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.973+0000 7f7e48862740 1 -- start start 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e48862740 1 Processor -- start 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e48862740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd4568600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e48862740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 0x55fcd4568cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e48862740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd4569830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e48862740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fcd43f8ab0 con 0x55fcd418e2f0 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e48862740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55fcd4466730 con 0x55fcd41b71b0 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e48862740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55fcd4560b30 con 0x55fcd41964d0 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e48862740 10 client.0 _ll_register_callbacks cb 0x55fcd44ba6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd4568600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd4568600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60410/0 (socket says 172.21.15.29:60410) 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e3affd640 1 -- 172.21.15.29:0/4114483201 learned_addr learned my addr 172.21.15.29:0/4114483201 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e3b7fe640 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd4569830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.974+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 0x55fcd4568cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e3affd640 1 -- 172.21.15.29:0/4114483201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 msgr2=0x55fcd4568cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e3affd640 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 0x55fcd4568cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e3affd640 1 -- 172.21.15.29:0/4114483201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 msgr2=0x55fcd4569830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e3affd640 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd4569830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e3affd640 1 -- 172.21.15.29:0/4114483201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd4222790 con 0x55fcd418e2f0 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e3affd640 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd4568600 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006010 tx=0x7f7e340051b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c901570fca06b35 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e1affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e1affd640 1 -- 172.21.15.29:0/4114483201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34014070 con 0x55fcd418e2f0 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e1affd640 1 -- 172.21.15.29:0/4114483201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f7e34004220 con 0x55fcd418e2f0 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e48862740 1 -- 172.21.15.29:0/4114483201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e340047a0 con 0x55fcd418e2f0 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e1affd640 1 -- 172.21.15.29:0/4114483201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7e3402d050 con 0x55fcd418e2f0 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e48862740 10 client.18717 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e48862740 1 -- 172.21.15.29:0/4114483201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55fcd429b620 con 0x55fcd418e2f0 2024-09-21T03:04:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.975+0000 7f7e1affd640 1 -- 172.21.15.29:0/4114483201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e340047a0 con 0x55fcd418e2f0 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.976+0000 7f7e1affd640 1 -- 172.21.15.29:0/4114483201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f7e34022460 con 0x55fcd418e2f0 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.976+0000 7f7e48862740 10 client.18717 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.976+0000 7f7e48862740 10 client.18717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.976+0000 7f7e48862740 20 client.18717 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.976+0000 7f7e48862740 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fcd4175d80 0x55fcd4575990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.976+0000 7f7e48862740 4 client.18717 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.976+0000 7f7e48862740 1 -- 172.21.15.29:0/4114483201 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcd44d3640 con 0x55fcd4175d80 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.976+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fcd4175d80 0x55fcd4575990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.977+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fcd4175d80 0x55fcd4575990 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db61b1b96832bda4 server_cookie=26eb37f59da659 in_seq=0 out_seq=0 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.977+0000 7f7e1affd640 10 client.18717 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.977+0000 7f7e1affd640 1 -- 172.21.15.29:0/4114483201 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fcd44d3640 con 0x55fcd4175d80 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.977+0000 7f7e1affd640 10 client.18717 handle_command_reply: tid=0 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.977+0000 7f7e48862740 1 client.18717 shutdown 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 -- 172.21.15.29:0/4114483201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 msgr2=0x55fcd4568600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd4568600 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006010 tx=0x7f7e340051b0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 -- 172.21.15.29:0/4114483201 shutdown_connections 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41b71b0 0x55fcd4569830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41964d0 0x55fcd4568cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fcd4175d80 0x55fcd4575990 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 --2- 172.21.15.29:0/4114483201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd4568600 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 -- 172.21.15.29:0/4114483201 >> 172.21.15.29:0/4114483201 conn(0x55fcd41ab300 msgr2=0x55fcd455fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e1affd640 0 client.18717 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 -- 172.21.15.29:0/4114483201 shutdown_connections 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 1 -- 172.21.15.29:0/4114483201 wait complete. 2024-09-21T03:04:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.978+0000 7f7e48862740 20 client.18717 trim_cache size 0 max 16384 2024-09-21T03:04:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e48862740 1 Processor -- start 2024-09-21T03:04:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e48862740 1 -- start start 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e48862740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd418e2f0 0x55fcd4567220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e48862740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd4570710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e48862740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd41b71b0 0x55fcd4560f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e48862740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c1ae740 con 0x55fcd41b71b0 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e48862740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c1b0280 con 0x55fcd41964d0 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e48862740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c1aee10 con 0x55fcd418e2f0 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd41b71b0 0x55fcd4560f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd41b71b0 0x55fcd4560f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60412/0 (socket says 172.21.15.29:60412) 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e3b7fe640 1 -- 172.21.15.29:0/3678240236 learned_addr learned my addr 172.21.15.29:0/3678240236 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.993+0000 7f7e3affd640 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd418e2f0 0x55fcd4567220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd4570710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e3b7fe640 1 -- 172.21.15.29:0/3678240236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd418e2f0 msgr2=0x55fcd4567220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e3b7fe640 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd418e2f0 0x55fcd4567220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e3b7fe640 1 -- 172.21.15.29:0/3678240236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 msgr2=0x55fcd4570710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e3b7fe640 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd4570710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e3b7fe640 1 -- 172.21.15.29:0/3678240236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd455fc90 con 0x55fcd41b71b0 2024-09-21T03:04:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd4570710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e3b7fe640 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd41b71b0 0x55fcd4560f30 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009a60 tx=0x7f7e2c0305c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdae1aa78bface62 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e317fa640 1 -- 172.21.15.29:0/3678240236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c04a040 con 0x55fcd41b71b0 2024-09-21T03:04:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e317fa640 1 -- 172.21.15.29:0/3678240236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e2c036070 con 0x55fcd41b71b0 2024-09-21T03:04:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.994+0000 7f7e317fa640 1 -- 172.21.15.29:0/3678240236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c0ca630 con 0x55fcd41b71b0 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 -- 172.21.15.29:0/3678240236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd41b71b0 msgr2=0x55fcd4560f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd41b71b0 0x55fcd4560f30 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009a60 tx=0x7f7e2c0305c0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 -- 172.21.15.29:0/3678240236 shutdown_connections 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd41b71b0 0x55fcd4560f30 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd4570710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 --2- 172.21.15.29:0/3678240236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd418e2f0 0x55fcd4567220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 -- 172.21.15.29:0/3678240236 >> 172.21.15.29:0/3678240236 conn(0x55fcd41ab300 msgr2=0x55fcd4566b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 -- 172.21.15.29:0/3678240236 shutdown_connections 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 -- 172.21.15.29:0/3678240236 wait complete. 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 -- start start 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.995+0000 7f7e48862740 1 Processor -- start 2024-09-21T03:04:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e48862740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd456d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e48862740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd456dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e48862740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41b71b0 0x55fcd456e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e48862740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c1afbb0 con 0x55fcd418e2f0 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e48862740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c1af4e0 con 0x55fcd41964d0 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e48862740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c1ac720 con 0x55fcd41b71b0 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e48862740 10 client.0 _ll_register_callbacks cb 0x7f7e3400cab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd456d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd456d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60428/0 (socket says 172.21.15.29:60428) 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 -- 172.21.15.29:0/2361981034 learned_addr learned my addr 172.21.15.29:0/2361981034 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3b7fe640 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41b71b0 0x55fcd456e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd456dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 -- 172.21.15.29:0/2361981034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41b71b0 msgr2=0x55fcd456e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41b71b0 0x55fcd456e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 -- 172.21.15.29:0/2361981034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 msgr2=0x55fcd456dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd456dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 -- 172.21.15.29:0/2361981034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd456f030 con 0x55fcd418e2f0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd456dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e3affd640 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd456d6a0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400f010 tx=0x7f7e34022690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80aa8396f2a25c86 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e1affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e1affd640 1 -- 172.21.15.29:0/2361981034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34014040 con 0x55fcd418e2f0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e1affd640 1 -- 172.21.15.29:0/2361981034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f7e34004040 con 0x55fcd418e2f0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.996+0000 7f7e48862740 1 -- 172.21.15.29:0/2361981034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e2c036070 con 0x55fcd418e2f0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e1affd640 1 -- 172.21.15.29:0/2361981034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7e34041050 con 0x55fcd418e2f0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e48862740 10 client.18723 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e48862740 1 -- 172.21.15.29:0/2361981034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55fcd45702c0 con 0x55fcd418e2f0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e1affd640 1 -- 172.21.15.29:0/2361981034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e2c036070 con 0x55fcd418e2f0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e1affd640 1 -- 172.21.15.29:0/2361981034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f7e340035f0 con 0x55fcd418e2f0 2024-09-21T03:04:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e48862740 10 client.18723 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e48862740 10 client.18723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:04:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e48862740 20 client.18723 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e48862740 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fcd414b1c0 0x55fcd45e23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e48862740 4 client.18723 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:04:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e48862740 1 -- 172.21.15.29:0/2361981034 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fcd44cf0f0 con 0x55fcd414b1c0 2024-09-21T03:04:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.997+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fcd414b1c0 0x55fcd45e23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.998+0000 7f7e3a7fc640 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fcd414b1c0 0x55fcd45e23e0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e2d3bb41ea162c1 server_cookie=6c2c6ab77f7476bf in_seq=0 out_seq=0 2024-09-21T03:04:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:10.998+0000 7f7e1affd640 10 client.18723 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e1affd640 1 -- 172.21.15.29:0/2361981034 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fcd44cf0f0 con 0x55fcd414b1c0 2024-09-21T03:04:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e1affd640 10 client.18723 handle_command_reply: tid=0 2024-09-21T03:04:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 client.18723 shutdown 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 -- 172.21.15.29:0/2361981034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 msgr2=0x55fcd456d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd456d6a0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400f010 tx=0x7f7e34022690 comp rx=0 tx=0).stop 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 -- 172.21.15.29:0/2361981034 shutdown_connections 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fcd41b71b0 0x55fcd456e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55fcd41964d0 0x55fcd456dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55fcd414b1c0 0x55fcd45e23e0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 --2- 172.21.15.29:0/2361981034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcd418e2f0 0x55fcd456d6a0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 -- 172.21.15.29:0/2361981034 >> 172.21.15.29:0/2361981034 conn(0x55fcd41ab300 msgr2=0x55fcd4558ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e1affd640 0 client.18723 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 -- 172.21.15.29:0/2361981034 shutdown_connections 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 1 -- 172.21.15.29:0/2361981034 wait complete. 2024-09-21T03:04:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.118+0000 7f7e48862740 20 client.18723 trim_cache size 0 max 16384 2024-09-21T03:04:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:04:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:04:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:04:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:04:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 -- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e280b6510 msgr2=0x7f7e280b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e280b6510 0x7f7e280b8a20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7e240058c0 tx=0x7f7e24005850 comp rx=0 tx=0).stop 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 -- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 msgr2=0x7f7e3c075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 0x7f7e3c075c50 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009dc0 tx=0x7f7e2c003ba0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 -- 172.21.15.29:0/743817658 shutdown_connections 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f7e280b6510 0x7f7e280b8a20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c105100 0x7f7e3c075c50 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7e3c104760 0x7f7e3c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 --2- 172.21.15.29:0/743817658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c1036e0 0x7f7e3c076e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 -- 172.21.15.29:0/743817658 >> 172.21.15.29:0/743817658 conn(0x7f7e3c0ff020 msgr2=0x7f7e3c107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 -- 172.21.15.29:0/743817658 shutdown_connections 2024-09-21T03:04:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:11.119+0000 7f7e41815640 1 -- 172.21.15.29:0/743817658 wait complete. 2024-09-21T03:04:11.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.51 MB/sec execute 588 sec latency 7968.302 ms 2024-09-21T03:04:11.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:11 smithi190 ceph-mon[33871]: pgmap v1142: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 2.6 MiB/s wr, 194 op/s 2024-09-21T03:04:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:11 smithi029 ceph-mon[31006]: pgmap v1142: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 2.6 MiB/s wr, 194 op/s 2024-09-21T03:04:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:11 smithi066 ceph-mon[33436]: pgmap v1142: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 2.6 MiB/s wr, 194 op/s 2024-09-21T03:04:12.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.50 MB/sec execute 589 sec latency 8968.411 ms 2024-09-21T03:04:13.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.49 MB/sec execute 590 sec latency 9968.464 ms 2024-09-21T03:04:13.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:13 smithi190 ceph-mon[33871]: pgmap v1143: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-09-21T03:04:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:13 smithi029 ceph-mon[31006]: pgmap v1143: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-09-21T03:04:14.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:13 smithi066 ceph-mon[33436]: pgmap v1143: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-09-21T03:04:14.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.48 MB/sec execute 591 sec latency 10968.574 ms 2024-09-21T03:04:15.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.47 MB/sec execute 592 sec latency 11968.696 ms 2024-09-21T03:04:15.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:15 smithi190 ceph-mon[33871]: pgmap v1144: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-09-21T03:04:15.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:04:15.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:15 smithi029 ceph-mon[31006]: pgmap v1144: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-09-21T03:04:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:15 smithi066 ceph-mon[33436]: pgmap v1144: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-09-21T03:04:16.154 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 -- 172.21.15.29:0/1314354798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4508105320 msgr2=0x7f450810d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 --2- 172.21.15.29:0/1314354798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4508105320 0x7f450810d870 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009900 tx=0x7f44fc02a410 comp rx=0 tx=0).stop 2024-09-21T03:04:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 -- 172.21.15.29:0/1314354798 shutdown_connections 2024-09-21T03:04:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 --2- 172.21.15.29:0/1314354798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f450810de30 0x7f4508110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 --2- 172.21.15.29:0/1314354798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4508105320 0x7f450810d870 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 --2- 172.21.15.29:0/1314354798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45081049d0 0x7f4508104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 -- 172.21.15.29:0/1314354798 >> 172.21.15.29:0/1314354798 conn(0x7f4508079740 msgr2=0x7f4508079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 -- 172.21.15.29:0/1314354798 shutdown_connections 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.703+0000 7f450e8f5640 1 -- 172.21.15.29:0/1314354798 wait complete. 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450e8f5640 1 Processor -- start 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450e8f5640 1 -- start start 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450e8f5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45081049d0 0x7f450819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450e8f5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4508105320 0x7f450819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450e8f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450810de30 0x7f450819fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450e8f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f45081139a0 con 0x7f450810de30 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450e8f5640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4508113820 con 0x7f45081049d0 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450e8f5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4508113b20 con 0x7f4508105320 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450ce92640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450810de30 0x7f450819fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450ce92640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450810de30 0x7f450819fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49552/0 (socket says 172.21.15.29:49552) 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f450ce92640 1 -- 172.21.15.29:0/2384065856 learned_addr learned my addr 172.21.15.29:0/2384065856 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f4507fff640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45081049d0 0x7f450819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.704+0000 7f45077fe640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4508105320 0x7f450819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f450ce92640 1 -- 172.21.15.29:0/2384065856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4508105320 msgr2=0x7f450819aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f450ce92640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4508105320 0x7f450819aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f450ce92640 1 -- 172.21.15.29:0/2384065856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45081049d0 msgr2=0x7f450819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f450ce92640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45081049d0 0x7f450819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f450ce92640 1 -- 172.21.15.29:0/2384065856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45081a03a0 con 0x7f450810de30 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f450ce92640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450810de30 0x7f450819fd30 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f450000cce0 tx=0x7f450000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f45057fa640 1 -- 172.21.15.29:0/2384065856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450000f450 con 0x7f450810de30 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f45057fa640 1 -- 172.21.15.29:0/2384065856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f450000fa80 con 0x7f450810de30 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45081a0690 con 0x7f450810de30 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f45057fa640 1 -- 172.21.15.29:0/2384065856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45000189c0 con 0x7f450810de30 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.705+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45081a0b50 con 0x7f450810de30 2024-09-21T03:04:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.706+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f450806c240 con 0x7f450810de30 2024-09-21T03:04:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.708+0000 7f45057fa640 1 -- 172.21.15.29:0/2384065856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f450000fc30 con 0x7f450810de30 2024-09-21T03:04:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.708+0000 7f45057fa640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b6540 0x7f44f00b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.709+0000 7f4507fff640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b6540 0x7f44f00b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.709+0000 7f45057fa640 1 -- 172.21.15.29:0/2384065856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f45000e7a80 con 0x7f450810de30 2024-09-21T03:04:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.709+0000 7f4507fff640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b6540 0x7f44f00b8a50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f450806c940 tx=0x7f44f8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.713+0000 7f45057fa640 1 -- 172.21.15.29:0/2384065856 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4500014030 con 0x7f450810de30 2024-09-21T03:04:16.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.46 MB/sec execute 593 sec latency 12968.751 ms 2024-09-21T03:04:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.943+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f450802e640 con 0x7f450810de30 2024-09-21T03:04:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.944+0000 7f45057fa640 1 -- 172.21.15.29:0/2384065856 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11971 (secure 0 0 0) 0x7f45000adf90 con 0x7f450810de30 2024-09-21T03:04:16.946 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:04:16.946 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"btime":"2024-09-21T03:02:24:667358+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:02:24.667336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:04:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-09-21T03:04:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b6540 msgr2=0x7f44f00b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b6540 0x7f44f00b8a50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f450806c940 tx=0x7f44f8009110 comp rx=0 tx=0).stop 2024-09-21T03:04:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450810de30 msgr2=0x7f450819fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450810de30 0x7f450819fd30 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f450000cce0 tx=0x7f450000a620 comp rx=0 tx=0).stop 2024-09-21T03:04:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 shutdown_connections 2024-09-21T03:04:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f44f00b6540 0x7f44f00b8a50 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450810de30 0x7f450819fd30 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4508105320 0x7f450819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 --2- 172.21.15.29:0/2384065856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45081049d0 0x7f450819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 >> 172.21.15.29:0/2384065856 conn(0x7f4508079740 msgr2=0x7f45080ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 shutdown_connections 2024-09-21T03:04:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:16.947+0000 7f450e8f5640 1 -- 172.21.15.29:0/2384065856 wait complete. 2024-09-21T03:04:17.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:17 smithi029 ceph-mon[31006]: pgmap v1145: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 748 KiB/s wr, 100 op/s 2024-09-21T03:04:17.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:17 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2384065856' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:17.726 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id fba12f0e --release --await' 2024-09-21T03:04:17.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id fba12f0e --release --await 2024-09-21T03:04:17.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72817 5.45 MB/sec execute 594 sec latency 13968.780 ms 2024-09-21T03:04:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:17 smithi190 ceph-mon[33871]: pgmap v1145: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 748 KiB/s wr, 100 op/s 2024-09-21T03:04:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:17 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2384065856' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:17.989 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:17 smithi066 ceph-mon[33436]: pgmap v1145: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 748 KiB/s wr, 100 op/s 2024-09-21T03:04:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:17 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2384065856' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 -- 172.21.15.29:0/1743216923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554104840 msgr2=0x7f3554104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 --2- 172.21.15.29:0/1743216923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554104840 0x7f3554104c50 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f354c009900 tx=0x7f354c02a3f0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 -- 172.21.15.29:0/1743216923 shutdown_connections 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 --2- 172.21.15.29:0/1743216923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f355410ddf0 0x7f3554110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 --2- 172.21.15.29:0/1743216923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554105270 0x7f355410d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 --2- 172.21.15.29:0/1743216923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554104840 0x7f3554104c50 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 -- 172.21.15.29:0/1743216923 >> 172.21.15.29:0/1743216923 conn(0x7f3554079730 msgr2=0x7f3554079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 -- 172.21.15.29:0/1743216923 shutdown_connections 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.654+0000 7f355a422640 1 -- 172.21.15.29:0/1743216923 wait complete. 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f355a422640 1 Processor -- start 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f355a422640 1 -- start start 2024-09-21T03:04:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f355a422640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554104840 0x7f355419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f355a422640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554105270 0x7f355419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f355a422640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f355410ddf0 0x7f355419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f355a422640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3554113990 con 0x7f3554105270 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f355a422640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3554113810 con 0x7f355410ddf0 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f355a422640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3554113b10 con 0x7f3554104840 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f35537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554105270 0x7f355419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f35537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554105270 0x7f355419aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49580/0 (socket says 172.21.15.29:49580) 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.655+0000 7f35537fe640 1 -- 172.21.15.29:0/3544821743 learned_addr learned my addr 172.21.15.29:0/3544821743 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f3553fff640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554104840 0x7f355419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35589bf640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f355410ddf0 0x7f355419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35537fe640 1 -- 172.21.15.29:0/3544821743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554104840 msgr2=0x7f355419a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35537fe640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554104840 0x7f355419a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35537fe640 1 -- 172.21.15.29:0/3544821743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f355410ddf0 msgr2=0x7f355419fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35537fe640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f355410ddf0 0x7f355419fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35537fe640 1 -- 172.21.15.29:0/3544821743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35541a0330 con 0x7f3554105270 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35589bf640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f355410ddf0 0x7f355419fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35537fe640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554105270 0x7f355419aa20 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f353c00c970 tx=0x7f353c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35517fa640 1 -- 172.21.15.29:0/3544821743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c004220 con 0x7f3554105270 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35517fa640 1 -- 172.21.15.29:0/3544821743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f353c007e40 con 0x7f3554105270 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f355a422640 1 -- 172.21.15.29:0/3544821743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35541a0620 con 0x7f3554105270 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f35517fa640 1 -- 172.21.15.29:0/3544821743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c0047a0 con 0x7f3554105270 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.656+0000 7f355a422640 1 -- 172.21.15.29:0/3544821743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35541a0b10 con 0x7f3554105270 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3560996740 1 Processor -- start 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3560996740 1 -- start start 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f35517fa640 1 -- 172.21.15.29:0/3544821743 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f353c0079b0 con 0x7f3554105270 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3560996740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c54043e520 0x55c5407e4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3560996740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c540415660 0x55c5407e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3560996740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3560996740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c5406cfdc0 con 0x55c54043e520 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3560996740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c54075c1f0 con 0x55c54041d840 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3560996740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c54074b620 con 0x55c540415660 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f35589bf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c54043e520 0x55c5407e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f35589bf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c54043e520 0x55c5407e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49582/0 (socket says 172.21.15.29:49582) 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f35589bf640 1 -- 172.21.15.29:0/267316332 learned_addr learned my addr 172.21.15.29:0/267316332 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f3553fff640 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c540415660 0x55c5407e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.659+0000 7f35537fe640 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35589bf640 1 -- 172.21.15.29:0/267316332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c540415660 msgr2=0x55c5407e4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35589bf640 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c540415660 0x55c5407e4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35589bf640 1 -- 172.21.15.29:0/267316332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 msgr2=0x55c5407e4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35589bf640 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35589bf640 1 -- 172.21.15.29:0/267316332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c540713fd0 con 0x55c54043e520 2024-09-21T03:04:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35537fe640 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35517fa640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f35400b6540 0x7f35400b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35589bf640 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c54043e520 0x55c5407e4510 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f3544008de0 tx=0x7f354400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c35092d4b095b6d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f3553fff640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f35400b6540 0x7f35400b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f35517fa640 1 -- 172.21.15.29:0/3544821743 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f353c0e7370 con 0x7f3554105270 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f354a7fc640 1 -- 172.21.15.29:0/267316332 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354400f830 con 0x55c54043e520 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f354a7fc640 1 -- 172.21.15.29:0/267316332 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f354400de20 con 0x55c54043e520 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.660+0000 7f354a7fc640 1 -- 172.21.15.29:0/267316332 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354400eda0 con 0x55c54043e520 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 -- 172.21.15.29:0/267316332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c54043e520 msgr2=0x55c5407e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3553fff640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f35400b6540 0x7f35400b8a50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f354c0057c0 tx=0x7f354c009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c54043e520 0x55c5407e4510 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f3544008de0 tx=0x7f354400d710 comp rx=0 tx=0).stop 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 -- 172.21.15.29:0/267316332 shutdown_connections 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c540415660 0x55c5407e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 --2- 172.21.15.29:0/267316332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c54043e520 0x55c5407e4510 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 -- 172.21.15.29:0/267316332 >> 172.21.15.29:0/267316332 conn(0x55c540432910 msgr2=0x55c5407e2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 -- 172.21.15.29:0/267316332 shutdown_connections 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 -- 172.21.15.29:0/267316332 wait complete. 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.661+0000 7f3560996740 1 -- start start 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3560996740 1 Processor -- start 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3560996740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407efc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3560996740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407f0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3560996740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c540800270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3560996740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c5406e6300 con 0x55c540415660 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3560996740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c54074f430 con 0x55c54041d840 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3560996740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c5405acbb0 con 0x55c54043e520 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3560996740 10 client.0 _ll_register_callbacks cb 0x55c540700120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3553fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3553fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407efc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49598/0 (socket says 172.21.15.29:49598) 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3553fff640 1 -- 172.21.15.29:0/3415314960 learned_addr learned my addr 172.21.15.29:0/3415314960 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f35537fe640 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407f0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f35589bf640 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c540800270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3553fff640 1 -- 172.21.15.29:0/3415314960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 msgr2=0x55c540800270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3553fff640 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c540800270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3553fff640 1 -- 172.21.15.29:0/3415314960 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 msgr2=0x55c5407f0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3553fff640 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407f0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.662+0000 7f3553fff640 1 -- 172.21.15.29:0/3415314960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5406d5780 con 0x55c540415660 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f3553fff640 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407efc00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f355406bc00 tx=0x7f354c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2e0802d36a97ac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f35337fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f35337fe640 1 -- 172.21.15.29:0/3415314960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c04e070 con 0x55c540415660 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f35337fe640 1 -- 172.21.15.29:0/3415314960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f354c04a920 con 0x55c540415660 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f3560996740 1 -- 172.21.15.29:0/3415314960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f354400de20 con 0x55c540415660 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f35337fe640 1 -- 172.21.15.29:0/3415314960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f354c05a050 con 0x55c540415660 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f3560996740 10 client.18741 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f3560996740 1 -- 172.21.15.29:0/3415314960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c5404a9b00 con 0x55c540415660 2024-09-21T03:04:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f35337fe640 1 -- 172.21.15.29:0/3415314960 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f354400de20 con 0x55c540415660 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.663+0000 7f35337fe640 1 -- 172.21.15.29:0/3415314960 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f354c03dbf0 con 0x55c540415660 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f3560996740 10 client.18741 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f3560996740 10 client.18741 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f3560996740 20 client.18741 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f3560996740 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c5403fcea0 0x55c540806380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f3560996740 4 client.18741 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f3560996740 1 -- 172.21.15.29:0/3415314960 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c54075e140 con 0x55c5403fcea0 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f35537fe640 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c5403fcea0 0x55c540806380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f35537fe640 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c5403fcea0 0x55c540806380 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=916dd7e63f3dbcd9 server_cookie=260df948a69ad2ca in_seq=0 out_seq=0 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.664+0000 7f35337fe640 10 client.18741 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f35337fe640 1 -- 172.21.15.29:0/3415314960 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c54075e140 con 0x55c5403fcea0 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f35337fe640 10 client.18741 handle_command_reply: tid=0 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 client.18741 shutdown 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 -- 172.21.15.29:0/3415314960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 msgr2=0x55c5407efc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407efc00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f355406bc00 tx=0x7f354c009260 comp rx=0 tx=0).stop 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 -- 172.21.15.29:0/3415314960 shutdown_connections 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c540800270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407f0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c5403fcea0 0x55c540806380 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 --2- 172.21.15.29:0/3415314960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407efc00 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 -- 172.21.15.29:0/3415314960 >> 172.21.15.29:0/3415314960 conn(0x55c540432910 msgr2=0x55c5407e76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f35337fe640 0 client.18741 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 -- 172.21.15.29:0/3415314960 shutdown_connections 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.665+0000 7f3560996740 1 -- 172.21.15.29:0/3415314960 wait complete. 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.666+0000 7f3560996740 20 client.18741 trim_cache size 0 max 16384 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3560996740 1 Processor -- start 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3560996740 1 -- start start 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3560996740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3560996740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3560996740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c5407e9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3560996740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f35541ae880 con 0x55c540415660 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3560996740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f35541b03c0 con 0x55c54041d840 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3560996740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f35541aef50 con 0x55c54043e520 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49602/0 (socket says 172.21.15.29:49602) 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 -- 172.21.15.29:0/3181499293 learned_addr learned my addr 172.21.15.29:0/3181499293 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f35537fe640 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f35589bf640 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c5407e9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 -- 172.21.15.29:0/3181499293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 msgr2=0x55c5407e9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c5407e9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 -- 172.21.15.29:0/3181499293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 msgr2=0x55c5407e95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 -- 172.21.15.29:0/3181499293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c540520960 con 0x55c540415660 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f35537fe640 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f35589bf640 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c5407e9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.683+0000 7f3553fff640 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f1ae0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f354c006010 tx=0x7f354c02aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a985ab10687dc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3549ffb640 1 -- 172.21.15.29:0/3181499293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c063070 con 0x55c540415660 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3549ffb640 1 -- 172.21.15.29:0/3181499293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f354c031e50 con 0x55c540415660 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3549ffb640 1 -- 172.21.15.29:0/3181499293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c04e040 con 0x55c540415660 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 -- 172.21.15.29:0/3181499293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 msgr2=0x55c5407f1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f1ae0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f354c006010 tx=0x7f354c02aa70 comp rx=0 tx=0).stop 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 -- 172.21.15.29:0/3181499293 shutdown_connections 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c5407e9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407e95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 --2- 172.21.15.29:0/3181499293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f1ae0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 -- 172.21.15.29:0/3181499293 >> 172.21.15.29:0/3181499293 conn(0x55c540432910 msgr2=0x55c5407e5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 -- 172.21.15.29:0/3181499293 shutdown_connections 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.684+0000 7f3560996740 1 -- 172.21.15.29:0/3181499293 wait complete. 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 1 -- start start 2024-09-21T03:04:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 1 Processor -- start 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407f90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c540869d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f35541afcf0 con 0x55c540415660 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f35541af620 con 0x55c54041d840 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f35541ac880 con 0x55c54043e520 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3560996740 10 client.0 _ll_register_callbacks cb 0x7f354c004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3553fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3553fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49614/0 (socket says 172.21.15.29:49614) 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f3553fff640 1 -- 172.21.15.29:0/83487461 learned_addr learned my addr 172.21.15.29:0/83487461 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.685+0000 7f35537fe640 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407f90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f35589bf640 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c540869d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3553fff640 1 -- 172.21.15.29:0/83487461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 msgr2=0x55c540869d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3553fff640 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c540869d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3553fff640 1 -- 172.21.15.29:0/83487461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 msgr2=0x55c5407f90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3553fff640 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407f90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3553fff640 1 -- 172.21.15.29:0/83487461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54086a3f0 con 0x55c540415660 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3553fff640 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f8b30 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f354c004650 tx=0x7f354c031a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bffd28310ac1c4ef server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f35337fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f35337fe640 1 -- 172.21.15.29:0/83487461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c063030 con 0x55c540415660 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f35337fe640 1 -- 172.21.15.29:0/83487461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f354c04a930 con 0x55c540415660 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3560996740 1 -- 172.21.15.29:0/83487461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f354c031e50 con 0x55c540415660 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f35337fe640 1 -- 172.21.15.29:0/83487461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f354c071050 con 0x55c540415660 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3560996740 10 client.18747 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.686+0000 7f3560996740 1 -- 172.21.15.29:0/83487461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c540871650 con 0x55c540415660 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f35337fe640 1 -- 172.21.15.29:0/83487461 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f354c031e50 con 0x55c540415660 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f35337fe640 1 -- 172.21.15.29:0/83487461 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f354c03dc60 con 0x55c540415660 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f3560996740 10 client.18747 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f3560996740 10 client.18747 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f3560996740 20 client.18747 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f3560996740 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c5403d2250 0x55c540873cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f3560996740 4 client.18747 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "fba12f0e", "release": true, "await": true}] 2024-09-21T03:04:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f3560996740 1 -- 172.21.15.29:0/83487461 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "fba12f0e", "release": true, "await": true}) -- 0x55c540759bf0 con 0x55c5403d2250 2024-09-21T03:04:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.687+0000 7f35537fe640 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c5403d2250 0x55c540873cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.688+0000 7f35537fe640 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c5403d2250 0x55c540873cc0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a9f7b182e1936ac server_cookie=a1a3d6250ff388b0 in_seq=0 out_seq=0 2024-09-21T03:04:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.688+0000 7f35337fe640 10 client.18747 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.750+0000 7f35337fe640 1 -- 172.21.15.29:0/83487461 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c540759bf0 con 0x55c5403d2250 2024-09-21T03:04:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.750+0000 7f35337fe640 10 client.18747 handle_command_reply: tid=0 2024-09-21T03:04:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 client.18747 shutdown 2024-09-21T03:04:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 -- 172.21.15.29:0/83487461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 msgr2=0x55c5407f8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f8b30 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f354c004650 tx=0x7f354c031a40 comp rx=0 tx=0).stop 2024-09-21T03:04:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 -- 172.21.15.29:0/83487461 shutdown_connections 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c54043e520 0x55c540869d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c54041d840 0x55c5407f90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c5403d2250 0x55c540873cc0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 --2- 172.21.15.29:0/83487461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c540415660 0x55c5407f8b30 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f35337fe640 0 client.18747 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 -- 172.21.15.29:0/83487461 >> 172.21.15.29:0/83487461 conn(0x55c540432910 msgr2=0x55c5407e5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 -- 172.21.15.29:0/83487461 shutdown_connections 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 1 -- 172.21.15.29:0/83487461 wait complete. 2024-09-21T03:04:18.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72819 5.45 MB/sec execute 595 sec latency 14968.808 ms 2024-09-21T03:04:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.751+0000 7f3560996740 20 client.18747 trim_cache size 0 max 16384 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 39, 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 96, 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "fba12f0e": { 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 96, 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.8, 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:18.755 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 -- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f35400b6540 msgr2=0x7f35400b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f35400b6540 0x7f35400b8a50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f354c0057c0 tx=0x7f354c009680 comp rx=0 tx=0).stop 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 -- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554105270 msgr2=0x7f355419aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554105270 0x7f355419aa20 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f353c00c970 tx=0x7f353c00ce30 comp rx=0 tx=0).stop 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 -- 172.21.15.29:0/3544821743 shutdown_connections 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f35400b6540 0x7f35400b8a50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f355410ddf0 0x7f355419fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3554105270 0x7f355419aa20 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 --2- 172.21.15.29:0/3544821743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554104840 0x7f355419a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 -- 172.21.15.29:0/3544821743 >> 172.21.15.29:0/3544821743 conn(0x7f3554079730 msgr2=0x7f35540ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 -- 172.21.15.29:0/3544821743 shutdown_connections 2024-09-21T03:04:18.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:18.752+0000 7f355a422640 1 -- 172.21.15.29:0/3544821743 wait complete. 2024-09-21T03:04:19.601 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'fba12f0e', total seconds elapsed: 23.5 2024-09-21T03:04:19.602 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.8 2024-09-21T03:04:19.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:04:19.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 72973 5.45 MB/sec execute 596 sec latency 14984.106 ms 2024-09-21T03:04:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:19 smithi190 ceph-mon[33871]: pgmap v1146: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 259 KiB/s wr, 59 op/s 2024-09-21T03:04:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:19 smithi029 ceph-mon[31006]: pgmap v1146: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 259 KiB/s wr, 59 op/s 2024-09-21T03:04:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:20.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:19 smithi066 ceph-mon[33436]: pgmap v1146: 129 pgs: 129 active+clean; 347 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 259 KiB/s wr, 59 op/s 2024-09-21T03:04:20.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.570+0000 7f881b36f640 1 -- 172.21.15.29:0/1010342594 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8814105310 msgr2=0x7f881410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:20.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.570+0000 7f881b36f640 1 --2- 172.21.15.29:0/1010342594 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8814105310 0x7f881410d950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8804009900 tx=0x7f880402a410 comp rx=0 tx=0).stop 2024-09-21T03:04:20.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.571+0000 7f881b36f640 1 -- 172.21.15.29:0/1010342594 shutdown_connections 2024-09-21T03:04:20.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.571+0000 7f881b36f640 1 --2- 172.21.15.29:0/1010342594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f881410de90 0x7f8814110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.571+0000 7f881b36f640 1 --2- 172.21.15.29:0/1010342594 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8814105310 0x7f881410d950 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.571+0000 7f881b36f640 1 --2- 172.21.15.29:0/1010342594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88141048e0 0x7f8814104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.571+0000 7f881b36f640 1 -- 172.21.15.29:0/1010342594 >> 172.21.15.29:0/1010342594 conn(0x7f8814079790 msgr2=0x7f8814079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:20.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.571+0000 7f881b36f640 1 -- 172.21.15.29:0/1010342594 shutdown_connections 2024-09-21T03:04:20.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.571+0000 7f881b36f640 1 -- 172.21.15.29:0/1010342594 wait complete. 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881b36f640 1 Processor -- start 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881b36f640 1 -- start start 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881b36f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88141048e0 0x7f881419c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881b36f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8814105310 0x7f881419cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881b36f640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f881410de90 0x7f88141a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881b36f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8814113960 con 0x7f88141048e0 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881b36f640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f88141137e0 con 0x7f881410de90 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881b36f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8814113ae0 con 0x7f8814105310 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881890a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8814105310 0x7f881419cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881890a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8814105310 0x7f881419cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52978/0 (socket says 172.21.15.29:52978) 2024-09-21T03:04:20.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881890a640 1 -- 172.21.15.29:0/1574328242 learned_addr learned my addr 172.21.15.29:0/1574328242 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:20.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.572+0000 7f881990c640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f881410de90 0x7f88141a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.573+0000 7f881890a640 1 -- 172.21.15.29:0/1574328242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f881410de90 msgr2=0x7f88141a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.573+0000 7f881890a640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f881410de90 0x7f88141a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.573+0000 7f881890a640 1 -- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88141048e0 msgr2=0x7f881419c610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.573+0000 7f881890a640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88141048e0 0x7f881419c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.573+0000 7f881890a640 1 -- 172.21.15.29:0/1574328242 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88141a2580 con 0x7f8814105310 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.573+0000 7f881890a640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8814105310 0x7f881419cbb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f880402a930 tx=0x7f8804003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.574+0000 7f880a7fc640 1 -- 172.21.15.29:0/1574328242 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880403f070 con 0x7f8814105310 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.574+0000 7f880a7fc640 1 -- 172.21.15.29:0/1574328242 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f880403baf0 con 0x7f8814105310 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.574+0000 7f880a7fc640 1 -- 172.21.15.29:0/1574328242 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804035030 con 0x7f8814105310 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.574+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88141a2810 con 0x7f8814105310 2024-09-21T03:04:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.574+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88141a8fd0 con 0x7f8814105310 2024-09-21T03:04:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.575+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8814104cf0 con 0x7f8814105310 2024-09-21T03:04:20.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.579+0000 7f880a7fc640 1 -- 172.21.15.29:0/1574328242 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f8804042070 con 0x7f8814105310 2024-09-21T03:04:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.580+0000 7f880a7fc640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88000b6590 0x7f88000b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.580+0000 7f880a7fc640 1 -- 172.21.15.29:0/1574328242 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f88041080a0 con 0x7f8814105310 2024-09-21T03:04:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.580+0000 7f881910b640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88000b6590 0x7f88000b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.581+0000 7f881910b640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88000b6590 0x7f88000b8aa0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f88100058c0 tx=0x7f8810005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.582+0000 7f880a7fc640 1 -- 172.21.15.29:0/1574328242 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f88040ce5b0 con 0x7f8814105310 2024-09-21T03:04:20.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 73124 5.45 MB/sec execute 597 sec latency 84.669 ms 2024-09-21T03:04:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.938+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f881419d740 con 0x7f8814105310 2024-09-21T03:04:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.939+0000 7f880a7fc640 1 -- 172.21.15.29:0/1574328242 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11971 (secure 0 0 0) 0x7f88040d2990 con 0x7f8814105310 2024-09-21T03:04:20.941 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:04:20.941 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"btime":"2024-09-21T03:02:24:667358+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:02:24.667336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:04:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88000b6590 msgr2=0x7f88000b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88000b6590 0x7f88000b8aa0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f88100058c0 tx=0x7f8810005850 comp rx=0 tx=0).stop 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8814105310 msgr2=0x7f881419cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8814105310 0x7f881419cbb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f880402a930 tx=0x7f8804003b90 comp rx=0 tx=0).stop 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 shutdown_connections 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f88000b6590 0x7f88000b8aa0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f881410de90 0x7f88141a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8814105310 0x7f881419cbb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 --2- 172.21.15.29:0/1574328242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88141048e0 0x7f881419c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 >> 172.21.15.29:0/1574328242 conn(0x7f8814079790 msgr2=0x7f881410c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 shutdown_connections 2024-09-21T03:04:20.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:20.942+0000 7f881b36f640 1 -- 172.21.15.29:0/1574328242 wait complete. 2024-09-21T03:04:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:21 smithi029 ceph-mon[31006]: pgmap v1147: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 311 KiB/s wr, 70 op/s 2024-09-21T03:04:21.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:21 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1574328242' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:21.716 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db / --timeout 90 --expiration 139.8 --await-for 45.0' 2024-09-21T03:04:21.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db / --timeout 90 --expiration 139.8 --await-for 45.0 2024-09-21T03:04:21.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 73255 5.45 MB/sec execute 598 sec latency 104.344 ms 2024-09-21T03:04:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:21 smithi190 ceph-mon[33871]: pgmap v1147: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 311 KiB/s wr, 70 op/s 2024-09-21T03:04:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:21 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1574328242' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:21 smithi066 ceph-mon[33436]: pgmap v1147: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 311 KiB/s wr, 70 op/s 2024-09-21T03:04:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:21 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1574328242' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.681+0000 7fd5c37de640 1 -- 172.21.15.29:0/4072655229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 msgr2=0x7fd5bc1061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:22.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.681+0000 7fd5c37de640 1 --2- 172.21.15.29:0/4072655229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 0x7fd5bc1061c0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009900 tx=0x7fd5b802a410 comp rx=0 tx=0).stop 2024-09-21T03:04:22.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.682+0000 7fd5c37de640 1 -- 172.21.15.29:0/4072655229 shutdown_connections 2024-09-21T03:04:22.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.682+0000 7fd5c37de640 1 --2- 172.21.15.29:0/4072655229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5bc10c040 0x7fd5bc112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.682+0000 7fd5c37de640 1 --2- 172.21.15.29:0/4072655229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5bc10b6f0 0x7fd5bc10bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.682+0000 7fd5c37de640 1 --2- 172.21.15.29:0/4072655229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 0x7fd5bc1061c0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.682+0000 7fd5c37de640 1 -- 172.21.15.29:0/4072655229 >> 172.21.15.29:0/4072655229 conn(0x7fd5bc1017e0 msgr2=0x7fd5bc103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:22.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.682+0000 7fd5c37de640 1 -- 172.21.15.29:0/4072655229 shutdown_connections 2024-09-21T03:04:22.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.682+0000 7fd5c37de640 1 -- 172.21.15.29:0/4072655229 wait complete. 2024-09-21T03:04:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.683+0000 7fd5c37de640 1 Processor -- start 2024-09-21T03:04:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.683+0000 7fd5c37de640 1 -- start start 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.683+0000 7fd5c37de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 0x7fd5bc19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.683+0000 7fd5c37de640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5bc10b6f0 0x7fd5bc19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.683+0000 7fd5c37de640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5bc10c040 0x7fd5bc1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.683+0000 7fd5c37de640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc115bb0 con 0x7fd5bc105db0 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.683+0000 7fd5c37de640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc115a30 con 0x7fd5bc10b6f0 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.683+0000 7fd5c37de640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc115d30 con 0x7fd5bc10c040 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 0x7fd5bc19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 0x7fd5bc19c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49650/0 (socket says 172.21.15.29:49650) 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 -- 172.21.15.29:0/3168420014 learned_addr learned my addr 172.21.15.29:0/3168420014 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5bc10c040 0x7fd5bc1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5bc10b6f0 0x7fd5bc19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 -- 172.21.15.29:0/3168420014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5bc10c040 msgr2=0x7fd5bc1a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5bc10c040 0x7fd5bc1a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 -- 172.21.15.29:0/3168420014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5bc10b6f0 msgr2=0x7fd5bc19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5bc10b6f0 0x7fd5bc19cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 -- 172.21.15.29:0/3168420014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5bc1a2590 con 0x7fd5bc105db0 2024-09-21T03:04:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5c157a640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 0x7fd5bc19c620 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009230 tx=0x7fd5b80319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.684+0000 7fd5b27fc640 1 -- 172.21.15.29:0/3168420014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b803f030 con 0x7fd5bc105db0 2024-09-21T03:04:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.685+0000 7fd5b27fc640 1 -- 172.21.15.29:0/3168420014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5b80043d0 con 0x7fd5bc105db0 2024-09-21T03:04:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.685+0000 7fd5b27fc640 1 -- 172.21.15.29:0/3168420014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8035070 con 0x7fd5bc105db0 2024-09-21T03:04:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.685+0000 7fd5c37de640 1 -- 172.21.15.29:0/3168420014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5bc1a2820 con 0x7fd5bc105db0 2024-09-21T03:04:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.685+0000 7fd5c37de640 1 -- 172.21.15.29:0/3168420014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5bc1a8fe0 con 0x7fd5bc105db0 2024-09-21T03:04:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.688+0000 7fd5c9d52740 1 Processor -- start 2024-09-21T03:04:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.688+0000 7fd5c9d52740 1 -- start start 2024-09-21T03:04:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3bc0840 0x55cdb3f6fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b97980 0x55cdb3f70350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f70890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3eddb90 con 0x55cdb3b9fb60 2024-09-21T03:04:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3eccfc0 con 0x55cdb3bc0840 2024-09-21T03:04:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3ed0dd0 con 0x55cdb3b97980 2024-09-21T03:04:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c0d79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f70890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c0d79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f70890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49662/0 (socket says 172.21.15.29:49662) 2024-09-21T03:04:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c0d79640 1 -- 172.21.15.29:0/915839343 learned_addr learned my addr 172.21.15.29:0/915839343 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/915839343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3bc0840 0x55cdb3f6fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c0d79640 1 -- 172.21.15.29:0/915839343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b97980 msgr2=0x55cdb3f70350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T03:04:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/915839343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b97980 0x55cdb3f70350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c0d79640 1 -- 172.21.15.29:0/915839343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3bc0840 msgr2=0x55cdb3f6fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/915839343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3bc0840 0x55cdb3f6fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5c0d79640 1 -- 172.21.15.29:0/915839343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdb3e54060 con 0x55cdb3b9fb60 2024-09-21T03:04:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.689+0000 7fd5b27fc640 1 -- 172.21.15.29:0/3168420014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd5b8042030 con 0x7fd5bc105db0 2024-09-21T03:04:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.690+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/915839343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f70890 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009bb0 tx=0x7fd5ac0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17dec946e765dda9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.690+0000 7fd5a77fe640 1 -- 172.21.15.29:0/915839343 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac012450 con 0x55cdb3b9fb60 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.690+0000 7fd5a77fe640 1 -- 172.21.15.29:0/915839343 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5ac012a80 con 0x55cdb3b9fb60 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.690+0000 7fd5a77fe640 1 -- 172.21.15.29:0/915839343 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac00f950 con 0x55cdb3b9fb60 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.691+0000 7fd5b27fc640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd5a80b6610 0x7fd5a80b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.691+0000 7fd5b27fc640 1 -- 172.21.15.29:0/3168420014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fd5b81079e0 con 0x7fd5bc105db0 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.691+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd5a80b6610 0x7fd5a80b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.691+0000 7fd5c9d52740 1 -- 172.21.15.29:0/915839343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b9fb60 msgr2=0x55cdb3f70890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.691+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/915839343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f70890 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009bb0 tx=0x7fd5ac0117b0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.692+0000 7fd5c9d52740 1 -- 172.21.15.29:0/915839343 shutdown_connections 2024-09-21T03:04:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.692+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/915839343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f70890 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.692+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/915839343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b97980 0x55cdb3f70350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.692+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/915839343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3bc0840 0x55cdb3f6fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.692+0000 7fd5c9d52740 1 -- 172.21.15.29:0/915839343 >> 172.21.15.29:0/915839343 conn(0x55cdb3bb4990 msgr2=0x55cdb3f6dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.692+0000 7fd5c9d52740 1 -- 172.21.15.29:0/915839343 shutdown_connections 2024-09-21T03:04:22.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.692+0000 7fd5c9d52740 1 -- 172.21.15.29:0/915839343 wait complete. 2024-09-21T03:04:22.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.692+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd5a80b6610 0x7fd5a80b8b20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4005fd0 tx=0x7fd5b4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:22.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.693+0000 7fd5c9d52740 1 -- start start 2024-09-21T03:04:22.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.693+0000 7fd5c9d52740 1 Processor -- start 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f7c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3bc0840 0x55cdb3f87910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3d2e7a0 con 0x55cdb3bc0840 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3e720a0 con 0x55cdb3b97980 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3f7e5f0 con 0x55cdb3b9fb60 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c9d52740 10 client.0 _ll_register_callbacks cb 0x55cdb3e4f3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c0d79640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f7c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c0d79640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f7c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53006/0 (socket says 172.21.15.29:53006) 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.694+0000 7fd5c0d79640 1 -- 172.21.15.29:0/671669792 learned_addr learned my addr 172.21.15.29:0/671669792 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.695+0000 7fd5c157a640 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.695+0000 7fd5c0d79640 1 -- 172.21.15.29:0/671669792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 msgr2=0x55cdb3f7ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.695+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.695+0000 7fd5c0d79640 1 -- 172.21.15.29:0/671669792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3bc0840 msgr2=0x55cdb3f87910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.695+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3bc0840 0x55cdb3f87910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.695+0000 7fd5c0d79640 1 -- 172.21.15.29:0/671669792 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdb3e60880 con 0x55cdb3b9fb60 2024-09-21T03:04:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.695+0000 7fd5c157a640 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.696+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f7c070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac0126b0 tx=0x7fd5ac0126e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec5b261a76dc57bd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.696+0000 7fd5a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-09-21T03:04:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.696+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/671669792 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac009de0 con 0x55cdb3b9fb60 2024-09-21T03:04:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.696+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/671669792 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fd5ac005230 con 0x55cdb3b9fb60 2024-09-21T03:04:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.696+0000 7fd5c9d52740 1 -- 172.21.15.29:0/671669792 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5ac012a80 con 0x55cdb3b9fb60 2024-09-21T03:04:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.696+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/671669792 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd5ac032050 con 0x55cdb3b9fb60 2024-09-21T03:04:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.697+0000 7fd5c9d52740 10 client.28171 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.697+0000 7fd5c9d52740 1 -- 172.21.15.29:0/671669792 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55cdb3c3ab90 con 0x55cdb3b9fb60 2024-09-21T03:04:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.697+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/671669792 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5ac029070 con 0x55cdb3b9fb60 2024-09-21T03:04:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.699+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/671669792 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fd5ac00f650 con 0x55cdb3b9fb60 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.699+0000 7fd5c9d52740 10 client.28171 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.699+0000 7fd5c9d52740 10 client.28171 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.699+0000 7fd5c9d52740 20 client.28171 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.699+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cdb3b7f1c0 0x55cdb3f8d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.699+0000 7fd5c9d52740 4 client.28171 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.699+0000 7fd5c9d52740 1 -- 172.21.15.29:0/671669792 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd5ac012a80 con 0x55cdb3b7f1c0 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.699+0000 7fd5c157a640 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cdb3b7f1c0 0x55cdb3f8d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.700+0000 7fd5c157a640 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cdb3b7f1c0 0x55cdb3f8d920 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=422dc5f28d879a14 server_cookie=eb40e205b46de989 in_seq=0 out_seq=0 2024-09-21T03:04:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.700+0000 7fd5a4ff9640 10 client.28171 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/671669792 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fd5ac012a80 con 0x55cdb3b7f1c0 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5a4ff9640 10 client.28171 handle_command_reply: tid=0 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 client.28171 shutdown 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 -- 172.21.15.29:0/671669792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 msgr2=0x55cdb3f7c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f7c070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac0126b0 tx=0x7fd5ac0126e0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 -- 172.21.15.29:0/671669792 shutdown_connections 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cdb3b7f1c0 0x55cdb3f8d920 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5a4ff9640 0 client.28171 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3bc0840 0x55cdb3f87910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f7c070 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/671669792 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.702+0000 7fd5c9d52740 1 -- 172.21.15.29:0/671669792 >> 172.21.15.29:0/671669792 conn(0x55cdb3bb4990 msgr2=0x55cdb3f72fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.703+0000 7fd5c9d52740 1 -- 172.21.15.29:0/671669792 shutdown_connections 2024-09-21T03:04:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.703+0000 7fd5c9d52740 1 -- 172.21.15.29:0/671669792 wait complete. 2024-09-21T03:04:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.703+0000 7fd5c9d52740 20 client.28171 trim_cache size 0 max 16384 2024-09-21T03:04:22.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.726+0000 7fd5c9d52740 1 Processor -- start 2024-09-21T03:04:22.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.726+0000 7fd5c9d52740 1 -- start start 2024-09-21T03:04:22.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f87880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f78a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1ae7c0 con 0x55cdb3b7f1c0 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1b0300 con 0x55cdb3b97980 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1aee90 con 0x55cdb3b9fb60 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c1d7b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f78a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c1d7b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f78a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53008/0 (socket says 172.21.15.29:53008) 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c1d7b640 1 -- 172.21.15.29:0/78979582 learned_addr learned my addr 172.21.15.29:0/78979582 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c157a640 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f87880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c1d7b640 1 -- 172.21.15.29:0/78979582 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 msgr2=0x55cdb3f7d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c1d7b640 1 -- 172.21.15.29:0/78979582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b7f1c0 msgr2=0x55cdb3f87880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f87880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.727+0000 7fd5c1d7b640 1 -- 172.21.15.29:0/78979582 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdb3f7cb50 con 0x55cdb3b9fb60 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.728+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f78a80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4003e30 tx=0x7fd5b4015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee4df186c5236af4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.728+0000 7fd5a6ffd640 1 -- 172.21.15.29:0/78979582 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4003e60 con 0x55cdb3b9fb60 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.728+0000 7fd5a6ffd640 1 -- 172.21.15.29:0/78979582 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5b401caf0 con 0x55cdb3b9fb60 2024-09-21T03:04:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.728+0000 7fd5a6ffd640 1 -- 172.21.15.29:0/78979582 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4011b90 con 0x55cdb3b9fb60 2024-09-21T03:04:22.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.729+0000 7fd5c9d52740 1 -- 172.21.15.29:0/78979582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 msgr2=0x55cdb3f78a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:22.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.729+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f78a80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4003e30 tx=0x7fd5b4015040 comp rx=0 tx=0).stop 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 -- 172.21.15.29:0/78979582 shutdown_connections 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f78a80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b97980 0x55cdb3f7d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/78979582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f87880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 -- 172.21.15.29:0/78979582 >> 172.21.15.29:0/78979582 conn(0x55cdb3bb4990 msgr2=0x55cdb3f7cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 -- 172.21.15.29:0/78979582 shutdown_connections 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 -- 172.21.15.29:0/78979582 wait complete. 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 -- start start 2024-09-21T03:04:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.730+0000 7fd5c9d52740 1 Processor -- start 2024-09-21T03:04:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f91110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b97980 0x55cdb3f8f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c9d52740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f8fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1afc30 con 0x55cdb3b97980 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1af560 con 0x55cdb3b9fb60 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c9d52740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1ac7a0 con 0x55cdb3b7f1c0 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c9d52740 10 client.0 _ll_register_callbacks cb 0x7fd5ac009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c1d7b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f8fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c157a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f91110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c1d7b640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f8fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:54656/0 (socket says 172.21.15.29:54656) 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c157a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f91110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53022/0 (socket says 172.21.15.29:53022) 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.731+0000 7fd5c1d7b640 1 -- 172.21.15.29:0/4183052963 learned_addr learned my addr 172.21.15.29:0/4183052963 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b97980 0x55cdb3f8f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c1d7b640 1 -- 172.21.15.29:0/4183052963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b7f1c0 msgr2=0x55cdb3f91110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f91110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c1d7b640 1 -- 172.21.15.29:0/4183052963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b97980 msgr2=0x55cdb3f8f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b97980 0x55cdb3f8f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c1d7b640 1 -- 172.21.15.29:0/4183052963 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdb3f902a0 con 0x55cdb3b9fb60 2024-09-21T03:04:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c0d79640 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b97980 0x55cdb3f8f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c1d7b640 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f8fb30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40098f0 tx=0x7fd5b4003e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba0bbd29c933f82b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:04:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/4183052963 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4034020 con 0x55cdb3b9fb60 2024-09-21T03:04:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/4183052963 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fd5b401c670 con 0x55cdb3b9fb60 2024-09-21T03:04:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5c9d52740 1 -- 172.21.15.29:0/4183052963 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5b401caf0 con 0x55cdb3b9fb60 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.732+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/4183052963 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd5b401c870 con 0x55cdb3b9fb60 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c9d52740 10 client.28004 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c9d52740 1 -- 172.21.15.29:0/4183052963 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55cdb3ff8fb0 con 0x55cdb3b9fb60 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/4183052963 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5b401caf0 con 0x55cdb3b9fb60 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/4183052963 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fd5b4025070 con 0x55cdb3b9fb60 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c9d52740 10 client.28004 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c9d52740 10 client.28004 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c9d52740 20 client.28004 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cdb3bc0840 0x55cdb3ffb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c9d52740 4 client.28004 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.8, "await_for": 45.0}] 2024-09-21T03:04:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c9d52740 1 -- 172.21.15.29:0/4183052963 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.8, "await_for": 45.0}) -- 0x55cdb3edfae0 con 0x55cdb3bc0840 2024-09-21T03:04:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.733+0000 7fd5c157a640 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cdb3bc0840 0x55cdb3ffb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.734+0000 7fd5a4ff9640 10 client.28004 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:22.734+0000 7fd5c157a640 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cdb3bc0840 0x55cdb3ffb7b0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c3f97d3d515db9 server_cookie=e97e3d4697dc8b85 in_seq=0 out_seq=0 2024-09-21T03:04:22.752 INFO:tasks.workunit.client.0.smithi029.stdout: 10 73364 5.45 MB/sec execute 599 sec latency 118.885 ms 2024-09-21T03:04:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.185+0000 7fd5a4ff9640 1 -- 172.21.15.29:0/4183052963 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cdb3edfae0 con 0x55cdb3bc0840 2024-09-21T03:04:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.185+0000 7fd5a4ff9640 10 client.28004 handle_command_reply: tid=0 2024-09-21T03:04:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.185+0000 7fd5c9d52740 1 client.28004 shutdown 2024-09-21T03:04:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 -- 172.21.15.29:0/4183052963 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b9fb60 msgr2=0x55cdb3f8fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f8fb30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40098f0 tx=0x7fd5b4003e60 comp rx=0 tx=0).stop 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 -- 172.21.15.29:0/4183052963 shutdown_connections 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55cdb3b9fb60 0x55cdb3f8fb30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55cdb3bc0840 0x55cdb3ffb7b0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdb3b97980 0x55cdb3f8f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 --2- 172.21.15.29:0/4183052963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdb3b7f1c0 0x55cdb3f91110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5a4ff9640 0 client.28004 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 -- 172.21.15.29:0/4183052963 >> 172.21.15.29:0/4183052963 conn(0x55cdb3bb4990 msgr2=0x55cdb3f70e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 -- 172.21.15.29:0/4183052963 shutdown_connections 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 1 -- 172.21.15.29:0/4183052963 wait complete. 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.186+0000 7fd5c9d52740 20 client.28004 trim_cache size 0 max 16384 2024-09-21T03:04:23.188 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 39, 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 98, 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "ae657674": { 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 98, 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.8, 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:04:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:23.190 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 -- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd5a80b6610 msgr2=0x7fd5a80b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd5a80b6610 0x7fd5a80b8b20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4005fd0 tx=0x7fd5b4009500 comp rx=0 tx=0).stop 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 -- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 msgr2=0x7fd5bc19c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 0x7fd5bc19c620 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009230 tx=0x7fd5b80319a0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 -- 172.21.15.29:0/3168420014 shutdown_connections 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd5a80b6610 0x7fd5a80b8b20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5bc10c040 0x7fd5bc1a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5bc10b6f0 0x7fd5bc19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 --2- 172.21.15.29:0/3168420014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc105db0 0x7fd5bc19c620 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 -- 172.21.15.29:0/3168420014 >> 172.21.15.29:0/3168420014 conn(0x7fd5bc1017e0 msgr2=0x7fd5bc10adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 -- 172.21.15.29:0/3168420014 shutdown_connections 2024-09-21T03:04:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:23.187+0000 7fd5c37de640 1 -- 172.21.15.29:0/3168420014 wait complete. 2024-09-21T03:04:23.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 600 sec 2024-09-21T03:04:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:23 smithi190 ceph-mon[33871]: pgmap v1148: 129 pgs: 129 active+clean; 351 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 639 KiB/s wr, 66 op/s 2024-09-21T03:04:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:23 smithi029 ceph-mon[31006]: pgmap v1148: 129 pgs: 129 active+clean; 351 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 639 KiB/s wr, 66 op/s 2024-09-21T03:04:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:23 smithi066 ceph-mon[33436]: pgmap v1148: 129 pgs: 129 active+clean; 351 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 639 KiB/s wr, 66 op/s 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 -- 172.21.15.29:0/518447456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103fc0 msgr2=0x7fc44410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 --2- 172.21.15.29:0/518447456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103fc0 0x7fc44410a960 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009c90 tx=0x7fc43c02a860 comp rx=0 tx=0).stop 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 -- 172.21.15.29:0/518447456 shutdown_connections 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 --2- 172.21.15.29:0/518447456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103fc0 0x7fc44410a960 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 --2- 172.21.15.29:0/518447456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc444103620 0x7fc444103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 --2- 172.21.15.29:0/518447456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4441025a0 0x7fc444102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 -- 172.21.15.29:0/518447456 >> 172.21.15.29:0/518447456 conn(0x7fc4440fdee0 msgr2=0x7fc444100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 -- 172.21.15.29:0/518447456 shutdown_connections 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.221+0000 7fc44afc6640 1 -- 172.21.15.29:0/518447456 wait complete. 2024-09-21T03:04:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc44afc6640 1 Processor -- start 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc44afc6640 1 -- start start 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc44afc6640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4441025a0 0x7fc44406b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc44afc6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103620 0x7fc44419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc44afc6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc444103fc0 0x7fc4441a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc44afc6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc44410e090 con 0x7fc444103620 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc44afc6640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc44410df10 con 0x7fc4441025a0 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc44afc6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc44410e210 con 0x7fc444103fc0 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc443fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103620 0x7fc44419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc443fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103620 0x7fc44419ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45696/0 (socket says 172.21.15.29:45696) 2024-09-21T03:04:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc443fff640 1 -- 172.21.15.29:0/2818849282 learned_addr learned my addr 172.21.15.29:0/2818849282 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc448d62640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4441025a0 0x7fc44406b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc449563640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc444103fc0 0x7fc4441a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc443fff640 1 -- 172.21.15.29:0/2818849282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc444103fc0 msgr2=0x7fc4441a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc443fff640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc444103fc0 0x7fc4441a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc443fff640 1 -- 172.21.15.29:0/2818849282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4441025a0 msgr2=0x7fc44406b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.222+0000 7fc443fff640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4441025a0 0x7fc44406b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.223+0000 7fc443fff640 1 -- 172.21.15.29:0/2818849282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4441a1960 con 0x7fc444103620 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.223+0000 7fc448d62640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4441025a0 0x7fc44406b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.223+0000 7fc443fff640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103620 0x7fc44419ebb0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc43000e9f0 tx=0x7fc43000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.223+0000 7fc441ffb640 1 -- 172.21.15.29:0/2818849282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43000cd80 con 0x7fc444103620 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.223+0000 7fc441ffb640 1 -- 172.21.15.29:0/2818849282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4300108c0 con 0x7fc444103620 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.223+0000 7fc44afc6640 1 -- 172.21.15.29:0/2818849282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4441a1c50 con 0x7fc444103620 2024-09-21T03:04:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.223+0000 7fc441ffb640 1 -- 172.21.15.29:0/2818849282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300187c0 con 0x7fc444103620 2024-09-21T03:04:24.228 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.223+0000 7fc44afc6640 1 -- 172.21.15.29:0/2818849282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4441a21a0 con 0x7fc444103620 2024-09-21T03:04:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.226+0000 7fc45201b740 1 Processor -- start 2024-09-21T03:04:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.226+0000 7fc45201b740 1 -- start start 2024-09-21T03:04:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.226+0000 7fc45201b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baed4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc45201b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baed54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc45201b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baed6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc45201b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c3bae476f0 con 0x55c3bab042f0 2024-09-21T03:04:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc45201b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c3bae36b20 con 0x55c3bab0c4d0 2024-09-21T03:04:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc45201b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c3bae3a930 con 0x55c3bab2d1b0 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baed54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baed54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45700/0 (socket says 172.21.15.29:45700) 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 -- 172.21.15.29:0/4211224293 learned_addr learned my addr 172.21.15.29:0/4211224293 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc449563640 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baed4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc441ffb640 1 -- 172.21.15.29:0/2818849282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fc430010430 con 0x7fc444103620 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 -- 172.21.15.29:0/4211224293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 msgr2=0x55c3baed4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baed4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 -- 172.21.15.29:0/4211224293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 msgr2=0x55c3baed6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baed6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 -- 172.21.15.29:0/4211224293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3badfc530 con 0x55c3bab042f0 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc449563640 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baed4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc448d62640 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baed54c0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fc4380115f0 tx=0x7fc438011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84454bf4b1e86565 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc42effd640 1 -- 172.21.15.29:0/4211224293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43801e070 con 0x55c3bab042f0 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc42effd640 1 -- 172.21.15.29:0/4211224293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc43800fad0 con 0x55c3bab042f0 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.227+0000 7fc42effd640 1 -- 172.21.15.29:0/4211224293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438019a90 con 0x55c3bab042f0 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc441ffb640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4340b6510 0x7fc4340b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc449563640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4340b6510 0x7fc4340b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc441ffb640 1 -- 172.21.15.29:0/2818849282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fc4300e79d0 con 0x7fc444103620 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc45201b740 1 -- 172.21.15.29:0/4211224293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 msgr2=0x55c3baed54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc45201b740 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baed54c0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fc4380115f0 tx=0x7fc438011ab0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc45201b740 1 -- 172.21.15.29:0/4211224293 shutdown_connections 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc45201b740 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baed6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc45201b740 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baed54c0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc45201b740 1 --2- 172.21.15.29:0/4211224293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baed4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.228+0000 7fc45201b740 1 -- 172.21.15.29:0/4211224293 >> 172.21.15.29:0/4211224293 conn(0x55c3bab21300 msgr2=0x55c3baed08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 -- 172.21.15.29:0/4211224293 shutdown_connections 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc449563640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4340b6510 0x7fc4340b8a20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc43c005e20 tx=0x7fc43c031910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 -- 172.21.15.29:0/4211224293 wait complete. 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 -- start start 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 Processor -- start 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baede6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baeded90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baedf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c3bad6eab0 con 0x55c3bab042f0 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x55c3baddc730 con 0x55c3bab0c4d0 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.229+0000 7fc45201b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c3baed6b30 con 0x55c3bab2d1b0 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc45201b740 10 client.0 _ll_register_callbacks cb 0x55c3bae306b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baede6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baede6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45712/0 (socket says 172.21.15.29:45712) 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 -- 172.21.15.29:0/4241827419 learned_addr learned my addr 172.21.15.29:0/4241827419 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc443fff640 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baeded90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc449563640 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baedf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 -- 172.21.15.29:0/4241827419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 msgr2=0x55c3baedf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baedf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 -- 172.21.15.29:0/4241827419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 msgr2=0x55c3baeded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baeded90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 -- 172.21.15.29:0/4241827419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3bab98790 con 0x55c3bab042f0 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc443fff640 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baeded90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc448d62640 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baede6d0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fc43800f8f0 tx=0x7fc43800f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9ac7adc2ee2f81 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc423fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc423fff640 1 -- 172.21.15.29:0/4241827419 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43801e040 con 0x55c3bab042f0 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc423fff640 1 -- 172.21.15.29:0/4241827419 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fc438004040 con 0x55c3bab042f0 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc45201b740 1 -- 172.21.15.29:0/4241827419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc43800fad0 con 0x55c3bab042f0 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.230+0000 7fc423fff640 1 -- 172.21.15.29:0/4241827419 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc43802f050 con 0x55c3bab042f0 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc45201b740 10 client.18774 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc45201b740 1 -- 172.21.15.29:0/4241827419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c3bac11620 con 0x55c3bab042f0 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc423fff640 1 -- 172.21.15.29:0/4241827419 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc43800fad0 con 0x55c3bab042f0 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc423fff640 1 -- 172.21.15.29:0/4241827419 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fc43802a030 con 0x55c3bab042f0 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc45201b740 10 client.18774 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc45201b740 10 client.18774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc45201b740 20 client.18774 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc45201b740 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3baaebd80 0x55c3baeebb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc45201b740 4 client.18774 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc45201b740 1 -- 172.21.15.29:0/4241827419 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3bae49640 con 0x55c3baaebd80 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.231+0000 7fc443fff640 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3baaebd80 0x55c3baeebb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.232+0000 7fc423fff640 10 client.18774 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.232+0000 7fc443fff640 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3baaebd80 0x55c3baeebb10 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30a1ec736ace2912 server_cookie=3ef2c12deb07252a in_seq=0 out_seq=0 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.232+0000 7fc423fff640 1 -- 172.21.15.29:0/4241827419 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c3bae49640 con 0x55c3baaebd80 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.232+0000 7fc423fff640 10 client.18774 handle_command_reply: tid=0 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.232+0000 7fc45201b740 1 client.18774 shutdown 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.232+0000 7fc45201b740 1 -- 172.21.15.29:0/4241827419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 msgr2=0x55c3baede6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.232+0000 7fc45201b740 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baede6d0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fc43800f8f0 tx=0x7fc43800f920 comp rx=0 tx=0).stop 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 1 -- 172.21.15.29:0/4241827419 shutdown_connections 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab2d1b0 0x55c3baedf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baeded90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3baaebd80 0x55c3baeebb10 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 1 --2- 172.21.15.29:0/4241827419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab042f0 0x55c3baede6d0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc423fff640 0 client.18774 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 1 -- 172.21.15.29:0/4241827419 >> 172.21.15.29:0/4241827419 conn(0x55c3bab21300 msgr2=0x55c3baed14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 1 -- 172.21.15.29:0/4241827419 shutdown_connections 2024-09-21T03:04:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 1 -- 172.21.15.29:0/4241827419 wait complete. 2024-09-21T03:04:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.233+0000 7fc45201b740 20 client.18774 trim_cache size 0 max 16384 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.247+0000 7fc45201b740 1 Processor -- start 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.247+0000 7fc45201b740 1 -- start start 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc45201b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab042f0 0x55c3baedd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc45201b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab0c4d0 0x55c3baecb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc45201b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baed0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc45201b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441ac660 con 0x55c3bab2d1b0 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc45201b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441ae1a0 con 0x55c3bab042f0 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc45201b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441acd30 con 0x55c3bab0c4d0 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc449563640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baed0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc443fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab0c4d0 0x55c3baecb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc449563640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baed0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45724/0 (socket says 172.21.15.29:45724) 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc448d62640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab042f0 0x55c3baedd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc449563640 1 -- 172.21.15.29:0/1254943556 learned_addr learned my addr 172.21.15.29:0/1254943556 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc443fff640 1 -- 172.21.15.29:0/1254943556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab042f0 msgr2=0x55c3baedd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc443fff640 1 --2- 172.21.15.29:0/1254943556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab042f0 0x55c3baedd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc443fff640 1 -- 172.21.15.29:0/1254943556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 msgr2=0x55c3baed0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc443fff640 1 --2- 172.21.15.29:0/1254943556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baed0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc443fff640 1 -- 172.21.15.29:0/1254943556 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3baecc0a0 con 0x55c3bab0c4d0 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc449563640 1 --2- 172.21.15.29:0/1254943556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baed0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc443fff640 1 --2- 172.21.15.29:0/1254943556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab0c4d0 0x55c3baecb5c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc4300aab00 tx=0x7fc4300a7f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=353cf67315bcd026 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc42e7fc640 1 -- 172.21.15.29:0/1254943556 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300ad3a0 con 0x55c3bab0c4d0 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc42e7fc640 1 -- 172.21.15.29:0/1254943556 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4300b22c0 con 0x55c3bab0c4d0 2024-09-21T03:04:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.248+0000 7fc42e7fc640 1 -- 172.21.15.29:0/1254943556 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300ac7c0 con 0x55c3bab0c4d0 2024-09-21T03:04:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 -- 172.21.15.29:0/1254943556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab0c4d0 msgr2=0x55c3baecb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 --2- 172.21.15.29:0/1254943556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab0c4d0 0x55c3baecb5c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc4300aab00 tx=0x7fc4300a7f90 comp rx=0 tx=0).stop 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 -- 172.21.15.29:0/1254943556 shutdown_connections 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 --2- 172.21.15.29:0/1254943556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baed0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 --2- 172.21.15.29:0/1254943556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab0c4d0 0x55c3baecb5c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 --2- 172.21.15.29:0/1254943556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab042f0 0x55c3baedd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 -- 172.21.15.29:0/1254943556 >> 172.21.15.29:0/1254943556 conn(0x55c3bab21300 msgr2=0x55c3baedcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 -- 172.21.15.29:0/1254943556 shutdown_connections 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.249+0000 7fc45201b740 1 -- 172.21.15.29:0/1254943556 wait complete. 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 1 -- start start 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 1 Processor -- start 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab042f0 0x55c3baee32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baee39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baee4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441adad0 con 0x55c3bab2d1b0 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441ad400 con 0x55c3bab0c4d0 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441aa640 con 0x55c3bab042f0 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc45201b740 10 client.0 _ll_register_callbacks cb 0x7fc43801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc449563640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baee4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc449563640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baee4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45734/0 (socket says 172.21.15.29:45734) 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc449563640 1 -- 172.21.15.29:0/3128346976 learned_addr learned my addr 172.21.15.29:0/3128346976 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc443fff640 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baee39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc448d62640 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab042f0 0x55c3baee32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc443fff640 1 -- 172.21.15.29:0/3128346976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab042f0 msgr2=0x55c3baee32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.250+0000 7fc443fff640 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab042f0 0x55c3baee32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc443fff640 1 -- 172.21.15.29:0/3128346976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 msgr2=0x55c3baee4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc443fff640 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baee4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc443fff640 1 -- 172.21.15.29:0/3128346976 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3baee4c70 con 0x55c3bab0c4d0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc449563640 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baee4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc443fff640 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baee39a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc430009380 tx=0x7fc4300a9fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f742d5e22cc88abd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc423fff640 10 client.0 ms_handle_connect on v2:172.21.15.66:3300/0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc423fff640 1 -- 172.21.15.29:0/3128346976 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300aed10 con 0x55c3bab0c4d0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc45201b740 1 -- 172.21.15.29:0/3128346976 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4300b22c0 con 0x55c3bab0c4d0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc423fff640 1 -- 172.21.15.29:0/3128346976 <== mon.2 v2:172.21.15.66:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fc4300a91e0 con 0x55c3bab0c4d0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc423fff640 1 -- 172.21.15.29:0/3128346976 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc4300b7200 con 0x55c3bab0c4d0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc45201b740 10 client.28019 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc45201b740 1 -- 172.21.15.29:0/3128346976 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c3baee5fb0 con 0x55c3bab0c4d0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc423fff640 1 -- 172.21.15.29:0/3128346976 <== mon.2 v2:172.21.15.66:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4300b22c0 con 0x55c3bab0c4d0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc423fff640 1 -- 172.21.15.29:0/3128346976 <== mon.2 v2:172.21.15.66:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fc4300b6740 con 0x55c3bab0c4d0 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc45201b740 10 client.28019 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.251+0000 7fc45201b740 10 client.28019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.252+0000 7fc45201b740 20 client.28019 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.252+0000 7fc45201b740 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3baac11c0 0x55c3baf586c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.252+0000 7fc45201b740 4 client.28019 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.252+0000 7fc45201b740 1 -- 172.21.15.29:0/3128346976 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c3bae450f0 con 0x55c3baac11c0 2024-09-21T03:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.252+0000 7fc448d62640 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3baac11c0 0x55c3baf586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.252+0000 7fc448d62640 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3baac11c0 0x55c3baf586c0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59aa0850ed693cfc server_cookie=56e71ca9627a653c in_seq=0 out_seq=0 2024-09-21T03:04:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.252+0000 7fc423fff640 10 client.28019 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:24.356 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ae657674', quiesce took 4.8 seconds. Will release after: 15.0 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc423fff640 1 -- 172.21.15.29:0/3128346976 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c3bae450f0 con 0x55c3baac11c0 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc423fff640 10 client.28019 handle_command_reply: tid=0 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 client.28019 shutdown 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 -- 172.21.15.29:0/3128346976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 msgr2=0x55c3baee39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baee39a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc430009380 tx=0x7fc4300a9fb0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 -- 172.21.15.29:0/3128346976 shutdown_connections 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x55c3baac11c0 0x55c3baf586c0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3bab2d1b0 0x55c3baee4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc423fff640 0 client.28019 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x55c3bab0c4d0 0x55c3baee39a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 --2- 172.21.15.29:0/3128346976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3bab042f0 0x55c3baee32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 -- 172.21.15.29:0/3128346976 >> 172.21.15.29:0/3128346976 conn(0x55c3bab21300 msgr2=0x55c3baed8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 -- 172.21.15.29:0/3128346976 shutdown_connections 2024-09-21T03:04:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 1 -- 172.21.15.29:0/3128346976 wait complete. 2024-09-21T03:04:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.359+0000 7fc45201b740 20 client.28019 trim_cache size 0 max 16384 2024-09-21T03:04:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:04:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:04:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:04:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 -- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4340b6510 msgr2=0x7fc4340b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4340b6510 0x7fc4340b8a20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc43c005e20 tx=0x7fc43c031910 comp rx=0 tx=0).stop 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 -- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103620 msgr2=0x7fc44419ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103620 0x7fc44419ebb0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc43000e9f0 tx=0x7fc43000eeb0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 -- 172.21.15.29:0/2818849282 shutdown_connections 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fc4340b6510 0x7fc4340b8a20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc444103fc0 0x7fc4441a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc444103620 0x7fc44419ebb0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 --2- 172.21.15.29:0/2818849282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4441025a0 0x7fc44406b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 -- 172.21.15.29:0/2818849282 >> 172.21.15.29:0/2818849282 conn(0x7fc4440fdee0 msgr2=0x7fc444106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 -- 172.21.15.29:0/2818849282 shutdown_connections 2024-09-21T03:04:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:24.360+0000 7fc44afc6640 1 -- 172.21.15.29:0/2818849282 wait complete. 2024-09-21T03:04:24.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 601 sec 2024-09-21T03:04:25.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 602 sec 2024-09-21T03:04:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:25 smithi190 ceph-mon[33871]: pgmap v1149: 129 pgs: 129 active+clean; 352 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1004 KiB/s wr, 92 op/s 2024-09-21T03:04:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:25 smithi029 ceph-mon[31006]: pgmap v1149: 129 pgs: 129 active+clean; 352 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1004 KiB/s wr, 92 op/s 2024-09-21T03:04:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:25 smithi066 ceph-mon[33436]: pgmap v1149: 129 pgs: 129 active+clean; 352 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1004 KiB/s wr, 92 op/s 2024-09-21T03:04:26.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 603 sec 2024-09-21T03:04:27.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 604 sec 2024-09-21T03:04:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:27 smithi190 ceph-mon[33871]: pgmap v1150: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-09-21T03:04:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:27 smithi029 ceph-mon[31006]: pgmap v1150: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-09-21T03:04:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:27 smithi066 ceph-mon[33436]: pgmap v1150: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-09-21T03:04:28.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 605 sec 2024-09-21T03:04:29.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 606 sec 2024-09-21T03:04:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:29 smithi190 ceph-mon[33871]: pgmap v1151: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-09-21T03:04:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:29 smithi029 ceph-mon[31006]: pgmap v1151: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-09-21T03:04:30.009 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:29 smithi066 ceph-mon[33436]: pgmap v1151: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-09-21T03:04:30.753 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 607 sec 2024-09-21T03:04:31.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 608 sec 2024-09-21T03:04:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:31 smithi190 ceph-mon[33871]: pgmap v1152: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-09-21T03:04:31.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:31 smithi029 ceph-mon[31006]: pgmap v1152: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-09-21T03:04:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:31 smithi066 ceph-mon[33436]: pgmap v1152: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-09-21T03:04:32.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 609 sec 2024-09-21T03:04:33.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 610 sec 2024-09-21T03:04:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:33 smithi190 ceph-mon[33871]: pgmap v1153: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-09-21T03:04:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:33 smithi029 ceph-mon[31006]: pgmap v1153: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-09-21T03:04:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:33 smithi066 ceph-mon[33436]: pgmap v1153: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-09-21T03:04:34.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 611 sec 2024-09-21T03:04:35.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 612 sec 2024-09-21T03:04:35.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:35 smithi190 ceph-mon[33871]: pgmap v1154: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 820 KiB/s wr, 125 op/s 2024-09-21T03:04:35.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:35 smithi029 ceph-mon[31006]: pgmap v1154: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 820 KiB/s wr, 125 op/s 2024-09-21T03:04:36.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:35 smithi066 ceph-mon[33436]: pgmap v1154: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 820 KiB/s wr, 125 op/s 2024-09-21T03:04:36.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 613 sec 2024-09-21T03:04:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 -- 172.21.15.29:0/3666727056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103620 msgr2=0x7f11c4103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 --2- 172.21.15.29:0/3666727056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103620 0x7f11c4103a30 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009ca0 tx=0x7f11b802a8c0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 -- 172.21.15.29:0/3666727056 shutdown_connections 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 --2- 172.21.15.29:0/3666727056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11c4103fc0 0x7f11c410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 --2- 172.21.15.29:0/3666727056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103620 0x7f11c4103a30 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 --2- 172.21.15.29:0/3666727056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11c41025a0 0x7f11c4102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 -- 172.21.15.29:0/3666727056 >> 172.21.15.29:0/3666727056 conn(0x7f11c40fdee0 msgr2=0x7f11c4100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 -- 172.21.15.29:0/3666727056 shutdown_connections 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 -- 172.21.15.29:0/3666727056 wait complete. 2024-09-21T03:04:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.460+0000 7f11c9939640 1 Processor -- start 2024-09-21T03:04:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c9939640 1 -- start start 2024-09-21T03:04:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c9939640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11c41025a0 0x7f11c4199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c9939640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11c4103620 0x7f11c419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c9939640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103fc0 0x7f11c419b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c9939640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410f190 con 0x7f11c4103fc0 2024-09-21T03:04:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c9939640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410f010 con 0x7f11c4103620 2024-09-21T03:04:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c9939640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410f310 con 0x7f11c41025a0 2024-09-21T03:04:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103fc0 0x7f11c419b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103fc0 0x7f11c419b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48096/0 (socket says 172.21.15.29:48096) 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c37fe640 1 -- 172.21.15.29:0/3136272069 learned_addr learned my addr 172.21.15.29:0/3136272069 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11c4103620 0x7f11c419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11c41025a0 0x7f11c4199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c37fe640 1 -- 172.21.15.29:0/3136272069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11c41025a0 msgr2=0x7f11c4199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c37fe640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11c41025a0 0x7f11c4199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c37fe640 1 -- 172.21.15.29:0/3136272069 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11c4103620 msgr2=0x7f11c419a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c37fe640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11c4103620 0x7f11c419a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.461+0000 7f11c37fe640 1 -- 172.21.15.29:0/3136272069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c419b900 con 0x7f11c4103fc0 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.462+0000 7f11c37fe640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103fc0 0x7f11c419b0e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00d740 tx=0x7f11bc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.462+0000 7f11c8937640 1 -- 172.21.15.29:0/3136272069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc017070 con 0x7f11c4103fc0 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.462+0000 7f11c8937640 1 -- 172.21.15.29:0/3136272069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11bc0044b0 con 0x7f11c4103fc0 2024-09-21T03:04:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.462+0000 7f11c9939640 1 -- 172.21.15.29:0/3136272069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c419bbf0 con 0x7f11c4103fc0 2024-09-21T03:04:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.462+0000 7f11c8937640 1 -- 172.21.15.29:0/3136272069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc003ae0 con 0x7f11c4103fc0 2024-09-21T03:04:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.462+0000 7f11c9939640 1 -- 172.21.15.29:0/3136272069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c41a22e0 con 0x7f11c4103fc0 2024-09-21T03:04:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11d08ab740 1 Processor -- start 2024-09-21T03:04:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11c8937640 1 -- 172.21.15.29:0/3136272069 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f11bc009c30 con 0x7f11c4103fc0 2024-09-21T03:04:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11d08ab740 1 -- start start 2024-09-21T03:04:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653687f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653687f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 0x5653687f9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11d08ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56536876a6f0 con 0x5653684272f0 2024-09-21T03:04:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11d08ab740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x565368759b20 con 0x5653684501b0 2024-09-21T03:04:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.465+0000 7f11d08ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56536875d930 con 0x56536842f4d0 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653687f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653687f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48112/0 (socket says 172.21.15.29:48112) 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 -- 172.21.15.29:0/3973665993 learned_addr learned my addr 172.21.15.29:0/3973665993 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 0x5653687f9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653687f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 -- 172.21.15.29:0/3973665993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 msgr2=0x5653687f9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 0x5653687f9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 -- 172.21.15.29:0/3973665993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 msgr2=0x5653687f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653687f7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c8937640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f11ac0b6590 0x7f11ac0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 -- 172.21.15.29:0/3973665993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56536871f530 con 0x5653684272f0 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f11ac0b6590 0x7f11ac0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653687f84c0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009c90 tx=0x7f11b40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9104720b344ecb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11c8937640 1 -- 172.21.15.29:0/3136272069 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f11bc0e6fe0 con 0x7f11c4103fc0 2024-09-21T03:04:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.466+0000 7f11a9ffb640 1 -- 172.21.15.29:0/3973665993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b4012450 con 0x5653684272f0 2024-09-21T03:04:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11a9ffb640 1 -- 172.21.15.29:0/3973665993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11b4012a80 con 0x5653684272f0 2024-09-21T03:04:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11a9ffb640 1 -- 172.21.15.29:0/3973665993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b400fae0 con 0x5653684272f0 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11d08ab740 1 -- 172.21.15.29:0/3973665993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 msgr2=0x5653687f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f11ac0b6590 0x7f11ac0b8aa0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f11b8005e30 tx=0x7f11b80318e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653687f84c0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009c90 tx=0x7f11b40117b0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11d08ab740 1 -- 172.21.15.29:0/3973665993 shutdown_connections 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 0x5653687f9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653687f84c0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3973665993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653687f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11d08ab740 1 -- 172.21.15.29:0/3973665993 >> 172.21.15.29:0/3973665993 conn(0x565368444300 msgr2=0x5653687f38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.467+0000 7f11d08ab740 1 -- 172.21.15.29:0/3973665993 shutdown_connections 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 -- 172.21.15.29:0/3973665993 wait complete. 2024-09-21T03:04:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 -- start start 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 Processor -- start 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653688015c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 0x565368801c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688027f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x565368691ab0 con 0x5653684272f0 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5653686ff730 con 0x5653684501b0 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.468+0000 7f11d08ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5653687f9b30 con 0x56536842f4d0 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11d08ab740 10 client.0 _ll_register_callbacks cb 0x5653687536b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653688015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653688015c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48114/0 (socket says 172.21.15.29:48114) 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 -- 172.21.15.29:0/3779406931 learned_addr learned my addr 172.21.15.29:0/3779406931 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 0x565368801c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c37fe640 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 -- 172.21.15.29:0/3779406931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 msgr2=0x565368801c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 0x565368801c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 -- 172.21.15.29:0/3779406931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 msgr2=0x5653688027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 -- 172.21.15.29:0/3779406931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653684bb790 con 0x5653684272f0 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c37fe640 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653688015c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009c60 tx=0x7f11b400fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3dbde16e0613e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11a2ffd640 1 -- 172.21.15.29:0/3779406931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b4029020 con 0x5653684272f0 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11a2ffd640 1 -- 172.21.15.29:0/3779406931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f11b4009d80 con 0x5653684272f0 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.469+0000 7f11d08ab740 1 -- 172.21.15.29:0/3779406931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11b4012a80 con 0x5653684272f0 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11a2ffd640 1 -- 172.21.15.29:0/3779406931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f11b4034050 con 0x5653684272f0 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11d08ab740 10 client.18792 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11d08ab740 1 -- 172.21.15.29:0/3779406931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x565368534620 con 0x5653684272f0 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11a2ffd640 1 -- 172.21.15.29:0/3779406931 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11b4012a80 con 0x5653684272f0 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11a2ffd640 1 -- 172.21.15.29:0/3779406931 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f11b40046c0 con 0x5653684272f0 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11d08ab740 10 client.18792 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11d08ab740 10 client.18792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11d08ab740 20 client.18792 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56536840ed80 0x56536880ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11d08ab740 4 client.18792 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11d08ab740 1 -- 172.21.15.29:0/3779406931 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56536876c640 con 0x56536840ed80 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.470+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56536840ed80 0x56536880ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.471+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56536840ed80 0x56536880ea90 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed35dbda2115e078 server_cookie=c16d109f390b81db in_seq=0 out_seq=0 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.471+0000 7f11a2ffd640 10 client.18792 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11a2ffd640 1 -- 172.21.15.29:0/3779406931 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56536876c640 con 0x56536840ed80 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11a2ffd640 10 client.18792 handle_command_reply: tid=0 2024-09-21T03:04:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 client.18792 shutdown 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 -- 172.21.15.29:0/3779406931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 msgr2=0x5653688015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653688015c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009c60 tx=0x7f11b400fa50 comp rx=0 tx=0).stop 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 -- 172.21.15.29:0/3779406931 shutdown_connections 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56536842f4d0 0x565368801c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x56536840ed80 0x56536880ea90 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3779406931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684272f0 0x5653688015c0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11a2ffd640 0 client.18792 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 -- 172.21.15.29:0/3779406931 >> 172.21.15.29:0/3779406931 conn(0x565368444300 msgr2=0x5653687f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 -- 172.21.15.29:0/3779406931 shutdown_connections 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 1 -- 172.21.15.29:0/3779406931 wait complete. 2024-09-21T03:04:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.472+0000 7f11d08ab740 20 client.18792 trim_cache size 0 max 16384 2024-09-21T03:04:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11d08ab740 1 Processor -- start 2024-09-21T03:04:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11d08ab740 1 -- start start 2024-09-21T03:04:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 0x5653687ffb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56536842f4d0 0x565368800050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684501b0 0x5653687f9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11d08ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41ac4a0 con 0x5653684501b0 2024-09-21T03:04:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11d08ab740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41adfe0 con 0x56536842f4d0 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11d08ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41acb70 con 0x5653684272f0 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684501b0 0x5653687f9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684501b0 0x5653687f9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48122/0 (socket says 172.21.15.29:48122) 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c37fe640 1 -- 172.21.15.29:0/3253780167 learned_addr learned my addr 172.21.15.29:0/3253780167 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 0x5653687ffb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c27fc640 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56536842f4d0 0x565368800050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c37fe640 1 -- 172.21.15.29:0/3253780167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 msgr2=0x5653687ffb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c37fe640 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 0x5653687ffb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c37fe640 1 -- 172.21.15.29:0/3253780167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56536842f4d0 msgr2=0x565368800050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c37fe640 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56536842f4d0 0x565368800050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.487+0000 7f11c37fe640 1 -- 172.21.15.29:0/3253780167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653687f9a10 con 0x5653684501b0 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.488+0000 7f11c37fe640 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684501b0 0x5653687f9ed0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0acd20 tx=0x7f11bc0ac500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872332c393290d2c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.488+0000 7f11a97fa640 1 -- 172.21.15.29:0/3253780167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc020040 con 0x5653684501b0 2024-09-21T03:04:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.488+0000 7f11a97fa640 1 -- 172.21.15.29:0/3253780167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11bc0ad490 con 0x5653684501b0 2024-09-21T03:04:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.488+0000 7f11a97fa640 1 -- 172.21.15.29:0/3253780167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc0e4bc0 con 0x5653684501b0 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 -- 172.21.15.29:0/3253780167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684501b0 msgr2=0x5653687f9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684501b0 0x5653687f9ed0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0acd20 tx=0x7f11bc0ac500 comp rx=0 tx=0).stop 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 -- 172.21.15.29:0/3253780167 shutdown_connections 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5653684501b0 0x5653687f9ed0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x56536842f4d0 0x565368800050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 --2- 172.21.15.29:0/3253780167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 0x5653687ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 -- 172.21.15.29:0/3253780167 >> 172.21.15.29:0/3253780167 conn(0x565368444300 msgr2=0x5653687f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 -- 172.21.15.29:0/3253780167 shutdown_connections 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 -- 172.21.15.29:0/3253780167 wait complete. 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.489+0000 7f11d08ab740 1 -- start start 2024-09-21T03:04:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11d08ab740 1 Processor -- start 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 0x5653688065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56536842f4d0 0x565368806c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11d08ab740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688077f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11d08ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41ad910 con 0x56536842f4d0 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11d08ab740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41ad240 con 0x5653684501b0 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11d08ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41aa480 con 0x5653684272f0 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11d08ab740 10 client.0 _ll_register_callbacks cb 0x7f11b4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56536842f4d0 0x565368806c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56536842f4d0 0x565368806c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48128/0 (socket says 172.21.15.29:48128) 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c27fc640 1 -- 172.21.15.29:0/4041114982 learned_addr learned my addr 172.21.15.29:0/4041114982 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c37fe640 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688077f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 0x5653688065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c27fc640 1 -- 172.21.15.29:0/4041114982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 msgr2=0x5653688065c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c27fc640 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 0x5653688065c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c27fc640 1 -- 172.21.15.29:0/4041114982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 msgr2=0x5653688077f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c27fc640 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688077f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.490+0000 7f11c27fc640 1 -- 172.21.15.29:0/4041114982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565368807f50 con 0x56536842f4d0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11c27fc640 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56536842f4d0 0x565368806c80 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f11b8031840 tx=0x7f11b802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7d95a63fe54ac2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11a2ffd640 1 -- 172.21.15.29:0/4041114982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8041040 con 0x56536842f4d0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11a2ffd640 1 -- 172.21.15.29:0/4041114982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f11b8004710 con 0x56536842f4d0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 1 -- 172.21.15.29:0/4041114982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11bc0ad490 con 0x56536842f4d0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11a2ffd640 1 -- 172.21.15.29:0/4041114982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f11b8062050 con 0x56536842f4d0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 10 client.18798 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 1 -- 172.21.15.29:0/4041114982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5653688092e0 con 0x56536842f4d0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11a2ffd640 1 -- 172.21.15.29:0/4041114982 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11bc0ad490 con 0x56536842f4d0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11a2ffd640 1 -- 172.21.15.29:0/4041114982 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f11b803d420 con 0x56536842f4d0 2024-09-21T03:04:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 10 client.18798 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 10 client.18798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T03:04:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 20 client.18798 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5653683e41c0 0x56536887b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 4 client.18798 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T03:04:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.491+0000 7f11d08ab740 1 -- 172.21.15.29:0/4041114982 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5653687680f0 con 0x5653683e41c0 2024-09-21T03:04:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.492+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5653683e41c0 0x56536887b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.492+0000 7f11a2ffd640 10 client.18798 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.492+0000 7f11c2ffd640 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5653683e41c0 0x56536887b5b0 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f40099b2d6e6fb1 server_cookie=7404a69cea9c3fa2 in_seq=0 out_seq=0 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11a2ffd640 1 -- 172.21.15.29:0/4041114982 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5653687680f0 con 0x5653683e41c0 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11a2ffd640 10 client.18798 handle_command_reply: tid=0 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 client.18798 shutdown 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 -- 172.21.15.29:0/4041114982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56536842f4d0 msgr2=0x565368806c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56536842f4d0 0x565368806c80 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f11b8031840 tx=0x7f11b802a8c0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 -- 172.21.15.29:0/4041114982 shutdown_connections 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5653684501b0 0x5653688077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x5653683e41c0 0x56536887b5b0 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56536842f4d0 0x565368806c80 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 --2- 172.21.15.29:0/4041114982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5653684272f0 0x5653688065c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11a2ffd640 0 client.18798 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 -- 172.21.15.29:0/4041114982 >> 172.21.15.29:0/4041114982 conn(0x565368444300 msgr2=0x5653687ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 -- 172.21.15.29:0/4041114982 shutdown_connections 2024-09-21T03:04:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 1 -- 172.21.15.29:0/4041114982 wait complete. 2024-09-21T03:04:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.534+0000 7f11d08ab740 20 client.18798 trim_cache size 0 max 16384 2024-09-21T03:04:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-09-21T03:04:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-09-21T03:04:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-09-21T03:04:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-09-21T03:04:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 -- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f11ac0b6590 msgr2=0x7f11ac0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f11ac0b6590 0x7f11ac0b8aa0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f11b8005e30 tx=0x7f11b80318e0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 -- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103fc0 msgr2=0x7f11c419b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103fc0 0x7f11c419b0e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00d740 tx=0x7f11bc00dc00 comp rx=0 tx=0).stop 2024-09-21T03:04:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 -- 172.21.15.29:0/3136272069 shutdown_connections 2024-09-21T03:04:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f11ac0b6590 0x7f11ac0b8aa0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c4103fc0 0x7f11c419b0e0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11c4103620 0x7f11c419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 --2- 172.21.15.29:0/3136272069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11c41025a0 0x7f11c4199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 -- 172.21.15.29:0/3136272069 >> 172.21.15.29:0/3136272069 conn(0x7f11c40fdee0 msgr2=0x7f11c4106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 -- 172.21.15.29:0/3136272069 shutdown_connections 2024-09-21T03:04:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-09-21T03:04:37.535+0000 7f11c9939640 1 -- 172.21.15.29:0/3136272069 wait complete. 2024-09-21T03:04:37.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 614 sec 2024-09-21T03:04:37.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:37 smithi190 ceph-mon[33871]: pgmap v1155: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 445 KiB/s wr, 95 op/s 2024-09-21T03:04:37.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:37 smithi029 ceph-mon[31006]: pgmap v1155: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 445 KiB/s wr, 95 op/s 2024-09-21T03:04:38.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:37 smithi066 ceph-mon[33436]: pgmap v1155: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 445 KiB/s wr, 95 op/s 2024-09-21T03:04:38.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 615 sec 2024-09-21T03:04:39.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:04:39.635 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:39.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 616 sec 2024-09-21T03:04:39.756 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:39 smithi029 ceph-mon[31006]: pgmap v1156: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 249 KiB/s wr, 50 op/s 2024-09-21T03:04:39.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:39 smithi190 ceph-mon[33871]: pgmap v1156: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 249 KiB/s wr, 50 op/s 2024-09-21T03:04:40.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:39 smithi066 ceph-mon[33436]: pgmap v1156: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 249 KiB/s wr, 50 op/s 2024-09-21T03:04:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.154+0000 7f5d4c362640 1 -- 172.21.15.29:0/1680243592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44102ae0 msgr2=0x7f5d44102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.154+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1680243592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44102ae0 0x7f5d44102ef0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009900 tx=0x7f5d4002a410 comp rx=0 tx=0).stop 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 -- 172.21.15.29:0/1680243592 shutdown_connections 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1680243592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d44108d20 0x7f5d4410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1680243592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d441083d0 0x7f5d441087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1680243592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44102ae0 0x7f5d44102ef0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 -- 172.21.15.29:0/1680243592 >> 172.21.15.29:0/1680243592 conn(0x7f5d44079790 msgr2=0x7f5d44079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 -- 172.21.15.29:0/1680243592 shutdown_connections 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 -- 172.21.15.29:0/1680243592 wait complete. 2024-09-21T03:04:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 Processor -- start 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.155+0000 7f5d4c362640 1 -- start start 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4c362640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d44102ae0 0x7f5d4419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4c362640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d441083d0 0x7f5d4419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4c362640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44108d20 0x7f5d4419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4c362640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44112890 con 0x7f5d44108d20 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4c362640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44112710 con 0x7f5d441083d0 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4c362640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44112a10 con 0x7f5d44102ae0 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44108d20 0x7f5d4419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44108d20 0x7f5d4419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48134/0 (socket says 172.21.15.29:48134) 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 -- 172.21.15.29:0/1639315821 learned_addr learned my addr 172.21.15.29:0/1639315821 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a0fe640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d44102ae0 0x7f5d4419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d498fd640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d441083d0 0x7f5d4419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 -- 172.21.15.29:0/1639315821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d44102ae0 msgr2=0x7f5d4419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d44102ae0 0x7f5d4419a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 -- 172.21.15.29:0/1639315821 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d441083d0 msgr2=0x7f5d4419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d441083d0 0x7f5d4419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 -- 172.21.15.29:0/1639315821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d441a02e0 con 0x7f5d44108d20 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d498fd640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d441083d0 0x7f5d4419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.156+0000 7f5d4a8ff640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44108d20 0x7f5d4419fc70 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00cce0 tx=0x7f5d3c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.157+0000 7f5d3b7fe640 1 -- 172.21.15.29:0/1639315821 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00f450 con 0x7f5d44108d20 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.157+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d441a05d0 con 0x7f5d44108d20 2024-09-21T03:04:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.157+0000 7f5d3b7fe640 1 -- 172.21.15.29:0/1639315821 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d3c00fa80 con 0x7f5d44108d20 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.157+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d441a6df0 con 0x7f5d44108d20 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.157+0000 7f5d3b7fe640 1 -- 172.21.15.29:0/1639315821 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c018a70 con 0x7f5d44108d20 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.158+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d44102ef0 con 0x7f5d44108d20 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.159+0000 7f5d3b7fe640 1 -- 172.21.15.29:0/1639315821 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f5d3c018c10 con 0x7f5d44108d20 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.160+0000 7f5d3b7fe640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d300b6540 0x7f5d300b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.160+0000 7f5d3b7fe640 1 -- 172.21.15.29:0/1639315821 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f5d3c0e7a60 con 0x7f5d44108d20 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.160+0000 7f5d4a0fe640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d300b6540 0x7f5d300b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.161+0000 7f5d4a0fe640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d300b6540 0x7f5d300b8a50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5d400057c0 tx=0x7f5d400058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.165+0000 7f5d3b7fe640 1 -- 172.21.15.29:0/1639315821 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5d3c014030 con 0x7f5d44108d20 2024-09-21T03:04:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.390+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d4410bb30 con 0x7f5d44108d20 2024-09-21T03:04:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.391+0000 7f5d3b7fe640 1 -- 172.21.15.29:0/1639315821 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11971 (secure 0 0 0) 0x7f5d3c0adf70 con 0x7f5d44108d20 2024-09-21T03:04:40.392 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:04:40.393 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"btime":"2024-09-21T03:02:24:667358+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:02:24.667336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:04:40.393 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d300b6540 msgr2=0x7f5d300b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d300b6540 0x7f5d300b8a50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5d400057c0 tx=0x7f5d400058d0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44108d20 msgr2=0x7f5d4419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44108d20 0x7f5d4419fc70 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00cce0 tx=0x7f5d3c00a620 comp rx=0 tx=0).stop 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 shutdown_connections 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5d300b6540 0x7f5d300b8a50 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d44108d20 0x7f5d4419fc70 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d441083d0 0x7f5d4419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 --2- 172.21.15.29:0/1639315821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d44102ae0 0x7f5d4419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.393+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 >> 172.21.15.29:0/1639315821 conn(0x7f5d44079790 msgr2=0x7f5d44101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.394+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 shutdown_connections 2024-09-21T03:04:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:40.394+0000 7f5d4c362640 1 -- 172.21.15.29:0/1639315821 wait complete. 2024-09-21T03:04:40.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 617 sec 2024-09-21T03:04:40.784 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:40 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1639315821' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:40.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:40 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1639315821' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:41.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:40 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1639315821' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:41.133 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --set-id ae657674 --release --await' 2024-09-21T03:04:41.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --set-id ae657674 --release --await 2024-09-21T03:04:41.396 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:41.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 618 sec 2024-09-21T03:04:41.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:41 smithi029 ceph-mon[31006]: pgmap v1157: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 190 KiB/s wr, 28 op/s 2024-09-21T03:04:41.873 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:41 smithi190 ceph-mon[33871]: pgmap v1157: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 190 KiB/s wr, 28 op/s 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 -- 172.21.15.29:0/3553888573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 msgr2=0x7f3e8c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 --2- 172.21.15.29:0/3553888573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 0x7f3e8c102e70 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009900 tx=0x7f3e7c02a410 comp rx=0 tx=0).stop 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 -- 172.21.15.29:0/3553888573 shutdown_connections 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 --2- 172.21.15.29:0/3553888573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e8c108cf0 0x7f3e8c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 --2- 172.21.15.29:0/3553888573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c1083a0 0x7f3e8c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 --2- 172.21.15.29:0/3553888573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 0x7f3e8c102e70 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 -- 172.21.15.29:0/3553888573 >> 172.21.15.29:0/3553888573 conn(0x7f3e8c079730 msgr2=0x7f3e8c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 -- 172.21.15.29:0/3553888573 shutdown_connections 2024-09-21T03:04:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.942+0000 7f3e9232c640 1 -- 172.21.15.29:0/3553888573 wait complete. 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e9232c640 1 Processor -- start 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e9232c640 1 -- start start 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e9232c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 0x7f3e8c199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e9232c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c1083a0 0x7f3e8c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e9232c640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e8c108cf0 0x7f3e8c19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e9232c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c112890 con 0x7f3e8c102a60 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e9232c640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c112710 con 0x7f3e8c108cf0 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e9232c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c112a10 con 0x7f3e8c1083a0 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 0x7f3e8c199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 0x7f3e8c199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48156/0 (socket says 172.21.15.29:48156) 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e8bfff640 1 -- 172.21.15.29:0/4252223570 learned_addr learned my addr 172.21.15.29:0/4252223570 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.943+0000 7f3e8b7fe640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c1083a0 0x7f3e8c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e908c9640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e8c108cf0 0x7f3e8c19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e8bfff640 1 -- 172.21.15.29:0/4252223570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c1083a0 msgr2=0x7f3e8c1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c1083a0 0x7f3e8c1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e8bfff640 1 -- 172.21.15.29:0/4252223570 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e8c108cf0 msgr2=0x7f3e8c19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e8c108cf0 0x7f3e8c19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e8bfff640 1 -- 172.21.15.29:0/4252223570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c19f270 con 0x7f3e8c102a60 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 0x7f3e8c199300 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009230 tx=0x7f3e7c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e897fa640 1 -- 172.21.15.29:0/4252223570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c03f030 con 0x7f3e8c102a60 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e9232c640 1 -- 172.21.15.29:0/4252223570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c19f500 con 0x7f3e8c102a60 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e897fa640 1 -- 172.21.15.29:0/4252223570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e7c003a80 con 0x7f3e8c102a60 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e897fa640 1 -- 172.21.15.29:0/4252223570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c035070 con 0x7f3e8c102a60 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.944+0000 7f3e9232c640 1 -- 172.21.15.29:0/4252223570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c1a5cc0 con 0x7f3e8c102a60 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e897fa640 1 -- 172.21.15.29:0/4252223570 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f3e7c042030 con 0x7f3e8c102a60 2024-09-21T03:04:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e988a0740 1 Processor -- start 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e988a0740 1 -- start start 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f39520 0x5651082e41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082e1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 0x5651082e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e988a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5651081cadc0 con 0x565107f10660 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e988a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x5651082571f0 con 0x565107f18840 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e988a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x565108246620 con 0x565107f39520 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082e1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.947+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082e1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48162/0 (socket says 172.21.15.29:48162) 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e8bfff640 1 -- 172.21.15.29:0/3612460988 learned_addr learned my addr 172.21.15.29:0/3612460988 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e908c9640 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f39520 0x5651082e41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e8b7fe640 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 0x5651082e1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e897fa640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3e740b6510 0x7f3e740b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e8bfff640 1 -- 172.21.15.29:0/3612460988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f39520 msgr2=0x5651082e41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f39520 0x5651082e41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e8bfff640 1 -- 172.21.15.29:0/3612460988 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 msgr2=0x5651082e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 0x5651082e1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e8bfff640 1 -- 172.21.15.29:0/3612460988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56510820efd0 con 0x565107f10660 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e908c9640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3e740b6510 0x7f3e740b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e897fa640 1 -- 172.21.15.29:0/4252223570 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f3e7c107c30 con 0x7f3e8c102a60 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082e1720 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0c5f90 tx=0x7f3e7c0c22f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2556f5f84b01ab49 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.948+0000 7f3e727fc640 1 -- 172.21.15.29:0/3612460988 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0cd390 con 0x565107f10660 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e908c9640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3e740b6510 0x7f3e740b8a20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400a4e0 tx=0x7f3e8400c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e727fc640 1 -- 172.21.15.29:0/3612460988 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e7c0c5ff0 con 0x565107f10660 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e727fc640 1 -- 172.21.15.29:0/3612460988 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0cd1b0 con 0x565107f10660 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 -- 172.21.15.29:0/3612460988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 msgr2=0x5651082e1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:41.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082e1720 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0c5f90 tx=0x7f3e7c0c22f0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 -- 172.21.15.29:0/3612460988 shutdown_connections 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 0x5651082e1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082e1720 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3612460988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f39520 0x5651082e41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 -- 172.21.15.29:0/3612460988 >> 172.21.15.29:0/3612460988 conn(0x565107f2d910 msgr2=0x5651082df610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 -- 172.21.15.29:0/3612460988 shutdown_connections 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.949+0000 7f3e988a0740 1 -- 172.21.15.29:0/3612460988 wait complete. 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.950+0000 7f3e988a0740 1 -- start start 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.950+0000 7f3e988a0740 1 Processor -- start 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.950+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f10660 0x5651082ef2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.950+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082ef880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.950+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x5651082f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.950+0000 7f3e988a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5651081e1300 con 0x565107f39520 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.950+0000 7f3e988a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x56510824a430 con 0x565107f10660 2024-09-21T03:04:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.950+0000 7f3e988a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5651080a7bb0 con 0x565107f18840 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e988a0740 10 client.0 _ll_register_callbacks cb 0x5651081fb120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x5651082f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x5651082f48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48174/0 (socket says 172.21.15.29:48174) 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 -- 172.21.15.29:0/3147117554 learned_addr learned my addr 172.21.15.29:0/3147117554 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e8b7fe640 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f10660 0x5651082ef2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 -- 172.21.15.29:0/3147117554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 msgr2=0x5651082ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082ef880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 -- 172.21.15.29:0/3147117554 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f10660 msgr2=0x5651082ef2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f10660 0x5651082ef2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 -- 172.21.15.29:0/3147117554 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651081d0780 con 0x565107f39520 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f10660 0x5651082ef2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e908c9640 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x5651082f48a0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f3e84019ea0 tx=0x7f3e8400e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4acb2a2d95876c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.951+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3147117554 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e840190d0 con 0x565107f39520 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3147117554 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f3e84019270 con 0x565107f39520 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 1 -- 172.21.15.29:0/3147117554 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565108259140 con 0x565107f39520 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3147117554 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3e84017420 con 0x565107f39520 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 10 client.18816 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 1 -- 172.21.15.29:0/3147117554 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x565107fa4b00 con 0x565107f39520 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3147117554 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x565108259140 con 0x565107f39520 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3147117554 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f3e84022690 con 0x565107f39520 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 10 client.18816 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 10 client.18816 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 20 client.18816 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565107ef7ea0 0x565108300f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 4 client.18816 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:04:41.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.952+0000 7f3e988a0740 1 -- 172.21.15.29:0/3147117554 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565108254bf0 con 0x565107ef7ea0 2024-09-21T03:04:41.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.953+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565107ef7ea0 0x565108300f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.953+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565107ef7ea0 0x565108300f70 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=652f99489b34187e server_cookie=7b92df7960d18c9f in_seq=0 out_seq=0 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.953+0000 7f3e6b7fe640 10 client.18816 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3147117554 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x565108254bf0 con 0x565107ef7ea0 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e6b7fe640 10 client.18816 handle_command_reply: tid=0 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 client.18816 shutdown 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 -- 172.21.15.29:0/3147117554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 msgr2=0x5651082f48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x5651082f48a0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f3e84019ea0 tx=0x7f3e8400e040 comp rx=0 tx=0).stop 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 -- 172.21.15.29:0/3147117554 shutdown_connections 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565107ef7ea0 0x565108300f70 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x5651082f48a0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e6b7fe640 0 client.18816 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3147117554 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f10660 0x5651082ef2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 -- 172.21.15.29:0/3147117554 >> 172.21.15.29:0/3147117554 conn(0x565107f2d910 msgr2=0x5651082e3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 -- 172.21.15.29:0/3147117554 shutdown_connections 2024-09-21T03:04:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 1 -- 172.21.15.29:0/3147117554 wait complete. 2024-09-21T03:04:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.954+0000 7f3e988a0740 20 client.18816 trim_cache size 0 max 16384 2024-09-21T03:04:41.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e988a0740 1 Processor -- start 2024-09-21T03:04:41.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e988a0740 1 -- start start 2024-09-21T03:04:41.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082f1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f39520 0x5651082f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e988a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1ae8a0 con 0x565107f10660 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e988a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1b03e0 con 0x565107f39520 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e988a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1aef70 con 0x565107f18840 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082f2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48180/0 (socket says 172.21.15.29:48180) 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 -- 172.21.15.29:0/89625334 learned_addr learned my addr 172.21.15.29:0/89625334 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8b7fe640 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 -- 172.21.15.29:0/89625334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 msgr2=0x5651082f1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082f1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 -- 172.21.15.29:0/89625334 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f39520 msgr2=0x5651082f1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e908c9640 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f39520 0x5651082f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f39520 0x5651082f1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 -- 172.21.15.29:0/89625334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56510801b960 con 0x565107f10660 2024-09-21T03:04:41.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8b7fe640 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:41.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e908c9640 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f39520 0x5651082f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:04:41.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.972+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082f2dc0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0c8070 tx=0x7f3e7c0b5040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a375493a474d4e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:41.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e71ffb640 1 -- 172.21.15.29:0/89625334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0cd390 con 0x565107f10660 2024-09-21T03:04:41.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e71ffb640 1 -- 172.21.15.29:0/89625334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e7c0cd550 con 0x565107f10660 2024-09-21T03:04:41.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e71ffb640 1 -- 172.21.15.29:0/89625334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0a40b0 con 0x565107f10660 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e988a0740 1 -- 172.21.15.29:0/89625334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 msgr2=0x5651082f2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e988a0740 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082f2dc0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0c8070 tx=0x7f3e7c0b5040 comp rx=0 tx=0).stop 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e988a0740 1 -- 172.21.15.29:0/89625334 shutdown_connections 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e988a0740 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f39520 0x5651082f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e988a0740 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f18840 0x5651082f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e988a0740 1 --2- 172.21.15.29:0/89625334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f10660 0x5651082f2dc0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.973+0000 7f3e988a0740 1 -- 172.21.15.29:0/89625334 >> 172.21.15.29:0/89625334 conn(0x565107f2d910 msgr2=0x5651082f26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.974+0000 7f3e988a0740 1 -- 172.21.15.29:0/89625334 shutdown_connections 2024-09-21T03:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.974+0000 7f3e988a0740 1 -- 172.21.15.29:0/89625334 wait complete. 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.974+0000 7f3e988a0740 1 -- start start 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.974+0000 7f3e988a0740 1 Processor -- start 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f10660 0x565108306ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 0x565108305760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e988a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x565108305ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e988a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1afd10 con 0x565107f39520 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e988a0740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1af640 con 0x565107f18840 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e988a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1ac880 con 0x565107f10660 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e988a0740 10 client.0 _ll_register_callbacks cb 0x7f3e84009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x565108305ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x565108305ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48184/0 (socket says 172.21.15.29:48184) 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 -- 172.21.15.29:0/3475277397 learned_addr learned my addr 172.21.15.29:0/3475277397 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f10660 0x565108306ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e8b7fe640 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 0x565108305760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 -- 172.21.15.29:0/3475277397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f10660 msgr2=0x565108306ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f10660 0x565108306ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 -- 172.21.15.29:0/3475277397 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 msgr2=0x565108305760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 0x565108305760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 -- 172.21.15.29:0/3475277397 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651083062e0 con 0x565107f39520 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e908c9640 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x565108305ca0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f3e84022ae0 tx=0x7f3e84016ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0e296db94db27e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.975+0000 7f3e6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3475277397 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84036030 con 0x565107f39520 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3475277397 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f3e8401f050 con 0x565107f39520 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 1 -- 172.21.15.29:0/3475277397 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e7c0cd550 con 0x565107f39520 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3475277397 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3e84040050 con 0x565107f39520 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 10 client.18822 fetch_fsmap learned FSMap version 39 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 1 -- 172.21.15.29:0/3475277397 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56510836c680 con 0x565107f39520 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3475277397 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e7c0cd550 con 0x565107f39520 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3475277397 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f3e84031440 con 0x565107f39520 2024-09-21T03:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 10 client.18822 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T03:04:41.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 10 client.18822 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:41.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 20 client.18822 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:41.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565107ecd250 0x56510836eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:41.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 4 client.18822 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ae657674", "release": true, "await": true}] 2024-09-21T03:04:41.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.976+0000 7f3e988a0740 1 -- 172.21.15.29:0/3475277397 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ae657674", "release": true, "await": true}) -- 0x565108252020 con 0x565107ecd250 2024-09-21T03:04:41.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.977+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565107ecd250 0x56510836eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:41.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.977+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565107ecd250 0x56510836eec0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9e3abf53cd476d server_cookie=71f98bdba4b683c in_seq=0 out_seq=0 2024-09-21T03:04:41.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:41.977+0000 7f3e6b7fe640 10 client.18822 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:42.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:41 smithi066 ceph-mon[33436]: pgmap v1157: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 190 KiB/s wr, 28 op/s 2024-09-21T03:04:42.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e6b7fe640 1 -- 172.21.15.29:0/3475277397 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+593 (crc 0 0 0) 0x565108252020 con 0x565107ecd250 2024-09-21T03:04:42.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e6b7fe640 10 client.18822 handle_command_reply: tid=0 2024-09-21T03:04:42.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 client.18822 shutdown 2024-09-21T03:04:42.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 -- 172.21.15.29:0/3475277397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 msgr2=0x565108305ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:42.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x565108305ca0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f3e84022ae0 tx=0x7f3e84016ca0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 -- 172.21.15.29:0/3475277397 shutdown_connections 2024-09-21T03:04:42.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x565107ecd250 0x56510836eec0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565107f39520 0x565108305ca0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e6b7fe640 0 client.18822 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x565107f18840 0x565108305760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 --2- 172.21.15.29:0/3475277397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565107f10660 0x565108306ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.039+0000 7f3e988a0740 1 -- 172.21.15.29:0/3475277397 >> 172.21.15.29:0/3475277397 conn(0x565107f2d910 msgr2=0x5651082ee0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.040+0000 7f3e988a0740 1 -- 172.21.15.29:0/3475277397 shutdown_connections 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.040+0000 7f3e988a0740 1 -- 172.21.15.29:0/3475277397 wait complete. 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.040+0000 7f3e988a0740 20 client.18822 trim_cache size 0 max 16384 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 39, 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 100, 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "ae657674": { 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 100, 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 139.8, 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-09-21T03:04:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:42.044 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.040+0000 7f3e9232c640 1 -- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3e740b6510 msgr2=0x7f3e740b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.040+0000 7f3e9232c640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3e740b6510 0x7f3e740b8a20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400a4e0 tx=0x7f3e8400c040 comp rx=0 tx=0).stop 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.040+0000 7f3e9232c640 1 -- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 msgr2=0x7f3e8c199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.040+0000 7f3e9232c640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 0x7f3e8c199300 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009230 tx=0x7f3e7c004270 comp rx=0 tx=0).stop 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.041+0000 7f3e9232c640 1 -- 172.21.15.29:0/4252223570 shutdown_connections 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.041+0000 7f3e9232c640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f3e740b6510 0x7f3e740b8a20 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.041+0000 7f3e9232c640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e8c108cf0 0x7f3e8c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.041+0000 7f3e9232c640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c1083a0 0x7f3e8c1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.041+0000 7f3e9232c640 1 --2- 172.21.15.29:0/4252223570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c102a60 0x7f3e8c199300 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.041+0000 7f3e9232c640 1 -- 172.21.15.29:0/4252223570 >> 172.21.15.29:0/4252223570 conn(0x7f3e8c079730 msgr2=0x7f3e8c101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.041+0000 7f3e9232c640 1 -- 172.21.15.29:0/4252223570 shutdown_connections 2024-09-21T03:04:42.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:42.041+0000 7f3e9232c640 1 -- 172.21.15.29:0/4252223570 wait complete. 2024-09-21T03:04:42.754 INFO:tasks.workunit.client.0.smithi029.stdout: 10 cleanup 619 sec 2024-09-21T03:04:42.898 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ae657674', total seconds elapsed: 23.3 2024-09-21T03:04:43.755 INFO:tasks.workunit.client.0.smithi029.stdout: 5 cleanup 620 sec 2024-09-21T03:04:43.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:43 smithi190 ceph-mon[33871]: pgmap v1158: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 34 KiB/s wr, 20 op/s 2024-09-21T03:04:43.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:43 smithi029 ceph-mon[31006]: pgmap v1158: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 34 KiB/s wr, 20 op/s 2024-09-21T03:04:43.964 INFO:tasks.workunit.client.0.smithi029.stdout: 0 cleanup 620 sec 2024-09-21T03:04:43.964 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-09-21T03:04:43.964 INFO:tasks.workunit.client.0.smithi029.stdout: Operation Count AvgLat MaxLat 2024-09-21T03:04:43.964 INFO:tasks.workunit.client.0.smithi029.stdout: ---------------------------------------- 2024-09-21T03:04:43.964 INFO:tasks.workunit.client.0.smithi029.stdout: NTCreateX 105307 19.561 30148.485 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Close 77229 0.195 356.006 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Rename 4445 57.701 30605.521 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Unlink 21305 19.601 30022.489 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Qpathinfo 95338 6.276 610.155 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Qfileinfo 16573 0.345 222.480 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Qfsinfo 17453 0.912 173.592 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Sfileinfo 8576 24.965 30158.250 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Find 36820 60.022 637.108 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: WriteX 51817 5.699 22497.323 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: ReadX 164418 0.007 5.888 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: LockX 340 2.682 50.128 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: UnlockX 340 2.397 39.764 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: Flush 7377 12.138 732.762 2024-09-21T03:04:43.965 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-09-21T03:04:43.966 INFO:tasks.workunit.client.0.smithi029.stdout:Throughput 5.44891 MB/sec 10 clients 10 procs max_latency=30605.527 ms 2024-09-21T03:04:43.966 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T03:04:43.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T03:04:44.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:43 smithi066 ceph-mon[33436]: pgmap v1158: 129 pgs: 129 active+clean; 348 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 34 KiB/s wr, 20 op/s 2024-09-21T03:04:45.462 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-09-21T03:04:45.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T03:04:45.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:45 smithi029 ceph-mon[31006]: pgmap v1159: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 79 KiB/s wr, 29 op/s 2024-09-21T03:04:45.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:45 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:45.801 DEBUG:teuthology.parallel:result is None 2024-09-21T03:04:45.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T03:04:45.838 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T03:04:45.839 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-21T03:04:45.854 INFO:tasks.quiescer:joining Quiescers 2024-09-21T03:04:45.854 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-21T03:04:45.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:04:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:45 smithi190 ceph-mon[33871]: pgmap v1159: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 79 KiB/s wr, 29 op/s 2024-09-21T03:04:45.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:45 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:45 smithi066 ceph-mon[33436]: pgmap v1159: 129 pgs: 129 active+clean; 349 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 79 KiB/s wr, 29 op/s 2024-09-21T03:04:46.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:45 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:46.136 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 -- 172.21.15.29:0/2137898501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890102ae0 msgr2=0x7f4890102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 --2- 172.21.15.29:0/2137898501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890102ae0 0x7f4890102ef0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f4880009900 tx=0x7f488002a410 comp rx=0 tx=0).stop 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 -- 172.21.15.29:0/2137898501 shutdown_connections 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 --2- 172.21.15.29:0/2137898501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4890108d20 0x7f489010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 --2- 172.21.15.29:0/2137898501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48901083d0 0x7f48901087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 --2- 172.21.15.29:0/2137898501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890102ae0 0x7f4890102ef0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 -- 172.21.15.29:0/2137898501 >> 172.21.15.29:0/2137898501 conn(0x7f4890079760 msgr2=0x7f4890079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 -- 172.21.15.29:0/2137898501 shutdown_connections 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.964+0000 7f4896dab640 1 -- 172.21.15.29:0/2137898501 wait complete. 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4896dab640 1 Processor -- start 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4896dab640 1 -- start start 2024-09-21T03:04:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4896dab640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4890102ae0 0x7f489019a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4896dab640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48901083d0 0x7f489019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4896dab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890108d20 0x7f489019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4896dab640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4890113960 con 0x7f4890108d20 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4896dab640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f48901137e0 con 0x7f48901083d0 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4896dab640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4890113ae0 con 0x7f4890102ae0 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4894b47640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4890102ae0 0x7f489019a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4895348640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890108d20 0x7f489019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4894b47640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4890102ae0 0x7f489019a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56868/0 (socket says 172.21.15.29:56868) 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4895348640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890108d20 0x7f489019fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35776/0 (socket says 172.21.15.29:35776) 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.965+0000 7f4894b47640 1 -- 172.21.15.29:0/2692684948 learned_addr learned my addr 172.21.15.29:0/2692684948 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f488ffff640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48901083d0 0x7f489019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4895348640 1 -- 172.21.15.29:0/2692684948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4890102ae0 msgr2=0x7f489019a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4895348640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4890102ae0 0x7f489019a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4895348640 1 -- 172.21.15.29:0/2692684948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48901083d0 msgr2=0x7f489019a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4895348640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48901083d0 0x7f489019a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4895348640 1 -- 172.21.15.29:0/2692684948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48901a0300 con 0x7f4890108d20 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4894b47640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4890102ae0 0x7f489019a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4895348640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890108d20 0x7f489019fbd0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f4888009fd0 tx=0x7f488800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f488dffb640 1 -- 172.21.15.29:0/2692684948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488800ce20 con 0x7f4890108d20 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48901a05f0 con 0x7f4890108d20 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f488dffb640 1 -- 172.21.15.29:0/2692684948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4888005020 con 0x7f4890108d20 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f488dffb640 1 -- 172.21.15.29:0/2692684948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888010690 con 0x7f4890108d20 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.966+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48901a6d50 con 0x7f4890108d20 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.967+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4890103d10 con 0x7f4890108d20 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.969+0000 7f488dffb640 1 -- 172.21.15.29:0/2692684948 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4888004a40 con 0x7f4890108d20 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.970+0000 7f488dffb640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48780b6510 0x7f48780b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:46.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.970+0000 7f4894b47640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48780b6510 0x7f48780b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.970+0000 7f488dffb640 1 -- 172.21.15.29:0/2692684948 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f48880e7d40 con 0x7f4890108d20 2024-09-21T03:04:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.970+0000 7f4894b47640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48780b6510 0x7f48780b8a20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4880032fd0 tx=0x7f48800095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:46.973+0000 7f488dffb640 1 -- 172.21.15.29:0/2692684948 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f48880ae250 con 0x7f4890108d20 2024-09-21T03:04:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.202+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4890102ef0 con 0x7f4890108d20 2024-09-21T03:04:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.203+0000 7f488dffb640 1 -- 172.21.15.29:0/2692684948 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11975 (secure 0 0 0) 0x7f48880b2630 con 0x7f4890108d20 2024-09-21T03:04:47.206 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:04:47.206 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":41,"btime":"2024-09-21T03:04:46:609757+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:04:46.396282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24599,24625,24613,24632,24637]},"id":1}]} 2024-09-21T03:04:47.206 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 41 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48780b6510 msgr2=0x7f48780b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48780b6510 0x7f48780b8a20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4880032fd0 tx=0x7f48800095c0 comp rx=0 tx=0).stop 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890108d20 msgr2=0x7f489019fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890108d20 0x7f489019fbd0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f4888009fd0 tx=0x7f488800ef10 comp rx=0 tx=0).stop 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 shutdown_connections 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f48780b6510 0x7f48780b8a20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4890108d20 0x7f489019fbd0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f48901083d0 0x7f489019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 --2- 172.21.15.29:0/2692684948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4890102ae0 0x7f489019a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 >> 172.21.15.29:0/2692684948 conn(0x7f4890079760 msgr2=0x7f48901015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 shutdown_connections 2024-09-21T03:04:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:47.205+0000 7f4896dab640 1 -- 172.21.15.29:0/2692684948 wait complete. 2024-09-21T03:04:47.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:47 smithi190 ceph-mon[33871]: pgmap v1160: 129 pgs: 129 active+clean; 296 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 265 KiB/s wr, 76 op/s 2024-09-21T03:04:47.907 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:47 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:47.907 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:47 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2692684948' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:47.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:47 smithi029 ceph-mon[31006]: pgmap v1160: 129 pgs: 129 active+clean; 296 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 265 KiB/s wr, 76 op/s 2024-09-21T03:04:47.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:47 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:47.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:47 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2692684948' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:47.945 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24637 quiesce db --cancel --all' 2024-09-21T03:04:47.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24637 quiesce db --cancel --all 2024-09-21T03:04:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:47 smithi066 ceph-mon[33436]: pgmap v1160: 129 pgs: 129 active+clean; 296 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 265 KiB/s wr, 76 op/s 2024-09-21T03:04:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:47 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:48.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:47 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2692684948' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:04:48.207 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:48.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 -- 172.21.15.29:0/291545719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 msgr2=0x7f61d010f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 --2- 172.21.15.29:0/291545719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 0x7f61d010f550 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009ca0 tx=0x7f61c802a8c0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 -- 172.21.15.29:0/291545719 shutdown_connections 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 --2- 172.21.15.29:0/291545719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 0x7f61d010f550 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 --2- 172.21.15.29:0/291545719 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61d01083d0 0x7f61d01087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 --2- 172.21.15.29:0/291545719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61d0102a90 0x7f61d0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 -- 172.21.15.29:0/291545719 >> 172.21.15.29:0/291545719 conn(0x7f61d0079750 msgr2=0x7f61d0079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 -- 172.21.15.29:0/291545719 shutdown_connections 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.757+0000 7f61d6d43640 1 -- 172.21.15.29:0/291545719 wait complete. 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.758+0000 7f61d6d43640 1 Processor -- start 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.758+0000 7f61d6d43640 1 -- start start 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.758+0000 7f61d6d43640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61d0102a90 0x7f61d0199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.758+0000 7f61d6d43640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61d01083d0 0x7f61d01998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.758+0000 7f61d6d43640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 0x7f61d019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.758+0000 7f61d6d43640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f61d0112890 con 0x7f61d0108d20 2024-09-21T03:04:48.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.758+0000 7f61d6d43640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f61d0112710 con 0x7f61d01083d0 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.758+0000 7f61d6d43640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f61d0112a10 con 0x7f61d0102a90 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 0x7f61d019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 0x7f61d019eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35796/0 (socket says 172.21.15.29:35796) 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 -- 172.21.15.29:0/2398113103 learned_addr learned my addr 172.21.15.29:0/2398113103 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d4adf640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61d0102a90 0x7f61d0199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61cffff640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61d01083d0 0x7f61d01998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 -- 172.21.15.29:0/2398113103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61d0102a90 msgr2=0x7f61d0199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61d0102a90 0x7f61d0199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 -- 172.21.15.29:0/2398113103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61d01083d0 msgr2=0x7f61d01998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61d01083d0 0x7f61d01998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 -- 172.21.15.29:0/2398113103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d019f1b0 con 0x7f61d0108d20 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d52e0640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 0x7f61d019eb40 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f61c8005cc0 tx=0x7f61c8032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61cdffb640 1 -- 172.21.15.29:0/2398113103 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8032e00 con 0x7f61d0108d20 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61cdffb640 1 -- 172.21.15.29:0/2398113103 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61c80044d0 con 0x7f61d0108d20 2024-09-21T03:04:48.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61cdffb640 1 -- 172.21.15.29:0/2398113103 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8030b70 con 0x7f61d0108d20 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.759+0000 7f61d6d43640 1 -- 172.21.15.29:0/2398113103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61d019f440 con 0x7f61d0108d20 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.760+0000 7f61d6d43640 1 -- 172.21.15.29:0/2398113103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d01a5cc0 con 0x7f61d0108d20 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.762+0000 7f61dd2b7740 1 Processor -- start 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.762+0000 7f61cdffb640 1 -- 172.21.15.29:0/2398113103 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f61c8030d10 con 0x7f61d0108d20 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61dd2b7740 1 -- start start 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 0x561e9352d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 0x561e9352d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9352dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561e934a3dd0 con 0x561e931655f0 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561e93493200 con 0x561e9315d410 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561e93497010 con 0x561e931862d0 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9352dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9352dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35802/0 (socket says 172.21.15.29:35802) 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61cffff640 1 -- 172.21.15.29:0/1709435330 learned_addr learned my addr 172.21.15.29:0/1709435330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61cffff640 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 0x561e9352d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61d4adf640 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 0x561e9352d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61cffff640 1 -- 172.21.15.29:0/1709435330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 msgr2=0x561e9352d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61cffff640 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 0x561e9352d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.763+0000 7f61cffff640 1 -- 172.21.15.29:0/1709435330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 msgr2=0x561e9352d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61cffff640 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 0x561e9352d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61cdffb640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61b80b65c0 0x7f61b80b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61cffff640 1 -- 172.21.15.29:0/1709435330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e931f18b0 con 0x561e931655f0 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61cdffb640 1 -- 172.21.15.29:0/2398113103 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f61c8108900 con 0x7f61d0108d20 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61cffff640 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9352dd90 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f61c4013910 tx=0x7f61c400e430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6667e08ac21d1e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61cffff640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61b80b65c0 0x7f61b80b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61b6ffd640 1 -- 172.21.15.29:0/1709435330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c400b910 con 0x561e931655f0 2024-09-21T03:04:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61b6ffd640 1 -- 172.21.15.29:0/1709435330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61c4011440 con 0x561e931655f0 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.764+0000 7f61b6ffd640 1 -- 172.21.15.29:0/1709435330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4010690 con 0x561e931655f0 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1709435330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 msgr2=0x561e9352dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61cffff640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61b80b65c0 0x7f61b80b8ad0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f61c4013430 tx=0x7f61c40109e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9352dd90 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f61c4013910 tx=0x7f61c400e430 comp rx=0 tx=0).stop 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1709435330 shutdown_connections 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9352dd90 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 0x561e9352d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/1709435330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 0x561e9352d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1709435330 >> 172.21.15.29:0/1709435330 conn(0x561e9317a6c0 msgr2=0x561e9352b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1709435330 shutdown_connections 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1709435330 wait complete. 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.765+0000 7f61dd2b7740 1 -- start start 2024-09-21T03:04:48.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61dd2b7740 1 Processor -- start 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 0x561e9353b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9353b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 0x561e93546f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561e932f4810 con 0x561e931655f0 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x561e9342e080 con 0x561e9315d410 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561e9353dc50 con 0x561e931862d0 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61dd2b7740 10 client.0 _ll_register_callbacks cb 0x561e9343e960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9353b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9353b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35810/0 (socket says 172.21.15.29:35810) 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61cffff640 1 -- 172.21.15.29:0/3688836692 learned_addr learned my addr 172.21.15.29:0/3688836692 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61d52e0640 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 0x561e93546f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.766+0000 7f61d4adf640 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 0x561e9353b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61cffff640 1 -- 172.21.15.29:0/3688836692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 msgr2=0x561e93546f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:48.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61cffff640 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 0x561e93546f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61cffff640 1 -- 172.21.15.29:0/3688836692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 msgr2=0x561e9353b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61cffff640 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 0x561e9353b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61cffff640 1 -- 172.21.15.29:0/3688836692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e93438460 con 0x561e931655f0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61cffff640 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9353b580 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f61c400ee50 tx=0x7f61c4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f93c42ede1c7d48 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61affff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61affff640 1 -- 172.21.15.29:0/3688836692 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4031070 con 0x561e931655f0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61affff640 1 -- 172.21.15.29:0/3688836692 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f61c4010640 con 0x561e931655f0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61dd2b7740 1 -- 172.21.15.29:0/3688836692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61c4011440 con 0x561e931655f0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61affff640 1 -- 172.21.15.29:0/3688836692 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f61c403d050 con 0x561e931655f0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61dd2b7740 10 client.18840 fetch_fsmap learned FSMap version 41 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61dd2b7740 1 -- 172.21.15.29:0/3688836692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x561e93549bd0 con 0x561e931655f0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.767+0000 7f61affff640 1 -- 172.21.15.29:0/3688836692 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61c4011440 con 0x561e931655f0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.768+0000 7f61affff640 1 -- 172.21.15.29:0/3688836692 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8386+0+0 (secure 0 0 0) 0x7f61c40115f0 con 0x561e931655f0 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.768+0000 7f61dd2b7740 10 client.18840 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T03:04:48.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.768+0000 7f61dd2b7740 10 client.18840 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.768+0000 7f61dd2b7740 20 client.18840 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.768+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561e93144c50 0x561e9354cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.768+0000 7f61dd2b7740 4 client.18840 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.768+0000 7f61dd2b7740 1 -- 172.21.15.29:0/3688836692 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e934a5d20 con 0x561e93144c50 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.768+0000 7f61d4adf640 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561e93144c50 0x561e9354cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61affff640 10 client.18840 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61d4adf640 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561e93144c50 0x561e9354cf30 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75cf7fd1bc6529cb server_cookie=f99c0c60017485ec in_seq=0 out_seq=0 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61affff640 1 -- 172.21.15.29:0/3688836692 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561e934a5d20 con 0x561e93144c50 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61affff640 10 client.18840 handle_command_reply: tid=0 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 client.18840 shutdown 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 -- 172.21.15.29:0/3688836692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 msgr2=0x561e9353b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9353b580 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f61c400ee50 tx=0x7f61c4015000 comp rx=0 tx=0).stop 2024-09-21T03:04:48.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 -- 172.21.15.29:0/3688836692 shutdown_connections 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e931862d0 0x561e93546f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561e93144c50 0x561e9354cf30 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931655f0 0x561e9353b580 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/3688836692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e9315d410 0x561e9353b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61affff640 0 client.18840 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.769+0000 7f61dd2b7740 1 -- 172.21.15.29:0/3688836692 >> 172.21.15.29:0/3688836692 conn(0x561e9317a6c0 msgr2=0x561e9352ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.770+0000 7f61dd2b7740 1 -- 172.21.15.29:0/3688836692 shutdown_connections 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.770+0000 7f61dd2b7740 1 -- 172.21.15.29:0/3688836692 wait complete. 2024-09-21T03:04:48.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.770+0000 7f61dd2b7740 20 client.18840 trim_cache size 0 max 16384 2024-09-21T03:04:48.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61dd2b7740 1 Processor -- start 2024-09-21T03:04:48.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61dd2b7740 1 -- start start 2024-09-21T03:04:48.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 0x561e9353ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e93546e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935473c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01ae710 con 0x561e931862d0 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01b02b0 con 0x561e931655f0 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01aede0 con 0x561e9315d410 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.787+0000 7f61d52e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d52e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935473c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35814/0 (socket says 172.21.15.29:35814) 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d52e0640 1 -- 172.21.15.29:0/1872309411 learned_addr learned my addr 172.21.15.29:0/1872309411 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61cffff640 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e93546e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d4adf640 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 0x561e9353ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d52e0640 1 -- 172.21.15.29:0/1872309411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 msgr2=0x561e9353ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d52e0640 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 0x561e9353ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d52e0640 1 -- 172.21.15.29:0/1872309411 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 msgr2=0x561e93546e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d52e0640 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e93546e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d52e0640 1 -- 172.21.15.29:0/1872309411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e93536690 con 0x561e931862d0 2024-09-21T03:04:48.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61d52e0640 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935473c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f61c8031040 tx=0x7f61c80ce5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1f8226d4b80203 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61b67fc640 1 -- 172.21.15.29:0/1872309411 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c80cee10 con 0x561e931862d0 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61b67fc640 1 -- 172.21.15.29:0/1872309411 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61c8036040 con 0x561e931862d0 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.788+0000 7f61b67fc640 1 -- 172.21.15.29:0/1872309411 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c80d7f00 con 0x561e931862d0 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1872309411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 msgr2=0x561e935473c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935473c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f61c8031040 tx=0x7f61c80ce5a0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1872309411 shutdown_connections 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935473c0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e93546e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/1872309411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 0x561e9353ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1872309411 >> 172.21.15.29:0/1872309411 conn(0x561e9317a6c0 msgr2=0x561e9353c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1872309411 shutdown_connections 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.789+0000 7f61dd2b7740 1 -- 172.21.15.29:0/1872309411 wait complete. 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 1 -- start start 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 1 Processor -- start 2024-09-21T03:04:48.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 0x561e9353f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e9353fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935b6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01afbe0 con 0x561e931862d0 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01af4e0 con 0x561e931655f0 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01ac6f0 con 0x561e9315d410 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61dd2b7740 10 client.0 _ll_register_callbacks cb 0x7f61c4013810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61d52e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61d52e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935b6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35830/0 (socket says 172.21.15.29:35830) 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61cffff640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e9353fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.790+0000 7f61d52e0640 1 -- 172.21.15.29:0/362867576 learned_addr learned my addr 172.21.15.29:0/362867576 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61d4adf640 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 0x561e9353f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61d52e0640 1 -- 172.21.15.29:0/362867576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 msgr2=0x561e9353f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61d52e0640 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 0x561e9353f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61d52e0640 1 -- 172.21.15.29:0/362867576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 msgr2=0x561e9353fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T03:04:48.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61d52e0640 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e9353fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61d52e0640 1 -- 172.21.15.29:0/362867576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e9354d2a0 con 0x561e931862d0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61cffff640 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e9353fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61d52e0640 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935b6860 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f61c81060d0 tx=0x7f61c8030630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3143e671fcb83a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61affff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61affff640 1 -- 172.21.15.29:0/362867576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c80e8070 con 0x561e931862d0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61affff640 1 -- 172.21.15.29:0/362867576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f61c80d7ea0 con 0x561e931862d0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61dd2b7740 1 -- 172.21.15.29:0/362867576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61c8036040 con 0x561e931862d0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61affff640 1 -- 172.21.15.29:0/362867576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f61c80cee10 con 0x561e931862d0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61dd2b7740 10 client.18846 fetch_fsmap learned FSMap version 41 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.791+0000 7f61dd2b7740 1 -- 172.21.15.29:0/362867576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x561e935b7f40 con 0x561e931862d0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61affff640 1 -- 172.21.15.29:0/362867576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61c8036040 con 0x561e931862d0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61affff640 1 -- 172.21.15.29:0/362867576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8386+0+0 (secure 0 0 0) 0x7f61c80ca060 con 0x561e931862d0 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61dd2b7740 10 client.18846 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61dd2b7740 10 client.18846 resolve_mds: validated gid 24637 aka daemon mds.i 2024-09-21T03:04:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61dd2b7740 20 client.18846 populate_metadata read hostname 'smithi029' 2024-09-21T03:04:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561e9311a000 0x561e935ba560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61dd2b7740 4 client.18846 mds_command: new command op to 24637 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-21T03:04:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61dd2b7740 1 -- 172.21.15.29:0/362867576 --> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x561e934a17d0 con 0x561e9311a000 2024-09-21T03:04:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61d4adf640 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561e9311a000 0x561e935ba560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61affff640 10 client.18846 ms_handle_connect on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.792+0000 7f61d4adf640 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561e9311a000 0x561e935ba560 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53bd860fb0640b79 server_cookie=d6c46b7b17ca6f57 in_seq=0 out_seq=0 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61affff640 1 -- 172.21.15.29:0/362867576 <== mds.0 v2:172.21.15.190:6838/3083654868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+81 (crc 0 0 0) 0x561e934a17d0 con 0x561e9311a000 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61affff640 10 client.18846 handle_command_reply: tid=0 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 client.18846 shutdown 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 -- 172.21.15.29:0/362867576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 msgr2=0x561e935b6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935b6860 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f61c81060d0 tx=0x7f61c8030630 comp rx=0 tx=0).stop 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 -- 172.21.15.29:0/362867576 shutdown_connections 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.190:6838/3083654868,v1:172.21.15.190:6839/3083654868] conn(0x561e9311a000 0x561e935ba560 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e931862d0 0x561e935b6860 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x561e931655f0 0x561e9353fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 --2- 172.21.15.29:0/362867576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e9315d410 0x561e9353f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 -- 172.21.15.29:0/362867576 >> 172.21.15.29:0/362867576 conn(0x561e9317a6c0 msgr2=0x561e9352ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61affff640 0 client.18846 ms_handle_reset on v2:172.21.15.190:6838/3083654868 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 -- 172.21.15.29:0/362867576 shutdown_connections 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 1 -- 172.21.15.29:0/362867576 wait complete. 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.793+0000 7f61dd2b7740 20 client.18846 trim_cache size 0 max 16384 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 41, 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24637, 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 100, 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": {} 2024-09-21T03:04:48.801 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 -- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61b80b65c0 msgr2=0x7f61b80b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61b80b65c0 0x7f61b80b8ad0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f61c4013430 tx=0x7f61c40109e0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 -- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 msgr2=0x7f61d019eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 0x7f61d019eb40 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f61c8005cc0 tx=0x7f61c8032a10 comp rx=0 tx=0).stop 2024-09-21T03:04:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 -- 172.21.15.29:0/2398113103 shutdown_connections 2024-09-21T03:04:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f61b80b65c0 0x7f61b80b8ad0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61d0108d20 0x7f61d019eb40 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61d01083d0 0x7f61d01998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 --2- 172.21.15.29:0/2398113103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61d0102a90 0x7f61d0199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:48.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 -- 172.21.15.29:0/2398113103 >> 172.21.15.29:0/2398113103 conn(0x7f61d0079750 msgr2=0x7f61d0104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:48.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 -- 172.21.15.29:0/2398113103 shutdown_connections 2024-09-21T03:04:48.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:48.794+0000 7f61d6d43640 1 -- 172.21.15.29:0/2398113103 wait complete. 2024-09-21T03:04:49.611 INFO:tasks.quiescer:done joining Quiescers 2024-09-21T03:04:49.611 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-09-21T03:04:49.622 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T03:04:49.631 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-21T03:04:49.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:04:49.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 -- 172.21.15.29:0/377549854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 msgr2=0x7f5ecc103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:49.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/377549854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 0x7f5ecc103a30 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009900 tx=0x7f5eb802a410 comp rx=0 tx=0).stop 2024-09-21T03:04:49.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 -- 172.21.15.29:0/377549854 shutdown_connections 2024-09-21T03:04:49.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/377549854 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ecc103fc0 0x7f5ecc10a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:49.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/377549854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 0x7f5ecc103a30 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:49.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/377549854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ecc1025a0 0x7f5ecc102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:49.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 -- 172.21.15.29:0/377549854 >> 172.21.15.29:0/377549854 conn(0x7f5ecc0fdee0 msgr2=0x7f5ecc100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:49.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 -- 172.21.15.29:0/377549854 shutdown_connections 2024-09-21T03:04:49.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.788+0000 7f5ed2c94640 1 -- 172.21.15.29:0/377549854 wait complete. 2024-09-21T03:04:49.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed2c94640 1 Processor -- start 2024-09-21T03:04:49.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed2c94640 1 -- start start 2024-09-21T03:04:49.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed2c94640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ecc1025a0 0x7f5ecc199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:49.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed2c94640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 0x7f5ecc19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:49.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed2c94640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ecc103fc0 0x7f5ecc19b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:49.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed2c94640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ecc10e0b0 con 0x7f5ecc103620 2024-09-21T03:04:49.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed2c94640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5ecc10df30 con 0x7f5ecc1025a0 2024-09-21T03:04:49.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed2c94640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ecc10e230 con 0x7f5ecc103fc0 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ecbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 0x7f5ecc19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ecbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 0x7f5ecc19a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35850/0 (socket says 172.21.15.29:35850) 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ecbfff640 1 -- 172.21.15.29:0/2788144906 learned_addr learned my addr 172.21.15.29:0/2788144906 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed0a30640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ecc1025a0 0x7f5ecc199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.789+0000 7f5ed1231640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ecc103fc0 0x7f5ecc19b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ecbfff640 1 -- 172.21.15.29:0/2788144906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ecc103fc0 msgr2=0x7f5ecc19b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ecbfff640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ecc103fc0 0x7f5ecc19b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ecbfff640 1 -- 172.21.15.29:0/2788144906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ecc1025a0 msgr2=0x7f5ecc199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ecbfff640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ecc1025a0 0x7f5ecc199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ecbfff640 1 -- 172.21.15.29:0/2788144906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ecc19b870 con 0x7f5ecc103620 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ecbfff640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 0x7f5ecc19a4a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009230 tx=0x7f5eb8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ec9ffb640 1 -- 172.21.15.29:0/2788144906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb803f030 con 0x7f5ecc103620 2024-09-21T03:04:49.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ec9ffb640 1 -- 172.21.15.29:0/2788144906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5eb8003a80 con 0x7f5ecc103620 2024-09-21T03:04:49.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ec9ffb640 1 -- 172.21.15.29:0/2788144906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8035070 con 0x7f5ecc103620 2024-09-21T03:04:49.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.790+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ecc19bb00 con 0x7f5ecc103620 2024-09-21T03:04:49.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.791+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ecc1a2070 con 0x7f5ecc103620 2024-09-21T03:04:49.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.792+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ecc1a22a0 con 0x7f5ecc103620 2024-09-21T03:04:49.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.794+0000 7f5ec9ffb640 1 -- 172.21.15.29:0/2788144906 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f5eb8042030 con 0x7f5ecc103620 2024-09-21T03:04:49.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.795+0000 7f5ec9ffb640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ebc0b6510 0x7f5ebc0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:49.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.795+0000 7f5ed0a30640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ebc0b6510 0x7f5ebc0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:49.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.795+0000 7f5ec9ffb640 1 -- 172.21.15.29:0/2788144906 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f5eb8108240 con 0x7f5ecc103620 2024-09-21T03:04:49.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.796+0000 7f5ed0a30640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ebc0b6510 0x7f5ebc0b8a20 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000bbe0 tx=0x7f5ec000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:49.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.798+0000 7f5ec9ffb640 1 -- 172.21.15.29:0/2788144906 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5eb80d2ad0 con 0x7f5ecc103620 2024-09-21T03:04:49.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:49 smithi190 ceph-mon[33871]: pgmap v1161: 129 pgs: 129 active+clean; 203 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 311 KiB/s wr, 130 op/s 2024-09-21T03:04:49.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:49 smithi029 ceph-mon[31006]: pgmap v1161: 129 pgs: 129 active+clean; 203 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 311 KiB/s wr, 130 op/s 2024-09-21T03:04:49.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.993+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ecc1a2470 con 0x7f5ecc103620 2024-09-21T03:04:49.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.994+0000 7f5ec9ffb640 1 -- 172.21.15.29:0/2788144906 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24773 (secure 0 0 0) 0x7f5eb80d7a10 con 0x7f5ecc103620 2024-09-21T03:04:49.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:04:49.997 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":86,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T03:02:24.075319+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:04:49.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ebc0b6510 msgr2=0x7f5ebc0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:49.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ebc0b6510 0x7f5ebc0b8a20 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000bbe0 tx=0x7f5ec000b390 comp rx=0 tx=0).stop 2024-09-21T03:04:49.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 msgr2=0x7f5ecc19a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:49.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 0x7f5ecc19a4a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009230 tx=0x7f5eb8004270 comp rx=0 tx=0).stop 2024-09-21T03:04:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 shutdown_connections 2024-09-21T03:04:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5ebc0b6510 0x7f5ebc0b8a20 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ecc103fc0 0x7f5ecc19b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ecc103620 0x7f5ecc19a4a0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 --2- 172.21.15.29:0/2788144906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ecc1025a0 0x7f5ecc199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 >> 172.21.15.29:0/2788144906 conn(0x7f5ecc0fdee0 msgr2=0x7f5ecc0ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 shutdown_connections 2024-09-21T03:04:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:49.997+0000 7f5ed2c94640 1 -- 172.21.15.29:0/2788144906 wait complete. 2024-09-21T03:04:50.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:49 smithi066 ceph-mon[33436]: pgmap v1161: 129 pgs: 129 active+clean; 203 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 311 KiB/s wr, 130 op/s 2024-09-21T03:04:50.011 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T02:31:38.302181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-21T02:36:22.103545+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-21T02:36:24.546927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-21T02:36:30.778080+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-21T03:04:50.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 -- 172.21.15.29:0/3206938479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 msgr2=0x7f218810b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 --2- 172.21.15.29:0/3206938479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 0x7f218810b8e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f2180009ca0 tx=0x7f218002a8c0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 -- 172.21.15.29:0/3206938479 shutdown_connections 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 --2- 172.21.15.29:0/3206938479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 0x7f218810b8e0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 --2- 172.21.15.29:0/3206938479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2188106ae0 0x7f2188108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 --2- 172.21.15.29:0/3206938479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2188101a50 0x7f2188101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 -- 172.21.15.29:0/3206938479 >> 172.21.15.29:0/3206938479 conn(0x7f21880fd690 msgr2=0x7f21880ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 -- 172.21.15.29:0/3206938479 shutdown_connections 2024-09-21T03:04:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.134+0000 7f218eb20640 1 -- 172.21.15.29:0/3206938479 wait complete. 2024-09-21T03:04:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.135+0000 7f218eb20640 1 Processor -- start 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.135+0000 7f218eb20640 1 -- start start 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.135+0000 7f218eb20640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2188101a50 0x7f2188199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.135+0000 7f218eb20640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2188106ae0 0x7f218819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.135+0000 7f218eb20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 0x7f218819b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.135+0000 7f218eb20640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f218810f1e0 con 0x7f2188109440 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.135+0000 7f218eb20640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f218810f060 con 0x7f2188106ae0 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.135+0000 7f218eb20640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f218810f360 con 0x7f2188101a50 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 0x7f218819b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 0x7f218819b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35868/0 (socket says 172.21.15.29:35868) 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 -- 172.21.15.29:0/896851098 learned_addr learned my addr 172.21.15.29:0/896851098 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f2187fff640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2188106ae0 0x7f218819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218c8bc640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2188101a50 0x7f2188199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 -- 172.21.15.29:0/896851098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2188101a50 msgr2=0x7f2188199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2188101a50 0x7f2188199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 -- 172.21.15.29:0/896851098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2188106ae0 msgr2=0x7f218819a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2188106ae0 0x7f218819a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 -- 172.21.15.29:0/896851098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218819b850 con 0x7f2188109440 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218d0bd640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 0x7f218819b030 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f2180005e30 tx=0x7f2180003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f2185ffb640 1 -- 172.21.15.29:0/896851098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180032e90 con 0x7f2188109440 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f2185ffb640 1 -- 172.21.15.29:0/896851098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21800044d0 con 0x7f2188109440 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218819bae0 con 0x7f2188109440 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.136+0000 7f2185ffb640 1 -- 172.21.15.29:0/896851098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180030ce0 con 0x7f2188109440 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.137+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21881a2230 con 0x7f2188109440 2024-09-21T03:04:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.138+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2188101e60 con 0x7f2188109440 2024-09-21T03:04:50.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.140+0000 7f2185ffb640 1 -- 172.21.15.29:0/896851098 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f2180004040 con 0x7f2188109440 2024-09-21T03:04:50.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.140+0000 7f2185ffb640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f21780b6510 0x7f21780b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.140+0000 7f2185ffb640 1 -- 172.21.15.29:0/896851098 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f2180108a90 con 0x7f2188109440 2024-09-21T03:04:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.141+0000 7f218c8bc640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f21780b6510 0x7f21780b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.141+0000 7f218c8bc640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f21780b6510 0x7f21780b8a20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f217c009b60 tx=0x7f217c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.145+0000 7f2185ffb640 1 -- 172.21.15.29:0/896851098 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f21800cefa0 con 0x7f2188109440 2024-09-21T03:04:50.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.332+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f218802e640 con 0x7f2188109440 2024-09-21T03:04:50.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.332+0000 7f2185ffb640 1 -- 172.21.15.29:0/896851098 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) ==== 93+0+10 (secure 0 0 0) 0x7f21800d3380 con 0x7f2188109440 2024-09-21T03:04:50.335 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-09-21T03:04:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f21780b6510 msgr2=0x7f21780b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f21780b6510 0x7f21780b8a20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f217c009b60 tx=0x7f217c005f90 comp rx=0 tx=0).stop 2024-09-21T03:04:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 msgr2=0x7f218819b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 0x7f218819b030 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f2180005e30 tx=0x7f2180003ba0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 shutdown_connections 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f21780b6510 0x7f21780b8a20 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2188109440 0x7f218819b030 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2188106ae0 0x7f218819a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 --2- 172.21.15.29:0/896851098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2188101a50 0x7f2188199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 >> 172.21.15.29:0/896851098 conn(0x7f21880fd690 msgr2=0x7f21881078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 shutdown_connections 2024-09-21T03:04:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.335+0000 7f218eb20640 1 -- 172.21.15.29:0/896851098 wait complete. 2024-09-21T03:04:50.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-21T03:04:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 -- 172.21.15.29:0/1461490422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b01036a0 msgr2=0x7f12b0103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 --2- 172.21.15.29:0/1461490422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b01036a0 0x7f12b0103b10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009900 tx=0x7f12a002a410 comp rx=0 tx=0).stop 2024-09-21T03:04:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 -- 172.21.15.29:0/1461490422 shutdown_connections 2024-09-21T03:04:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 --2- 172.21.15.29:0/1461490422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12b01050c0 0x7f12b010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 --2- 172.21.15.29:0/1461490422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12b0104720 0x7f12b0104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 --2- 172.21.15.29:0/1461490422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b01036a0 0x7f12b0103b10 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 -- 172.21.15.29:0/1461490422 >> 172.21.15.29:0/1461490422 conn(0x7f12b00fefe0 msgr2=0x7f12b0101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 -- 172.21.15.29:0/1461490422 shutdown_connections 2024-09-21T03:04:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.466+0000 7f12b55ad640 1 -- 172.21.15.29:0/1461490422 wait complete. 2024-09-21T03:04:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12b55ad640 1 Processor -- start 2024-09-21T03:04:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12b55ad640 1 -- start start 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12b55ad640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12b01036a0 0x7f12b0076e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12b55ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b0104720 0x7f12b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12b55ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12b01050c0 0x7f12b0075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12b55ad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f12b010f190 con 0x7f12b0104720 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12b55ad640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f12b010f010 con 0x7f12b01036a0 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12b55ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f12b010f310 con 0x7f12b01050c0 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b0104720 0x7f12b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12aeffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12b01036a0 0x7f12b0076e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b0104720 0x7f12b0075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35894/0 (socket says 172.21.15.29:35894) 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12a7fff640 1 -- 172.21.15.29:0/2004566703 learned_addr learned my addr 172.21.15.29:0/2004566703 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.467+0000 7f12aeffd640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12b01036a0 0x7f12b0076e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.29:36280/0 (socket says 172.21.15.29:36280) 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12af7fe640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12b01050c0 0x7f12b0075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12aeffd640 1 -- 172.21.15.29:0/2004566703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12b01050c0 msgr2=0x7f12b0075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12aeffd640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12b01050c0 0x7f12b0075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12aeffd640 1 -- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b0104720 msgr2=0x7f12b0075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12aeffd640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b0104720 0x7f12b0075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12aeffd640 1 -- 172.21.15.29:0/2004566703 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b0076380 con 0x7f12b01036a0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12a7fff640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b0104720 0x7f12b0075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12aeffd640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12b01036a0 0x7f12b0076e50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f12a0005760 tx=0x7f12a0005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12acff9640 1 -- 172.21.15.29:0/2004566703 <== mon.2 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a003f070 con 0x7f12b01036a0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12acff9640 1 -- 172.21.15.29:0/2004566703 <== mon.2 v2:172.21.15.66:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f12a0003a80 con 0x7f12b01036a0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b01a3ed0 con 0x7f12b01036a0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12acff9640 1 -- 172.21.15.29:0/2004566703 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0035070 con 0x7f12b01036a0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.468+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b01a4280 con 0x7f12b01036a0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.469+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b010a870 con 0x7f12b01036a0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.471+0000 7f12acff9640 1 -- 172.21.15.29:0/2004566703 <== mon.2 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f12a0003c30 con 0x7f12b01036a0 2024-09-21T03:04:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.472+0000 7f12acff9640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f129c0b6510 0x7f129c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.472+0000 7f12a7fff640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f129c0b6510 0x7f129c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.472+0000 7f12acff9640 1 -- 172.21.15.29:0/2004566703 <== mon.2 v2:172.21.15.66:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f12a0107ed0 con 0x7f12b01036a0 2024-09-21T03:04:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.472+0000 7f12a7fff640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f129c0b6510 0x7f129c0b8a20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1298011500 tx=0x7f1298005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.475+0000 7f12acff9640 1 -- 172.21.15.29:0/2004566703 <== mon.2 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f12a00ce380 con 0x7f12b01036a0 2024-09-21T03:04:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.678+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f12b002e7d0 con 0x7f12b01036a0 2024-09-21T03:04:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.679+0000 7f12acff9640 1 -- 172.21.15.29:0/2004566703 <== mon.2 v2:172.21.15.66:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) ==== 104+0+11 (secure 0 0 0) 0x7f12b002e7d0 con 0x7f12b01036a0 2024-09-21T03:04:50.682 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-09-21T03:04:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f129c0b6510 msgr2=0x7f129c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f129c0b6510 0x7f129c0b8a20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1298011500 tx=0x7f1298005f90 comp rx=0 tx=0).stop 2024-09-21T03:04:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12b01036a0 msgr2=0x7f12b0076e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12b01036a0 0x7f12b0076e50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f12a0005760 tx=0x7f12a0005e70 comp rx=0 tx=0).stop 2024-09-21T03:04:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 shutdown_connections 2024-09-21T03:04:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f129c0b6510 0x7f129c0b8a20 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12b01050c0 0x7f12b0075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12b0104720 0x7f12b0075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 --2- 172.21.15.29:0/2004566703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12b01036a0 0x7f12b0076e50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 >> 172.21.15.29:0/2004566703 conn(0x7f12b00fefe0 msgr2=0x7f12b0107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 shutdown_connections 2024-09-21T03:04:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.682+0000 7f12b55ad640 1 -- 172.21.15.29:0/2004566703 wait complete. 2024-09-21T03:04:50.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-21T03:04:50.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 -- 172.21.15.29:0/4258125025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c102580 msgr2=0x7ffb7c1029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 --2- 172.21.15.29:0/4258125025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c102580 0x7ffb7c1029f0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009ca0 tx=0x7ffb7802a8c0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 -- 172.21.15.29:0/4258125025 shutdown_connections 2024-09-21T03:04:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 --2- 172.21.15.29:0/4258125025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c103fa0 0x7ffb7c10a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 --2- 172.21.15.29:0/4258125025 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c103600 0x7ffb7c103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 --2- 172.21.15.29:0/4258125025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c102580 0x7ffb7c1029f0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 -- 172.21.15.29:0/4258125025 >> 172.21.15.29:0/4258125025 conn(0x7ffb7c0fdee0 msgr2=0x7ffb7c100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 -- 172.21.15.29:0/4258125025 shutdown_connections 2024-09-21T03:04:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 -- 172.21.15.29:0/4258125025 wait complete. 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 Processor -- start 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.812+0000 7ffb83a25640 1 -- start start 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb83a25640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c102580 0x7ffb7c06af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb83a25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c103600 0x7ffb7c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb83a25640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c103fa0 0x7ffb7c1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb83a25640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c10e030 con 0x7ffb7c103fa0 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb83a25640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c10deb0 con 0x7ffb7c102580 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb83a25640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c10e1b0 con 0x7ffb7c103600 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c103fa0 0x7ffb7c1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c103fa0 0x7ffb7c1a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35916/0 (socket says 172.21.15.29:35916) 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 -- 172.21.15.29:0/2470036503 learned_addr learned my addr 172.21.15.29:0/2470036503 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb80fc0640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c103600 0x7ffb7c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb817c1640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c102580 0x7ffb7c06af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 -- 172.21.15.29:0/2470036503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c103600 msgr2=0x7ffb7c19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c103600 0x7ffb7c19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 -- 172.21.15.29:0/2470036503 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c102580 msgr2=0x7ffb7c06af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c102580 0x7ffb7c06af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 -- 172.21.15.29:0/2470036503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7c1a18b0 con 0x7ffb7c103fa0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb81fc2640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c103fa0 0x7ffb7c1a10b0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400d750 tx=0x7ffb7400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb727fc640 1 -- 172.21.15.29:0/2470036503 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74017070 con 0x7ffb7c103fa0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.813+0000 7ffb727fc640 1 -- 172.21.15.29:0/2470036503 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb740054b0 con 0x7ffb7c103fa0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.814+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7c1a1ba0 con 0x7ffb7c103fa0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.814+0000 7ffb727fc640 1 -- 172.21.15.29:0/2470036503 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74004b90 con 0x7ffb7c103fa0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.814+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb7c1a20f0 con 0x7ffb7c103fa0 2024-09-21T03:04:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.815+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb7c109750 con 0x7ffb7c103fa0 2024-09-21T03:04:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.816+0000 7ffb727fc640 1 -- 172.21.15.29:0/2470036503 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7ffb74004d30 con 0x7ffb7c103fa0 2024-09-21T03:04:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.817+0000 7ffb727fc640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb680b6510 0x7ffb680b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:50.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.817+0000 7ffb817c1640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb680b6510 0x7ffb680b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:50.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.818+0000 7ffb727fc640 1 -- 172.21.15.29:0/2470036503 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7ffb740e70b0 con 0x7ffb7c103fa0 2024-09-21T03:04:50.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.818+0000 7ffb817c1640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb680b6510 0x7ffb680b8a20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ffb780094e0 tx=0x7ffb780318e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:50.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:50.822+0000 7ffb727fc640 1 -- 172.21.15.29:0/2470036503 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ffb740ad5c0 con 0x7ffb7c103fa0 2024-09-21T03:04:50.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:50 smithi029 ceph-mon[31006]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:50.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:50 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2788144906' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T03:04:50.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:50 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:04:50.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:50 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/896851098' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T03:04:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:50 smithi066 ceph-mon[33436]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:50 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2788144906' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T03:04:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:50 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:04:51.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:50 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/896851098' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T03:04:51.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.025+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffb7c063fc0 con 0x7ffb7c103fa0 2024-09-21T03:04:51.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.025+0000 7ffb727fc640 1 -- 172.21.15.29:0/2470036503 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) ==== 100+0+11 (secure 0 0 0) 0x7ffb740b19a0 con 0x7ffb7c103fa0 2024-09-21T03:04:51.028 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-09-21T03:04:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb680b6510 msgr2=0x7ffb680b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb680b6510 0x7ffb680b8a20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ffb780094e0 tx=0x7ffb780318e0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c103fa0 msgr2=0x7ffb7c1a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c103fa0 0x7ffb7c1a10b0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400d750 tx=0x7ffb7400dc10 comp rx=0 tx=0).stop 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 shutdown_connections 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7ffb680b6510 0x7ffb680b8a20 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb7c103fa0 0x7ffb7c1a10b0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c103600 0x7ffb7c19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 --2- 172.21.15.29:0/2470036503 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c102580 0x7ffb7c06af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 >> 172.21.15.29:0/2470036503 conn(0x7ffb7c0fdee0 msgr2=0x7ffb7c106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 shutdown_connections 2024-09-21T03:04:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.028+0000 7ffb83a25640 1 -- 172.21.15.29:0/2470036503 wait complete. 2024-09-21T03:04:51.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-21T03:04:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:50 smithi190 ceph-mon[33871]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T03:04:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:50 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2788144906' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T03:04:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:50 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T03:04:51.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:50 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/896851098' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T03:04:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 -- 172.21.15.29:0/1974853371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd640102c40 msgr2=0x7fd640105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 --2- 172.21.15.29:0/1974853371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd640102c40 0x7fd640105060 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd630009860 tx=0x7fd63002a340 comp rx=0 tx=0).stop 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 -- 172.21.15.29:0/1974853371 shutdown_connections 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 --2- 172.21.15.29:0/1974853371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6401082c0 0x7fd64010a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 --2- 172.21.15.29:0/1974853371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6401056a0 0x7fd640107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 --2- 172.21.15.29:0/1974853371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd640102c40 0x7fd640105060 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 -- 172.21.15.29:0/1974853371 >> 172.21.15.29:0/1974853371 conn(0x7fd6400fd690 msgr2=0x7fd6400ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 -- 172.21.15.29:0/1974853371 shutdown_connections 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.161+0000 7fd645cf2640 1 -- 172.21.15.29:0/1974853371 wait complete. 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd645cf2640 1 Processor -- start 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd645cf2640 1 -- start start 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd645cf2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd640102c40 0x7fd640197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd645cf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6401056a0 0x7fd640198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:51.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd645cf2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6401082c0 0x7fd640198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd645cf2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd64010cf90 con 0x7fd6401056a0 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd645cf2640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fd64010ce10 con 0x7fd6401082c0 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd645cf2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd64010d110 con 0x7fd640102c40 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6401056a0 0x7fd640198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6401056a0 0x7fd640198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35930/0 (socket says 172.21.15.29:35930) 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63effd640 1 -- 172.21.15.29:0/271332992 learned_addr learned my addr 172.21.15.29:0/271332992 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63ffff640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6401082c0 0x7fd640198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63f7fe640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd640102c40 0x7fd640197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63effd640 1 -- 172.21.15.29:0/271332992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd640102c40 msgr2=0x7fd640197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63effd640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd640102c40 0x7fd640197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63effd640 1 -- 172.21.15.29:0/271332992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6401082c0 msgr2=0x7fd640198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63effd640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6401082c0 0x7fd640198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.162+0000 7fd63effd640 1 -- 172.21.15.29:0/271332992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd640199630 con 0x7fd6401056a0 2024-09-21T03:04:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.163+0000 7fd63effd640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6401056a0 0x7fd640198260 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd63400e9c0 tx=0x7fd63400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.163+0000 7fd63cff9640 1 -- 172.21.15.29:0/271332992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63400cde0 con 0x7fd6401056a0 2024-09-21T03:04:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.163+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd640199920 con 0x7fd6401056a0 2024-09-21T03:04:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.163+0000 7fd63cff9640 1 -- 172.21.15.29:0/271332992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd6340035f0 con 0x7fd6401056a0 2024-09-21T03:04:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.163+0000 7fd63cff9640 1 -- 172.21.15.29:0/271332992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634010690 con 0x7fd6401056a0 2024-09-21T03:04:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.163+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6401a0010 con 0x7fd6401056a0 2024-09-21T03:04:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.164+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6400ff1b0 con 0x7fd6401056a0 2024-09-21T03:04:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.166+0000 7fd63cff9640 1 -- 172.21.15.29:0/271332992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fd634004000 con 0x7fd6401056a0 2024-09-21T03:04:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.166+0000 7fd63cff9640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd6280b6510 0x7fd6280b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.167+0000 7fd63f7fe640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd6280b6510 0x7fd6280b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.167+0000 7fd63cff9640 1 -- 172.21.15.29:0/271332992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fd6340e7a70 con 0x7fd6401056a0 2024-09-21T03:04:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.167+0000 7fd63f7fe640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd6280b6510 0x7fd6280b8a20 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd63002a930 tx=0x7fd630000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:51.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.170+0000 7fd63cff9640 1 -- 172.21.15.29:0/271332992 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd6340adf80 con 0x7fd6401056a0 2024-09-21T03:04:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.363+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd64002e7d0 con 0x7fd6401056a0 2024-09-21T03:04:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.364+0000 7fd63cff9640 1 -- 172.21.15.29:0/271332992 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) ==== 103+0+11 (secure 0 0 0) 0x7fd6340b2360 con 0x7fd6401056a0 2024-09-21T03:04:51.366 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-09-21T03:04:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd6280b6510 msgr2=0x7fd6280b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd6280b6510 0x7fd6280b8a20 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd63002a930 tx=0x7fd630000bc0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6401056a0 msgr2=0x7fd640198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6401056a0 0x7fd640198260 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd63400e9c0 tx=0x7fd63400ee80 comp rx=0 tx=0).stop 2024-09-21T03:04:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 shutdown_connections 2024-09-21T03:04:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fd6280b6510 0x7fd6280b8a20 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6401082c0 0x7fd640198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6401056a0 0x7fd640198260 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 --2- 172.21.15.29:0/271332992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd640102c40 0x7fd640197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.366+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 >> 172.21.15.29:0/271332992 conn(0x7fd6400fd690 msgr2=0x7fd640103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:51.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.367+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 shutdown_connections 2024-09-21T03:04:51.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.367+0000 7fd645cf2640 1 -- 172.21.15.29:0/271332992 wait complete. 2024-09-21T03:04:51.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-21T03:04:51.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 -- 172.21.15.29:0/863625008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c102570 msgr2=0x7fb65c1029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 --2- 172.21.15.29:0/863625008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c102570 0x7fb65c1029e0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fb650009900 tx=0x7fb65002a410 comp rx=0 tx=0).stop 2024-09-21T03:04:51.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 -- 172.21.15.29:0/863625008 shutdown_connections 2024-09-21T03:04:51.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 --2- 172.21.15.29:0/863625008 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb65c103f90 0x7fb65c10a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 --2- 172.21.15.29:0/863625008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb65c1035f0 0x7fb65c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 --2- 172.21.15.29:0/863625008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c102570 0x7fb65c1029e0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 -- 172.21.15.29:0/863625008 >> 172.21.15.29:0/863625008 conn(0x7fb65c0fdeb0 msgr2=0x7fb65c100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:51.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 -- 172.21.15.29:0/863625008 shutdown_connections 2024-09-21T03:04:51.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 -- 172.21.15.29:0/863625008 wait complete. 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.495+0000 7fb663fda640 1 Processor -- start 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb663fda640 1 -- start start 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb663fda640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb65c102570 0x7fb65c199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb663fda640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb65c1035f0 0x7fb65c19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb663fda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c103f90 0x7fb65c19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb663fda640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c10e0b0 con 0x7fb65c103f90 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb663fda640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c10df30 con 0x7fb65c1035f0 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb663fda640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c10e230 con 0x7fb65c102570 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb662577640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c103f90 0x7fb65c19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb662577640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c103f90 0x7fb65c19b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35956/0 (socket says 172.21.15.29:35956) 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb662577640 1 -- 172.21.15.29:0/1390498973 learned_addr learned my addr 172.21.15.29:0/1390498973 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:04:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb661d76640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb65c102570 0x7fb65c199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb661575640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb65c1035f0 0x7fb65c19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb662577640 1 -- 172.21.15.29:0/1390498973 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb65c102570 msgr2=0x7fb65c199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb662577640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb65c102570 0x7fb65c199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb662577640 1 -- 172.21.15.29:0/1390498973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb65c1035f0 msgr2=0x7fb65c19a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb662577640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb65c1035f0 0x7fb65c19a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb662577640 1 -- 172.21.15.29:0/1390498973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c19b8e0 con 0x7fb65c103f90 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.496+0000 7fb661575640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb65c1035f0 0x7fb65c19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.497+0000 7fb662577640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c103f90 0x7fb65c19b0c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb65400eaa0 tx=0x7fb65400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.497+0000 7fb64effd640 1 -- 172.21.15.29:0/1390498973 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65400cdb0 con 0x7fb65c103f90 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.497+0000 7fb64effd640 1 -- 172.21.15.29:0/1390498973 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb654005020 con 0x7fb65c103f90 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.497+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65c19bbd0 con 0x7fb65c103f90 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.497+0000 7fb64effd640 1 -- 172.21.15.29:0/1390498973 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6540116d0 con 0x7fb65c103f90 2024-09-21T03:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.497+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65c1a22c0 con 0x7fb65c103f90 2024-09-21T03:04:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.498+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb65c1a2490 con 0x7fb65c103f90 2024-09-21T03:04:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.500+0000 7fb64effd640 1 -- 172.21.15.29:0/1390498973 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fb6540051d0 con 0x7fb65c103f90 2024-09-21T03:04:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.500+0000 7fb64effd640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6480b6590 0x7fb6480b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:04:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.500+0000 7fb64effd640 1 -- 172.21.15.29:0/1390498973 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fb6540e7bb0 con 0x7fb65c103f90 2024-09-21T03:04:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.500+0000 7fb661d76640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6480b6590 0x7fb6480b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:04:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.501+0000 7fb661d76640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6480b6590 0x7fb6480b8aa0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb650005760 tx=0x7fb650031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:04:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.503+0000 7fb64effd640 1 -- 172.21.15.29:0/1390498973 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb654016030 con 0x7fb65c103f90 2024-09-21T03:04:51.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.760+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fb65c1a2660 con 0x7fb65c103f90 2024-09-21T03:04:51.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.769+0000 7fb64effd640 1 -- 172.21.15.29:0/1390498973 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+833520 (secure 0 0 0) 0x7fb6540ae140 con 0x7fb65c103f90 2024-09-21T03:04:51.780 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:04:51.780 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:51.780 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 20, 2024-09-21T03:04:51.780 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.780 INFO:teuthology.orchestra.run.smithi029.stdout: "active_gid": 14152, 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "active_name": "x", 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "active_addrs": { 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6800", 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 257624116 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 257624116 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:51.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "active_addr": "172.21.15.29:6801/257624116", 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "active_change": "2024-09-21T02:28:37.826720+0000", 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "active_mgr_features": 4540719139924082687, 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "available": true, 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "standbys": [ 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "gid": 24118, 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "z", 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_features": 4540719139924082687, 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-09-21T03:04:51.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.783 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.784 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.786 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.787 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.788 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.789 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.790 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-09-21T03:04:51.791 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.792 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.795 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.797 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.798 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.799 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-09-21T03:04:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-09-21T03:04:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.802 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-09-21T03:04:51.803 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T03:04:51.804 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T03:04:51.805 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.806 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.807 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.808 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.809 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:51.810 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.811 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.812 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.813 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.814 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-09-21T03:04:51.816 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.819 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-09-21T03:04:51.820 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.821 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:51.822 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.823 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.824 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.825 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.826 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.827 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.828 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.829 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-09-21T03:04:51.830 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.831 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.833 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-09-21T03:04:51.836 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-09-21T03:04:51.837 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nginx": { 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nginx", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nginx container image", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.838 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.839 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.840 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_oauth2_proxy": { 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "oauth2-proxy container image", 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-09-21T03:04:51.841 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.842 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.843 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_samba": { 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_samba", 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Samba/SMB container image", 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_samba_metrics": { 2024-09-21T03:04:51.844 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_samba_metrics", 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.845 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.847 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-09-21T03:04:51.848 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-09-21T03:04:51.849 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-09-21T03:04:51.850 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-09-21T03:04:51.851 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "grafana_dashboards_path": { 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "grafana_dashboards_path", 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-09-21T03:04:51.856 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.858 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.865 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.869 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T03:04:51.870 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.871 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T03:04:51.872 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.873 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.874 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:51.876 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.877 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_keepalive_count_max": { 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.878 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_keepalive_interval": { 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_keepalive_interval", 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.879 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.880 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.881 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.883 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.884 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:51.885 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.886 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.887 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.888 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.889 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.890 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.891 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.892 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T03:04:51.893 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T03:04:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.896 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.897 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.898 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.899 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.899 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.899 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.902 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.903 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.904 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.905 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.906 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.907 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.908 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.909 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.910 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.911 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.912 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.913 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.916 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.917 INFO:teuthology.orchestra.run.smithi029.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T03:04:51.918 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T03:04:51.918 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.918 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.918 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.918 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "[]", 2024-09-21T03:04:51.918 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.918 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.918 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.919 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "{}", 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.920 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.921 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.922 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.923 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T03:04:51.924 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T03:04:51.925 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T03:04:51.926 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T03:04:51.927 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.928 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.930 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.931 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.933 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.936 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.937 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.939 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.940 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.941 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.942 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.943 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.945 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.946 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.947 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.950 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.951 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.952 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.953 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.956 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-09-21T03:04:51.957 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-09-21T03:04:51.958 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "sso_oauth2": { 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sso_oauth2", 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.961 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.963 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.964 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.965 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.966 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.967 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.968 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.970 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.972 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.973 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-09-21T03:04:51.973 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.973 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.973 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.973 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-09-21T03:04:51.973 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.973 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.973 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.974 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.975 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T03:04:51.976 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.977 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T03:04:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.981 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.985 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.986 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.987 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.988 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.989 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.990 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-09-21T03:04:51.991 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.992 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:51.995 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.996 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-09-21T03:04:51.997 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-09-21T03:04:51.998 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:51.999 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T03:04:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T03:04:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.002 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.003 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.004 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.005 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.006 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.007 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.008 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.009 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.011 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.012 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.013 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.014 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.015 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.017 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.017 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.017 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.017 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.017 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.017 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.017 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.018 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.019 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.020 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.021 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.022 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.023 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.024 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.026 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.027 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.028 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.031 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.032 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.033 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.034 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.035 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.036 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.039 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.040 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.042 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.043 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.044 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.045 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.046 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.047 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.048 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.049 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.050 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.051 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.052 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.053 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.056 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.057 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-09-21T03:04:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-09-21T03:04:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-09-21T03:04:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T03:04:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.064 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.065 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.066 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-09-21T03:04:52.067 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.068 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.071 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.072 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.073 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.074 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.075 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.076 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.077 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.078 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.079 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.080 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.081 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.083 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.084 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.085 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.086 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.087 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.088 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.089 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-09-21T03:04:52.090 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.091 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.092 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.093 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.096 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.097 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-09-21T03:04:52.098 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.099 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-09-21T03:04:52.103 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-09-21T03:04:52.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.103 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.103 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.106 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.107 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.108 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.109 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.110 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.111 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.112 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.113 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.114 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.115 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.116 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.117 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-09-21T03:04:52.118 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.119 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.120 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.121 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.124 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.125 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.126 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.127 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.128 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.129 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.131 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.132 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.133 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.134 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.135 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.136 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout: "max_requests": { 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_requests", 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.140 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.141 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.142 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.143 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.144 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.145 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.146 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.148 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.148 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.148 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.148 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.148 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.148 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.149 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.150 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.151 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.152 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.153 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.156 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.158 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.160 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.163 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.165 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.166 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.167 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.168 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.169 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.170 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.171 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.172 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.173 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.175 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smb", 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "internal_store_backend": { 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "internal_store_backend", 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.176 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.177 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.181 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "update_orchestration": { 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "update_orchestration", 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.184 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.186 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.187 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.188 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.189 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.190 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.190 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.190 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.190 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.190 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.190 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.190 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.191 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.192 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.194 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.195 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.196 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.199 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.200 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.201 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.204 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.205 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-09-21T03:04:52.206 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.207 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.208 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.209 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.210 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.211 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.212 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.213 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.214 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-09-21T03:04:52.215 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.216 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T03:04:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.218 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.218 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-09-21T03:04:52.218 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-09-21T03:04:52.218 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.218 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.218 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.219 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.219 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.219 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.219 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.219 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T03:04:52.219 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.220 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.220 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.220 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-09-21T03:04:52.220 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-09-21T03:04:52.220 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.220 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.221 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.221 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.221 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.221 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.221 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.221 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T03:04:52.222 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.222 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.222 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.222 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-09-21T03:04:52.222 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-09-21T03:04:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.224 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T03:04:52.224 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.224 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.224 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.224 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-09-21T03:04:52.224 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-09-21T03:04:52.225 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.225 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.225 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.225 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.225 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.225 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.226 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.226 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.226 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.226 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.226 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.226 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-09-21T03:04:52.227 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-09-21T03:04:52.227 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.227 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.227 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.227 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.227 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.228 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.228 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.228 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.228 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.228 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.228 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.229 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-09-21T03:04:52.229 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-09-21T03:04:52.229 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.229 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.229 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.229 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T03:04:52.230 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.230 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.230 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.230 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.230 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.230 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.231 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.231 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-09-21T03:04:52.231 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-09-21T03:04:52.231 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.231 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.231 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.232 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.232 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.232 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.232 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.232 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.232 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.233 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.233 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.233 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.233 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.233 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.233 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.234 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.234 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-09-21T03:04:52.234 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-09-21T03:04:52.234 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.234 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.234 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-09-21T03:04:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-09-21T03:04:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-09-21T03:04:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-09-21T03:04:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-09-21T03:04:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-09-21T03:04:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-09-21T03:04:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.241 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.241 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.241 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.241 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.241 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.241 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.242 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.242 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.242 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.242 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.242 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.243 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.243 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.243 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.243 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.243 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.243 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.243 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.243 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.244 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.244 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.244 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.244 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.244 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.244 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.244 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.247 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.247 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.247 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.247 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.247 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.247 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.247 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.248 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.249 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.249 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.249 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.249 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.249 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.249 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.249 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.250 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.250 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.250 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.250 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.250 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.250 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.250 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.250 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.251 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.254 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.255 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.256 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.257 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.258 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.259 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.260 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.261 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.262 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.263 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.264 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.265 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.266 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.268 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.269 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.270 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_no_wait": { 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.271 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.272 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.273 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.274 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.275 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.277 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.278 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.280 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.281 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.282 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.283 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-09-21T03:04:52.284 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.285 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "gid": 24128, 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "y", 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_features": 4540719139924082687, 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.286 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.287 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.288 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.289 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.290 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.291 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.292 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.293 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.294 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.295 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.296 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-09-21T03:04:52.297 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.306 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T03:04:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-09-21T03:04:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-09-21T03:04:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.319 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.320 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-09-21T03:04:52.321 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-09-21T03:04:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.324 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.325 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.326 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.327 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.328 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-09-21T03:04:52.330 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.331 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.332 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.333 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.334 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.335 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T03:04:52.336 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-09-21T03:04:52.337 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.338 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-09-21T03:04:52.339 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-09-21T03:04:52.340 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-09-21T03:04:52.341 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.342 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.344 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.345 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.345 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nginx": { 2024-09-21T03:04:52.345 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nginx", 2024-09-21T03:04:52.345 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.345 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.345 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.346 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T03:04:52.346 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.346 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.346 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.346 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nginx container image", 2024-09-21T03:04:52.346 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.347 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.347 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.347 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-09-21T03:04:52.347 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-09-21T03:04:52.347 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.348 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.349 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-09-21T03:04:52.350 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.350 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.350 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.350 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_oauth2_proxy": { 2024-09-21T03:04:52.350 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T03:04:52.351 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.351 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.351 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.351 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T03:04:52.351 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.351 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.352 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.352 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "oauth2-proxy container image", 2024-09-21T03:04:52.352 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.352 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.352 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.353 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-09-21T03:04:52.353 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-09-21T03:04:52.353 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.353 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.353 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.353 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T03:04:52.354 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.354 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.354 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.354 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:52.354 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.354 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.355 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.355 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-09-21T03:04:52.355 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-09-21T03:04:52.355 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.355 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.355 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_samba": { 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_samba", 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.356 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.357 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T03:04:52.357 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.357 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.357 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.357 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Samba/SMB container image", 2024-09-21T03:04:52.357 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_samba_metrics": { 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_samba_metrics", 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T03:04:52.358 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.359 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.360 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.360 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T03:04:52.360 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.360 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.360 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.361 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-09-21T03:04:52.361 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.361 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.361 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.361 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-09-21T03:04:52.362 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-09-21T03:04:52.362 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.362 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.362 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.362 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.362 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.363 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.364 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.364 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.364 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.364 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.364 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.364 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.365 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.366 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.367 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.367 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-09-21T03:04:52.367 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.367 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.367 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.367 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T03:04:52.368 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.368 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.368 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.368 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-09-21T03:04:52.368 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-09-21T03:04:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-09-21T03:04:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T03:04:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.371 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "grafana_dashboards_path": { 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "grafana_dashboards_path", 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.372 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-09-21T03:04:52.373 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.374 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.375 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.376 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.377 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.378 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.379 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.380 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.381 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.382 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.383 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.384 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.385 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.386 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-09-21T03:04:52.387 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-09-21T03:04:52.388 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T03:04:52.389 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.390 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.391 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.392 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.395 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.396 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.397 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_keepalive_count_max": { 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T03:04:52.398 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_keepalive_interval": { 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_keepalive_interval", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.399 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.400 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.401 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.404 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.405 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.407 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.408 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.409 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-09-21T03:04:52.410 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.411 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.412 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.413 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.414 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.416 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.418 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.419 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.424 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.426 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.427 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.428 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.429 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.430 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.431 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.432 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.433 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.434 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.435 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "[]", 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.436 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "{}", 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.437 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.442 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.444 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.445 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.446 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.447 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.448 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.449 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.450 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.451 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.452 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.453 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.454 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.455 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.456 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.457 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.459 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.460 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.461 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.461 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-09-21T03:04:52.461 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-09-21T03:04:52.461 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.461 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.461 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.462 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.462 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.462 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.462 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.462 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.462 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.463 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.464 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.466 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.467 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.468 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.469 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.470 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.471 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.472 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.473 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.474 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.476 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "sso_oauth2": { 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sso_oauth2", 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.478 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-09-21T03:04:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.480 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.481 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-09-21T03:04:52.482 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.483 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.484 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.485 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.486 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.487 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.487 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.487 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.487 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.487 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.487 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.487 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.488 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.488 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.488 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.488 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.488 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.488 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.488 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.488 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.489 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.489 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.489 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.489 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.489 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.489 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.489 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.490 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.490 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.490 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.490 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.490 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.490 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.491 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-09-21T03:04:52.491 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-09-21T03:04:52.491 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.491 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.491 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.491 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-09-21T03:04:52.491 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.491 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.492 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.492 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T03:04:52.492 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.492 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.492 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.492 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-09-21T03:04:52.493 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-09-21T03:04:52.493 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.493 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.493 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.493 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-09-21T03:04:52.493 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.493 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.494 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.495 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.497 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.498 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-09-21T03:04:52.500 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.501 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.502 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.503 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.506 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.508 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.510 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.512 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T03:04:52.513 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.516 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.517 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-09-21T03:04:52.519 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.520 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.521 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-09-21T03:04:52.522 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-09-21T03:04:52.523 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.524 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.527 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.528 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.532 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.533 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.535 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.536 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.537 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.539 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.540 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.541 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.542 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.543 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.544 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.545 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.546 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-09-21T03:04:52.547 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.548 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.549 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.550 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.551 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.552 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-09-21T03:04:52.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-09-21T03:04:52.554 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-09-21T03:04:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-09-21T03:04:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-09-21T03:04:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.564 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.566 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.567 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.568 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.569 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.570 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.572 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.573 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.577 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.578 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.579 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.580 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.581 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.582 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.583 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.585 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.586 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.587 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.588 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.592 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.593 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.594 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.595 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.598 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.599 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.600 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.601 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.602 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.604 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.606 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.607 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.608 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.609 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.610 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.611 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.612 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.613 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.614 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.615 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.616 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.617 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.618 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.619 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.620 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.621 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.622 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.623 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.624 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.625 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.626 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.627 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-09-21T03:04:52.628 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-09-21T03:04:52.629 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.630 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-09-21T03:04:52.631 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-09-21T03:04:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-09-21T03:04:52.634 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.635 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.636 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.637 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.639 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.640 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.641 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.642 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.643 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.644 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-09-21T03:04:52.645 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.646 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "max_requests": { 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_requests", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.654 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.655 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-09-21T03:04:52.656 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.656 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.656 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.656 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.656 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.656 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.657 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.657 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.657 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.657 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.657 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.657 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.657 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.658 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.658 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.658 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.658 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.658 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.658 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.659 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.659 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.659 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.659 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.659 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.659 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.659 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.660 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.660 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.660 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.660 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.660 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.660 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.660 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.661 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.662 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.663 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.665 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.666 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.667 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.668 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.669 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.670 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.671 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-09-21T03:04:52.672 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.673 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.674 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.675 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.676 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.677 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.678 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.679 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.680 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.681 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.682 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.683 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.684 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.686 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.687 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.688 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.689 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.690 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.691 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smb", 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.692 INFO:teuthology.orchestra.run.smithi029.stdout: "internal_store_backend": { 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "internal_store_backend", 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.693 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.694 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.695 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.696 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.697 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.698 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "update_orchestration": { 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "update_orchestration", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.699 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.701 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.702 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.703 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.704 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.707 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.708 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.710 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.712 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.713 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.716 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.717 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.718 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.719 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.720 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.721 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.722 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.723 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.724 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.725 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.727 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.728 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.730 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.731 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.733 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.734 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.735 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.737 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.738 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.739 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.740 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.741 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.742 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.743 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.744 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.745 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.747 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.750 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.752 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.754 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.755 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.756 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.757 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.758 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.759 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.760 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.763 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.764 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.765 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_no_wait": { 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.766 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.767 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.768 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.769 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.770 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.771 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.772 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.773 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.774 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.777 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-09-21T03:04:52.778 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-09-21T03:04:52.779 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "modules": [ 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "dashboard", 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "iostat", 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "nfs", 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "restful" 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-09-21T03:04:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.782 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.783 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.784 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.785 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.786 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-09-21T03:04:52.788 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.789 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.790 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.791 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.792 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.793 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.794 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.795 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-09-21T03:04:52.796 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-09-21T03:04:52.797 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-09-21T03:04:52.798 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-09-21T03:04:52.799 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.800 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.801 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.802 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.804 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.806 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.807 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.808 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-09-21T03:04:52.810 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.811 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.812 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.813 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.814 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.815 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-09-21T03:04:52.816 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-09-21T03:04:52.817 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-09-21T03:04:52.818 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-09-21T03:04:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.820 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.821 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.822 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-09-21T03:04:52.823 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.824 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.825 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T03:04:52.826 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.827 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.828 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:52.829 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.830 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.831 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.832 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.833 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-09-21T03:04:52.834 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-09-21T03:04:52.835 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nginx": { 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nginx", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nginx container image", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.836 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.837 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-09-21T03:04:52.838 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_oauth2_proxy": { 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T03:04:52.839 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "oauth2-proxy container image", 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T03:04:52.840 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.841 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_samba": { 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_samba", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.842 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Samba/SMB container image", 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_samba_metrics": { 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_samba_metrics", 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.843 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T03:04:52.844 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.845 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.846 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.847 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T03:04:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.849 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.851 INFO:teuthology.orchestra.run.smithi029.stdout: "grafana_dashboards_path": { 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "grafana_dashboards_path", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-09-21T03:04:52.852 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.853 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.854 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.855 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.856 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.857 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.858 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.859 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.860 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.861 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T03:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-09-21T03:04:52.863 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.864 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.865 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.866 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.866 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.866 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.866 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.866 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.866 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-09-21T03:04:52.866 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.867 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.867 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.867 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-09-21T03:04:52.867 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-09-21T03:04:52.867 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.867 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.868 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.868 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-09-21T03:04:52.868 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.868 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.868 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.868 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-09-21T03:04:52.868 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-09-21T03:04:52.868 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.869 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-09-21T03:04:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T03:04:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.870 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-09-21T03:04:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-09-21T03:04:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.871 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.872 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T03:04:52.872 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.872 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.872 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.872 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T03:04:52.872 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.873 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.874 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.875 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.876 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.877 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-09-21T03:04:52.878 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.879 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_keepalive_count_max": { 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.881 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_keepalive_interval": { 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_keepalive_interval", 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-09-21T03:04:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-09-21T03:04:52.883 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-09-21T03:04:52.884 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-09-21T03:04:52.885 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-09-21T03:04:52.886 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.889 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.890 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.891 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.892 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.893 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.894 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T03:04:52.895 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.896 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.897 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.898 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.899 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T03:04:52.901 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T03:04:52.902 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T03:04:52.903 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.904 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.906 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.907 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.908 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.909 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.910 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.911 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.912 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.913 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.914 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.916 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "[]", 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.917 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "{}", 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.918 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.919 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.921 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.922 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.924 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.925 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T03:04:52.926 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T03:04:52.927 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T03:04:52.928 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.929 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.930 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T03:04:52.931 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T03:04:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T03:04:52.936 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.937 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.938 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T03:04:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.940 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.942 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.943 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.944 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.945 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.946 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.947 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.948 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.949 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.950 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.951 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.952 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.953 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.954 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.955 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.956 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "sso_oauth2": { 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sso_oauth2", 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.957 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:52.959 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.960 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-09-21T03:04:52.961 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.963 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.964 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.965 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.966 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.967 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.968 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.969 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.970 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.971 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.972 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.973 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-09-21T03:04:52.974 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:52.976 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.976 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.976 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.976 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.976 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.976 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.976 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.978 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.979 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.980 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.981 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.982 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.983 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.984 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.985 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-09-21T03:04:52.986 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.987 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T03:04:52.988 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.989 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.990 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.991 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:52.992 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-09-21T03:04:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-09-21T03:04:52.994 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:52.995 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.996 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.997 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.998 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-09-21T03:04:52.999 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.000 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.001 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.002 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.003 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.004 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.005 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.006 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.008 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.009 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.010 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.011 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.012 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.013 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-09-21T03:04:53.014 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.015 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.016 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.017 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.018 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.019 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.020 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.021 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.022 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.023 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.024 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.025 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.026 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-09-21T03:04:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.028 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.029 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.031 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T03:04:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.034 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.035 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.036 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.036 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.036 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.036 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.036 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.036 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.037 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.039 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.040 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.041 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.042 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.043 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.044 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.045 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.046 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.048 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.049 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.050 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.051 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.052 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.053 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.055 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.056 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.057 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.058 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.058 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.058 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.058 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.058 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.058 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.059 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.059 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.059 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.059 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.059 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.059 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.060 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.060 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.060 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.060 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.060 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.060 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.060 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.060 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.061 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.061 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.061 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.061 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.061 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.061 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.061 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.061 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.062 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.062 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.062 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.062 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.062 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.062 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.062 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.063 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.063 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.063 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.063 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.063 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.063 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.063 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.064 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.064 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.064 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-09-21T03:04:53.064 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-09-21T03:04:53.064 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.064 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.065 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.066 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.075 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.076 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.077 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.078 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.079 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.080 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.081 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.082 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.083 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.084 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:53.085 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.086 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-09-21T03:04:53.087 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.091 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.092 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.093 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.094 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.095 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:53.096 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.097 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.098 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.101 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.102 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.103 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.104 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-09-21T03:04:53.105 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T03:04:53.106 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:53.107 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.108 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.110 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.111 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.112 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-09-21T03:04:53.113 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.115 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.116 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.117 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.118 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.119 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.120 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.122 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.123 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-09-21T03:04:53.124 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-09-21T03:04:53.125 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.126 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.127 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.128 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.129 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.130 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "max_requests": { 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_requests", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.131 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-09-21T03:04:53.132 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-09-21T03:04:53.133 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.134 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.135 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.136 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.137 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.138 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.139 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.140 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.141 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.142 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.143 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.144 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.145 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.152 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.153 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.154 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.155 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-09-21T03:04:53.156 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-09-21T03:04:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-09-21T03:04:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-09-21T03:04:53.159 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.160 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.161 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.162 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.163 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.164 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.165 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.166 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.167 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smb", 2024-09-21T03:04:53.168 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "internal_store_backend": { 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "internal_store_backend", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.169 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.170 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.171 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.172 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.173 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.174 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.175 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "update_orchestration": { 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "update_orchestration", 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.176 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.177 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.178 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.179 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.180 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.181 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.182 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.183 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.184 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.185 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.186 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.187 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.188 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.189 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.190 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.191 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.192 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.193 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.194 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.195 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.196 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.197 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.198 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:53.199 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.200 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.201 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.202 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.203 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.204 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.207 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.208 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.210 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.211 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.212 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.213 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.214 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.215 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-09-21T03:04:53.216 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.219 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.220 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.221 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.222 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.223 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.225 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.226 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.227 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.228 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.229 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.230 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.233 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.234 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.235 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.236 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.237 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.238 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.239 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.240 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.241 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.242 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T03:04:53.243 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_no_wait": { 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-09-21T03:04:53.244 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.245 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-09-21T03:04:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-09-21T03:04:53.247 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-09-21T03:04:53.248 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.249 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.250 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.251 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-09-21T03:04:53.252 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.253 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "sqlite3_killpoint": { 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sqlite3_killpoint", 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "dev", 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-09-21T03:04:53.254 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.256 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-09-21T03:04:53.257 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.258 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "services": { 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "dashboard": "https://172.21.15.29:8443/" 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "always_on_modules": { 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "octopus": [ 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-09-21T03:04:53.259 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "pacific": [ 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-09-21T03:04:53.260 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "quincy": [ 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "reef": [ 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-09-21T03:04:53.261 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "squid": [ 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-09-21T03:04:53.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "last_failure_osd_epoch": 3, 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "active_clients": [ 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.263 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2606589568 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "libcephsqlite", 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1819393002 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.264 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2180096491 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-09-21T03:04:53.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T03:04:53.266 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-09-21T03:04:53.266 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-09-21T03:04:53.266 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2895165666 2024-09-21T03:04:53.266 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.266 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:53.266 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:53.266 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T03:04:53.266 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:53.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6480b6590 msgr2=0x7fb6480b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:53.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6480b6590 0x7fb6480b8aa0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb650005760 tx=0x7fb650031400 comp rx=0 tx=0).stop 2024-09-21T03:04:53.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c103f90 msgr2=0x7fb65c19b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c103f90 0x7fb65c19b0c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb65400eaa0 tx=0x7fb65400ef60 comp rx=0 tx=0).stop 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 shutdown_connections 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fb6480b6590 0x7fb6480b8aa0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb65c103f90 0x7fb65c19b0c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb65c1035f0 0x7fb65c19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 --2- 172.21.15.29:0/1390498973 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb65c102570 0x7fb65c199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 >> 172.21.15.29:0/1390498973 conn(0x7fb65c0fdeb0 msgr2=0x7fb65c0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 shutdown_connections 2024-09-21T03:04:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:04:51.775+0000 7fb663fda640 1 -- 172.21.15.29:0/1390498973 wait complete. 2024-09-21T03:04:53.309 DEBUG:tasks.check_counter:Getting stats from a 2024-09-21T03:04:53.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.a perf dump 2024-09-21T03:04:53.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:51 smithi029 ceph-mon[31006]: pgmap v1162: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 324 KiB/s wr, 175 op/s 2024-09-21T03:04:53.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:51 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2004566703' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:51 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2470036503' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:51 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/271332992' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:52 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1390498973' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T03:04:53.311 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:51 smithi190 ceph-mon[33871]: pgmap v1162: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 324 KiB/s wr, 175 op/s 2024-09-21T03:04:53.311 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:51 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2004566703' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.311 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:51 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2470036503' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.311 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:51 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/271332992' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.311 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:52 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1390498973' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T03:04:53.311 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:51 smithi066 ceph-mon[33436]: pgmap v1162: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 324 KiB/s wr, 175 op/s 2024-09-21T03:04:53.312 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:51 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2004566703' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.312 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:51 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2470036503' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.312 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:51 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/271332992' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T03:04:53.312 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:52 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1390498973' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T03:04:53.576 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:53.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:53 smithi029 ceph-mon[31006]: pgmap v1163: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 170 op/s 2024-09-21T03:04:54.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:53 smithi066 ceph-mon[33436]: pgmap v1163: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 170 op/s 2024-09-21T03:04:54.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:53 smithi190 ceph-mon[33871]: pgmap v1163: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 170 op/s 2024-09-21T03:04:54.222 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:54.222 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:04:54.222 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 2595, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2576, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 10078556, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 955204, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 2, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.323561271, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.146534500, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.100420419, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.067124753, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2575, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.095549466, 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000037106 2024-09-21T03:04:54.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1018928, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 284224 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 1579, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 1924, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 38150144, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 1200920, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 7, 2024-09-21T03:04:54.224 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 5, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.214694544, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.087007096, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.071731947, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.042426415, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 1923, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.070507640, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000036665 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.225 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 3120, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 700320 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 2040, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2042, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 37270974, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 648721, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 7, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 5, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.220902136, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.077509389, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.075254912, 2024-09-21T03:04:54.226 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.054188436, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2040, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.070660344, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000034637 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.227 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 1, 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-MDSRank": { 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 9759, 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.253452277, 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000025971 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-PurgeQueue": { 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-09-21T03:04:54.228 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "request": 0, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "reply": 0, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "reply_latency": { 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_reply": 0, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "forward": 0, 2024-09-21T03:04:54.229 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_complete": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_keys": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_commit": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_split": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_merge": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes": 8067, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_top": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_bottom": 8067, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pin_tail": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pinned": 8067, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_expired": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_with_caps": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": 0, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "subtrees": 1, 2024-09-21T03:04:54.230 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_hit": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_forward": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_discover": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_remote_ino": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_lock": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "load_cent": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "q": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "exported": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "exported_inodes": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "imported": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "imported_inodes": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_dir_fetch": 0, 2024-09-21T03:04:54.231 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_peer_discover": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rfiles": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rbytes": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rsnaps": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_set_tag": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_file_inodes": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_inode_file_caps": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T03:04:54.232 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_grant": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps_dirty": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_cap_release": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "process_request_cap_release": 0 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache": { 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update_receipt": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_try_discover": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_send_discover": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_handle_discover": 0, 2024-09-21T03:04:54.233 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_delayed": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_created": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_enqueued": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_reintegrated": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_migrated": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_processing": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_started": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_completed": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_quiesce_path": 0, 2024-09-21T03:04:54.234 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_quiesce_inode": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_exportdir": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_flush": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragstats": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_inodestats": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "uninline_started": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "uninline_succeeded": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "uninline_write_failed": 0 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log": { 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "evlrg": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "evadd": 0, 2024-09-21T03:04:54.235 INFO:teuthology.orchestra.run.smithi029.stdout: "evex": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "evtrm": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "ev": 52430, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "evexg": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "evexd": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "segadd": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "segex": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "segtrm": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "seg": 67, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "segmjr": 6, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "segexg": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "segexd": 0, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "expos": 4194304, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "wrpos": 87083021, 2024-09-21T03:04:54.236 INFO:teuthology.orchestra.run.smithi029.stdout: "rdpos": 87080798, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "jlat": { 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "replayed": 52430 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_mem": { 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "ino": 13, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "ino+": 8093, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "ino-": 8080, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "dir": 11, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "dir+": 42, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "dir-": 31, 2024-09-21T03:04:54.237 INFO:teuthology.orchestra.run.smithi029.stdout: "dn": 8067, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "dn+": 8313, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "dn-": 246, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "cap": 0, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "cap+": 0, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "cap-": 0, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "rss": 128500, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "heap": 328089600 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_server": { 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_client_request": 0, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_server_request": 0, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_request": 0, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_session": 0, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_peer_request": 0, 2024-09-21T03:04:54.238 INFO:teuthology.orchestra.run.smithi029.stdout: "req_create_latency": { 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getattr_latency": { 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getfilelock_latency": { 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.239 INFO:teuthology.orchestra.run.smithi029.stdout: "req_link_latency": { 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookup_latency": { 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookuphash_latency": { 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupino_latency": { 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupname_latency": { 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupparent_latency": { 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.242 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupsnap_latency": { 2024-09-21T03:04:54.242 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.242 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.242 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.242 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lssnap_latency": { 2024-09-21T03:04:54.242 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.242 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mkdir_latency": { 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mknod_latency": { 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mksnap_latency": { 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "req_open_latency": { 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "req_readdir_latency": { 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rename_latency": { 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "req_renamesnap_latency": { 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "req_snapdiff_latency": { 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.245 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmdir_latency": { 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmsnap_latency": { 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmxattr_latency": { 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setattr_latency": { 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setdirlayout_latency": { 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setfilelock_latency": { 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setlayout_latency": { 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setxattr_latency": { 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "req_symlink_latency": { 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.248 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "req_unlink_latency": { 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getvxattr_latency": { 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sessions": { 2024-09-21T03:04:54.249 INFO:teuthology.orchestra.run.smithi029.stdout: "session_count": 1, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "session_add": 2, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "session_remove": 1, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_open": 1, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_stale": 0, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "total_load": 0, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "average_load": 0, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "avg_session_uptime": 2751, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "mdthresh_evicted": 0 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:04:54.250 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-09-21T03:04:54.251 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:04:54.252 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 224873, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 76, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 8760, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 164, 2024-09-21T03:04:54.253 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 5529712, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 32396, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter": { 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "op_active": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "op_laggy": 0, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send": 5724, 2024-09-21T03:04:54.254 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send_bytes": 21, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "op_resend": 3, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "op_reply": 5724, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "op_latency": { 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 5724, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 6.023907304, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.001052394 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "op_inflight": 0, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "oplen_avg": { 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 5724, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 5725 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "op": 5724, 2024-09-21T03:04:54.255 INFO:teuthology.orchestra.run.smithi029.stdout: "op_r": 5724, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "op_w": 0, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "op_rmw": 0, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "op_pg": 0, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_stat": 3215, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_create": 0, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_read": 2507, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_write": 0, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writefull": 0, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writesame": 0, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_append": 0, 2024-09-21T03:04:54.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_zero": 0, 2024-09-21T03:04:54.258 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_truncate": 0, 2024-09-21T03:04:54.258 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_delete": 0, 2024-09-21T03:04:54.258 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_mapext": 0, 2024-09-21T03:04:54.258 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_sparse_read": 0, 2024-09-21T03:04:54.258 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_clonerange": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_getxattr": 1, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_setxattr": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_rmxattr": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_call": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_watch": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_notify": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_other": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_active": 0, 2024-09-21T03:04:54.259 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_send": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_resend": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_ping": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_active": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_send": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_resend": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_active": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_send": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_resend": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_active": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_send": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_resend": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "command_active": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "command_send": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "command_resend": 0, 2024-09-21T03:04:54.260 INFO:teuthology.orchestra.run.smithi029.stdout: "map_epoch": 86, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "map_full": 0, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "map_inc": 5, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_sessions": 9, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_open": 11, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_close": 2, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_laggy": 0, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_wr": 0, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_rd": 2, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_del": 0 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "oft": { 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_objs": 0, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_updates": 0, 2024-09-21T03:04:54.261 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_removes": 0 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "purge_queue": { 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops": 0, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing": 0, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_high_water": 0, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed_ops": 0, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed": 0, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_item_in_journal": 0 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-09-21T03:04:54.262 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 6210, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 84912922, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 6210, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 6210, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 84912922, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_bytes": { 2024-09-21T03:04:54.263 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 5724, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 82891354, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 5724, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 82891354, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.264 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_ops": { 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 1024, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 5724, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 5724, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 5724, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 5724, 2024-09-21T03:04:54.265 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-09-21T03:04:54.266 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle-0x55b18e9ef450": { 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-09-21T03:04:54.267 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "trackedop": { 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_ops_count": 0 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:54.268 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:55.823 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:55 smithi029 ceph-mon[31006]: pgmap v1164: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 322 KiB/s wr, 175 op/s 2024-09-21T03:04:55.860 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-09-21T03:04:55.860 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-09-21T03:04:55.860 DEBUG:tasks.check_counter:Getting stats from d 2024-09-21T03:04:55.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.d perf dump 2024-09-21T03:04:56.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:55 smithi066 ceph-mon[33436]: pgmap v1164: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 322 KiB/s wr, 175 op/s 2024-09-21T03:04:56.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:55 smithi190 ceph-mon[33871]: pgmap v1164: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 322 KiB/s wr, 175 op/s 2024-09-21T03:04:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:57.059 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:57.059 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:04:57.059 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 467, 2024-09-21T03:04:57.059 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 462, 2024-09-21T03:04:57.059 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 908000, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 286352, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.071282723, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.036407535, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.041901573, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 461, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.020156832, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000043724 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 908000, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 286352 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 2, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 353, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 320, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 222864, 2024-09-21T03:04:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 2, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.029803423, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.024914923, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.001609161, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 351, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.015017072, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000042783 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 222864 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 4, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 4, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 2960, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 448, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-09-21T03:04:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.002217048, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000050051, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.002158249, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.003909869, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.001303289 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 2960, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:04:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:04:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:04:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148744, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 30, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-09-21T03:04:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:04:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:04:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-09-21T03:04:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-09-21T03:04:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 968, 2024-09-21T03:04:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-09-21T03:04:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-09-21T03:04:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-09-21T03:04:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-09-21T03:04:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-09-21T03:04:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:04:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-09-21T03:04:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 469, 2024-09-21T03:04:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 841960, 2024-09-21T03:04:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 469, 2024-09-21T03:04:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-09-21T03:04:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-09-21T03:04:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 469, 2024-09-21T03:04:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 841960, 2024-09-21T03:04:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:57.073 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:04:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:04:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:57 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:57 smithi066 ceph-mon[33436]: pgmap v1165: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 278 KiB/s wr, 166 op/s 2024-09-21T03:04:57.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:04:57.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:04:57.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:57 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.864 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:57 smithi029 ceph-mon[31006]: pgmap v1165: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 278 KiB/s wr, 166 op/s 2024-09-21T03:04:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T03:04:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T03:04:57.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:57 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' 2024-09-21T03:04:57.875 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:57 smithi190 ceph-mon[33871]: pgmap v1165: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 278 KiB/s wr, 166 op/s 2024-09-21T03:04:57.894 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-21T03:04:57.894 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-21T03:04:57.894 DEBUG:tasks.check_counter:Getting stats from g 2024-09-21T03:04:57.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.g perf dump 2024-09-21T03:04:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:58.699 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:04:58.699 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:04:58.699 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 2152, 2024-09-21T03:04:58.699 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2133, 2024-09-21T03:04:58.699 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 1578912, 2024-09-21T03:04:58.699 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 814974, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 5, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.281295593, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.140457267, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.083242543, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.050367551, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2132, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.087490185, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000041036 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.700 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1019472, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 285504 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 3403, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 3749, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 3363432, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 1782749, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 5, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 4, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.309682073, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.134852131, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.064827470, 2024-09-21T03:04:58.701 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.089227012, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3748, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.149067359, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000039772 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 3120, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 702128 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.702 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 4, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 6, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 1629, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 1284, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 7, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 4, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.002456670, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000134677, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.001988088, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000105221, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 4, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.002854272, 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000713568 2024-09-21T03:04:58.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 1, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-MDSRank": { 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-09-21T03:04:58.704 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 8539, 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.227177738, 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000026604 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-PurgeQueue": { 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-09-21T03:04:58.705 INFO:teuthology.orchestra.run.smithi029.stdout: "request": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "reply": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "reply_latency": { 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_reply": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "forward": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_complete": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_keys": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_commit": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_split": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_merge": 0, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes": 276, 2024-09-21T03:04:58.706 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_top": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_bottom": 276, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pin_tail": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pinned": 276, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_expired": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_with_caps": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "subtrees": 1, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_hit": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_forward": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_discover": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_remote_ino": 0, 2024-09-21T03:04:58.707 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_lock": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "load_cent": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "q": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "exported": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "exported_inodes": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "imported": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "imported_inodes": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_dir_fetch": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_peer_discover": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rfiles": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rbytes": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rsnaps": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_set_tag": 0, 2024-09-21T03:04:58.708 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_file_inodes": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_inode_file_caps": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_grant": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps_dirty": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_cap_release": 0, 2024-09-21T03:04:58.709 INFO:teuthology.orchestra.run.smithi029.stdout: "process_request_cap_release": 0 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache": { 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update_receipt": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_try_discover": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_send_discover": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_handle_discover": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_delayed": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_created": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_enqueued": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_reintegrated": 0, 2024-09-21T03:04:58.710 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_migrated": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_processing": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_started": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_completed": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_quiesce_path": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_quiesce_inode": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_exportdir": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_flush": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragstats": 0, 2024-09-21T03:04:58.711 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_inodestats": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "uninline_started": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "uninline_succeeded": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "uninline_write_failed": 0 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log": { 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "evlrg": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "evadd": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "evex": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "evtrm": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "ev": 2273, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "evexg": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "evexd": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "segadd": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "segex": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "segtrm": 0, 2024-09-21T03:04:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: "seg": 4, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "segmjr": 2, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "segexg": 0, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "segexd": 0, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "expos": 4194304, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "wrpos": 6692004, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "rdpos": 6690866, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "jlat": { 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "replayed": 2273 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_mem": { 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "ino": 6, 2024-09-21T03:04:58.713 INFO:teuthology.orchestra.run.smithi029.stdout: "ino+": 291, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "ino-": 285, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "dir": 4, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "dir+": 17, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "dir-": 13, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "dn": 276, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "dn+": 298, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "dn-": 22, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "cap": 0, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "cap+": 0, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "cap-": 0, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "rss": 121000, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "heap": 320749568 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.714 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_server": { 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_client_request": 0, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_server_request": 0, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_request": 0, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_session": 0, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_peer_request": 0, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "req_create_latency": { 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getattr_latency": { 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.715 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getfilelock_latency": { 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "req_link_latency": { 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookup_latency": { 2024-09-21T03:04:58.716 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.717 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.717 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.717 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookuphash_latency": { 2024-09-21T03:04:58.717 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.717 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.717 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupino_latency": { 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupname_latency": { 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupparent_latency": { 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.718 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupsnap_latency": { 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lssnap_latency": { 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.719 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mkdir_latency": { 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mknod_latency": { 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mksnap_latency": { 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.720 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "req_open_latency": { 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "req_readdir_latency": { 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rename_latency": { 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.721 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "req_renamesnap_latency": { 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "req_snapdiff_latency": { 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmdir_latency": { 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.722 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmsnap_latency": { 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmxattr_latency": { 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setattr_latency": { 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.723 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setdirlayout_latency": { 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setfilelock_latency": { 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setlayout_latency": { 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.724 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setxattr_latency": { 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "req_symlink_latency": { 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "req_unlink_latency": { 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.725 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getvxattr_latency": { 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sessions": { 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "session_count": 1, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "session_add": 1, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "session_remove": 0, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_open": 1, 2024-09-21T03:04:58.726 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_stale": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "total_load": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "average_load": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "avg_session_uptime": 2753, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "mdthresh_evicted": 0 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:04:58.727 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:04:58.728 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 198264, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 57, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-09-21T03:04:58.729 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 8760, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 164, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 207024, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 1155, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-09-21T03:04:58.730 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter": { 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "op_active": 0, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "op_laggy": 0, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send": 5067, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send_bytes": 21, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "op_resend": 3, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "op_reply": 5067, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "op_latency": { 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 5067, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 4.025938165, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000794540 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.731 INFO:teuthology.orchestra.run.smithi029.stdout: "op_inflight": 0, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "oplen_avg": { 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 5067, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 5068 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "op": 5067, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "op_r": 5067, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "op_w": 0, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "op_rmw": 0, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "op_pg": 0, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_stat": 3217, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_create": 0, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_read": 1848, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_write": 0, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writefull": 0, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writesame": 0, 2024-09-21T03:04:58.732 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_append": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_zero": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_truncate": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_delete": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_mapext": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_sparse_read": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_clonerange": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_getxattr": 1, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_setxattr": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_rmxattr": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_call": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_watch": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_notify": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls": 0, 2024-09-21T03:04:58.733 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_other": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_active": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_send": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_resend": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_ping": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_active": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_send": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_resend": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_active": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_send": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_resend": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_active": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_send": 0, 2024-09-21T03:04:58.734 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_resend": 0, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "command_active": 0, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "command_send": 0, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "command_resend": 0, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "map_epoch": 86, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "map_full": 0, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "map_inc": 5, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_sessions": 6, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_open": 8, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_close": 2, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_laggy": 0, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_wr": 0, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_rd": 2, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_del": 0 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.735 INFO:teuthology.orchestra.run.smithi029.stdout: "oft": { 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_objs": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_updates": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_removes": 0 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "purge_queue": { 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_high_water": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed_ops": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed": 0, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_item_in_journal": 0 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.736 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 5555, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 4416067, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 5555, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 5555, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 4416067, 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:58.737 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_bytes": { 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 5067, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 2497705, 2024-09-21T03:04:58.738 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 5067, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 2497705, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_ops": { 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 1024, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-09-21T03:04:58.739 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 5067, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 5067, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 5067, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 5067, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-09-21T03:04:58.740 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle-0x55b201c2b6c0": { 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-09-21T03:04:58.741 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "trackedop": { 2024-09-21T03:04:58.742 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_ops_count": 0 2024-09-21T03:04:58.743 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:04:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:04:59.433 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-09-21T03:04:59.434 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-09-21T03:04:59.434 DEBUG:tasks.check_counter:Getting stats from j 2024-09-21T03:04:59.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.j perf dump 2024-09-21T03:04:59.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:04:59 smithi029 ceph-mon[31006]: pgmap v1166: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 71 KiB/s wr, 120 op/s 2024-09-21T03:04:59.693 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:04:59.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:04:59 smithi190 ceph-mon[33871]: pgmap v1166: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 71 KiB/s wr, 120 op/s 2024-09-21T03:05:00.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:04:59 smithi066 ceph-mon[33436]: pgmap v1166: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 71 KiB/s wr, 120 op/s 2024-09-21T03:05:00.397 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T03:05:00.397 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:00.397 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 4, 2024-09-21T03:05:00.397 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 4, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 2960, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 448, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.003271237, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000045162, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.002236388, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.003994194, 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.001331398 2024-09-21T03:05:00.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 2960, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 468, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 463, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 908272, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 286992, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-09-21T03:05:00.399 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.069425434, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.036061577, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.042390793, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 462, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.018453843, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000039943 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 908272, 2024-09-21T03:05:00.400 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 286992 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 2, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 354, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 320, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 223040, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 2, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.031798857, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.026695794, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.001847176, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:00.401 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 352, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.015009076, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000042639 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 223040 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-09-21T03:05:00.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:00.403 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:00.404 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148744, 2024-09-21T03:05:00.405 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 30, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:00.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 968, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 470, 2024-09-21T03:05:00.407 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 842090, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 470, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 470, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 842090, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T03:05:00.408 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T03:05:01.140 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-21T03:05:01.141 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-21T03:05:01.141 DEBUG:tasks.check_counter:Getting stats from b 2024-09-21T03:05:01.141 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.b perf dump 2024-09-21T03:05:01.412 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T03:05:01.805 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:01 smithi066 ceph-mon[33436]: pgmap v1167: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 KiB/s wr, 64 op/s 2024-09-21T03:05:01.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:01 smithi190 ceph-mon[33871]: pgmap v1167: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 KiB/s wr, 64 op/s 2024-09-21T03:05:01.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:01 smithi029 ceph-mon[31006]: pgmap v1167: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 KiB/s wr, 64 op/s 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 37969, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 25353, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 9806272, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 31883062, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 13, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 5, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 2.576172069, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 1.130649960, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 1.551696682, 2024-09-21T03:05:02.086 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.221105512, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 25351, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.654199479, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000025805 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 47993, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.008980428, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000187 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:02.087 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 160107, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 178161, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 48650693, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 113324144, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 12, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 7, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 12.439252322, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 6.016380474, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 8.023860091, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.087070632, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 178160, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 3.349469900, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000018800 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:02.088 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 328090, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.059892403, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000182 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 1047584, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 289936 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 7793, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 7950, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 2372164, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 16517387, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 13, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 6, 2024-09-21T03:05:02.089 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.720147877, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.310938574, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.177867337, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.198853541, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 7949, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.280093179, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000035236 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 860, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000172178, 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000200 2024-09-21T03:05:02.090 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 6656, 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 798688 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "cct": { 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "total_workers": 1, 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "unhealthy_workers": 0 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "finisher-MDSRank": { 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "queue_len": 0, 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "complete_latency": { 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 15525, 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 9.483619082, 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000610861 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "finisher-PurgeQueue": { 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "queue_len": 0, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "complete_latency": { 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 8979, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.483624464, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000053861 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "mds": { 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "request": 140611, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "reply": 138001, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "reply_latency": { 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 138001, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 278.634264827, 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.002019074 2024-09-21T03:05:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "slow_reply": 0, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "forward": 2610, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_fetch_complete": 0, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_fetch_keys": 0, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_commit": 11, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_split": 0, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_merge": 0, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes": 16, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_top": 2, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_bottom": 3, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_pin_tail": 11, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_pinned": 12, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_expired": 52, 2024-09-21T03:05:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_with_caps": 2, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "caps": 2, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "subtrees": 4, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse": 265604, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_hit": 262326, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_forward": 2144, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_discover": 14, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_remote_ino": 0, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_lock": 0, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "load_cent": 0, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "q": 0, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "exported": 2, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "exported_inodes": 1, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "imported": 3, 2024-09-21T03:05:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "imported_inodes": 61, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_dir_fetch": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_peer_discover": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rfiles": 66, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rbytes": 12312210, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rsnaps": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_set_tag": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_file_inodes": 0, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_inode_file_caps": 2, 2024-09-21T03:05:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_revoke": 9594, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_grant": 12973, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_caps": 15102, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_caps_dirty": 4353, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_cap_release": 462, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "process_request_cap_release": 11563 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache": { 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_update": 285, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_update_receipt": 29, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_try_discover": 20, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_send_discover": 26, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_handle_discover": 303, 2024-09-21T03:05:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays_delayed": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_created": 3309, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_enqueued": 3309, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_reintegrated": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_migrated": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_processing": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_started": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_completed": 0, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_quiesce_path": 25, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_quiesce_inode": 1365, 2024-09-21T03:05:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_exportdir": 2, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_flush": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_fragstats": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_inodestats": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_started": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_succeeded": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_write_failed": 0 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log": { 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "evlrg": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "evadd": 21435, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "evex": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "evtrm": 0, 2024-09-21T03:05:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "ev": 21435, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "evexg": 0, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "evexd": 0, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "segadd": 32, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "segex": 0, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "segtrm": 0, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "seg": 32, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "segmjr": 3, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "segexg": 0, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "segexd": 0, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "expos": 4194304, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "wrpos": 46526398, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "rdpos": 0, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "jlat": { 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 7614, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 34.041380289, 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.004470893 2024-09-21T03:05:02.099 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "replayed": 0 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_mem": { 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "ino": 21, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "ino+": 3356, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "ino-": 3335, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "dir": 20, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "dir+": 56, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "dir-": 36, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "dn": 16, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "dn+": 3915, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "dn-": 3899, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "cap": 2, 2024-09-21T03:05:02.100 INFO:teuthology.orchestra.run.smithi066.stdout: "cap+": 3318, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "cap-": 3316, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "rss": 128508, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "heap": 335425536 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_server": { 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "dispatch_client_request": 147171, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "dispatch_server_request": 1025, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_request": 140611, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_session": 477, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_peer_request": 2689, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "req_create_latency": { 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 3248, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 174.276871923, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.053656672 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getattr_latency": { 2024-09-21T03:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 122603, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 31.732418640, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000258822 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getfilelock_latency": { 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "req_link_latency": { 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.102 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookup_latency": { 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 1723, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 7.786231986, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.004518997 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookuphash_latency": { 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupino_latency": { 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupname_latency": { 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.103 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupparent_latency": { 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupsnap_latency": { 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.104 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lssnap_latency": { 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mkdir_latency": { 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mknod_latency": { 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.105 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mksnap_latency": { 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "req_open_latency": { 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 2166, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 1.429727544, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000660077 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "req_readdir_latency": { 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 4274, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 17.371850445, 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.004064541 2024-09-21T03:05:02.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rename_latency": { 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 565, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.808532533, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.001431031 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "req_renamesnap_latency": { 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "req_snapdiff_latency": { 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmdir_latency": { 2024-09-21T03:05:02.107 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 9, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.054869486, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.006096609 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmsnap_latency": { 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmxattr_latency": { 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setattr_latency": { 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 1, 2024-09-21T03:05:02.108 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 21.964463242, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 21.964463242 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setdirlayout_latency": { 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setfilelock_latency": { 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 112, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.038844845, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000346828 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setlayout_latency": { 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setxattr_latency": { 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "req_symlink_latency": { 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "req_unlink_latency": { 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 3300, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 23.170454183, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.007021349 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.110 INFO:teuthology.orchestra.run.smithi066.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getvxattr_latency": { 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_sessions": { 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "session_count": 1, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "session_add": 3, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "session_remove": 2, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "sessions_open": 1, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "sessions_stale": 0, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "total_load": 12185, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "average_load": 12185, 2024-09-21T03:05:02.111 INFO:teuthology.orchestra.run.smithi066.stdout: "avg_session_uptime": 1558, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "mdthresh_evicted": 0 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "mempool": { 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bloom_filter_items": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:02.112 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:02.113 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_items": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_anon_bytes": 296406, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_anon_items": 106, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_bytes": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_items": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:02.114 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_bytes": 8760, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_items": 164, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "pgmap_items": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_co_bytes": 86232, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_co_items": 368, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_1_items": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_2_items": 0 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter": { 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "op_active": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "op_laggy": 0, 2024-09-21T03:05:02.115 INFO:teuthology.orchestra.run.smithi066.stdout: "op_send": 16985, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "op_send_bytes": 42996736, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "op_resend": 29, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "op_reply": 16985, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "op_latency": { 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 16985, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 81.671280354, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.004808435 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "op_inflight": 0, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "oplen_avg": { 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 16985, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 17974 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "op": 16985, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "op_r": 838, 2024-09-21T03:05:02.116 INFO:teuthology.orchestra.run.smithi066.stdout: "op_w": 16147, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "op_rmw": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "op_pg": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_stat": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_create": 10, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_read": 838, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_write": 8452, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_writefull": 495, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_writesame": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_append": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_zero": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_truncate": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_delete": 6631, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_mapext": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_sparse_read": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_clonerange": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_getxattr": 0, 2024-09-21T03:05:02.117 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_setxattr": 20, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_rmxattr": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_call": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_watch": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_notify": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_pgls": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_other": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_active": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_send": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_resend": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_ping": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_active": 0, 2024-09-21T03:05:02.118 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_send": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_resend": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_active": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_send": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_resend": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_active": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_send": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_resend": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "command_active": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "command_send": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "command_resend": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "map_epoch": 86, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "map_full": 0, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "map_inc": 5, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_sessions": 12, 2024-09-21T03:05:02.119 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_session_open": 15, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_session_close": 3, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_laggy": 0, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_wr": 1050, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_rd": 0, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_del": 478 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "oft": { 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_objs": 1, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_updates": 0, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_removes": 0 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "purge_queue": { 2024-09-21T03:05:02.120 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_ops": 0, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_ops_high_water": 10, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing": 0, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_high_water": 8, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executed_ops": 6609, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executed": 3309, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_item_in_journal": 0 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 104857600, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 205865, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 42271093, 2024-09-21T03:05:02.121 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 205865, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 205865, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 42271093, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-objecter_bytes": { 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 104857600, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 16985, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 43357345, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 16985, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 43357345, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-objecter_ops": { 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:02.123 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 1024, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 16985, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 16985, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 16985, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 16985, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.124 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 3758096384, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 3309, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 360609, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 3309, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 838, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 360609, 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:02.125 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-write_buf_throttle-0x56309d3b95f0": { 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 3758096384, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 21435, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 42332094, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 21435, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 7614, 2024-09-21T03:05:02.126 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 42332094, 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: "trackedop": { 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: "slow_ops_count": 1287 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:02.127 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-09-21T03:05:03.212 INFO:tasks.check_counter:Daemon mds.b mds.exported=2 2024-09-21T03:05:03.212 INFO:tasks.check_counter:Daemon mds.b mds.imported=3 2024-09-21T03:05:03.212 DEBUG:tasks.check_counter:Getting stats from e 2024-09-21T03:05:03.212 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.e perf dump 2024-09-21T03:05:03.471 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T03:05:03.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:03 smithi190 ceph-mon[33871]: pgmap v1168: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-21T03:05:03.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:03 smithi029 ceph-mon[31006]: pgmap v1168: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-21T03:05:03.905 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:03 smithi066 ceph-mon[33436]: pgmap v1168: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-21T03:05:04.030 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-09-21T03:05:04.030 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:04.030 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 281, 2024-09-21T03:05:04.030 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 283, 2024-09-21T03:05:04.030 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 8959615, 2024-09-21T03:05:04.030 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 89377, 2024-09-21T03:05:04.030 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 7, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 3, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.035731491, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.013542037, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.010969349, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.007600045, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 281, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.012340322, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000043915 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 199, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 195, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 12562534, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 73581, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 5, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 4, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.028429657, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.009360003, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.017762723, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.003546856, 2024-09-21T03:05:04.032 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 194, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.009738003, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000050195 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 207056, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 28368 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 198, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 230, 2024-09-21T03:05:04.033 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 21023901, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 210559, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 6, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 5, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.034074652, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.008364442, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.019544754, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.004580110, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 229, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.012407576, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000054181 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.034 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 3120, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 149184 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "cct": { 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "total_workers": 1, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "unhealthy_workers": 0 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "finisher-MDSRank": { 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "queue_len": 0, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "complete_latency": { 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 1045, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.059456934, 2024-09-21T03:05:04.035 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000056896 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "finisher-PurgeQueue": { 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "queue_len": 0, 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "complete_latency": { 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "mds": { 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "request": 0, 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "reply": 0, 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "reply_latency": { 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.036 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "slow_reply": 0, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "forward": 0, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_fetch_complete": 0, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_fetch_keys": 0, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_commit": 0, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_split": 0, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_merge": 0, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes": 3690, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_top": 2052, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_bottom": 1638, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_pin_tail": 0, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_pinned": 758, 2024-09-21T03:05:04.037 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_expired": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_with_caps": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "caps": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "subtrees": 2, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_hit": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_forward": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_discover": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_remote_ino": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_lock": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "load_cent": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "q": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "exported": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "exported_inodes": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "imported": 0, 2024-09-21T03:05:04.038 INFO:teuthology.orchestra.run.smithi066.stdout: "imported_inodes": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_dir_fetch": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_peer_discover": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rfiles": 66, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rbytes": 12312210, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rsnaps": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_set_tag": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_file_inodes": 0, 2024-09-21T03:05:04.039 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_inode_file_caps": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_grant": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_caps": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_caps_dirty": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_cap_release": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "process_request_cap_release": 0 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache": { 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_update": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_update_receipt": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_try_discover": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_send_discover": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_handle_discover": 0, 2024-09-21T03:05:04.040 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays_delayed": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_created": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_enqueued": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_reintegrated": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_migrated": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_processing": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_started": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_completed": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_quiesce_path": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_quiesce_inode": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:05:04.041 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_exportdir": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_flush": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_fragstats": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_inodestats": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_started": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_succeeded": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_write_failed": 0 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log": { 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "evlrg": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "evadd": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "evex": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "evtrm": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "ev": 20318, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "evexg": 0, 2024-09-21T03:05:04.042 INFO:teuthology.orchestra.run.smithi066.stdout: "evexd": 0, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "segadd": 0, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "segex": 0, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "segtrm": 0, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "seg": 2, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "segmjr": 11, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "segexg": 0, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "segexd": 0, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "expos": 337343608, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "wrpos": 340224400, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "rdpos": 340222344, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "jlat": { 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.043 INFO:teuthology.orchestra.run.smithi066.stdout: "replayed": 20318 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_mem": { 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "ino": 26, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "ino+": 3542, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "ino-": 3516, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "dir": 25, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "dir+": 113, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "dir-": 88, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "dn": 3690, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "dn+": 4747, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "dn-": 1057, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "cap": 0, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "cap+": 0, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "cap-": 0, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "rss": 121416, 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: "heap": 324943872 2024-09-21T03:05:04.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_server": { 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "dispatch_client_request": 0, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "dispatch_server_request": 0, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_request": 0, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_session": 0, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_peer_request": 0, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "req_create_latency": { 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getattr_latency": { 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getfilelock_latency": { 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "req_link_latency": { 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookup_latency": { 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.046 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookuphash_latency": { 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupino_latency": { 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupname_latency": { 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupparent_latency": { 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.047 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupsnap_latency": { 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lssnap_latency": { 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.048 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mkdir_latency": { 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mknod_latency": { 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mksnap_latency": { 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "req_open_latency": { 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "req_readdir_latency": { 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rename_latency": { 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "req_renamesnap_latency": { 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "req_snapdiff_latency": { 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmdir_latency": { 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.051 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmsnap_latency": { 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmxattr_latency": { 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setattr_latency": { 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setdirlayout_latency": { 2024-09-21T03:05:04.052 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setfilelock_latency": { 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setlayout_latency": { 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setxattr_latency": { 2024-09-21T03:05:04.053 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "req_symlink_latency": { 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "req_unlink_latency": { 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:05:04.054 INFO:teuthology.orchestra.run.smithi066.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getvxattr_latency": { 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_sessions": { 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "session_count": 1, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "session_add": 1, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "session_remove": 0, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "sessions_open": 0, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "sessions_stale": 0, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "total_load": 0, 2024-09-21T03:05:04.055 INFO:teuthology.orchestra.run.smithi066.stdout: "average_load": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "avg_session_uptime": 2738, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "mdthresh_evicted": 0 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "mempool": { 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bloom_filter_items": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:04.056 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:04.057 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_items": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_anon_bytes": 220776, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_anon_items": 62, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:04.058 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_bytes": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_items": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_bytes": 8760, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_items": 164, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "pgmap_items": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_co_bytes": 2600707, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_co_items": 15707, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_1_items": 0, 2024-09-21T03:05:04.059 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_2_items": 0 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter": { 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "op_active": 0, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "op_laggy": 0, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "op_send": 616, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "op_send_bytes": 26, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "op_resend": 4, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "op_reply": 616, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "op_latency": { 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 616, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.938991976, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.001524337 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.060 INFO:teuthology.orchestra.run.smithi066.stdout: "op_inflight": 0, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "oplen_avg": { 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 616, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 617 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "op": 616, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "op_r": 616, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "op_w": 0, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "op_rmw": 0, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "op_pg": 0, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_stat": 319, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_create": 0, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_read": 294, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_write": 0, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_writefull": 0, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_writesame": 0, 2024-09-21T03:05:04.061 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_append": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_zero": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_truncate": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_delete": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_mapext": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_sparse_read": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_clonerange": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_getxattr": 2, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_setxattr": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_rmxattr": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_call": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_watch": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_notify": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:05:04.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_pgls": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_other": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_active": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_send": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_resend": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_ping": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_active": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_send": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_resend": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_active": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_send": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_resend": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_active": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_send": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_resend": 0, 2024-09-21T03:05:04.063 INFO:teuthology.orchestra.run.smithi066.stdout: "command_active": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "command_send": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "command_resend": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "map_epoch": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "map_full": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "map_inc": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_sessions": 9, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_session_open": 9, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_session_close": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_laggy": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_wr": 0, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_rd": 2, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_del": 0 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "oft": { 2024-09-21T03:05:04.064 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_objs": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_updates": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_removes": 0 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "purge_queue": { 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_ops": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_high_water": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executed_ops": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executed": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_item_in_journal": 0 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:04.065 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 104857600, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 674, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 42479518, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 674, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 674, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 42479518, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-objecter_bytes": { 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 104857600, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 616, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 42163967, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 616, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 42163967, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.067 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-objecter_ops": { 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 1024, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 616, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 616, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 616, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 616, 2024-09-21T03:05:04.068 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 3758096384, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:04.069 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-write_buf_throttle-0x555c6b0d5ba0": { 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 3758096384, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:04.070 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 0, 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 0, 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "trackedop": { 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: "slow_ops_count": 0 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:04.071 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-09-21T03:05:05.220 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-09-21T03:05:05.220 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-09-21T03:05:05.220 DEBUG:tasks.check_counter:Getting stats from h 2024-09-21T03:05:05.221 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.h perf dump 2024-09-21T03:05:05.483 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T03:05:05.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:05 smithi190 ceph-mon[33871]: pgmap v1169: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 KiB/s wr, 16 op/s 2024-09-21T03:05:05.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:05 smithi029 ceph-mon[31006]: pgmap v1169: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 KiB/s wr, 16 op/s 2024-09-21T03:05:05.891 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:05 smithi066 ceph-mon[33436]: pgmap v1169: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 KiB/s wr, 16 op/s 2024-09-21T03:05:06.015 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-09-21T03:05:06.015 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:06.015 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 31385, 2024-09-21T03:05:06.015 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 19360, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 9229545, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 7257709, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 10, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 5, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 2.029440176, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.900527935, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 1.425120608, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.055510590, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 19359, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.440107984, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000022734 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.016 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 45662, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.008214496, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000179 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 1033024, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 293040 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 5783, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 7300, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 1794187, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 4476308, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 14, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 6, 2024-09-21T03:05:06.017 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.588449650, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.290019747, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.364641118, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.005672460, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 7299, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.191947605, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000026297 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 12037, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.002316991, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000192 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.018 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 3280, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 796320 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 947, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 764, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 289605, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 358117, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 10, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 6, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.078892550, 2024-09-21T03:05:06.019 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.031018075, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.031302768, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.017430464, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 762, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.050160899, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000065827 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 446, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000090218, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000202 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T03:05:06.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "cct": { 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "total_workers": 1, 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "unhealthy_workers": 0 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "finisher-MDSRank": { 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "queue_len": 0, 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "complete_latency": { 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 2066, 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.594457673, 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000287733 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: "finisher-PurgeQueue": { 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "queue_len": 0, 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "complete_latency": { 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 1151, 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.049748286, 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000043221 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "mds": { 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "request": 4117, 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "reply": 2270, 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "reply_latency": { 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 2270, 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 39.313700665, 2024-09-21T03:05:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.017318810 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "slow_reply": 0, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "forward": 1847, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_fetch_complete": 0, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_fetch_keys": 0, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_commit": 11, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_split": 0, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_merge": 0, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes": 17, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_top": 2, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_bottom": 15, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_pin_tail": 0, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_pinned": 13, 2024-09-21T03:05:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_expired": 50, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_with_caps": 0, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "caps": 0, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "subtrees": 4, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse": 7594, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_hit": 3774, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_forward": 1847, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_discover": 20, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_remote_ino": 0, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_lock": 0, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "load_cent": 0, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "q": 0, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "exported": 1, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "exported_inodes": 0, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "imported": 1, 2024-09-21T03:05:06.024 INFO:teuthology.orchestra.run.smithi066.stdout: "imported_inodes": 3, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_dir_fetch": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_peer_discover": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rfiles": 66, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rbytes": 12312210, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rsnaps": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_set_tag": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_file_inodes": 0, 2024-09-21T03:05:06.025 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_inode_file_caps": 0, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_revoke": 1015, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_grant": 789, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_caps": 1252, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_caps_dirty": 165, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_cap_release": 94, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "process_request_cap_release": 1086 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache": { 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_update": 0, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_update_receipt": 220, 2024-09-21T03:05:06.026 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_try_discover": 116, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_send_discover": 228, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_handle_discover": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays_delayed": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_created": 273, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_enqueued": 273, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_reintegrated": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_migrated": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_processing": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:05:06.027 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_started": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_completed": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_quiesce_path": 25, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_quiesce_inode": 559, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_exportdir": 1, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_flush": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_fragstats": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_inodestats": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_started": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_succeeded": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_write_failed": 0 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log": { 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "evlrg": 0, 2024-09-21T03:05:06.028 INFO:teuthology.orchestra.run.smithi066.stdout: "evadd": 2273, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "evex": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "evtrm": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "ev": 2273, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "evexg": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "evexd": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "segadd": 4, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "segex": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "segtrm": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "seg": 4, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "segmjr": 2, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "segexg": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "segexd": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "expos": 4194304, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "wrpos": 6692004, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "rdpos": 0, 2024-09-21T03:05:06.029 INFO:teuthology.orchestra.run.smithi066.stdout: "jlat": { 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 919, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 3.870387192, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.004211520 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "replayed": 0 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_mem": { 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "ino": 22, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "ino+": 320, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "ino-": 298, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "dir": 21, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "dir+": 148, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "dir-": 127, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "dn": 17, 2024-09-21T03:05:06.030 INFO:teuthology.orchestra.run.smithi066.stdout: "dn+": 463, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "dn-": 446, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "cap": 0, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "cap+": 276, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "cap-": 276, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "rss": 123392, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "heap": 331100160 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_server": { 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "dispatch_client_request": 8019, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "dispatch_server_request": 5, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_request": 4117, 2024-09-21T03:05:06.031 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_session": 108, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_peer_request": 1309, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "req_create_latency": { 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 270, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 33.283464511, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.123272090 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getattr_latency": { 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getfilelock_latency": { 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.032 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "req_link_latency": { 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookup_latency": { 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 1037, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 3.292830299, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.003175342 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookuphash_latency": { 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupino_latency": { 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupname_latency": { 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupparent_latency": { 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupsnap_latency": { 2024-09-21T03:05:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lssnap_latency": { 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mkdir_latency": { 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mknod_latency": { 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.036 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mksnap_latency": { 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "req_open_latency": { 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 144, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.082319069, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000571660 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "req_readdir_latency": { 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 384, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.170127489, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000443040 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.037 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rename_latency": { 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 90, 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.634127520, 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.007045861 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "req_renamesnap_latency": { 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "req_snapdiff_latency": { 2024-09-21T03:05:06.038 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmdir_latency": { 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmsnap_latency": { 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.039 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmxattr_latency": { 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setattr_latency": { 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setdirlayout_latency": { 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.040 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setfilelock_latency": { 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 72, 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.021971883, 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000305165 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setlayout_latency": { 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setxattr_latency": { 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "req_symlink_latency": { 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "req_unlink_latency": { 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 273, 2024-09-21T03:05:06.042 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 1.828859894, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.006699120 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getvxattr_latency": { 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_sessions": { 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "session_count": 1, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "session_add": 1, 2024-09-21T03:05:06.043 INFO:teuthology.orchestra.run.smithi066.stdout: "session_remove": 0, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "sessions_open": 1, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "sessions_stale": 0, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "total_load": 197, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "average_load": 197, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "avg_session_uptime": 1561, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "mdthresh_evicted": 0 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "mempool": { 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "bloom_filter_items": 0, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:06.044 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:06.045 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_items": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:06.046 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_anon_bytes": 287303, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_anon_items": 100, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_bytes": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_items": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:06.047 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_bytes": 8760, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_items": 164, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "pgmap_items": 0, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_co_bytes": 89409, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_co_items": 383, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_1_items": 0, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_2_items": 0 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter": { 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "op_active": 0, 2024-09-21T03:05:06.048 INFO:teuthology.orchestra.run.smithi066.stdout: "op_laggy": 0, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "op_send": 2171, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "op_send_bytes": 2583762, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "op_resend": 33, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "op_reply": 2171, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "op_latency": { 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 2171, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 10.741905449, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.004947906 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "op_inflight": 0, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "oplen_avg": { 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 2171, 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 2437 2024-09-21T03:05:06.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "op": 2171, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "op_r": 155, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "op_w": 2016, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "op_rmw": 0, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "op_pg": 0, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_stat": 0, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_create": 10, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_read": 155, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_write": 1074, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_writefull": 150, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_writesame": 0, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_append": 0, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_zero": 0, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_truncate": 0, 2024-09-21T03:05:06.050 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_delete": 558, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_mapext": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_sparse_read": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_clonerange": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_getxattr": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_setxattr": 20, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_rmxattr": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_call": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_watch": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_notify": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:05:06.051 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_pgls": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_other": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_active": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_send": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_resend": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_ping": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_active": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_send": 0, 2024-09-21T03:05:06.052 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_resend": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_active": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_send": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_resend": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_active": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_send": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_resend": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "command_active": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "command_send": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "command_resend": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "map_epoch": 86, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "map_full": 0, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "map_inc": 5, 2024-09-21T03:05:06.053 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_sessions": 12, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_session_open": 13, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_session_close": 1, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_laggy": 0, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_wr": 327, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_rd": 0, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_del": 143 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "oft": { 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_objs": 1, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_updates": 0, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_removes": 0 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.054 INFO:teuthology.orchestra.run.smithi066.stdout: "purge_queue": { 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_ops": 0, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_ops_high_water": 10, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing": 0, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_high_water": 5, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executed_ops": 546, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executed": 273, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_item_in_journal": 0 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 104857600, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 38111, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 7853668, 2024-09-21T03:05:06.055 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 38111, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 38111, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 7853668, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-objecter_bytes": { 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 104857600, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:06.056 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 2171, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 2613519, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 2171, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 2613519, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-objecter_ops": { 2024-09-21T03:05:06.057 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 1024, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 2171, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 2171, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 2171, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 2171, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.058 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 3758096384, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 273, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 29757, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 273, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 155, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 29757, 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:06.059 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-write_buf_throttle-0x558b117a5520": { 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 3758096384, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 2273, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 2497700, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 2273, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:06.060 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 919, 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 2497700, 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: "trackedop": { 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: "slow_ops_count": 538 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:06.061 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-09-21T03:05:06.759 INFO:tasks.check_counter:Daemon mds.h mds.exported=1 2024-09-21T03:05:06.759 INFO:tasks.check_counter:Daemon mds.h mds.imported=1 2024-09-21T03:05:06.759 DEBUG:tasks.check_counter:Getting stats from k 2024-09-21T03:05:06.759 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.k perf dump 2024-09-21T03:05:07.016 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 497, 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 478, 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 1022818, 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 291108, 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 6, 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 1, 2024-09-21T03:05:07.533 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.078841448, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.038368538, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.050584083, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.000025843, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 477, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.025544036, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000053551 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.534 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 1021984, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 290784 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 1563, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 1918, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 384434, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 1202929, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 5, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 4, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.185800397, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.087090597, 2024-09-21T03:05:07.535 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.031833886, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.050579737, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 1916, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.065276495, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000034069 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 706208 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:07.536 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_messages": 3502, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages": 3502, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_bytes": 3399386, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_bytes": 1112870, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_created_connections": 4, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_active_connections": 3, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_total_time": 0.391483951, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_send_time": 0.178973486, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_recv_time": 0.083969481, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_running_fast_dispatch_time": 0.099546684, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 3501, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.130815126, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000037365 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.537 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_recv_encrypted_bytes": 2960, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "cct": { 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "total_workers": 1, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "unhealthy_workers": 0 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "finisher-MDSRank": { 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "queue_len": 0, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "complete_latency": { 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 8422, 2024-09-21T03:05:07.538 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.238702223, 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000028342 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "finisher-PurgeQueue": { 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "queue_len": 0, 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "complete_latency": { 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "mds": { 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "request": 0, 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "reply": 0, 2024-09-21T03:05:07.539 INFO:teuthology.orchestra.run.smithi066.stdout: "reply_latency": { 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "slow_reply": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "forward": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_fetch_complete": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_fetch_keys": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_commit": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_split": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_merge": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes": 207, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_top": 0, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_bottom": 207, 2024-09-21T03:05:07.540 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_pin_tail": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_pinned": 207, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_expired": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "inodes_with_caps": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "caps": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "subtrees": 1, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_hit": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_forward": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_discover": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_remote_ino": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "traverse_lock": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "load_cent": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "q": 0, 2024-09-21T03:05:07.541 INFO:teuthology.orchestra.run.smithi066.stdout: "exported": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "exported_inodes": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "imported": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "imported_inodes": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_dir_fetch": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "openino_peer_discover": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rfiles": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rbytes": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "root_rsnaps": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_set_tag": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:05:07.542 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "scrub_file_inodes": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_inode_file_caps": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_grant": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_caps": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_caps_dirty": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_cap_release": 0, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "process_request_cap_release": 0 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.543 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache": { 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_update": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_update_receipt": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_try_discover": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_send_discover": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "dir_handle_discover": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays_delayed": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_created": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_enqueued": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_reintegrated": 0, 2024-09-21T03:05:07.544 INFO:teuthology.orchestra.run.smithi066.stdout: "strays_migrated": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_processing": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_started": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_completed": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_quiesce_path": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_quiesce_inode": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_exportdir": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_flush": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_fragstats": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "ireq_inodestats": 0, 2024-09-21T03:05:07.545 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_started": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_succeeded": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "uninline_write_failed": 0 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log": { 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "evlrg": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "evadd": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "evex": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "evtrm": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "ev": 2047, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "evexg": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "evexd": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "segadd": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "segex": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "segtrm": 0, 2024-09-21T03:05:07.546 INFO:teuthology.orchestra.run.smithi066.stdout: "seg": 3, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "segmjr": 2, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "segexg": 0, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "segexd": 0, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "expos": 4194304, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "wrpos": 6552785, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rdpos": 6551647, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "jlat": { 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "replayed": 2047 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_mem": { 2024-09-21T03:05:07.547 INFO:teuthology.orchestra.run.smithi066.stdout: "ino": 5, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "ino+": 222, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "ino-": 217, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "dir": 3, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "dir+": 16, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "dir-": 13, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "dn": 207, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "dn+": 235, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "dn-": 28, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "cap": 0, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "cap+": 0, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "cap-": 0, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rss": 117676, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "heap": 317476864 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.548 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_server": { 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "dispatch_client_request": 0, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "dispatch_server_request": 0, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_request": 0, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_client_session": 0, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "handle_peer_request": 0, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "req_create_latency": { 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getattr_latency": { 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getfilelock_latency": { 2024-09-21T03:05:07.549 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "req_link_latency": { 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookup_latency": { 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.550 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookuphash_latency": { 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupino_latency": { 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupname_latency": { 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupparent_latency": { 2024-09-21T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lookupsnap_latency": { 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "req_lssnap_latency": { 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.552 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mkdir_latency": { 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mknod_latency": { 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "req_mksnap_latency": { 2024-09-21T03:05:07.553 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "req_open_latency": { 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "req_readdir_latency": { 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rename_latency": { 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.554 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "req_renamesnap_latency": { 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "req_snapdiff_latency": { 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmdir_latency": { 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.555 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmsnap_latency": { 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: "req_rmxattr_latency": { 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setattr_latency": { 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setdirlayout_latency": { 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setfilelock_latency": { 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.557 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setlayout_latency": { 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "req_setxattr_latency": { 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "req_symlink_latency": { 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "req_unlink_latency": { 2024-09-21T03:05:07.558 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "req_getvxattr_latency": { 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_sessions": { 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "session_count": 1, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "session_add": 1, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "session_remove": 0, 2024-09-21T03:05:07.559 INFO:teuthology.orchestra.run.smithi066.stdout: "sessions_open": 1, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "sessions_stale": 0, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "total_load": 0, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "average_load": 0, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "avg_session_uptime": 2745, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "mdthresh_evicted": 0 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "mempool": { 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "bloom_filter_items": 0, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:07.560 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:07.561 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_items": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:07.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_anon_bytes": 190968, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_anon_items": 55, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_bytes": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_items": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_bytes": 8760, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_items": 164, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:07.563 INFO:teuthology.orchestra.run.smithi066.stdout: "pgmap_items": 0, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_co_bytes": 156560, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_co_items": 868, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_1_items": 0, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "unittest_2_items": 0 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter": { 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "op_active": 0, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "op_laggy": 0, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "op_send": 5061, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "op_send_bytes": 21, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "op_resend": 3, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "op_reply": 5061, 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "op_latency": { 2024-09-21T03:05:07.564 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 5061, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 4.748957920, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000938343 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "op_inflight": 0, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "oplen_avg": { 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 5061, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 5062 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "op": 5061, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "op_r": 5061, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "op_w": 0, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "op_rmw": 0, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "op_pg": 0, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_stat": 3233, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_create": 0, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_read": 1826, 2024-09-21T03:05:07.565 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_write": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_writefull": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_writesame": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_append": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_zero": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_truncate": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_delete": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_mapext": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_sparse_read": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_clonerange": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_getxattr": 1, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_setxattr": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:05:07.566 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_rmxattr": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_call": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_watch": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_notify": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_pgls": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "osdop_other": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_active": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_send": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_resend": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "linger_ping": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_active": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_send": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "poolop_resend": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_active": 0, 2024-09-21T03:05:07.567 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_send": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "poolstat_resend": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_active": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_send": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "statfs_resend": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "command_active": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "command_send": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "command_resend": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "map_epoch": 86, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "map_full": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "map_inc": 5, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_sessions": 5, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_session_open": 6, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_session_close": 1, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_laggy": 0, 2024-09-21T03:05:07.568 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_wr": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_rd": 2, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_del": 0 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "oft": { 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_objs": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_updates": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "omap_total_removes": 0 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "purge_queue": { 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_ops": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executing_high_water": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executed_ops": 0, 2024-09-21T03:05:07.569 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_executed": 0, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "pq_item_in_journal": 0 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 104857600, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 5558, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 4277994, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 5558, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 5558, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 4277994, 2024-09-21T03:05:07.570 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-objecter_bytes": { 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 104857600, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 5061, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 2358486, 2024-09-21T03:05:07.571 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 5061, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 2358486, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-objecter_ops": { 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 1024, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:07.572 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 5061, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 5061, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 5061, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 5061, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 3758096384, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:07.573 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "throttle-write_buf_throttle-0x564c6ca31520": { 2024-09-21T03:05:07.574 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "max": 3758096384, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "get_started": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "get": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "get_sum": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "take": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "take_sum": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "put": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "put_sum": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "wait": { 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "avgcount": 0, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "sum": 0.000000000, 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: "avgtime": 0.000000000 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-09-21T03:05:07.576 INFO:teuthology.orchestra.run.smithi066.stdout: "trackedop": { 2024-09-21T03:05:07.576 INFO:teuthology.orchestra.run.smithi066.stdout: "slow_ops_count": 0 2024-09-21T03:05:07.576 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-09-21T03:05:07.576 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-09-21T03:05:07.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:07 smithi066 ceph-mon[33436]: pgmap v1170: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T03:05:07.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:07 smithi190 ceph-mon[33871]: pgmap v1170: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T03:05:07.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:07 smithi029 ceph-mon[31006]: pgmap v1170: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T03:05:08.354 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-09-21T03:05:08.354 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-09-21T03:05:08.354 DEBUG:tasks.check_counter:Getting stats from c 2024-09-21T03:05:08.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.c perf dump 2024-09-21T03:05:08.670 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T03:05:09.675 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 6738, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 7742, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 2035172, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 6772382, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 16, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 5, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.566658420, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.273465380, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.310171654, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.030455297, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 7741, 2024-09-21T03:05:09.676 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.180854716, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000023363 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 11930, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.002327041, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000195 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 2960, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 731, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 898, 2024-09-21T03:05:09.677 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 232520, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 1083329, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 12, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 6, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.086763136, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.044970696, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.028807808, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.012507774, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 896, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.043563282, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000048619 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 448, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000085201, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000190 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 796064 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 29556, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 17515, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 8769043, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 4247386, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 8, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 6, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 1.932820363, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.963257659, 2024-09-21T03:05:09.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 1.278237413, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.018634744, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 17514, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.466244957, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000026621 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 43868, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.008509546, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000193 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1044944, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 293680 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.680 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1986, 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.555099596, 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000279506 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1002, 2024-09-21T03:05:09.681 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.039942939, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000039863 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 4128, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 2228, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2228, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 44.417998353, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.019936264 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 1900, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-09-21T03:05:09.682 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 11, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 17, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 1, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 16, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 13, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 50, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 1, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 1, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 4, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 7169, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 3713, 2024-09-21T03:05:09.683 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 1900, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 21, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 1, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 1, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 11, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 66, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 12312210, 2024-09-21T03:05:09.684 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 1023, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 822, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 1347, 2024-09-21T03:05:09.685 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 115, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 88, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 814 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 220, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 116, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 229, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 205, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 205, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-09-21T03:05:09.686 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_path": 25, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_inode": 637, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 1, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_started": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_succeeded": 0, 2024-09-21T03:05:09.687 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_write_failed": 0 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 2047, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 2047, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 3, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 3, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 2, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-09-21T03:05:09.688 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 6552785, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 898, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 2.288052398, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002547942 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 22, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 252, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 230, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 21, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 148, 2024-09-21T03:05:09.689 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 127, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 17, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 361, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 344, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 1, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 209, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 208, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 122060, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 330121216 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 7514, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 5, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 4128, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 18, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 1136, 2024-09-21T03:05:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 194, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 19.622879556, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.101148863 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000573087, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000573087 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.691 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1172, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 3.260769915, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002782226 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-09-21T03:05:09.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.693 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.694 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.694 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-09-21T03:05:09.694 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.694 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.694 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.694 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.695 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 303, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.196228130, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000647617 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 283, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.178449680, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000630564 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 70, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.470529597, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.006721851 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-09-21T03:05:09.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-09-21T03:05:09.697 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.698 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.699 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 205, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 20.688568388, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.100919845 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.700 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 1, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 0, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 193, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 193, 2024-09-21T03:05:09.701 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 1563, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:09.702 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:09.703 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 287613, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 101, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:09.704 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 8760, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 164, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 89729, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 385, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-09-21T03:05:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 1913, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 2437270, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 33, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 1913, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1913, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 6.732825157, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.003519511 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1913, 2024-09-21T03:05:09.706 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 2117 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 1913, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 137, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 1776, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 0, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 10, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 137, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 1035, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 141, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-09-21T03:05:09.707 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 422, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 20, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-09-21T03:05:09.708 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 86, 2024-09-21T03:05:09.709 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 5, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 12, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 15, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 3, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 251, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 0, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 101 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-09-21T03:05:09.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 10, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 5, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 410, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 205, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 37021, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 7674932, 2024-09-21T03:05:09.711 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 37021, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 37021, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 7674932, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:09.712 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 1913, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 2459615, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 1913, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 2459615, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-09-21T03:05:09.713 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 1913, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 1913, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 1913, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 1913, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 205, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 22345, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 205, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 137, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 22345, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.715 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x561f0ba65380": { 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 2047, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 2358481, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 2047, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 898, 2024-09-21T03:05:09.716 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 2358481, 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: "trackedop": { 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 607 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:09.717 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-21T03:05:09.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:09 smithi190 ceph-mon[33871]: pgmap v1171: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T03:05:09.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:09 smithi029 ceph-mon[31006]: pgmap v1171: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T03:05:10.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:09 smithi066 ceph-mon[33436]: pgmap v1171: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T03:05:10.858 INFO:tasks.check_counter:Daemon mds.c mds.exported=1 2024-09-21T03:05:10.858 INFO:tasks.check_counter:Daemon mds.c mds.imported=1 2024-09-21T03:05:10.858 DEBUG:tasks.check_counter:Getting stats from f 2024-09-21T03:05:10.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.f perf dump 2024-09-21T03:05:11.119 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T03:05:11.686 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 258533, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 296200, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 78275012, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 207211042, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 14, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 5, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 18.689318277, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 9.615801636, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 10.417223683, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.476301382, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 296198, 2024-09-21T03:05:11.687 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 6.816459572, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000023013 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 517967, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.084729348, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000163 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1584, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 1440 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 15633, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 15796, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 4854113, 2024-09-21T03:05:11.688 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 46229111, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 12, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 7, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 1.439878470, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.649563990, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.314561327, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.417841697, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 15796, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.641668722, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000040622 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 443, 2024-09-21T03:05:11.689 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000091961, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000207 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1042960, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 295104 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 80749, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 78296, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 20859069, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 36856192, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 9, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 6, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 5.896484259, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 3.018203115, 2024-09-21T03:05:11.690 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 3.189783650, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.301862706, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 78294, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1.756768804, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000022438 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 139362, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.022076113, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000158 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T03:05:11.691 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 797872 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 34194, 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 20.534265435, 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000600522 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.692 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 18482, 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1.226870039, 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000066381 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 219217, 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 218738, 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 218738, 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 397.443748192, 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001816985 2024-09-21T03:05:11.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 479, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 11, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 15, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 0, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 1, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 14, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 14, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 52, 2024-09-21T03:05:11.694 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 1, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 1, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 3, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 434065, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 398626, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 477, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 18, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 135, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 4, 2024-09-21T03:05:11.695 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 21, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 4, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 110, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 66, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 12312210, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:05:11.696 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 15606, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 23758, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 24059, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 6412, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 744, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 26821 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 3, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 215, 2024-09-21T03:05:11.697 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 112, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 221, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 10, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 8057, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 8057, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:05:11.698 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_path": 25, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_inode": 1824, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 4, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_started": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_succeeded": 0, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_write_failed": 0 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.699 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 52430, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 52430, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 67, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 67, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 6, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-09-21T03:05:11.700 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 87083021, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 16826, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 63.032563349, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.003746140 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 19, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 8102, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 8083, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 18, 2024-09-21T03:05:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 153, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 135, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 15, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 12294, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 12279, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 1, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 8067, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 8066, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 135948, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 339623936 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 257120, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 16, 2024-09-21T03:05:11.702 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 219217, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 555, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 9559, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 7968, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 170.226267826, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.021363738 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 179498, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 48.154313733, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000268272 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.703 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 8720, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 15.675161177, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001797610 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.704 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.706 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-09-21T03:05:11.706 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.706 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.706 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.706 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-09-21T03:05:11.706 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-09-21T03:05:11.707 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3743, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 2.716994501, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000725886 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.708 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 9703, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 30.458957926, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.003139127 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 912, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1.976829832, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002167576 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.709 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 10, 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.045035354, 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.004503535 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.710 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 17.833685938, 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 17.833685938 2024-09-21T03:05:11.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 136, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.047388614, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000348445 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.712 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 8047, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 110.309113291, 2024-09-21T03:05:11.713 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.013708104 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 2, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 1, 2024-09-21T03:05:11.714 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 18272, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 18272, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 1566, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:11.715 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:11.716 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:11.717 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 312261, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 109, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 8760, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 164, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:11.718 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 79041, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 427, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 37368, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 84248554, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 33, 2024-09-21T03:05:11.719 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 37368, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 37368, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 163.438738110, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.004373761 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 37368, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 38927 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 37368, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 1331, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 36037, 2024-09-21T03:05:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 10, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 1331, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 18157, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 816, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 16135, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 0, 2024-09-21T03:05:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 20, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-09-21T03:05:11.722 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 86, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 5, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 12, 2024-09-21T03:05:11.723 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 12, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 1666, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 0, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 792 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 10, 2024-09-21T03:05:11.724 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 9, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 16104, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 8057, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 354911, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 72016074, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 354911, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:11.725 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 354911, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 72016074, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:11.726 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 37368, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 85126687, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 37368, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 85126687, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-09-21T03:05:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 37368, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 37368, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 37368, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 37368, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:05:11.728 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 8057, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 878133, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 8057, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 1331, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 878133, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.729 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x563ec89352b0": { 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 52430, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 82888717, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 52430, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 16826, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 82888717, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:11.730 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:11.731 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:11.731 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:11.731 INFO:teuthology.orchestra.run.smithi190.stdout: "trackedop": { 2024-09-21T03:05:11.731 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 1696 2024-09-21T03:05:11.731 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:11.731 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-21T03:05:11.731 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:11 smithi190 ceph-mon[33871]: pgmap v1172: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-21T03:05:11.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:11 smithi029 ceph-mon[31006]: pgmap v1172: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-21T03:05:12.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:11 smithi066 ceph-mon[33436]: pgmap v1172: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-21T03:05:12.804 INFO:tasks.check_counter:Daemon mds.f mds.exported=4 2024-09-21T03:05:12.804 INFO:tasks.check_counter:Daemon mds.f mds.imported=4 2024-09-21T03:05:12.804 DEBUG:tasks.check_counter:Getting stats from i 2024-09-21T03:05:12.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.i perf dump 2024-09-21T03:05:13.077 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T03:05:13.604 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:13 smithi190 ceph-mon[33871]: pgmap v1173: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 1585209, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 1734805, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 470102467, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 1137811904, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 31, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 7, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 89.955104509, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 47.872312587, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 49.820207223, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 1.918787053, 2024-09-21T03:05:13.657 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1734798, 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 52.184038355, 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000030080 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3148779, 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.478180891, 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000151 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1118784, 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 328608 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.658 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 66950, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 79320, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 19495804, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 171022702, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 11, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 6, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 8.807046790, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 4.725698912, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 1.935714343, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 2.020659677, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 79322, 2024-09-21T03:05:13.659 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 32.039921903, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000403922 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 43625, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.011123363, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000254 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 3120, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 728128 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 128712, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 144254, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 31680453, 2024-09-21T03:05:13.660 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 166801713, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 330, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 8, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 11.591855045, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 5.525863143, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 4.684676276, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 1.910616079, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 144255, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 29.981673394, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000207838 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:13.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 187247, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.030691785, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000163 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 30896, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 1856 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 98508, 2024-09-21T03:05:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 113.284038348, 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001149998 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 46413, 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 9.509495244, 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000204888 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 1415352, 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 1408215, 2024-09-21T03:05:13.663 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1408215, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 3770.222513335, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002677306 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 7137, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 5208, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 37, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 9, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 18, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 10, 2024-09-21T03:05:13.664 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 24, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 219, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 2, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 2, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 4, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 2732865, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 2718425, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 2455, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 1, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 11, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 9, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 185, 2024-09-21T03:05:13.665 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 8, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 22, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 66, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 12312210, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:05:13.666 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 10, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 71539, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 96707, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 112656, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 33880, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 991, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 88119 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-09-21T03:05:13.667 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 32, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 1, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 392, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 9, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 23620, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 23611, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-09-21T03:05:13.668 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_path": 25, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_inode": 8641, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 9, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0, 2024-09-21T03:05:13.669 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_started": 0, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_succeeded": 0, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_write_failed": 0 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 160965, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 159486, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 159486, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 1479, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 77, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 273, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 271, 2024-09-21T03:05:13.670 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 271, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 2, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 1, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 1, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 337343608, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 340229951, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 51844, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 333.007070460, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.006423251 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.671 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 30, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 23815, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 23785, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 29, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 135, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 106, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 37, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 39071, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 39034, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 2, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 23810, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 23808, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 158760, 2024-09-21T03:05:13.672 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 378417152 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 1461301, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 12590, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 1415352, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 2540, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 26252, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 23675, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 988.308890235, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.041744831 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1291582, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1855.590442932, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001436680 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.674 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 13863, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 112.865012645, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.008141456 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-09-21T03:05:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.676 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 119, 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.322076803, 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002706527 2024-09-21T03:05:13.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 16025, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 31.628452381, 2024-09-21T03:05:13.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001973694 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 32810, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 189.479061172, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.005775039 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 5684, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 176.947676822, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.031130836 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 105, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.786760040, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.007492952 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 11, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 169.066249203, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 15.369659018 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 804, 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1.207360337, 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001501691 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 11, 2024-09-21T03:05:13.682 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.028700369, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002609124 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 23526, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 243.991830396, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.010371156 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:05:13.683 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 4, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 3, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-09-21T03:05:13.684 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 84838, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 84838, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 1569, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 360, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 360, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:13.685 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:13.686 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 288202, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 108, 2024-09-21T03:05:13.687 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 8760, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 164, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 138955, 2024-09-21T03:05:13.688 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 1749, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 125923, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 361367895, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 37, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 125923, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 125923, 2024-09-21T03:05:13.689 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1898.015575571, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.015072826 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 125923, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 162322 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 125923, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 1885, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 124038, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 5101, 2024-09-21T03:05:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 14654, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 1885, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 53729, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 1354, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 47808, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 22215, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:05:13.691 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-09-21T03:05:13.692 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 86, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 11, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 12, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 12, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-09-21T03:05:13.693 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 12306, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 0, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 3270 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 10, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-09-21T03:05:13.694 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 9, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 47137, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 23611, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 1780867, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 360960465, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 1780867, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:13.695 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 1780867, 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 360960465, 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:13.696 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 125923, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 363940814, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 125923, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 363940814, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:13.697 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 125923, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 125923, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 125923, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 125923, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 23611, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 2572919, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 23611, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 1885, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 2572919, 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:13.699 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x557fdb9df5f0": { 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 160965, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 336035647, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 160965, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 51844, 2024-09-21T03:05:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 336035647, 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: "trackedop": { 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 8020 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:13.701 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-21T03:05:13.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:13 smithi066 ceph-mon[33436]: pgmap v1173: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-21T03:05:13.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:13 smithi029 ceph-mon[31006]: pgmap v1173: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-21T03:05:14.448 INFO:tasks.check_counter:Daemon mds.i mds.exported=9 2024-09-21T03:05:14.448 INFO:tasks.check_counter:Daemon mds.i mds.imported=8 2024-09-21T03:05:14.448 DEBUG:tasks.check_counter:Getting stats from l 2024-09-21T03:05:14.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph daemon mds.l perf dump 2024-09-21T03:05:14.698 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T03:05:15.289 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-21T03:05:15.289 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T03:05:15.289 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 4519, 2024-09-21T03:05:15.289 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 4521, 2024-09-21T03:05:15.289 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 26451161, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 1437049, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 7, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 3, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.421680954, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.175331296, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.101718110, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.118797030, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4519, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.144965299, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000032079 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.290 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 1262, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 1241, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 14189246, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 534254, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 5, 2024-09-21T03:05:15.291 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 4, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.149498611, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.063328616, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.073282174, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.019175790, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1241, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.051666109, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000041632 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1025488, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 292512 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 258, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 615, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 4259206, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 880000, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 7, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 4, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.051000321, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.028698422, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.010570867, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.007091611, 2024-09-21T03:05:15.293 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 613, 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.024848213, 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000040535 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 798592 2024-09-21T03:05:15.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 9407, 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.226884879, 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000024118 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 0, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 0, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.296 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 3319, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 3319, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 3319, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2024-09-21T03:05:15.297 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 1, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-09-21T03:05:15.298 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T03:05:15.299 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 0 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-09-21T03:05:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-09-21T03:05:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_path": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_inode": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_started": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_succeeded": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "uninline_write_failed": 0 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-09-21T03:05:15.302 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 21435, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 32, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 3, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-09-21T03:05:15.303 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 46526398, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 46525260, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 21435 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 13, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 3337, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 3324, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 11, 2024-09-21T03:05:15.304 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 36, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 25, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 3319, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 3464, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 145, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 0, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 0, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 0, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 121688, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 321732608 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-09-21T03:05:15.305 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 0, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 0, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 0, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 0, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 0, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.306 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.307 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.308 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-09-21T03:05:15.309 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.310 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.310 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.310 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-09-21T03:05:15.310 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.310 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-09-21T03:05:15.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.312 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-09-21T03:05:15.313 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.314 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.315 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 3, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 2, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 2, 2024-09-21T03:05:15.317 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 0, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 0, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 3989, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T03:05:15.318 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T03:05:15.319 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-09-21T03:05:15.320 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 217567, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 72, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-09-21T03:05:15.321 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 8760, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 164, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 2301072, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 13404, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-09-21T03:05:15.322 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 5535, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 21, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 3, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 5535, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 5535, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 3.781900886, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000683270 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 5535, 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 5536 2024-09-21T03:05:15.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 5535, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 5535, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 3271, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 2262, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-09-21T03:05:15.324 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 1, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-09-21T03:05:15.325 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-09-21T03:05:15.326 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 86, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 5, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 8, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 9, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 1, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 0, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 2, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 0 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 0, 2024-09-21T03:05:15.327 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-09-21T03:05:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 6035, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 44327892, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 6035, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 6035, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 44327892, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 5535, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 42332099, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 5535, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 42332099, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 5535, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 5535, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 5535, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 5535, 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x55b252a75450": { 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-09-21T03:05:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "trackedop": { 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 0 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-21T03:05:15.334 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-21T03:05:15.846 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:15 smithi190 ceph-mon[33871]: pgmap v1174: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-21T03:05:15.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:15 smithi029 ceph-mon[31006]: pgmap v1174: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-21T03:05:16.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:15 smithi066 ceph-mon[33436]: pgmap v1174: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-21T03:05:16.242 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-09-21T03:05:16.242 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-09-21T03:05:16.242 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-21T03:05:16.252 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-21T03:05:16.252 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T03:05:16.252 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T03:05:16.281 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T03:05:16.281 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T03:05:16.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T03:05:16.654 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 -- 172.21.15.29:0/174547732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998105290 msgr2=0x7f099810d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 --2- 172.21.15.29:0/174547732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998105290 0x7f099810d8d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f0984009900 tx=0x7f098402a410 comp rx=0 tx=0).stop 2024-09-21T03:05:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 -- 172.21.15.29:0/174547732 shutdown_connections 2024-09-21T03:05:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 --2- 172.21.15.29:0/174547732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f099810de10 0x7f0998110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 --2- 172.21.15.29:0/174547732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998105290 0x7f099810d8d0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 --2- 172.21.15.29:0/174547732 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0998104860 0x7f0998104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 -- 172.21.15.29:0/174547732 >> 172.21.15.29:0/174547732 conn(0x7f0998079740 msgr2=0x7f0998079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 -- 172.21.15.29:0/174547732 shutdown_connections 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 -- 172.21.15.29:0/174547732 wait complete. 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.172+0000 7f099f1b2640 1 Processor -- start 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099f1b2640 1 -- start start 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099f1b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998104860 0x7f099819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099f1b2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0998105290 0x7f099819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099f1b2640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099810de10 0x7f099819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099f1b2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0998113940 con 0x7f0998104860 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099f1b2640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f09981137c0 con 0x7f099810de10 2024-09-21T03:05:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099f1b2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0998113ac0 con 0x7f0998105290 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099cf4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998104860 0x7f099819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099cf4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998104860 0x7f099819a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57810/0 (socket says 172.21.15.29:57810) 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099cf4e640 1 -- 172.21.15.29:0/947968969 learned_addr learned my addr 172.21.15.29:0/947968969 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099d74f640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099810de10 0x7f099819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f0997fff640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0998105290 0x7f099819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099cf4e640 1 -- 172.21.15.29:0/947968969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0998105290 msgr2=0x7f099819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099cf4e640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0998105290 0x7f099819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099cf4e640 1 -- 172.21.15.29:0/947968969 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099810de10 msgr2=0x7f099819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099cf4e640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099810de10 0x7f099819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f099cf4e640 1 -- 172.21.15.29:0/947968969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09981a0380 con 0x7f0998104860 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.173+0000 7f0997fff640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0998105290 0x7f099819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:05:17.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.174+0000 7f099cf4e640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998104860 0x7f099819a410 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f098c00d770 tx=0x7f098c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.174+0000 7f0995ffb640 1 -- 172.21.15.29:0/947968969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c017070 con 0x7f0998104860 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.174+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09981a0670 con 0x7f0998104860 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.174+0000 7f0995ffb640 1 -- 172.21.15.29:0/947968969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f098c009c30 con 0x7f0998104860 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.174+0000 7f0995ffb640 1 -- 172.21.15.29:0/947968969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c003c50 con 0x7f0998104860 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.174+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09981a6dd0 con 0x7f0998104860 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.175+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f099806c1e0 con 0x7f0998104860 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.176+0000 7f0995ffb640 1 -- 172.21.15.29:0/947968969 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f098c003df0 con 0x7f0998104860 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.177+0000 7f0995ffb640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09880b6510 0x7f09880b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.177+0000 7f0997fff640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09880b6510 0x7f09880b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.177+0000 7f0995ffb640 1 -- 172.21.15.29:0/947968969 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f098c0e75a0 con 0x7f0998104860 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.178+0000 7f0997fff640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09880b6510 0x7f09880b8a20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0984006010 tx=0x7f0984000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.180+0000 7f0995ffb640 1 -- 172.21.15.29:0/947968969 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f098c009de0 con 0x7f0998104860 2024-09-21T03:05:17.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.381+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f099819b540 con 0x7f0998104860 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.381+0000 7f0995ffb640 1 -- 172.21.15.29:0/947968969 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) ==== 81+0+589 (secure 0 0 0) 0x7f098c0b6020 con 0x7f0998104860 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3488265372 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3273694575 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1830887273 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/313500774 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/2882183575 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/2882183575 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3241279793 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:17.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/595746591 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:17.384 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1927307007 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:17.384 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/595746591 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.383+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09880b6510 msgr2=0x7f09880b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.383+0000 7f099f1b2640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09880b6510 0x7f09880b8a20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0984006010 tx=0x7f0984000bc0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.383+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998104860 msgr2=0x7f099819a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.383+0000 7f099f1b2640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998104860 0x7f099819a410 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f098c00d770 tx=0x7f098c00dc30 comp rx=0 tx=0).stop 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.384+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 shutdown_connections 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.384+0000 7f099f1b2640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f09880b6510 0x7f09880b8a20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.384+0000 7f099f1b2640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099810de10 0x7f099819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.384+0000 7f099f1b2640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0998105290 0x7f099819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.384+0000 7f099f1b2640 1 --2- 172.21.15.29:0/947968969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0998104860 0x7f099819a410 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.384+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 >> 172.21.15.29:0/947968969 conn(0x7f0998079740 msgr2=0x7f09980ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.384+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 shutdown_connections 2024-09-21T03:05:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:17.384+0000 7f099f1b2640 1 -- 172.21.15.29:0/947968969 wait complete. 2024-09-21T03:05:17.793 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:17 smithi029 ceph-mon[31006]: pgmap v1175: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-21T03:05:17.793 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:17 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/947968969' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T03:05:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:17 smithi190 ceph-mon[33871]: pgmap v1175: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-21T03:05:17.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:17 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/947968969' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T03:05:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:17 smithi066 ceph-mon[33436]: pgmap v1175: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-21T03:05:18.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:17 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/947968969' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T03:05:18.124 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T03:05:18.124 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T03:05:18.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T03:05:18.444 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:19.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.992+0000 7f15a39e5640 1 -- 172.21.15.29:0/3129201902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c1083d0 msgr2=0x7f159c1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:19.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.992+0000 7f15a39e5640 1 --2- 172.21.15.29:0/3129201902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c1083d0 0x7f159c1087e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f1590009900 tx=0x7f159002a410 comp rx=0 tx=0).stop 2024-09-21T03:05:19.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.992+0000 7f15a39e5640 1 -- 172.21.15.29:0/3129201902 shutdown_connections 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.992+0000 7f15a39e5640 1 --2- 172.21.15.29:0/3129201902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f159c108d20 0x7f159c10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.992+0000 7f15a39e5640 1 --2- 172.21.15.29:0/3129201902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c1083d0 0x7f159c1087e0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.992+0000 7f15a39e5640 1 --2- 172.21.15.29:0/3129201902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f159c102ae0 0x7f159c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.992+0000 7f15a39e5640 1 -- 172.21.15.29:0/3129201902 >> 172.21.15.29:0/3129201902 conn(0x7f159c079760 msgr2=0x7f159c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.993+0000 7f15a39e5640 1 -- 172.21.15.29:0/3129201902 shutdown_connections 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.993+0000 7f15a39e5640 1 -- 172.21.15.29:0/3129201902 wait complete. 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.993+0000 7f15a39e5640 1 Processor -- start 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.993+0000 7f15a39e5640 1 -- start start 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a39e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c102ae0 0x7f159c0ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a39e5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f159c1083d0 0x7f159c0ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a39e5640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f159c108d20 0x7f159c1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a39e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f159c112840 con 0x7f159c102ae0 2024-09-21T03:05:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a39e5640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1126c0 con 0x7f159c108d20 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a39e5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1129c0 con 0x7f159c1083d0 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c102ae0 0x7f159c0ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c102ae0 0x7f159c0ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57830/0 (socket says 172.21.15.29:57830) 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 -- 172.21.15.29:0/2522657983 learned_addr learned my addr 172.21.15.29:0/2522657983 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a0f80640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f159c1083d0 0x7f159c0ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1f82640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f159c108d20 0x7f159c1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 -- 172.21.15.29:0/2522657983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f159c1083d0 msgr2=0x7f159c0ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f159c1083d0 0x7f159c0ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 -- 172.21.15.29:0/2522657983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f159c108d20 msgr2=0x7f159c1000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f159c108d20 0x7f159c1000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 -- 172.21.15.29:0/2522657983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c1a8640 con 0x7f159c102ae0 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1f82640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f159c108d20 0x7f159c1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.994+0000 7f15a1781640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c102ae0 0x7f159c0ff670 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f158c00d950 tx=0x7f158c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.995+0000 7f159a7fc640 1 -- 172.21.15.29:0/2522657983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c017070 con 0x7f159c102ae0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.995+0000 7f159a7fc640 1 -- 172.21.15.29:0/2522657983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f158c009c30 con 0x7f159c102ae0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.995+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159c1a8930 con 0x7f159c102ae0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.995+0000 7f159a7fc640 1 -- 172.21.15.29:0/2522657983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c00f620 con 0x7f159c102ae0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.995+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f159c1a8e90 con 0x7f159c102ae0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.996+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f159c103d10 con 0x7f159c102ae0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.997+0000 7f159a7fc640 1 -- 172.21.15.29:0/2522657983 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f158c010460 con 0x7f159c102ae0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.998+0000 7f159a7fc640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15840b6590 0x7f15840b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.999+0000 7f15a0f80640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15840b6590 0x7f15840b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.999+0000 7f159a7fc640 1 -- 172.21.15.29:0/2522657983 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f158c0e7260 con 0x7f159c102ae0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:18.999+0000 7f15a0f80640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15840b6590 0x7f15840b8aa0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1590004770 tx=0x7f15900058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.003+0000 7f159a7fc640 1 -- 172.21.15.29:0/2522657983 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f158c0ad710 con 0x7f159c102ae0 2024-09-21T03:05:19.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.204+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f159c02e640 con 0x7f159c102ae0 2024-09-21T03:05:19.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.205+0000 7f159a7fc640 1 -- 172.21.15.29:0/2522657983 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) ==== 81+0+589 (secure 0 0 0) 0x7f158c009de0 con 0x7f159c102ae0 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3488265372 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3273694575 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1830887273 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/313500774 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/2882183575 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/2882183575 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3241279793 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/595746591 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1927307007 2024-09-22T02:28:37.826591+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/595746591 2024-09-22T02:28:10.590358+0000 2024-09-21T03:05:19.207 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2024-09-21T03:05:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.207+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15840b6590 msgr2=0x7f15840b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15840b6590 0x7f15840b8aa0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1590004770 tx=0x7f15900058d0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c102ae0 msgr2=0x7f159c0ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c102ae0 0x7f159c0ff670 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f158c00d950 tx=0x7f158c00de10 comp rx=0 tx=0).stop 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 shutdown_connections 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f15840b6590 0x7f15840b8aa0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f159c108d20 0x7f159c1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f159c1083d0 0x7f159c0ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 --2- 172.21.15.29:0/2522657983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f159c102ae0 0x7f159c0ff670 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 >> 172.21.15.29:0/2522657983 conn(0x7f159c079760 msgr2=0x7f159c101b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 shutdown_connections 2024-09-21T03:05:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:19.208+0000 7f15a39e5640 1 -- 172.21.15.29:0/2522657983 wait complete. 2024-09-21T03:05:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:19 smithi190 ceph-mon[33871]: pgmap v1176: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T03:05:19.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:19 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2522657983' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T03:05:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:19 smithi029 ceph-mon[31006]: pgmap v1176: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T03:05:19.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:19 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2522657983' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T03:05:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:19 smithi066 ceph-mon[33436]: pgmap v1176: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T03:05:20.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:19 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2522657983' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T03:05:20.060 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi029.front.sepia.ceph.com... 2024-09-21T03:05:20.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T03:05:20.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-21T03:05:20.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi029.stderr:ceph-fuse[85391]: fuse finished with error 0 and tester_r 0 2024-09-21T03:05:20.118 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:05:21.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:21 smithi190 ceph-mon[33871]: pgmap v1177: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-21T03:05:21.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:21 smithi029 ceph-mon[31006]: pgmap v1177: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-21T03:05:22.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:21 smithi066 ceph-mon[33436]: pgmap v1177: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-21T03:05:23.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:23 smithi190 ceph-mon[33871]: pgmap v1178: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2024-09-21T03:05:23.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:23 smithi029 ceph-mon[31006]: pgmap v1178: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2024-09-21T03:05:24.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:23 smithi066 ceph-mon[33436]: pgmap v1178: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2024-09-21T03:05:25.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:25 smithi190 ceph-mon[33871]: pgmap v1179: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-21T03:05:25.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:25 smithi029 ceph-mon[31006]: pgmap v1179: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-21T03:05:26.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:25 smithi066 ceph-mon[33436]: pgmap v1179: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-21T03:05:26.220 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T03:05:26.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T03:05:26.220 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-21T03:05:26.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T03:05:26.265 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-09-21T03:05:26.284 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-21T03:05:26.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T03:05:26.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T03:05:26.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T03:05:26.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-09-21T03:05:26.455 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T03:05:26.464 INFO:tasks.cephadm:Teardown begin 2024-09-21T03:05:26.465 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-21T03:05:26.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:05:26.729 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:27.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 -- 172.21.15.29:0/3319094155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 msgr2=0x7f2fa4110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 --2- 172.21.15.29:0/3319094155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 0x7f2fa4110620 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009ca0 tx=0x7f2f9c02a8c0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 -- 172.21.15.29:0/3319094155 shutdown_connections 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 --2- 172.21.15.29:0/3319094155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 0x7f2fa4110620 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 --2- 172.21.15.29:0/3319094155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fa41094a0 0x7f2fa41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 --2- 172.21.15.29:0/3319094155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fa4103b60 0x7f2fa4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 -- 172.21.15.29:0/3319094155 >> 172.21.15.29:0/3319094155 conn(0x7f2fa407b850 msgr2=0x7f2fa407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 -- 172.21.15.29:0/3319094155 shutdown_connections 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.289+0000 7f2fac576640 1 -- 172.21.15.29:0/3319094155 wait complete. 2024-09-21T03:05:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fac576640 1 Processor -- start 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fac576640 1 -- start start 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fac576640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fa4103b60 0x7f2fa419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fac576640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fa41094a0 0x7f2fa419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fac576640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 0x7f2fa419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fac576640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa41139b0 con 0x7f2fa4109df0 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fac576640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4113830 con 0x7f2fa41094a0 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fac576640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4113b30 con 0x7f2fa4103b60 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faab13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 0x7f2fa419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faab13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 0x7f2fa419fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59392/0 (socket says 172.21.15.29:59392) 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faab13640 1 -- 172.21.15.29:0/2303751767 learned_addr learned my addr 172.21.15.29:0/2303751767 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faa312640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fa4103b60 0x7f2fa419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2fa9b11640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fa41094a0 0x7f2fa419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faab13640 1 -- 172.21.15.29:0/2303751767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fa4103b60 msgr2=0x7f2fa419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faab13640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fa4103b60 0x7f2fa419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faab13640 1 -- 172.21.15.29:0/2303751767 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fa41094a0 msgr2=0x7f2fa419a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faab13640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fa41094a0 0x7f2fa419a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faab13640 1 -- 172.21.15.29:0/2303751767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa41a0370 con 0x7f2fa4109df0 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.290+0000 7f2faa312640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fa4103b60 0x7f2fa419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.291+0000 7f2faab13640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 0x7f2fa419fc40 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c005cc0 tx=0x7f2f9c032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.291+0000 7f2f9b7fe640 1 -- 172.21.15.29:0/2303751767 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c032e00 con 0x7f2fa4109df0 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.291+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa41a0600 con 0x7f2fa4109df0 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.291+0000 7f2f9b7fe640 1 -- 172.21.15.29:0/2303751767 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f9c0044d0 con 0x7f2fa4109df0 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.291+0000 7f2f9b7fe640 1 -- 172.21.15.29:0/2303751767 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c030b70 con 0x7f2fa4109df0 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.291+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa41a6dc0 con 0x7f2fa4109df0 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.292+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa4103f70 con 0x7f2fa4109df0 2024-09-21T03:05:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.294+0000 7f2f9b7fe640 1 -- 172.21.15.29:0/2303751767 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f2f9c030d10 con 0x7f2fa4109df0 2024-09-21T03:05:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.295+0000 7f2f9b7fe640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2f900b6590 0x7f2f900b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.295+0000 7f2faa312640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2f900b6590 0x7f2f900b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.295+0000 7f2f9b7fe640 1 -- 172.21.15.29:0/2303751767 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f2f9c108a20 con 0x7f2fa4109df0 2024-09-21T03:05:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.296+0000 7f2faa312640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2f900b6590 0x7f2f900b8aa0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0005fd0 tx=0x7f2fa000b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.298+0000 7f2f9b7fe640 1 -- 172.21.15.29:0/2303751767 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2f9c0cef30 con 0x7f2fa4109df0 2024-09-21T03:05:27.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.554+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fa419b530 con 0x7f2fa4109df0 2024-09-21T03:05:27.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.555+0000 7f2f9b7fe640 1 -- 172.21.15.29:0/2303751767 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11976 (secure 0 0 0) 0x7f2fa419b530 con 0x7f2fa4109df0 2024-09-21T03:05:27.557 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:05:27.557 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"btime":"2024-09-21T03:04:49:618327+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:04:48.728569+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T03:05:27.557 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2024-09-21T03:05:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2f900b6590 msgr2=0x7f2f900b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2f900b6590 0x7f2f900b8aa0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0005fd0 tx=0x7f2fa000b5f0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 msgr2=0x7f2fa419fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 0x7f2fa419fc40 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c005cc0 tx=0x7f2f9c032a10 comp rx=0 tx=0).stop 2024-09-21T03:05:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 shutdown_connections 2024-09-21T03:05:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f2f900b6590 0x7f2f900b8aa0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa4109df0 0x7f2fa419fc40 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fa41094a0 0x7f2fa419a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 --2- 172.21.15.29:0/2303751767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fa4103b60 0x7f2fa419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.557+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 >> 172.21.15.29:0/2303751767 conn(0x7f2fa407b850 msgr2=0x7f2fa4107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.558+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 shutdown_connections 2024-09-21T03:05:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:27.558+0000 7f2fac576640 1 -- 172.21.15.29:0/2303751767 wait complete. 2024-09-21T03:05:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:27 smithi190 ceph-mon[33871]: pgmap v1180: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T03:05:27.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:27 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2303751767' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:05:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:27 smithi029 ceph-mon[31006]: pgmap v1180: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T03:05:27.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:27 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2303751767' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:05:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:27 smithi066 ceph-mon[33436]: pgmap v1180: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-21T03:05:28.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:27 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2303751767' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:05:28.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T03:05:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:29.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.289+0000 7f73d554d640 1 -- 172.21.15.29:0/4019080237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d01083a0 msgr2=0x7f73d01087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:29.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.289+0000 7f73d554d640 1 --2- 172.21.15.29:0/4019080237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d01083a0 0x7f73d01087b0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f73b8009900 tx=0x7f73b802a410 comp rx=0 tx=0).stop 2024-09-21T03:05:29.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.290+0000 7f73d554d640 1 -- 172.21.15.29:0/4019080237 shutdown_connections 2024-09-21T03:05:29.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.290+0000 7f73d554d640 1 --2- 172.21.15.29:0/4019080237 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73d0108cf0 0x7f73d010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.290+0000 7f73d554d640 1 --2- 172.21.15.29:0/4019080237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d01083a0 0x7f73d01087b0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.290+0000 7f73d554d640 1 --2- 172.21.15.29:0/4019080237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73d0102a60 0x7f73d0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.290+0000 7f73d554d640 1 -- 172.21.15.29:0/4019080237 >> 172.21.15.29:0/4019080237 conn(0x7f73d0079730 msgr2=0x7f73d0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:29.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.290+0000 7f73d554d640 1 -- 172.21.15.29:0/4019080237 shutdown_connections 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.290+0000 7f73d554d640 1 -- 172.21.15.29:0/4019080237 wait complete. 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.290+0000 7f73d554d640 1 Processor -- start 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73d554d640 1 -- start start 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73d554d640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73d0102a60 0x7f73d019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73d554d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73d01083a0 0x7f73d019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73d554d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d0108cf0 0x7f73d019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73d554d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73d0113930 con 0x7f73d0108cf0 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73d554d640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f73d01137b0 con 0x7f73d0102a60 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73d554d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f73d0113ab0 con 0x7f73d01083a0 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73cf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d0108cf0 0x7f73d019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73cf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d0108cf0 0x7f73d019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59408/0 (socket says 172.21.15.29:59408) 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73cf7fe640 1 -- 172.21.15.29:0/160987576 learned_addr learned my addr 172.21.15.29:0/160987576 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73ceffd640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73d0102a60 0x7f73d019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.291+0000 7f73ce7fc640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73d01083a0 0x7f73d019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73cf7fe640 1 -- 172.21.15.29:0/160987576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73d01083a0 msgr2=0x7f73d019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73cf7fe640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73d01083a0 0x7f73d019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73cf7fe640 1 -- 172.21.15.29:0/160987576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73d0102a60 msgr2=0x7f73d019a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73cf7fe640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73d0102a60 0x7f73d019a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73cf7fe640 1 -- 172.21.15.29:0/160987576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d01a0210 con 0x7f73d0108cf0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73cf7fe640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d0108cf0 0x7f73d019fba0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f73c000eaa0 tx=0x7f73c000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73c7fff640 1 -- 172.21.15.29:0/160987576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0004d20 con 0x7f73d0108cf0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73c7fff640 1 -- 172.21.15.29:0/160987576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73c0005020 con 0x7f73d0108cf0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d01a0500 con 0x7f73d0108cf0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73c7fff640 1 -- 172.21.15.29:0/160987576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c00116d0 con 0x7f73d0108cf0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.292+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d01a6d20 con 0x7f73d0108cf0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.293+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d0102e70 con 0x7f73d0108cf0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.295+0000 7f73c7fff640 1 -- 172.21.15.29:0/160987576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f73c00051d0 con 0x7f73d0108cf0 2024-09-21T03:05:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.296+0000 7f73c7fff640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73bc0b65c0 0x7f73bc0b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.296+0000 7f73ceffd640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73bc0b65c0 0x7f73bc0b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.296+0000 7f73c7fff640 1 -- 172.21.15.29:0/160987576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7f73c00e7b20 con 0x7f73d0108cf0 2024-09-21T03:05:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.296+0000 7f73ceffd640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73bc0b65c0 0x7f73bc0b8ad0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f73c800abe0 tx=0x7f73c800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.299+0000 7f73c7fff640 1 -- 172.21.15.29:0/160987576 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f73c0016030 con 0x7f73d0108cf0 2024-09-21T03:05:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.530+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73d010ab40 con 0x7f73d0108cf0 2024-09-21T03:05:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.531+0000 7f73c7fff640 1 -- 172.21.15.29:0/160987576 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11976 (secure 0 0 0) 0x7f73c00ae050 con 0x7f73d0108cf0 2024-09-21T03:05:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:05:29.532 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"btime":"2024-09-21T03:04:49:618327+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14721,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/502969019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":502969019},{"type":"v1","addr":"172.21.15.29:6835","nonce":502969019}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2096055950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2096055950},{"type":"v1","addr":"172.21.15.29:6837","nonce":2096055950}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:36:26.562695+0000","modified":"2024-09-21T03:04:48.728569+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24637,"mds_1":24632,"mds_2":24625,"mds_3":24613,"mds_4":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/886485890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":886485890},{"type":"v1","addr":"172.21.15.29:6839","nonce":886485890}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14787":{"gid":14787,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/1488187270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":1488187270},{"type":"v1","addr":"172.21.15.29:6841","nonce":1488187270}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6833/2291646688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6832","nonce":2291646688},{"type":"v1","addr":"172.21.15.66:6833","nonce":2291646688}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/3478087331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3478087331},{"type":"v1","addr":"172.21.15.190:6833","nonce":3478087331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.66:6837/3184195773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6836","nonce":3184195773},{"type":"v1","addr":"172.21.15.66:6837","nonce":3184195773}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":19,"addr":"172.21.15.190:6835/131555194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":131555194},{"type":"v1","addr":"172.21.15.190:6835","nonce":131555194}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24625":{"gid":24625,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.190:6837/3758896499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3758896499},{"type":"v1","addr":"172.21.15.190:6837","nonce":3758896499}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24632":{"gid":24632,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.66:6839/3618856137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6838","nonce":3618856137},{"type":"v1","addr":"172.21.15.66:6839","nonce":3618856137}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6839/3083654868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":3083654868},{"type":"v1","addr":"172.21.15.190:6839","nonce":3083654868}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27782":{"gid":27782,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.66:6835/3193047193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6834","nonce":3193047193},{"type":"v1","addr":"172.21.15.66:6835","nonce":3193047193}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24637,"qdb_cluster":[24637,24632,24613,24625,24599]},"id":1}]} 2024-09-21T03:05:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73bc0b65c0 msgr2=0x7f73bc0b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73bc0b65c0 0x7f73bc0b8ad0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f73c800abe0 tx=0x7f73c800a2d0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d0108cf0 msgr2=0x7f73d019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d0108cf0 0x7f73d019fba0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f73c000eaa0 tx=0x7f73c000eea0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 shutdown_connections 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f73bc0b65c0 0x7f73bc0b8ad0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73d0108cf0 0x7f73d019fba0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73d01083a0 0x7f73d019a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 --2- 172.21.15.29:0/160987576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f73d0102a60 0x7f73d019a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 >> 172.21.15.29:0/160987576 conn(0x7f73d0079730 msgr2=0x7f73d01052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 shutdown_connections 2024-09-21T03:05:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:29.534+0000 7f73d554d640 1 -- 172.21.15.29:0/160987576 wait complete. 2024-09-21T03:05:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:29 smithi190 ceph-mon[33871]: pgmap v1181: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-21T03:05:29.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:29 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/160987576' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:05:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:29 smithi029 ceph-mon[31006]: pgmap v1181: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-21T03:05:29.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:29 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/160987576' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:05:30.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:29 smithi066 ceph-mon[33436]: pgmap v1181: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-21T03:05:30.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:29 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/160987576' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T03:05:30.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T03:05:30.497 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 -- 172.21.15.29:0/1315578913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 msgr2=0x7fafe81087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 --2- 172.21.15.29:0/1315578913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 0x7fafe81087b0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009900 tx=0x7fafd002a410 comp rx=0 tx=0).stop 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 -- 172.21.15.29:0/1315578913 shutdown_connections 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 --2- 172.21.15.29:0/1315578913 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe8108cf0 0x7fafe810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 --2- 172.21.15.29:0/1315578913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 0x7fafe81087b0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 --2- 172.21.15.29:0/1315578913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8102a60 0x7fafe8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 -- 172.21.15.29:0/1315578913 >> 172.21.15.29:0/1315578913 conn(0x7fafe8079730 msgr2=0x7fafe8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 -- 172.21.15.29:0/1315578913 shutdown_connections 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.205+0000 7fafeefba640 1 -- 172.21.15.29:0/1315578913 wait complete. 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.206+0000 7fafeefba640 1 Processor -- start 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.206+0000 7fafeefba640 1 -- start start 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.206+0000 7fafeefba640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe8102a60 0x7fafe819b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.206+0000 7fafeefba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 0x7fafe819b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.206+0000 7fafeefba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8108cf0 0x7fafe819bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.206+0000 7fafeefba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fafe8112810 con 0x7fafe81083a0 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.206+0000 7fafeefba640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fafe8112690 con 0x7fafe8102a60 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.206+0000 7fafeefba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fafe8112990 con 0x7fafe8108cf0 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 0x7fafe819b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 0x7fafe819b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59422/0 (socket says 172.21.15.29:59422) 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 -- 172.21.15.29:0/4121334531 learned_addr learned my addr 172.21.15.29:0/4121334531 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafecd56640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe8102a60 0x7fafe819b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafed557640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8108cf0 0x7fafe819bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 -- 172.21.15.29:0/4121334531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8108cf0 msgr2=0x7fafe819bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8108cf0 0x7fafe819bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 -- 172.21.15.29:0/4121334531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe8102a60 msgr2=0x7fafe819b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe8102a60 0x7fafe819b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 -- 172.21.15.29:0/4121334531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe81a64d0 con 0x7fafe81083a0 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe7fff640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 0x7fafe819b780 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fafd0005940 tx=0x7fafd0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe5ffb640 1 -- 172.21.15.29:0/4121334531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd003f030 con 0x7fafe81083a0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe5ffb640 1 -- 172.21.15.29:0/4121334531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fafd0004440 con 0x7fafe81083a0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe81a6760 con 0x7fafe81083a0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.207+0000 7fafe5ffb640 1 -- 172.21.15.29:0/4121334531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0035070 con 0x7fafe81083a0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.208+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe81a6c90 con 0x7fafe81083a0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.209+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe8102e70 con 0x7fafe81083a0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.211+0000 7fafe5ffb640 1 -- 172.21.15.29:0/4121334531 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fafd0042030 con 0x7fafe81083a0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.212+0000 7fafe5ffb640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fafd40b65c0 0x7fafd40b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.212+0000 7fafecd56640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fafd40b65c0 0x7fafd40b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.212+0000 7fafe5ffb640 1 -- 172.21.15.29:0/4121334531 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fafd0107e90 con 0x7fafe81083a0 2024-09-21T03:05:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.213+0000 7fafecd56640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fafd40b65c0 0x7fafd40b8ad0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fafe806b9c0 tx=0x7fafd800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:31.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.215+0000 7fafe5ffb640 1 -- 172.21.15.29:0/4121334531 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fafd00ce340 con 0x7fafe81083a0 2024-09-21T03:05:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.405+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafe810ab20 con 0x7fafe81083a0 2024-09-21T03:05:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.407+0000 7fafe5ffb640 1 -- 172.21.15.29:0/4121334531 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24773 (secure 0 0 0) 0x7fafd00d2720 con 0x7fafe81083a0 2024-09-21T03:05:31.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T03:05:31.409 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":86,"fsid":"e1049baa-77c0-11ef-baf5-efdc52797490","created":"2024-09-21T02:27:47.925089+0000","modified":"2024-09-21T03:02:24.075319+0000","last_up_change":"2024-09-21T02:35:33.157210+0000","last_in_change":"2024-09-21T02:35:09.861751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T02:31:38.302181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:36:22.103545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:36:24.546927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:36:30.778080+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"daf314e1-5c4c-4bd4-8fcf-7ec6a7592d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6803","nonce":3066733118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6805","nonce":3066733118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6809","nonce":3066733118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3066733118},{"type":"v1","addr":"172.21.15.29:6807","nonce":3066733118}]},"public_addr":"172.21.15.29:6803/3066733118","cluster_addr":"172.21.15.29:6805/3066733118","heartbeat_back_addr":"172.21.15.29:6809/3066733118","heartbeat_front_addr":"172.21.15.29:6807/3066733118","state":["exists","up"]},{"osd":1,"uuid":"53d7d39c-63c1-4b1c-905d-cbbad4c7e516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6801","nonce":3928353915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6803","nonce":3928353915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6807","nonce":3928353915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":3928353915},{"type":"v1","addr":"172.21.15.66:6805","nonce":3928353915}]},"public_addr":"172.21.15.66:6801/3928353915","cluster_addr":"172.21.15.66:6803/3928353915","heartbeat_back_addr":"172.21.15.66:6807/3928353915","heartbeat_front_addr":"172.21.15.66:6805/3928353915","state":["exists","up"]},{"osd":2,"uuid":"70a0ab7a-bfb1-44ca-b2e1-98d6bc3bb492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6801","nonce":334233995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6803","nonce":334233995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6807","nonce":334233995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":334233995},{"type":"v1","addr":"172.21.15.190:6805","nonce":334233995}]},"public_addr":"172.21.15.190:6801/334233995","cluster_addr":"172.21.15.190:6803/334233995","heartbeat_back_addr":"172.21.15.190:6807/334233995","heartbeat_front_addr":"172.21.15.190:6805/334233995","state":["exists","up"]},{"osd":3,"uuid":"2c4edcdb-857c-4f2a-a290-ba400aea7930","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6811","nonce":1263922693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6813","nonce":1263922693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6817","nonce":1263922693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1263922693},{"type":"v1","addr":"172.21.15.29:6815","nonce":1263922693}]},"public_addr":"172.21.15.29:6811/1263922693","cluster_addr":"172.21.15.29:6813/1263922693","heartbeat_back_addr":"172.21.15.29:6817/1263922693","heartbeat_front_addr":"172.21.15.29:6815/1263922693","state":["exists","up"]},{"osd":4,"uuid":"f3c143be-ba4f-4bd0-b14d-5c09661d58a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6809","nonce":3255192229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6811","nonce":3255192229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6815","nonce":3255192229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3255192229},{"type":"v1","addr":"172.21.15.66:6813","nonce":3255192229}]},"public_addr":"172.21.15.66:6809/3255192229","cluster_addr":"172.21.15.66:6811/3255192229","heartbeat_back_addr":"172.21.15.66:6815/3255192229","heartbeat_front_addr":"172.21.15.66:6813/3255192229","state":["exists","up"]},{"osd":5,"uuid":"53d4db37-3682-40a4-ba57-c957525f8a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6809","nonce":4042925750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6811","nonce":4042925750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6815","nonce":4042925750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":4042925750},{"type":"v1","addr":"172.21.15.190:6813","nonce":4042925750}]},"public_addr":"172.21.15.190:6809/4042925750","cluster_addr":"172.21.15.190:6811/4042925750","heartbeat_back_addr":"172.21.15.190:6815/4042925750","heartbeat_front_addr":"172.21.15.190:6813/4042925750","state":["exists","up"]},{"osd":6,"uuid":"22b00286-ed8f-4f2d-954b-c055126544f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6819","nonce":2439227427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6821","nonce":2439227427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6825","nonce":2439227427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2439227427},{"type":"v1","addr":"172.21.15.29:6823","nonce":2439227427}]},"public_addr":"172.21.15.29:6819/2439227427","cluster_addr":"172.21.15.29:6821/2439227427","heartbeat_back_addr":"172.21.15.29:6825/2439227427","heartbeat_front_addr":"172.21.15.29:6823/2439227427","state":["exists","up"]},{"osd":7,"uuid":"8b2ea8fa-8bdb-4065-9c6b-be4157c46f31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6817","nonce":3852321903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6819","nonce":3852321903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6823","nonce":3852321903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":3852321903},{"type":"v1","addr":"172.21.15.66:6821","nonce":3852321903}]},"public_addr":"172.21.15.66:6817/3852321903","cluster_addr":"172.21.15.66:6819/3852321903","heartbeat_back_addr":"172.21.15.66:6823/3852321903","heartbeat_front_addr":"172.21.15.66:6821/3852321903","state":["exists","up"]},{"osd":8,"uuid":"a0995b89-dfe7-4dc5-b927-6d57c8e71d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6817","nonce":2392824828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6819","nonce":2392824828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6823","nonce":2392824828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2392824828},{"type":"v1","addr":"172.21.15.190:6821","nonce":2392824828}]},"public_addr":"172.21.15.190:6817/2392824828","cluster_addr":"172.21.15.190:6819/2392824828","heartbeat_back_addr":"172.21.15.190:6823/2392824828","heartbeat_front_addr":"172.21.15.190:6821/2392824828","state":["exists","up"]},{"osd":9,"uuid":"33205fd3-a7ec-4cba-a969-c2b1ee9d0bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6827","nonce":1846064827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6829","nonce":1846064827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6833","nonce":1846064827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1846064827},{"type":"v1","addr":"172.21.15.29:6831","nonce":1846064827}]},"public_addr":"172.21.15.29:6827/1846064827","cluster_addr":"172.21.15.29:6829/1846064827","heartbeat_back_addr":"172.21.15.29:6833/1846064827","heartbeat_front_addr":"172.21.15.29:6831/1846064827","state":["exists","up"]},{"osd":10,"uuid":"e6234f49-76e4-49a0-9468-5dba3e2f378a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6825","nonce":2015526133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6827","nonce":2015526133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6831","nonce":2015526133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2015526133},{"type":"v1","addr":"172.21.15.66:6829","nonce":2015526133}]},"public_addr":"172.21.15.66:6825/2015526133","cluster_addr":"172.21.15.66:6827/2015526133","heartbeat_back_addr":"172.21.15.66:6831/2015526133","heartbeat_front_addr":"172.21.15.66:6829/2015526133","state":["exists","up"]},{"osd":11,"uuid":"75c747c5-0647-4baa-b2fa-a9651588604f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6825","nonce":2545559018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6827","nonce":2545559018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6831","nonce":2545559018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2545559018},{"type":"v1","addr":"172.21.15.190:6829","nonce":2545559018}]},"public_addr":"172.21.15.190:6825/2545559018","cluster_addr":"172.21.15.190:6827/2545559018","heartbeat_back_addr":"172.21.15.190:6831/2545559018","heartbeat_front_addr":"172.21.15.190:6829/2545559018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:30:43.832199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:10.235496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:31:34.990064+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:02.312240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:28.853766+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:32:53.858422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:21.368513+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:33:47.701643+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:13.512755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:34:40.540946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:06.356730+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:35:31.348350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3488265372":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3273694575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/1830887273":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/313500774":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/2882183575":"2024-09-22T02:28:37.826591+0000","172.21.15.29:0/3241279793":"2024-09-22T02:28:10.590358+0000","172.21.15.29:6801/595746591":"2024-09-22T02:28:10.590358+0000","172.21.15.29:0/1927307007":"2024-09-22T02:28:37.826591+0000","172.21.15.29:6800/595746591":"2024-09-22T02:28:10.590358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:05:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fafd40b65c0 msgr2=0x7fafd40b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fafd40b65c0 0x7fafd40b8ad0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fafe806b9c0 tx=0x7fafd800b390 comp rx=0 tx=0).stop 2024-09-21T03:05:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 msgr2=0x7fafe819b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 0x7fafe819b780 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fafd0005940 tx=0x7fafd0004040 comp rx=0 tx=0).stop 2024-09-21T03:05:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 shutdown_connections 2024-09-21T03:05:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fafd40b65c0 0x7fafd40b8ad0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8108cf0 0x7fafe819bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe81083a0 0x7fafe819b780 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 --2- 172.21.15.29:0/4121334531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe8102a60 0x7fafe819b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 >> 172.21.15.29:0/4121334531 conn(0x7fafe8079730 msgr2=0x7fafe8104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 shutdown_connections 2024-09-21T03:05:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:31.409+0000 7fafeefba640 1 -- 172.21.15.29:0/4121334531 wait complete. 2024-09-21T03:05:31.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:31 smithi029 ceph-mon[31006]: pgmap v1182: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-09-21T03:05:31.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:31 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4121334531' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T03:05:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:31 smithi190 ceph-mon[33871]: pgmap v1182: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-09-21T03:05:31.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:31 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4121334531' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T03:05:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:31 smithi066 ceph-mon[33436]: pgmap v1182: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-09-21T03:05:32.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:31 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4121334531' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T03:05:32.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs fail cephfs 2024-09-21T03:05:32.438 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 -- 172.21.15.29:0/1391183659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb601049b0 msgr2=0x7fcb60104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 --2- 172.21.15.29:0/1391183659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb601049b0 0x7fcb60104dc0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009ca0 tx=0x7fcb5002a8c0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 -- 172.21.15.29:0/1391183659 shutdown_connections 2024-09-21T03:05:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 --2- 172.21.15.29:0/1391183659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb6010de10 0x7fcb60110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 --2- 172.21.15.29:0/1391183659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb60105300 0x7fcb6010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 --2- 172.21.15.29:0/1391183659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb601049b0 0x7fcb60104dc0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 -- 172.21.15.29:0/1391183659 >> 172.21.15.29:0/1391183659 conn(0x7fcb60079730 msgr2=0x7fcb60079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 -- 172.21.15.29:0/1391183659 shutdown_connections 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.969+0000 7fcb67e17640 1 -- 172.21.15.29:0/1391183659 wait complete. 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb67e17640 1 Processor -- start 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb67e17640 1 -- start start 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb67e17640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb601049b0 0x7fcb6019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb67e17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb60105300 0x7fcb6019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb67e17640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6010de10 0x7fcb6019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb67e17640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcb60113970 con 0x7fcb60105300 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb67e17640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601137f0 con 0x7fcb601049b0 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb67e17640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb60113af0 con 0x7fcb6010de10 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb653b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb60105300 0x7fcb6019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb653b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb60105300 0x7fcb6019a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59450/0 (socket says 172.21.15.29:59450) 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb653b2640 1 -- 172.21.15.29:0/2818971141 learned_addr learned my addr 172.21.15.29:0/2818971141 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.970+0000 7fcb65bb3640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb601049b0 0x7fcb6019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb663b4640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6010de10 0x7fcb6019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb653b2640 1 -- 172.21.15.29:0/2818971141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6010de10 msgr2=0x7fcb6019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb653b2640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6010de10 0x7fcb6019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb653b2640 1 -- 172.21.15.29:0/2818971141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb601049b0 msgr2=0x7fcb6019a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb653b2640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb601049b0 0x7fcb6019a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb653b2640 1 -- 172.21.15.29:0/2818971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb601a0350 con 0x7fcb60105300 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb663b4640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6010de10 0x7fcb6019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb653b2640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb60105300 0x7fcb6019a980 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400d740 tx=0x7fcb5400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb5effd640 1 -- 172.21.15.29:0/2818971141 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54017070 con 0x7fcb60105300 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb5effd640 1 -- 172.21.15.29:0/2818971141 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb5400de50 con 0x7fcb60105300 2024-09-21T03:05:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb601a0640 con 0x7fcb60105300 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb5effd640 1 -- 172.21.15.29:0/2818971141 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5400f620 con 0x7fcb60105300 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.971+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb601a6da0 con 0x7fcb60105300 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.972+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb60104dc0 con 0x7fcb60105300 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.975+0000 7fcb5effd640 1 -- 172.21.15.29:0/2818971141 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7fcb5401a070 con 0x7fcb60105300 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.976+0000 7fcb5effd640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb480b6590 0x7fcb480b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.976+0000 7fcb65bb3640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb480b6590 0x7fcb480b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.976+0000 7fcb5effd640 1 -- 172.21.15.29:0/2818971141 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9121+0+0 (secure 0 0 0) 0x7fcb540e6ff0 con 0x7fcb60105300 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.976+0000 7fcb65bb3640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb480b6590 0x7fcb480b8aa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcb50006010 tx=0x7fcb50031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:32.979+0000 7fcb5effd640 1 -- 172.21.15.29:0/2818971141 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcb540ad580 con 0x7fcb60105300 2024-09-21T03:05:33.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.206+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcb6019b510 con 0x7fcb60105300 2024-09-21T03:05:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.209+0000 7fcb5effd640 1 -- 172.21.15.29:0/2818971141 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) ==== 161+0+0 (secure 0 0 0) 0x7fcb6019b510 con 0x7fcb60105300 2024-09-21T03:05:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-21T03:05:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.213+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb480b6590 msgr2=0x7fcb480b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.213+0000 7fcb67e17640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb480b6590 0x7fcb480b8aa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcb50006010 tx=0x7fcb50031000 comp rx=0 tx=0).stop 2024-09-21T03:05:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.213+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb60105300 msgr2=0x7fcb6019a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.213+0000 7fcb67e17640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb60105300 0x7fcb6019a980 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400d740 tx=0x7fcb5400dc00 comp rx=0 tx=0).stop 2024-09-21T03:05:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.213+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 shutdown_connections 2024-09-21T03:05:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.213+0000 7fcb67e17640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7fcb480b6590 0x7fcb480b8aa0 secure :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcb50006010 tx=0x7fcb50031000 comp rx=0 tx=0).stop 2024-09-21T03:05:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.214+0000 7fcb67e17640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6010de10 0x7fcb6019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.214+0000 7fcb67e17640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb60105300 0x7fcb6019a980 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.214+0000 7fcb67e17640 1 --2- 172.21.15.29:0/2818971141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb601049b0 0x7fcb6019a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.214+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 >> 172.21.15.29:0/2818971141 conn(0x7fcb60079730 msgr2=0x7fcb600ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.214+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 shutdown_connections 2024-09-21T03:05:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:33.214+0000 7fcb67e17640 1 -- 172.21.15.29:0/2818971141 wait complete. 2024-09-21T03:05:33.593 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a[31002]: 2024-09-21T03:05:33.207+0000 7f64f922b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T03:05:33.594 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a[72143]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.594 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a[72143]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.594 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mds[72154]: -- 172.21.15.29:0/1273433457 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55823bd7f860 con 0x55823ff27180 2024-09-21T03:05:33.594 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a[72143]: starting mds.a at 2024-09-21T03:05:33.594 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g[71834]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.594 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g[71834]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.594 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mds[71838]: -- 172.21.15.29:0/2964223431 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de2fb23860 con 0x55de33ccb180 2024-09-21T03:05:33.594 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g[71834]: starting mds.g at 2024-09-21T03:05:33.595 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c[62423]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.595 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c[62423]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.595 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mds[62427]: -- 172.21.15.190:0/371535914 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ba441e8ea0 con 0x55ba48387a80 2024-09-21T03:05:33.595 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mds[62427]: -- 172.21.15.190:0/371535914 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba441df860 con 0x55ba48387a80 2024-09-21T03:05:33.595 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c[62423]: starting mds.c at 2024-09-21T03:05:33.595 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: starting mds.e at 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f[62151]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f[62151]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mds[62155]: -- 172.21.15.190:0/3530544957 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d9e0855860 con 0x55d9e49fda80 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f[62151]: starting mds.f at 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k[60698]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k[60698]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.596 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k[60698]: starting mds.k at 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l[61877]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l[61877]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mds[61881]: -- 172.21.15.190:0/1042644426 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae9dca9860 con 0x55aea1e51600 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l[61877]: starting mds.l at 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b[61527]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b[61527]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mds[61531]: -- 172.21.15.66:0/3913667676 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a10265860 con 0x563a1440da80 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b[61527]: starting mds.b at 2024-09-21T03:05:33.597 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.598 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.598 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mds[62699]: --2- 172.21.15.190:0/2570703604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x5561589b1180 0x5561595a0000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x5561589c2b70 tx=0x5561588e8540 comp rx=0 tx=0).write_message sending message m=0x5561588768c0 seq=3 auth(proto 2 2 bytes epoch 0) 2024-09-21T03:05:33.598 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mds[62699]: -- 172.21.15.190:0/2570703604 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556154809860 con 0x5561589b1180 2024-09-21T03:05:33.598 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: starting mds.i at 2024-09-21T03:05:33.598 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-h[61255]: ignoring --setuser ceph since I am not root 2024-09-21T03:05:33.598 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-h[61255]: ignoring --setgroup ceph since I am not root 2024-09-21T03:05:33.598 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mds[61259]: -- 172.21.15.66:0/1406597571 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a692fcd860 con 0x55a697175180 2024-09-21T03:05:33.598 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-h[61255]: starting mds.h at 2024-09-21T03:05:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mon[33871]: pgmap v1183: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-21T03:05:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2818971141' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T03:05:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mon[33871]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T03:05:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mon[33871]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T03:05:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2818971141' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T03:05:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mon[33871]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T03:05:33.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:33 smithi190 ceph-mon[33871]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T03:05:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mon[31006]: pgmap v1183: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-21T03:05:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2818971141' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T03:05:33.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mon[31006]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T03:05:33.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mon[31006]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T03:05:33.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2818971141' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T03:05:33.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mon[31006]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T03:05:33.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:33 smithi029 ceph-mon[31006]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T03:05:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mon[33436]: pgmap v1183: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-21T03:05:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2818971141' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T03:05:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mon[33436]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T03:05:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mon[33436]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T03:05:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2818971141' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T03:05:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mon[33436]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T03:05:34.010 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:33 smithi066 ceph-mon[33436]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T03:05:34.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-21T03:05:34.414 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:34.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 -- 172.21.15.29:0/2911414577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53341094a0 msgr2=0x7f53341098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:34.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 --2- 172.21.15.29:0/2911414577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53341094a0 0x7f53341098b0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f5324009900 tx=0x7f532402a410 comp rx=0 tx=0).stop 2024-09-21T03:05:34.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 -- 172.21.15.29:0/2911414577 shutdown_connections 2024-09-21T03:05:34.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 --2- 172.21.15.29:0/2911414577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5334109df0 0x7f5334110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:34.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 --2- 172.21.15.29:0/2911414577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53341094a0 0x7f53341098b0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:34.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 --2- 172.21.15.29:0/2911414577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5334103b60 0x7f5334103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:34.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 -- 172.21.15.29:0/2911414577 >> 172.21.15.29:0/2911414577 conn(0x7f533407b850 msgr2=0x7f533407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 -- 172.21.15.29:0/2911414577 shutdown_connections 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 -- 172.21.15.29:0/2911414577 wait complete. 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.930+0000 7f533bce7640 1 Processor -- start 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533bce7640 1 -- start start 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533bce7640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5334103b60 0x7f533419c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533bce7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53341094a0 0x7f533419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533bce7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5334109df0 0x7f53341a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533bce7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5334113960 con 0x7f5334109df0 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533bce7640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f53341137e0 con 0x7f5334103b60 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533bce7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5334113ae0 con 0x7f53341094a0 2024-09-21T03:05:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533a284640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5334109df0 0x7f53341a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533a284640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5334109df0 0x7f53341a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56732/0 (socket says 172.21.15.29:56732) 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533a284640 1 -- 172.21.15.29:0/4265312120 learned_addr learned my addr 172.21.15.29:0/4265312120 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f5339a83640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5334103b60 0x7f533419c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f5339282640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53341094a0 0x7f533419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533a284640 1 -- 172.21.15.29:0/4265312120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53341094a0 msgr2=0x7f533419cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533a284640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53341094a0 0x7f533419cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533a284640 1 -- 172.21.15.29:0/4265312120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5334103b60 msgr2=0x7f533419c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533a284640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5334103b60 0x7f533419c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.931+0000 7f533a284640 1 -- 172.21.15.29:0/4265312120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53341a2560 con 0x7f5334109df0 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.932+0000 7f533a284640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5334109df0 0x7f53341a1e30 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f532c00eaa0 tx=0x7f532c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.932+0000 7f532affd640 1 -- 172.21.15.29:0/4265312120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c00cdb0 con 0x7f5334109df0 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.932+0000 7f532affd640 1 -- 172.21.15.29:0/4265312120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f532c005020 con 0x7f5334109df0 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.932+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53341a2850 con 0x7f5334109df0 2024-09-21T03:05:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.932+0000 7f532affd640 1 -- 172.21.15.29:0/4265312120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c0116d0 con 0x7f5334109df0 2024-09-21T03:05:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.932+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53341a8fb0 con 0x7f5334109df0 2024-09-21T03:05:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.933+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5334103f70 con 0x7f5334109df0 2024-09-21T03:05:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.935+0000 7f532affd640 1 -- 172.21.15.29:0/4265312120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f532c0051d0 con 0x7f5334109df0 2024-09-21T03:05:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.935+0000 7f532affd640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53200b6590 0x7f53200b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.935+0000 7f5339a83640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53200b6590 0x7f53200b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.935+0000 7f532affd640 1 -- 172.21.15.29:0/4265312120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9551+0+0 (secure 0 0 0) 0x7f532c0e7cc0 con 0x7f5334109df0 2024-09-21T03:05:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.936+0000 7f5339a83640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53200b6590 0x7f53200b8aa0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5330009bd0 tx=0x7f5330005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:34.939+0000 7f532affd640 1 -- 172.21.15.29:0/4265312120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f532c016030 con 0x7f5334109df0 2024-09-21T03:05:35.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.166+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f53341a9290 con 0x7f5334109df0 2024-09-21T03:05:35.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.217+0000 7f532affd640 1 -- 172.21.15.29:0/4265312120 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v45) ==== 104+0+0 (secure 0 0 0) 0x7f532c0b2490 con 0x7f5334109df0 2024-09-21T03:05:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.220+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53200b6590 msgr2=0x7f53200b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.220+0000 7f533bce7640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53200b6590 0x7f53200b8aa0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5330009bd0 tx=0x7f5330005f90 comp rx=0 tx=0).stop 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.220+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5334109df0 msgr2=0x7f53341a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.220+0000 7f533bce7640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5334109df0 0x7f53341a1e30 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f532c00eaa0 tx=0x7f532c00ef60 comp rx=0 tx=0).stop 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.220+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 shutdown_connections 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.221+0000 7f533bce7640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f53200b6590 0x7f53200b8aa0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.221+0000 7f533bce7640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5334109df0 0x7f53341a1e30 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.221+0000 7f533bce7640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53341094a0 0x7f533419cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.221+0000 7f533bce7640 1 --2- 172.21.15.29:0/4265312120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5334103b60 0x7f533419c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.221+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 >> 172.21.15.29:0/4265312120 conn(0x7f533407b850 msgr2=0x7f53341072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.221+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 shutdown_connections 2024-09-21T03:05:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:35.221+0000 7f533bce7640 1 -- 172.21.15.29:0/4265312120 wait complete. 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6832/3654066742,v1:172.21.15.66:6835/3654066742] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6834/549179463,v1:172.21.15.66:6837/549179463] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6836/3186083612,v1:172.21.15.66:6839/3186083612] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.66:6833/2622084082,v1:172.21.15.66:6838/2622084082] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6833/1054884265,v1:172.21.15.190:6836/1054884265] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6834/981705968,v1:172.21.15.190:6837/981705968] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6832/1012246974,v1:172.21.15.190:6835/1012246974] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.190:6838/452514659,v1:172.21.15.190:6839/452514659] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.29:6838/1323710127,v1:172.21.15.29:6840/1323710127] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: mds.? [v2:172.21.15.29:6839/4128773166,v1:172.21.15.29:6841/4128773166] up:boot 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T03:05:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: pgmap v1185: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 10 op/s 2024-09-21T03:05:35.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:35 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4265312120' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T03:05:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6832/3654066742,v1:172.21.15.66:6835/3654066742] up:boot 2024-09-21T03:05:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6834/549179463,v1:172.21.15.66:6837/549179463] up:boot 2024-09-21T03:05:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6836/3186083612,v1:172.21.15.66:6839/3186083612] up:boot 2024-09-21T03:05:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.66:6833/2622084082,v1:172.21.15.66:6838/2622084082] up:boot 2024-09-21T03:05:35.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6833/1054884265,v1:172.21.15.190:6836/1054884265] up:boot 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6834/981705968,v1:172.21.15.190:6837/981705968] up:boot 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6832/1012246974,v1:172.21.15.190:6835/1012246974] up:boot 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.190:6838/452514659,v1:172.21.15.190:6839/452514659] up:boot 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.29:6838/1323710127,v1:172.21.15.29:6840/1323710127] up:boot 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: mds.? [v2:172.21.15.29:6839/4128773166,v1:172.21.15.29:6841/4128773166] up:boot 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T03:05:35.511 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T03:05:35.512 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: pgmap v1185: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 10 op/s 2024-09-21T03:05:35.512 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:35 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4265312120' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6832/3654066742,v1:172.21.15.66:6835/3654066742] up:boot 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6834/549179463,v1:172.21.15.66:6837/549179463] up:boot 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6836/3186083612,v1:172.21.15.66:6839/3186083612] up:boot 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.66:6833/2622084082,v1:172.21.15.66:6838/2622084082] up:boot 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6833/1054884265,v1:172.21.15.190:6836/1054884265] up:boot 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6834/981705968,v1:172.21.15.190:6837/981705968] up:boot 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6832/1012246974,v1:172.21.15.190:6835/1012246974] up:boot 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.190:6838/452514659,v1:172.21.15.190:6839/452514659] up:boot 2024-09-21T03:05:35.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.29:6838/1323710127,v1:172.21.15.29:6840/1323710127] up:boot 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: mds.? [v2:172.21.15.29:6839/4128773166,v1:172.21.15.29:6841/4128773166] up:boot 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='mgr.14152 172.21.15.29:0/4194408430' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: pgmap v1185: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 10 op/s 2024-09-21T03:05:35.625 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:35 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4265312120' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T03:05:36.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-21T03:05:36.357 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:36.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:36 smithi029 ceph-mon[31006]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T03:05:36.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:36 smithi029 ceph-mon[31006]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T03:05:36.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:36 smithi029 ceph-mon[31006]: Cluster is now healthy 2024-09-21T03:05:36.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:36 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/4265312120' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T03:05:36.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:36 smithi029 ceph-mon[31006]: fsmap 12 up:standby 2024-09-21T03:05:36.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:36 smithi066 ceph-mon[33436]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T03:05:36.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:36 smithi066 ceph-mon[33436]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T03:05:36.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:36 smithi066 ceph-mon[33436]: Cluster is now healthy 2024-09-21T03:05:36.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:36 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/4265312120' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T03:05:36.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:36 smithi066 ceph-mon[33436]: fsmap 12 up:standby 2024-09-21T03:05:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:36 smithi190 ceph-mon[33871]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T03:05:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:36 smithi190 ceph-mon[33871]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T03:05:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:36 smithi190 ceph-mon[33871]: Cluster is now healthy 2024-09-21T03:05:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:36 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/4265312120' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T03:05:36.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:36 smithi190 ceph-mon[33871]: fsmap 12 up:standby 2024-09-21T03:05:36.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 -- 172.21.15.29:0/1111086934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8108cf0 msgr2=0x7f32c810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:36.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 --2- 172.21.15.29:0/1111086934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8108cf0 0x7f32c810f520 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009ca0 tx=0x7f32c002a8c0 comp rx=0 tx=0).stop 2024-09-21T03:05:36.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 -- 172.21.15.29:0/1111086934 shutdown_connections 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 --2- 172.21.15.29:0/1111086934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8108cf0 0x7f32c810f520 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 --2- 172.21.15.29:0/1111086934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32c81083a0 0x7f32c81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 --2- 172.21.15.29:0/1111086934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32c8102a60 0x7f32c8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 -- 172.21.15.29:0/1111086934 >> 172.21.15.29:0/1111086934 conn(0x7f32c8079730 msgr2=0x7f32c8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 -- 172.21.15.29:0/1111086934 shutdown_connections 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.948+0000 7f32d066a640 1 -- 172.21.15.29:0/1111086934 wait complete. 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32d066a640 1 Processor -- start 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32d066a640 1 -- start start 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32d066a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8102a60 0x7f32c819b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32d066a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32c81083a0 0x7f32c819b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32d066a640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32c8108cf0 0x7f32c819bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32d066a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32c8112860 con 0x7f32c8102a60 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32d066a640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f32c81126e0 con 0x7f32c8108cf0 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32d066a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f32c81129e0 con 0x7f32c81083a0 2024-09-21T03:05:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32ce406640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8102a60 0x7f32c819b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32ce406640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8102a60 0x7f32c819b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56736/0 (socket says 172.21.15.29:56736) 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.949+0000 7f32ce406640 1 -- 172.21.15.29:0/1041412859 learned_addr learned my addr 172.21.15.29:0/1041412859 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32cdc05640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32c81083a0 0x7f32c819b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32cec07640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32c8108cf0 0x7f32c819bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32ce406640 1 -- 172.21.15.29:0/1041412859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32c81083a0 msgr2=0x7f32c819b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32ce406640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32c81083a0 0x7f32c819b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32ce406640 1 -- 172.21.15.29:0/1041412859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32c8108cf0 msgr2=0x7f32c819bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32ce406640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32c8108cf0 0x7f32c819bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32ce406640 1 -- 172.21.15.29:0/1041412859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c81a63d0 con 0x7f32c8102a60 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32cec07640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32c8108cf0 0x7f32c819bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32cdc05640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32c81083a0 0x7f32c819b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32ce406640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8102a60 0x7f32c819b110 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f32c400d770 tx=0x7f32c400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32bf7fe640 1 -- 172.21.15.29:0/1041412859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4017070 con 0x7f32c8102a60 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c81a66c0 con 0x7f32c8102a60 2024-09-21T03:05:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32bf7fe640 1 -- 172.21.15.29:0/1041412859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32c40054b0 con 0x7f32c8102a60 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32bf7fe640 1 -- 172.21.15.29:0/1041412859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4004c60 con 0x7f32c8102a60 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.950+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c81a6bb0 con 0x7f32c8102a60 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.951+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c8102e70 con 0x7f32c8102a60 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.953+0000 7f32bf7fe640 1 -- 172.21.15.29:0/1041412859 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f32c4004e00 con 0x7f32c8102a60 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.954+0000 7f32bf7fe640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f32b40b6510 0x7f32b40b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.954+0000 7f32cdc05640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f32b40b6510 0x7f32b40b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.954+0000 7f32bf7fe640 1 -- 172.21.15.29:0/1041412859 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9551+0+0 (secure 0 0 0) 0x7f32c40e7740 con 0x7f32c8102a60 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.955+0000 7f32cdc05640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f32b40b6510 0x7f32b40b8a20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f32c806a260 tx=0x7f32b8005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:36.957+0000 7f32bf7fe640 1 -- 172.21.15.29:0/1041412859 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f32c40adab0 con 0x7f32c8102a60 2024-09-21T03:05:37.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.147+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f32c802e640 con 0x7f32c8102a60 2024-09-21T03:05:37.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.224+0000 7f32bf7fe640 1 -- 172.21.15.29:0/1041412859 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v88) ==== 181+0+0 (secure 0 0 0) 0x7f32c40b1e90 con 0x7f32c8102a60 2024-09-21T03:05:37.226 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2024-09-21T03:05:37.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f32b40b6510 msgr2=0x7f32b40b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:37.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f32b40b6510 0x7f32b40b8a20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f32c806a260 tx=0x7f32b8005ec0 comp rx=0 tx=0).stop 2024-09-21T03:05:37.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8102a60 msgr2=0x7f32c819b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:37.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8102a60 0x7f32c819b110 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f32c400d770 tx=0x7f32c400dc30 comp rx=0 tx=0).stop 2024-09-21T03:05:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 shutdown_connections 2024-09-21T03:05:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f32b40b6510 0x7f32b40b8a20 secure :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f32c806a260 tx=0x7f32b8005ec0 comp rx=0 tx=0).stop 2024-09-21T03:05:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32c8108cf0 0x7f32c819bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32c81083a0 0x7f32c819b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.228+0000 7f32d066a640 1 --2- 172.21.15.29:0/1041412859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c8102a60 0x7f32c819b110 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.229+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 >> 172.21.15.29:0/1041412859 conn(0x7f32c8079730 msgr2=0x7f32c81043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.229+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 shutdown_connections 2024-09-21T03:05:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:37.230+0000 7f32d066a640 1 -- 172.21.15.29:0/1041412859 wait complete. 2024-09-21T03:05:37.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:37 smithi066 ceph-mon[33436]: pgmap v1186: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 9 op/s 2024-09-21T03:05:37.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:37 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1041412859' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:37.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:37 smithi190 ceph-mon[33871]: pgmap v1186: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 9 op/s 2024-09-21T03:05:37.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:37 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1041412859' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:37.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:37 smithi029 ceph-mon[31006]: pgmap v1186: 129 pgs: 129 active+clean; 133 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 9 op/s 2024-09-21T03:05:37.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:37 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1041412859' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:38.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-21T03:05:38.265 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:38.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:38 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/1041412859' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:38.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:38 smithi066 ceph-mon[33436]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T03:05:38.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:38 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/1041412859' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:38.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:38 smithi190 ceph-mon[33871]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T03:05:38.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:38 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/1041412859' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:38.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:38 smithi029 ceph-mon[31006]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 -- 172.21.15.29:0/2976594989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50108d20 msgr2=0x7f4b5010f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 --2- 172.21.15.29:0/2976594989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50108d20 0x7f4b5010f550 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009ca0 tx=0x7f4b4802a8c0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 -- 172.21.15.29:0/2976594989 shutdown_connections 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 --2- 172.21.15.29:0/2976594989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50108d20 0x7f4b5010f550 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 --2- 172.21.15.29:0/2976594989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b501083d0 0x7f4b501087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 --2- 172.21.15.29:0/2976594989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b50102ae0 0x7f4b50102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 -- 172.21.15.29:0/2976594989 >> 172.21.15.29:0/2976594989 conn(0x7f4b50079760 msgr2=0x7f4b50079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 -- 172.21.15.29:0/2976594989 shutdown_connections 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 -- 172.21.15.29:0/2976594989 wait complete. 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.010+0000 7f4b56629640 1 Processor -- start 2024-09-21T03:05:39.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b56629640 1 -- start start 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b56629640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50102ae0 0x7f4b5019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b56629640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b501083d0 0x7f4b5019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b56629640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b50108d20 0x7f4b5019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b56629640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4b50113990 con 0x7f4b50102ae0 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b56629640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f4b50113810 con 0x7f4b50108d20 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b56629640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b50113b10 con 0x7f4b501083d0 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50102ae0 0x7f4b5019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50102ae0 0x7f4b5019a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56746/0 (socket says 172.21.15.29:56746) 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b4ffff640 1 -- 172.21.15.29:0/2472640952 learned_addr learned my addr 172.21.15.29:0/2472640952 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b54bc6640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b50108d20 0x7f4b5019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b4f7fe640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b501083d0 0x7f4b5019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b4ffff640 1 -- 172.21.15.29:0/2472640952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b501083d0 msgr2=0x7f4b5019a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b4ffff640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b501083d0 0x7f4b5019a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b4ffff640 1 -- 172.21.15.29:0/2472640952 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b50108d20 msgr2=0x7f4b5019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:39.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.011+0000 7f4b4ffff640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b50108d20 0x7f4b5019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.012+0000 7f4b4ffff640 1 -- 172.21.15.29:0/2472640952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b501a0390 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.012+0000 7f4b54bc6640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b50108d20 0x7f4b5019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.012+0000 7f4b4ffff640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50102ae0 0x7f4b5019a420 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000d950 tx=0x7f4b4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.012+0000 7f4b4d7fa640 1 -- 172.21.15.29:0/2472640952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b40017070 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.012+0000 7f4b4d7fa640 1 -- 172.21.15.29:0/2472640952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4b40009c30 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.012+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b501a0680 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.012+0000 7f4b4d7fa640 1 -- 172.21.15.29:0/2472640952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4000f620 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.012+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b501a6de0 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.013+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b50103d10 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.015+0000 7f4b4d7fa640 1 -- 172.21.15.29:0/2472640952 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f4b40005030 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.016+0000 7f4b4d7fa640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b380b6590 0x7f4b380b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.016+0000 7f4b4f7fe640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b380b6590 0x7f4b380b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.016+0000 7f4b4d7fa640 1 -- 172.21.15.29:0/2472640952 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9026+0+0 (secure 0 0 0) 0x7f4b400e7180 con 0x7f4b50102ae0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.017+0000 7f4b4f7fe640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b380b6590 0x7f4b380b8aa0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4b50079c10 tx=0x7f4b4400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.019+0000 7f4b4d7fa640 1 -- 172.21.15.29:0/2472640952 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4b400ad770 con 0x7f4b50102ae0 2024-09-21T03:05:39.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.208+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4b50102ef0 con 0x7f4b50102ae0 2024-09-21T03:05:39.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.232+0000 7f4b4d7fa640 1 -- 172.21.15.29:0/2472640952 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v89) ==== 169+0+0 (secure 0 0 0) 0x7f4b400e4e20 con 0x7f4b50102ae0 2024-09-21T03:05:39.234 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2024-09-21T03:05:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.236+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b380b6590 msgr2=0x7f4b380b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.236+0000 7f4b56629640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b380b6590 0x7f4b380b8aa0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4b50079c10 tx=0x7f4b4400b410 comp rx=0 tx=0).stop 2024-09-21T03:05:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.236+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50102ae0 msgr2=0x7f4b5019a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.236+0000 7f4b56629640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50102ae0 0x7f4b5019a420 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000d950 tx=0x7f4b4000de10 comp rx=0 tx=0).stop 2024-09-21T03:05:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.237+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 shutdown_connections 2024-09-21T03:05:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.237+0000 7f4b56629640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f4b380b6590 0x7f4b380b8aa0 secure :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4b50079c10 tx=0x7f4b4400b410 comp rx=0 tx=0).stop 2024-09-21T03:05:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.237+0000 7f4b56629640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b50108d20 0x7f4b5019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.237+0000 7f4b56629640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b501083d0 0x7f4b5019a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.237+0000 7f4b56629640 1 --2- 172.21.15.29:0/2472640952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b50102ae0 0x7f4b5019a420 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:39.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.237+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 >> 172.21.15.29:0/2472640952 conn(0x7f4b50079760 msgr2=0x7f4b50104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:39.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.237+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 shutdown_connections 2024-09-21T03:05:39.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:39.238+0000 7f4b56629640 1 -- 172.21.15.29:0/2472640952 wait complete. 2024-09-21T03:05:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:39 smithi029 ceph-mon[31006]: pgmap v1188: 97 pgs: 97 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:39 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2472640952' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:39.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:39 smithi066 ceph-mon[33436]: pgmap v1188: 97 pgs: 97 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:39.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:39 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2472640952' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:39.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:39 smithi190 ceph-mon[33871]: pgmap v1188: 97 pgs: 97 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:39.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:39 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2472640952' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:40.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid e1049baa-77c0-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-21T03:05:40.328 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T03:05:40.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:40 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/2472640952' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:40.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:40 smithi029 ceph-mon[31006]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T03:05:40.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:40 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/2472640952' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:40.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:40 smithi066 ceph-mon[33436]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T03:05:40.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:40 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/2472640952' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:40.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:40 smithi190 ceph-mon[33871]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T03:05:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 -- 172.21.15.29:0/1449119579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd01052b0 msgr2=0x7f5dd010d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/1449119579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd01052b0 0x7f5dd010d8f0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009900 tx=0x7f5dc402a410 comp rx=0 tx=0).stop 2024-09-21T03:05:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 -- 172.21.15.29:0/1449119579 shutdown_connections 2024-09-21T03:05:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/1449119579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dd010de30 0x7f5dd0110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/1449119579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd01052b0 0x7f5dd010d8f0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/1449119579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5dd0104880 0x7f5dd0104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 -- 172.21.15.29:0/1449119579 >> 172.21.15.29:0/1449119579 conn(0x7f5dd0079730 msgr2=0x7f5dd0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 -- 172.21.15.29:0/1449119579 shutdown_connections 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 -- 172.21.15.29:0/1449119579 wait complete. 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.071+0000 7f5dd5e59640 1 Processor -- start 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dd5e59640 1 -- start start 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dd5e59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd0104880 0x7f5dd019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dd5e59640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dd01052b0 0x7f5dd019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dd5e59640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5dd010de30 0x7f5dd019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dd5e59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd0113920 con 0x7f5dd0104880 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dd5e59640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd01137a0 con 0x7f5dd010de30 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dd5e59640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd0113aa0 con 0x7f5dd01052b0 2024-09-21T03:05:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd0104880 0x7f5dd019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd0104880 0x7f5dd019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56764/0 (socket says 172.21.15.29:56764) 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcf7fe640 1 -- 172.21.15.29:0/3985287216 learned_addr learned my addr 172.21.15.29:0/3985287216 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcffff640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5dd010de30 0x7f5dd019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dceffd640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dd01052b0 0x7f5dd019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcf7fe640 1 -- 172.21.15.29:0/3985287216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dd01052b0 msgr2=0x7f5dd019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcf7fe640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dd01052b0 0x7f5dd019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcf7fe640 1 -- 172.21.15.29:0/3985287216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5dd010de30 msgr2=0x7f5dd019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcf7fe640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5dd010de30 0x7f5dd019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcf7fe640 1 -- 172.21.15.29:0/3985287216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd01a0280 con 0x7f5dd0104880 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dcffff640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5dd010de30 0x7f5dd019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.072+0000 7f5dceffd640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dd01052b0 0x7f5dd019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.073+0000 7f5dcf7fe640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd0104880 0x7f5dd019a3d0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c970 tx=0x7f5dc000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.073+0000 7f5dccff9640 1 -- 172.21.15.29:0/3985287216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0007c10 con 0x7f5dd0104880 2024-09-21T03:05:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.073+0000 7f5dccff9640 1 -- 172.21.15.29:0/3985287216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dc0007db0 con 0x7f5dd0104880 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.073+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd01a0570 con 0x7f5dd0104880 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.073+0000 7f5dccff9640 1 -- 172.21.15.29:0/3985287216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc000f7d0 con 0x7f5dd0104880 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.073+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd01a6d90 con 0x7f5dd0104880 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.074+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd0104c90 con 0x7f5dd0104880 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.075+0000 7f5dccff9640 1 -- 172.21.15.29:0/3985287216 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 154512+0+0 (secure 0 0 0) 0x7f5dc0018480 con 0x7f5dd0104880 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.076+0000 7f5dccff9640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5db80b65c0 0x7f5db80b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.076+0000 7f5dceffd640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5db80b65c0 0x7f5db80b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.076+0000 7f5dccff9640 1 -- 172.21.15.29:0/3985287216 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8496+0+0 (secure 0 0 0) 0x7f5dc00e7730 con 0x7f5dd0104880 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.077+0000 7f5dceffd640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5db80b65c0 0x7f5db80b8ad0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4006010 tx=0x7f5dc4005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:05:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.081+0000 7f5dccff9640 1 -- 172.21.15.29:0/3985287216 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5dc00adf50 con 0x7f5dd0104880 2024-09-21T03:05:41.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:41.272+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5dd019b500 con 0x7f5dd0104880 2024-09-21T03:05:41.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:41 smithi029 ceph-mon[31006]: pgmap v1190: 33 pgs: 33 active+clean; 577 KiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:41.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:41 smithi066 ceph-mon[33436]: pgmap v1190: 33 pgs: 33 active+clean; 577 KiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:41.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:41 smithi190 ceph-mon[33871]: pgmap v1190: 33 pgs: 33 active+clean; 577 KiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:42.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.247+0000 7f5dccff9640 1 -- 172.21.15.29:0/3985287216 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v90) ==== 178+0+0 (secure 0 0 0) 0x7f5dc00b2330 con 0x7f5dd0104880 2024-09-21T03:05:42.249 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data_ec' removed 2024-09-21T03:05:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.251+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5db80b65c0 msgr2=0x7f5db80b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.251+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5db80b65c0 0x7f5db80b8ad0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4006010 tx=0x7f5dc4005de0 comp rx=0 tx=0).stop 2024-09-21T03:05:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.251+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd0104880 msgr2=0x7f5dd019a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:05:42.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.251+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd0104880 0x7f5dd019a3d0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c970 tx=0x7f5dc000ce30 comp rx=0 tx=0).stop 2024-09-21T03:05:42.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.253+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 shutdown_connections 2024-09-21T03:05:42.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.253+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:6800/257624116,v1:172.21.15.29:6801/257624116] conn(0x7f5db80b65c0 0x7f5db80b8ad0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.253+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5dd010de30 0x7f5dd019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.253+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dd01052b0 0x7f5dd019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:42.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.253+0000 7f5dd5e59640 1 --2- 172.21.15.29:0/3985287216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dd0104880 0x7f5dd019a3d0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:05:42.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.253+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 >> 172.21.15.29:0/3985287216 conn(0x7f5dd0079730 msgr2=0x7f5dd00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:05:42.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.254+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 shutdown_connections 2024-09-21T03:05:42.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T03:05:42.254+0000 7f5dd5e59640 1 -- 172.21.15.29:0/3985287216 wait complete. 2024-09-21T03:05:42.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:42 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3985287216' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:42.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:42 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3985287216' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:42.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:42 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3985287216' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T03:05:43.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T03:05:43.185 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T03:05:43.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T03:05:43.335 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T03:05:43.335 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T03:05:43.351 DEBUG:teuthology.orchestra.run.smithi066:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T03:05:43.367 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T03:05:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:43 smithi029 ceph-mon[31006]: from='client.? 172.21.15.29:0/3985287216' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:43 smithi029 ceph-mon[31006]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T03:05:43.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:43 smithi029 ceph-mon[31006]: pgmap v1192: 1 pgs: 1 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:43.383 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T03:05:43.383 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T03:05:43.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.a 2024-09-21T03:05:43.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:43 smithi066 ceph-mon[33436]: from='client.? 172.21.15.29:0/3985287216' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:43.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:43 smithi066 ceph-mon[33436]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T03:05:43.510 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:43 smithi066 ceph-mon[33436]: pgmap v1192: 1 pgs: 1 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:43.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:43 smithi190 ceph-mon[33871]: from='client.? 172.21.15.29:0/3985287216' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T03:05:43.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:43 smithi190 ceph-mon[33871]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T03:05:43.624 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:43 smithi190 ceph-mon[33871]: pgmap v1192: 1 pgs: 1 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T03:05:43.728 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:43 smithi029 systemd[1]: Stopping Ceph mon.a for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:05:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:43 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a[31002]: 2024-09-21T03:05:43.726+0000 7f64fea36640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:05:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:43 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a[31002]: 2024-09-21T03:05:43.726+0000 7f64fea36640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T03:05:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:43 smithi029 podman[116716]: 2024-09-21 03:05:43.870350029 +0000 UTC m=+0.286467811 container stop d52d85c4a57f56fdc11a6559214e6a2e6d2b58e7892a5a010198d42608a0d6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T03:05:44.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:43 smithi029 podman[116716]: 2024-09-21 03:05:43.896085454 +0000 UTC m=+0.312203227 container died d52d85c4a57f56fdc11a6559214e6a2e6d2b58e7892a5a010198d42608a0d6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T03:05:44.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:44 smithi029 podman[116716]: 2024-09-21 03:05:44.374109476 +0000 UTC m=+0.790227248 container cleanup d52d85c4a57f56fdc11a6559214e6a2e6d2b58e7892a5a010198d42608a0d6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T03:05:44.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:44 smithi029 bash[116716]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a 2024-09-21T03:05:44.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 21 03:05:44 smithi029 podman[116729]: 2024-09-21 03:05:44.602634174 +0000 UTC m=+0.725517737 container remove d52d85c4a57f56fdc11a6559214e6a2e6d2b58e7892a5a010198d42608a0d6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T03:05:44.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.a.service' 2024-09-21T03:05:44.921 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:05:44.921 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T03:05:44.921 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-21T03:05:44.921 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.b 2024-09-21T03:05:45.260 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:44 smithi066 systemd[1]: Stopping Ceph mon.b for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:05:46.153 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-b[33432]: 2024-09-21T03:05:46.028+0000 7f6f3332f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:05:46.153 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-b[33432]: 2024-09-21T03:05:46.029+0000 7f6f3332f640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T03:05:46.437 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:46 smithi066 podman[68094]: 2024-09-21 03:05:46.152038945 +0000 UTC m=+0.245530779 container died e664fe5b69d1c3f3844b32a5eecda02055ad45941f1da97b87540db8b4fed250 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-b, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T03:05:46.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:46 smithi066 podman[68094]: 2024-09-21 03:05:46.538578358 +0000 UTC m=+0.632070191 container cleanup e664fe5b69d1c3f3844b32a5eecda02055ad45941f1da97b87540db8b4fed250 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T03:05:46.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:46 smithi066 bash[68094]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-b 2024-09-21T03:05:47.137 INFO:journalctl@ceph.mon.b.smithi066.stdout:Sep 21 03:05:47 smithi066 podman[68134]: 2024-09-21 03:05:47.025481239 +0000 UTC m=+0.871456084 container remove e664fe5b69d1c3f3844b32a5eecda02055ad45941f1da97b87540db8b4fed250 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:05:47.759 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.b.service' 2024-09-21T03:05:47.827 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:05:47.827 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-21T03:05:47.827 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-21T03:05:47.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.c 2024-09-21T03:05:48.124 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:47 smithi190 systemd[1]: Stopping Ceph mon.c for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:05:48.537 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:48 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-c[33867]: 2024-09-21T03:05:48.175+0000 7f142984c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:05:48.537 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:48 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-c[33867]: 2024-09-21T03:05:48.175+0000 7f142984c640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T03:05:48.537 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:48 smithi190 podman[71172]: 2024-09-21 03:05:48.248489328 +0000 UTC m=+0.194486310 container died 898283fdb943c25456375dc5247ec6a154fd68c5a1d84d2a35029ddf649d99f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-c, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T03:05:48.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:48 smithi190 podman[71172]: 2024-09-21 03:05:48.627540619 +0000 UTC m=+0.573537598 container cleanup 898283fdb943c25456375dc5247ec6a154fd68c5a1d84d2a35029ddf649d99f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:05:48.874 INFO:journalctl@ceph.mon.c.smithi190.stdout:Sep 21 03:05:48 smithi190 bash[71172]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mon-c 2024-09-21T03:05:49.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mon.c.service' 2024-09-21T03:05:49.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:05:49.261 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-21T03:05:49.261 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T03:05:49.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x 2024-09-21T03:05:49.600 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:49 smithi029 systemd[1]: Stopping Ceph mgr.x for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:05:49.882 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:49 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x[31236]: 2024-09-21T03:05:49.597+0000 7fabfae34640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:05:49.883 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:49 smithi029 podman[116839]: 2024-09-21 03:05:49.629992072 +0000 UTC m=+0.166386896 container died d8db2ab5cded05173c066343f091af730c1d9486c1a551d46029653b4377abe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:05:50.308 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:50 smithi029 podman[116839]: 2024-09-21 03:05:50.06252654 +0000 UTC m=+0.598921394 container cleanup d8db2ab5cded05173c066343f091af730c1d9486c1a551d46029653b4377abe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T03:05:50.309 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:50 smithi029 bash[116839]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x 2024-09-21T03:05:50.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x.service' 2024-09-21T03:05:50.572 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:50 smithi029 podman[116854]: 2024-09-21 03:05:50.308019693 +0000 UTC m=+0.671729837 container remove d8db2ab5cded05173c066343f091af730c1d9486c1a551d46029653b4377abe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T03:05:50.573 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:50 smithi029 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x.service: Deactivated successfully. 2024-09-21T03:05:50.573 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:50 smithi029 systemd[1]: Stopped Ceph mgr.x for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T03:05:50.573 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Sep 21 03:05:50 smithi029 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.x.service: Consumed 58.163s CPU time. 2024-09-21T03:05:50.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:05:50.587 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T03:05:50.587 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T03:05:50.587 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.y 2024-09-21T03:05:50.903 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Sep 21 03:05:50 smithi066 systemd[1]: Stopping Ceph mgr.y for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:05:51.184 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Sep 21 03:05:50 smithi066 podman[70920]: 2024-09-21 03:05:50.902863703 +0000 UTC m=+0.115010370 container died cf829af23cc11e0b42d6b07bc79e9733fec1e809a430d6cee9013a8e1d592917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:05:51.510 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Sep 21 03:05:51 smithi066 podman[70920]: 2024-09-21 03:05:51.290755701 +0000 UTC m=+0.502902368 container cleanup cf829af23cc11e0b42d6b07bc79e9733fec1e809a430d6cee9013a8e1d592917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-y, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T03:05:51.510 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Sep 21 03:05:51 smithi066 bash[70920]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-y 2024-09-21T03:05:51.885 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.y.service' 2024-09-21T03:05:51.920 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:05:51.921 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T03:05:51.921 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-21T03:05:51.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.z 2024-09-21T03:05:52.246 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Sep 21 03:05:51 smithi190 systemd[1]: Stopping Ceph mgr.z for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:05:52.552 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Sep 21 03:05:52 smithi190 podman[71297]: 2024-09-21 03:05:52.245261512 +0000 UTC m=+0.140417981 container died a0293db2ee1aa7caa076e33199aafbc1c903a9aabbede0b37a2ed32b0d5001ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-z, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T03:05:52.874 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Sep 21 03:05:52 smithi190 podman[71297]: 2024-09-21 03:05:52.641510034 +0000 UTC m=+0.536666491 container cleanup a0293db2ee1aa7caa076e33199aafbc1c903a9aabbede0b37a2ed32b0d5001ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-z, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T03:05:52.874 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Sep 21 03:05:52 smithi190 bash[71297]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mgr-z 2024-09-21T03:05:53.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mgr.z.service' 2024-09-21T03:05:53.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:05:53.104 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-21T03:05:53.104 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T03:05:53.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.0 2024-09-21T03:05:53.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:05:53 smithi029 systemd[1]: Stopping Ceph osd.0 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:05:53.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:05:53 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0[46236]: 2024-09-21T03:05:53.453+0000 7fb67431a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:05:53.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:05:53 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0[46236]: 2024-09-21T03:05:53.453+0000 7fb67431a640 -1 osd.0 90 *** Got signal Terminated *** 2024-09-21T03:05:53.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:05:53 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0[46236]: 2024-09-21T03:05:53.453+0000 7fb67431a640 -1 osd.0 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:05:59.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:05:59 smithi029 podman[116963]: 2024-09-21 03:05:59.046756979 +0000 UTC m=+5.742923506 container died 17d5d9cf7bda844ba88f2d8f0048167ba90299c8d68d28096a09b102887924a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T03:05:59.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:05:59 smithi029 podman[116963]: 2024-09-21 03:05:59.435562797 +0000 UTC m=+6.131729331 container cleanup 17d5d9cf7bda844ba88f2d8f0048167ba90299c8d68d28096a09b102887924a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T03:05:59.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:05:59 smithi029 bash[116963]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0 2024-09-21T03:06:00.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:05:59 smithi029 podman[116978]: 2024-09-21 03:05:59.739246355 +0000 UTC m=+0.686147592 container remove 17d5d9cf7bda844ba88f2d8f0048167ba90299c8d68d28096a09b102887924a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T03:06:00.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:06:00 smithi029 podman[117069]: 2024-09-21 03:05:59.978241519 +0000 UTC m=+0.016435895 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:06:00.382 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:06:00 smithi029 podman[117069]: 2024-09-21 03:06:00.099650653 +0000 UTC m=+0.137845058 container create 1ae791648087a52ff83c23d4152dbb5b1db5bbfedd3c4bfda8908458b902a72e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T03:06:00.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:06:00 smithi029 podman[117069]: 2024-09-21 03:06:00.381146102 +0000 UTC m=+0.419340475 container init 1ae791648087a52ff83c23d4152dbb5b1db5bbfedd3c4bfda8908458b902a72e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:06:00.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:06:00 smithi029 podman[117069]: 2024-09-21 03:06:00.384957003 +0000 UTC m=+0.423151370 container start 1ae791648087a52ff83c23d4152dbb5b1db5bbfedd3c4bfda8908458b902a72e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:00.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:06:00 smithi029 podman[117069]: 2024-09-21 03:06:00.498676102 +0000 UTC m=+0.536870483 container attach 1ae791648087a52ff83c23d4152dbb5b1db5bbfedd3c4bfda8908458b902a72e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T03:06:00.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 21 03:06:00 smithi029 podman[117069]: 2024-09-21 03:06:00.592023946 +0000 UTC m=+0.630218314 container died 1ae791648087a52ff83c23d4152dbb5b1db5bbfedd3c4bfda8908458b902a72e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T03:06:01.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.0.service' 2024-09-21T03:06:01.296 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:06:01.296 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T03:06:01.296 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T03:06:01.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.1 2024-09-21T03:06:01.510 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:01 smithi066 systemd[1]: Stopping Ceph osd.1 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:06:02.010 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:01 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1[37383]: 2024-09-21T03:06:01.620+0000 7fc4fc883640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:06:02.010 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:01 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1[37383]: 2024-09-21T03:06:01.620+0000 7fc4fc883640 -1 osd.1 90 *** Got signal Terminated *** 2024-09-21T03:06:02.010 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:01 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1[37383]: 2024-09-21T03:06:01.620+0000 7fc4fc883640 -1 osd.1 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:06:07.464 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:07 smithi066 podman[71055]: 2024-09-21 03:06:07.16811945 +0000 UTC m=+5.678868282 container died b629e6e8dde2be16fc550167320ad7c37eb4a3c633c53e5c3774ec6ae069cb5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T03:06:07.760 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:07 smithi066 podman[71055]: 2024-09-21 03:06:07.55491971 +0000 UTC m=+6.065668539 container cleanup b629e6e8dde2be16fc550167320ad7c37eb4a3c633c53e5c3774ec6ae069cb5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:06:07.760 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:07 smithi066 bash[71055]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1 2024-09-21T03:06:08.227 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:07 smithi066 podman[71071]: 2024-09-21 03:06:07.864670877 +0000 UTC m=+0.688803442 container remove b629e6e8dde2be16fc550167320ad7c37eb4a3c633c53e5c3774ec6ae069cb5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T03:06:08.510 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:08 smithi066 podman[71159]: 2024-09-21 03:06:08.129183952 +0000 UTC m=+0.016418168 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:06:08.510 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:08 smithi066 podman[71159]: 2024-09-21 03:06:08.265673379 +0000 UTC m=+0.152907586 container create ddf683973d1d85b19121200ccaa36f50b7e7597aa37cd1b948445b05e6a0340c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T03:06:09.010 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:08 smithi066 podman[71159]: 2024-09-21 03:06:08.5181978 +0000 UTC m=+0.405432011 container init ddf683973d1d85b19121200ccaa36f50b7e7597aa37cd1b948445b05e6a0340c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T03:06:09.010 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:08 smithi066 podman[71159]: 2024-09-21 03:06:08.521528495 +0000 UTC m=+0.408762702 container start ddf683973d1d85b19121200ccaa36f50b7e7597aa37cd1b948445b05e6a0340c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T03:06:09.010 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:08 smithi066 podman[71159]: 2024-09-21 03:06:08.618537539 +0000 UTC m=+0.505771746 container attach ddf683973d1d85b19121200ccaa36f50b7e7597aa37cd1b948445b05e6a0340c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T03:06:09.010 INFO:journalctl@ceph.osd.1.smithi066.stdout:Sep 21 03:06:08 smithi066 podman[71159]: 2024-09-21 03:06:08.720757018 +0000 UTC m=+0.607991228 container died ddf683973d1d85b19121200ccaa36f50b7e7597aa37cd1b948445b05e6a0340c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T03:06:09.378 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.1.service' 2024-09-21T03:06:09.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:06:09.423 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T03:06:09.423 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T03:06:09.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.2 2024-09-21T03:06:09.823 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:09 smithi190 systemd[1]: Stopping Ceph osd.2 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:06:10.124 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:09 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2[38327]: 2024-09-21T03:06:09.821+0000 7f3f02980640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:06:10.124 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:09 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2[38327]: 2024-09-21T03:06:09.821+0000 7f3f02980640 -1 osd.2 90 *** Got signal Terminated *** 2024-09-21T03:06:10.124 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:09 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2[38327]: 2024-09-21T03:06:09.821+0000 7f3f02980640 -1 osd.2 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:06:15.699 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:15 smithi190 podman[71423]: 2024-09-21 03:06:15.416091369 +0000 UTC m=+5.802010534 container died f6d8c94129b77816e831e867e5d1fee08cf33ff54318659e6a4808cfd24f1846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:16.074 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:15 smithi190 podman[71423]: 2024-09-21 03:06:15.797167944 +0000 UTC m=+6.183087122 container cleanup f6d8c94129b77816e831e867e5d1fee08cf33ff54318659e6a4808cfd24f1846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True) 2024-09-21T03:06:16.075 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:15 smithi190 bash[71423]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2 2024-09-21T03:06:16.374 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:16 smithi190 podman[71441]: 2024-09-21 03:06:16.073764411 +0000 UTC m=+0.655077713 container remove f6d8c94129b77816e831e867e5d1fee08cf33ff54318659e6a4808cfd24f1846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True) 2024-09-21T03:06:16.653 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:16 smithi190 podman[71529]: 2024-09-21 03:06:16.284083194 +0000 UTC m=+0.015884199 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:06:16.654 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:16 smithi190 podman[71529]: 2024-09-21 03:06:16.408100818 +0000 UTC m=+0.139901811 container create b4e9010cefc4616c66279c7447fcd138909300ded3091ff2e61e8cd81fe5db39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T03:06:17.124 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:16 smithi190 podman[71529]: 2024-09-21 03:06:16.652466486 +0000 UTC m=+0.384267479 container init b4e9010cefc4616c66279c7447fcd138909300ded3091ff2e61e8cd81fe5db39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T03:06:17.124 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:16 smithi190 podman[71529]: 2024-09-21 03:06:16.656167724 +0000 UTC m=+0.387968719 container start b4e9010cefc4616c66279c7447fcd138909300ded3091ff2e61e8cd81fe5db39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T03:06:17.124 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:16 smithi190 podman[71529]: 2024-09-21 03:06:16.727506543 +0000 UTC m=+0.459307537 container attach b4e9010cefc4616c66279c7447fcd138909300ded3091ff2e61e8cd81fe5db39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T03:06:17.124 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:16 smithi190 podman[71529]: 2024-09-21 03:06:16.863714888 +0000 UTC m=+0.595515895 container died b4e9010cefc4616c66279c7447fcd138909300ded3091ff2e61e8cd81fe5db39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T03:06:17.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.2.service' 2024-09-21T03:06:17.476 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:17 smithi190 podman[71552]: 2024-09-21 03:06:17.432171865 +0000 UTC m=+0.559572600 container remove b4e9010cefc4616c66279c7447fcd138909300ded3091ff2e61e8cd81fe5db39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T03:06:17.476 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:17 smithi190 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-21T03:06:17.476 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:17 smithi190 systemd[1]: Stopped Ceph osd.2 for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T03:06:17.476 INFO:journalctl@ceph.osd.2.smithi190.stdout:Sep 21 03:06:17 smithi190 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.2.service: Consumed 3min 13.363s CPU time. 2024-09-21T03:06:17.493 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:06:17.493 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T03:06:17.494 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T03:06:17.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.3 2024-09-21T03:06:17.826 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:17 smithi029 systemd[1]: Stopping Ceph osd.3 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:06:18.132 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T03:06:17.824+0000 7f4ce895d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:06:18.132 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T03:06:17.824+0000 7f4ce895d640 -1 osd.3 90 *** Got signal Terminated *** 2024-09-21T03:06:18.132 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:17 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T03:06:17.824+0000 7f4ce895d640 -1 osd.3 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:06:20.382 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:19 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T03:06:19.896+0000 7f4ce4f48640 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:56.739295+0000 front 2024-09-21T03:05:56.739229+0000 (oldest deadline 2024-09-21T03:06:19.639384+0000) 2024-09-21T03:06:20.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:20.422+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:20.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:20.314+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:21.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:20 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:20.633+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:21.382 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:20 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T03:06:20.898+0000 7f4ce4f48640 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:56.739295+0000 front 2024-09-21T03:05:56.739229+0000 (oldest deadline 2024-09-21T03:06:19.639384+0000) 2024-09-21T03:06:21.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:21.281+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:21.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:21.454+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:22.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:21 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:21.596+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:22.132 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:21 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3[52782]: 2024-09-21T03:06:21.867+0000 7f4ce4f48640 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:56.739295+0000 front 2024-09-21T03:05:56.739229+0000 (oldest deadline 2024-09-21T03:06:19.639384+0000) 2024-09-21T03:06:22.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:21 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:21.987+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:22.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:22.242+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:22.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:22.471+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:23.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:22 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:22.554+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:23.236 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:22 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:22.945+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:23.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:23.201+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:23.577 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:23 smithi029 podman[117183]: 2024-09-21 03:06:23.235560815 +0000 UTC m=+5.535968324 container died 4f61b525a23b25c242944e907c603172762c96f7e23d1c554e49454cf918de23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T03:06:23.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:23.491+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:23.882 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:23 smithi029 podman[117183]: 2024-09-21 03:06:23.685428429 +0000 UTC m=+5.985835936 container cleanup 4f61b525a23b25c242944e907c603172762c96f7e23d1c554e49454cf918de23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:06:23.882 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:23 smithi029 bash[117183]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3 2024-09-21T03:06:24.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:23 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:23.603+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:24.309 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:23 smithi029 podman[117198]: 2024-09-21 03:06:23.881509001 +0000 UTC m=+0.638126375 container remove 4f61b525a23b25c242944e907c603172762c96f7e23d1c554e49454cf918de23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T03:06:24.309 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:23 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:23.916+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:24.458 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:24.171+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:24.597 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:24 smithi029 podman[117290]: 2024-09-21 03:06:24.210959984 +0000 UTC m=+0.016454820 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:06:24.597 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:24 smithi029 podman[117290]: 2024-09-21 03:06:24.333649888 +0000 UTC m=+0.139144720 container create 71d9ef74f96e6380253afe405b547c8854eebefe1431a6987fbc33902865af61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T03:06:24.597 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:24 smithi029 podman[117290]: 2024-09-21 03:06:24.596362638 +0000 UTC m=+0.401857471 container init 71d9ef74f96e6380253afe405b547c8854eebefe1431a6987fbc33902865af61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T03:06:24.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:24.455+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:24.882 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:24 smithi029 podman[117290]: 2024-09-21 03:06:24.599993687 +0000 UTC m=+0.405488523 container start 71d9ef74f96e6380253afe405b547c8854eebefe1431a6987fbc33902865af61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:06:24.882 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:24 smithi029 podman[117290]: 2024-09-21 03:06:24.706942901 +0000 UTC m=+0.512437735 container attach 71d9ef74f96e6380253afe405b547c8854eebefe1431a6987fbc33902865af61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:06:24.883 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 21 03:06:24 smithi029 podman[117290]: 2024-09-21 03:06:24.795693188 +0000 UTC m=+0.601188022 container died 71d9ef74f96e6380253afe405b547c8854eebefe1431a6987fbc33902865af61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T03:06:25.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:24 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:24.592+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:25.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:24 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:24.921+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:25.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:25.184+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:25.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.3.service' 2024-09-21T03:06:25.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:06:25.563 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T03:06:25.563 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T03:06:25.563 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.4 2024-09-21T03:06:25.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:25.473+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:25.885 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:25 smithi066 systemd[1]: Stopping Ceph osd.4 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:06:25.885 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:25 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:25.601+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:26.260 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:25 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4[43893]: 2024-09-21T03:06:25.882+0000 7ff88b27a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:06:26.260 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:25 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4[43893]: 2024-09-21T03:06:25.883+0000 7ff88b27a640 -1 osd.4 90 *** Got signal Terminated *** 2024-09-21T03:06:26.260 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:25 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4[43893]: 2024-09-21T03:06:25.883+0000 7ff88b27a640 -1 osd.4 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:06:26.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:25 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:25.956+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:26.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:26.176+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:26.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:26.464+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:27.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:26 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:26.563+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:27.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:26 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:26.927+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:27.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:27 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:27.217+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:27.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:27 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:27.500+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:27.874 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:27 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:27.475+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:28.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:27 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:27.604+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:28.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:27 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:27.918+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:28.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:27 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:27.918+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:28.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:28 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:28.090+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:28.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:28 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:28.478+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:28.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:28 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:28.487+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:28.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:28 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:28.245+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:29.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:28 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:28.653+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:29.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:28 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:28.653+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:29.300 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:29 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:29.043+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:29.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:28 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:28.898+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:29.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:28 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:28.898+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:29.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:29 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:29.252+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:29.528 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:29 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:29.442+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:29.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:29 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:29.526+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:30.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:29 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:29.651+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:30.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:29 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:29.651+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:30.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:29 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:29.965+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:30.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:29 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:29.867+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:30.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:29 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:29.867+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:30.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:30 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:30.088+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:30.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:30 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:30.294+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:30.570 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:30 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:30.447+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:30.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:30 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:30.568+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:30.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:30 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:30.568+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:31.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:30 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:30.686+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:31.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:30 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:30.686+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:31.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:30 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:30.980+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:31.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:30 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:30.867+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:31.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:30 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:30.867+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:31.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:31 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:31.072+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:31.510 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:31 smithi066 podman[71261]: 2024-09-21 03:06:31.307060652 +0000 UTC m=+5.540585329 container died 709db2b8fb7ad84efece11604e5cdd023fef05d22a3e904ce65e63983967140f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T03:06:31.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:31 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:31.266+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:31.521 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:31 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:31.449+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:31.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:31 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:31.519+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:31.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:31 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:31.519+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:32.010 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:31 smithi066 podman[71261]: 2024-09-21 03:06:31.658849452 +0000 UTC m=+5.892374129 container cleanup 709db2b8fb7ad84efece11604e5cdd023fef05d22a3e904ce65e63983967140f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T03:06:32.010 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:31 smithi066 bash[71261]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4 2024-09-21T03:06:32.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:31 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:31.667+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:32.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:31 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:31.667+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:32.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:31 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:31.933+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:32.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:31 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:31.832+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:32.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:31 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:31.832+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:32.442 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:32 smithi066 podman[71290]: 2024-09-21 03:06:32.060656011 +0000 UTC m=+0.752053677 container remove 709db2b8fb7ad84efece11604e5cdd023fef05d22a3e904ce65e63983967140f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T03:06:32.443 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:32 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:32.096+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:32.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:32 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:32.283+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:32.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:32 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:32.436+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:32.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:32 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:32.506+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:32.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:32 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:32.506+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:32.723 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:32 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:32.679+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:32.723 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:32 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:32.679+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:32.723 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:32 smithi066 podman[71378]: 2024-09-21 03:06:32.344659483 +0000 UTC m=+0.016705633 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:06:32.724 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:32 smithi066 podman[71378]: 2024-09-21 03:06:32.479492903 +0000 UTC m=+0.151539043 container create 1a091874663b2f5974e9b70e82837f2d46a10714276167d325b17929f688012a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T03:06:33.010 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:32 smithi066 podman[71378]: 2024-09-21 03:06:32.722574146 +0000 UTC m=+0.394620302 container init 1a091874663b2f5974e9b70e82837f2d46a10714276167d325b17929f688012a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T03:06:33.010 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:32 smithi066 podman[71378]: 2024-09-21 03:06:32.72655447 +0000 UTC m=+0.398600607 container start 1a091874663b2f5974e9b70e82837f2d46a10714276167d325b17929f688012a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T03:06:33.010 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:32 smithi066 podman[71378]: 2024-09-21 03:06:32.822382497 +0000 UTC m=+0.494428647 container attach 1a091874663b2f5974e9b70e82837f2d46a10714276167d325b17929f688012a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:33.010 INFO:journalctl@ceph.osd.4.smithi066.stdout:Sep 21 03:06:32 smithi066 podman[71378]: 2024-09-21 03:06:32.920840921 +0000 UTC m=+0.592887068 container died 1a091874663b2f5974e9b70e82837f2d46a10714276167d325b17929f688012a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-4-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T03:06:33.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:32 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:32.911+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:33.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:32 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:32.871+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:33.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:32 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:32.871+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:33.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:33.081+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:33.550 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:33.474+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:33.550 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:33.480+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:33.551 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:33.480+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:33.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:33.294+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:33.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:33.294+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:00.949999+0000 front 2024-09-21T03:06:00.950143+0000 (oldest deadline 2024-09-21T03:06:32.750938+0000) 2024-09-21T03:06:33.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.4.service' 2024-09-21T03:06:33.684 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:06:33.684 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T03:06:33.685 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T03:06:33.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.5 2024-09-21T03:06:33.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:33 smithi190 systemd[1]: Stopping Ceph osd.5 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:06:34.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:33.691+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:34.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:33 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:33.691+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:34.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:33.929+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:34.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:33.829+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:34.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:33 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:33.829+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:34.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:33.976+0000 7f1a35752640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:06:34.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:33.976+0000 7f1a35752640 -1 osd.5 90 *** Got signal Terminated *** 2024-09-21T03:06:34.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:33 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:33.976+0000 7f1a35752640 -1 osd.5 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:06:34.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:34 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:34.263+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:34.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:34 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:34.263+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:00.949999+0000 front 2024-09-21T03:06:00.950143+0000 (oldest deadline 2024-09-21T03:06:32.750938+0000) 2024-09-21T03:06:34.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:34 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:34.110+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:34.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:34 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:34.431+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:34.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:34 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:34.465+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:34.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:34 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:34.465+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:35.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:34 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:34.682+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:35.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:34 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:34.682+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:35.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:34 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:34.914+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:35.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:34 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:34.863+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:35.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:34 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:34.863+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:35.504 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:35 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:35.386+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:35.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:35 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:35.230+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:35.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:35 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:35.230+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:00.949999+0000 front 2024-09-21T03:06:00.950143+0000 (oldest deadline 2024-09-21T03:06:32.750938+0000) 2024-09-21T03:06:35.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:35 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:35.078+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:35.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:35 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:35.502+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:35.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:35 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:35.502+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:36.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:35 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:35.692+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:36.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:35 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:35.692+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:36.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:35 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:35.960+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:36.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:35 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:35.868+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:36.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:35 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:35.868+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:36.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:36 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:36.052+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:36.549 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:36 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:36.365+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:36.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:36 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:36.196+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:36.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:36 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:36.196+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:00.949999+0000 front 2024-09-21T03:06:00.950143+0000 (oldest deadline 2024-09-21T03:06:32.750938+0000) 2024-09-21T03:06:36.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:36 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:36.547+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:36.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:36 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:36.547+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:37.009 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:36 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:36.726+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:37.010 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:36 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:36.726+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:37.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:36 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:36.976+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:37.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:36 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:36.882+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:37.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:36 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:36.882+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:37.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:37 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:37.020+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:37.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:37 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:37.020+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:37.598 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:37 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:37.327+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:37.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:37 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:37.226+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:37.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:37 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:37.226+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:00.949999+0000 front 2024-09-21T03:06:00.950143+0000 (oldest deadline 2024-09-21T03:06:32.750938+0000) 2024-09-21T03:06:37.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:37 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:37.597+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:37.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:37 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:37.597+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:38.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:37 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:37.975+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:38.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:37 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:37.844+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:38.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:37 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:37.844+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:38.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:37 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:37.844+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:38.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:37 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:37.989+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:38.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:37 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:37.989+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:38.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:37 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:37.759+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:38.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:37 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:37.759+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:38.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:38 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:38.297+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:38.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:38 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:38.297+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:38.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:38 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:38.237+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:54.549073+0000 front 2024-09-21T03:05:54.549324+0000 (oldest deadline 2024-09-21T03:06:19.848983+0000) 2024-09-21T03:06:38.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:38 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5[44854]: 2024-09-21T03:06:38.237+0000 7f1a31d3d640 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:00.949999+0000 front 2024-09-21T03:06:00.950143+0000 (oldest deadline 2024-09-21T03:06:32.750938+0000) 2024-09-21T03:06:39.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:38 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:38.633+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:39.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:38 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:38.633+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:39.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:38 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:38.633+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:39.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:38 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:38.970+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:39.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:38 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:38.970+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:39.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:38 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:38.793+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:39.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:38 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:38.793+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:39.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:38 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:38.995+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:39.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:38 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:38.995+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.883157+0000 front 2024-09-21T03:06:12.883139+0000 (oldest deadline 2024-09-21T03:06:38.183092+0000) 2024-09-21T03:06:39.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:38 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:38.892+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:39.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:38 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:38.892+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:39.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:38 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:38.892+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:39.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:39 smithi190 podman[71644]: 2024-09-21 03:06:39.534323754 +0000 UTC m=+5.663367746 container died 49aa38da51f236ee6402f65f5e13e1a46ef4d61de59b7ad1f89fb7a891a2d2cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T03:06:39.603 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:39 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:39.290+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:39.603 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:39 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:39.290+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:39.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:39 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:39.601+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:39.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:39 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:39.601+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:39.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:39 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:39.601+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:40.221 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:39 smithi190 podman[71644]: 2024-09-21 03:06:39.935678465 +0000 UTC m=+6.064722493 container cleanup 49aa38da51f236ee6402f65f5e13e1a46ef4d61de59b7ad1f89fb7a891a2d2cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:06:40.221 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:39 smithi190 bash[71644]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5 2024-09-21T03:06:40.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:39 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:39.984+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:40.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:39 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:39.984+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:40.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:39 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:39.760+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:40.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:39 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:39.760+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:40.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:39 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:39.760+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:40.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:39 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:39.969+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:40.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:39 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:39.969+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.883157+0000 front 2024-09-21T03:06:12.883139+0000 (oldest deadline 2024-09-21T03:06:38.183092+0000) 2024-09-21T03:06:40.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:39 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:39.919+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:40.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:39 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:39.919+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:40.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:39 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:39.919+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:40.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:40 smithi190 podman[71663]: 2024-09-21 03:06:40.220387488 +0000 UTC m=+0.682881793 container remove 49aa38da51f236ee6402f65f5e13e1a46ef4d61de59b7ad1f89fb7a891a2d2cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:40.544 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:40 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:40.328+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:40.544 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:40 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:40.328+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:40.808 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:40 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:40.635+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:40.808 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:40 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:40.636+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:40.808 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:40 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:40.636+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:40.808 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:40 smithi190 podman[71750]: 2024-09-21 03:06:40.446344496 +0000 UTC m=+0.015952889 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:06:40.808 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:40 smithi190 podman[71750]: 2024-09-21 03:06:40.546353037 +0000 UTC m=+0.115961466 container create 2ed860dfc9e146d7cb41ba10fd71228670ab6647e60956b7fa74de3ca3ea1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, ceph=True) 2024-09-21T03:06:40.808 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:40 smithi190 podman[71750]: 2024-09-21 03:06:40.807453408 +0000 UTC m=+0.377061794 container init 2ed860dfc9e146d7cb41ba10fd71228670ab6647e60956b7fa74de3ca3ea1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T03:06:41.124 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:40 smithi190 podman[71750]: 2024-09-21 03:06:40.810831349 +0000 UTC m=+0.380439747 container start 2ed860dfc9e146d7cb41ba10fd71228670ab6647e60956b7fa74de3ca3ea1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T03:06:41.124 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:40 smithi190 podman[71750]: 2024-09-21 03:06:40.907451455 +0000 UTC m=+0.477059861 container attach 2ed860dfc9e146d7cb41ba10fd71228670ab6647e60956b7fa74de3ca3ea1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T03:06:41.124 INFO:journalctl@ceph.osd.5.smithi190.stdout:Sep 21 03:06:41 smithi190 podman[71750]: 2024-09-21 03:06:41.000993935 +0000 UTC m=+0.570602323 container died 2ed860dfc9e146d7cb41ba10fd71228670ab6647e60956b7fa74de3ca3ea1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:41.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:40 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:40.976+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:41.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:40 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:40.976+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:41.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:40 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:40.768+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:41.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:40 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:40.768+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:41.261 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:40 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:40.768+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:41.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:40.981+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:41.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:40.981+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.883157+0000 front 2024-09-21T03:06:12.883139+0000 (oldest deadline 2024-09-21T03:06:38.183092+0000) 2024-09-21T03:06:41.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:40.907+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:41.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:40.907+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:41.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:40 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:40.907+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:41.596 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:41 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:41.310+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:41.596 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:41 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:41.310+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:41.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.5.service' 2024-09-21T03:06:41.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:06:41.649 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T03:06:41.649 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T03:06:41.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.6 2024-09-21T03:06:41.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:41 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:41.649+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:41.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:41 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:41.649+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:41.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:41 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:41.649+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:41.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:41 smithi029 systemd[1]: Stopping Ceph osd.6 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:06:42.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:41 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:41.938+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:42.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:41 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:41.938+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:42.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:41 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:41.815+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:42.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:41 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:41.815+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:42.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:41 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:41.815+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:42.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:41 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:41.956+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:42.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:41 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:41.956+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:42.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:41 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:41.956+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:42.383 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:41 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:41.971+0000 7ff4aa7c3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:06:42.383 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:41 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:41.971+0000 7ff4aa7c3640 -1 osd.6 90 *** Got signal Terminated *** 2024-09-21T03:06:42.383 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:41 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:41.971+0000 7ff4aa7c3640 -1 osd.6 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:06:42.383 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:42.030+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:42.383 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:42.030+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.883157+0000 front 2024-09-21T03:06:12.883139+0000 (oldest deadline 2024-09-21T03:06:38.183092+0000) 2024-09-21T03:06:42.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:42 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:42.325+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:42.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:42 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:42.325+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:43.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:42 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:42.659+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:43.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:42 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:42.659+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:43.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:42 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:42.659+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:43.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:42 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:42.936+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:43.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:42 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:42.936+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:43.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:42 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:42.847+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:43.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:42 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:42.847+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:43.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:42 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:42.847+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:43.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:42.995+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:43.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:42.995+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.883157+0000 front 2024-09-21T03:06:12.883139+0000 (oldest deadline 2024-09-21T03:06:38.183092+0000) 2024-09-21T03:06:43.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:42.937+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:43.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:42.937+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:43.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:42.937+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:43.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:43 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:43.331+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:43.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:43 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:43.331+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:43.953 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:43 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:43.897+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:43.953 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:43 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:43.897+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:43.953 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:43 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:43.897+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:44.023 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:43 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:43.900+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:44.023 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:43 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:43.900+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:44.023 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:43 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:43.900+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:44.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:43 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:43.665+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:44.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:43 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:43.665+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:44.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:43 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:43.665+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:44.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:43 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:43.951+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:44.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:43 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:43.951+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:44.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:44 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:44.022+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:44.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:44 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:44.022+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.883157+0000 front 2024-09-21T03:06:12.883139+0000 (oldest deadline 2024-09-21T03:06:38.183092+0000) 2024-09-21T03:06:44.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:44 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:44.330+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:44.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:44 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:44.330+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:45.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:44 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:44.694+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:45.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:44 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:44.694+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:45.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:44 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:44.694+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:45.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:44 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:44.979+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:45.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:44 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:44.979+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:45.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:44 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:44.932+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:45.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:44 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:44.932+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:45.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:44 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:44.932+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:45.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:45 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:45.001+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:45.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:45 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:45.001+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.883157+0000 front 2024-09-21T03:06:12.883139+0000 (oldest deadline 2024-09-21T03:06:38.183092+0000) 2024-09-21T03:06:45.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:44 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:44.907+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:45.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:44 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:44.907+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:45.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:44 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:44.907+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:45.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:44 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:44.907+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:45.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:45 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:45.320+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:45.719 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:45 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:45.320+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:45.719 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:45 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:45.320+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:46.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:45 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:45.679+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:46.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:45 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:45.679+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:46.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:45 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:45.679+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:46.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:46.011+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:46.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:46.011+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:46.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:45 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:45.919+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:46.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:45 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:45.919+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:46.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:45 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:45.919+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:46.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:46 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:46.030+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:06.581238+0000 front 2024-09-21T03:06:06.581059+0000 (oldest deadline 2024-09-21T03:06:29.381760+0000) 2024-09-21T03:06:46.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:46 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6[59085]: 2024-09-21T03:06:46.030+0000 7ff4a6dae640 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.883157+0000 front 2024-09-21T03:06:12.883139+0000 (oldest deadline 2024-09-21T03:06:38.183092+0000) 2024-09-21T03:06:46.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:45 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:45.884+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:46.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:45 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:45.884+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:46.382 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:45 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:45.884+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:46.383 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:45 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:45.884+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:46.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:46 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:46.323+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:46.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:46 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:46.323+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:46.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:46 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:46.323+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:47.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:46 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:46.682+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:47.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:46 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:46.682+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:47.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:46 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:46.682+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:47.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:46 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:46.871+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:47.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:46 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:46.871+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:47.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:46 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:46.871+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:47.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:46 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:46.871+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:47.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:47 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:47.056+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:47.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:47 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:47.056+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:47.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:46.919+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:47.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:46.919+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:47.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:46.919+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:47.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:46.919+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:47.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:47 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:47.308+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:47.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:47 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:47.308+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:47.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:47 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:47.308+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:47.632 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:47 smithi029 podman[117392]: 2024-09-21 03:06:47.33425222 +0000 UTC m=+5.486324870 container died 9ed1f1be55f70be1530c8651b3e2d260dc6ddac4db04644058832859f7495d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:48.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:47 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:47.642+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:48.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:47 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:47.642+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:48.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:47 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:47.642+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:48.133 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:47 smithi029 podman[117392]: 2024-09-21 03:06:47.85157253 +0000 UTC m=+6.003645107 container cleanup 9ed1f1be55f70be1530c8651b3e2d260dc6ddac4db04644058832859f7495d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:06:48.133 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:47 smithi029 bash[117392]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6 2024-09-21T03:06:48.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:47 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:47.822+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:48.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:47 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:47.822+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:48.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:47 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:47.822+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:48.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:47 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:47.822+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:48.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:48.098+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:48.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:48.098+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:48.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:48.098+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:22.326973+0000 front 2024-09-21T03:06:22.326998+0000 (oldest deadline 2024-09-21T03:06:47.627143+0000) 2024-09-21T03:06:48.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:47 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:47.946+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:48.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:47 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:47.946+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:48.261 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:47 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:47.946+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:48.261 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:47 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:47.946+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:48.601 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:48 smithi029 podman[117421]: 2024-09-21 03:06:48.16308911 +0000 UTC m=+0.820464081 container remove 9ed1f1be55f70be1530c8651b3e2d260dc6ddac4db04644058832859f7495d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T03:06:48.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:48 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:48.321+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:48.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:48 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:48.321+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:48.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:48 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:48.321+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:48.882 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:48 smithi029 podman[117509]: 2024-09-21 03:06:48.502965652 +0000 UTC m=+0.016424943 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:06:48.882 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:48 smithi029 podman[117509]: 2024-09-21 03:06:48.616458842 +0000 UTC m=+0.129918132 container create cc06abeac72f4ca4e6dccea949d82a82e70ab802c6c9234134c9ad6cbafbebe8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T03:06:48.882 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:48 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:48.782+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:48.882 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:48 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:48.782+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:48.883 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:48 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:48.782+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:48.883 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:48 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:48.782+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:49.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:48 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:48.685+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:49.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:48 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:48.685+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:49.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:48 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:48.685+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:49.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:49 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:49.105+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:49.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:49 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:49.105+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:49.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:49 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:49.105+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:22.326973+0000 front 2024-09-21T03:06:22.326998+0000 (oldest deadline 2024-09-21T03:06:47.627143+0000) 2024-09-21T03:06:49.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:48.920+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:49.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:48.920+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:49.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:48.920+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:49.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:48 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:48.920+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:49.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:48 smithi029 podman[117509]: 2024-09-21 03:06:48.887436956 +0000 UTC m=+0.400896247 container init cc06abeac72f4ca4e6dccea949d82a82e70ab802c6c9234134c9ad6cbafbebe8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T03:06:49.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:48 smithi029 podman[117509]: 2024-09-21 03:06:48.891314342 +0000 UTC m=+0.404773633 container start cc06abeac72f4ca4e6dccea949d82a82e70ab802c6c9234134c9ad6cbafbebe8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:49.382 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:48 smithi029 podman[117509]: 2024-09-21 03:06:48.979823263 +0000 UTC m=+0.493282562 container attach cc06abeac72f4ca4e6dccea949d82a82e70ab802c6c9234134c9ad6cbafbebe8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T03:06:49.383 INFO:journalctl@ceph.osd.6.smithi029.stdout:Sep 21 03:06:49 smithi029 podman[117509]: 2024-09-21 03:06:49.084564021 +0000 UTC m=+0.598023325 container died cc06abeac72f4ca4e6dccea949d82a82e70ab802c6c9234134c9ad6cbafbebe8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T03:06:49.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:49 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:49.299+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:49.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:49 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:49.299+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:49.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:49 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:49.299+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:49.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.6.service' 2024-09-21T03:06:49.860 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:06:49.860 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T03:06:49.860 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T03:06:49.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.7 2024-09-21T03:06:50.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:49 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:49.734+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:50.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:49 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:49.734+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:50.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:49 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:49.734+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:50.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:49 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:49.792+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:50.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:49 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:49.792+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:50.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:49 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:49.792+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:50.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:49 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:49.792+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:50.184 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:49 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:49.881+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:50.184 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:49 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:49.881+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:50.184 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:49 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:49.881+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:50.184 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:49 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:49.881+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:50.185 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:49 smithi066 systemd[1]: Stopping Ceph osd.7 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:06:50.185 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:50.129+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:50.185 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:50.129+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:50.185 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:50.129+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:22.326973+0000 front 2024-09-21T03:06:22.326998+0000 (oldest deadline 2024-09-21T03:06:47.627143+0000) 2024-09-21T03:06:50.185 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:50.183+0000 7f6a6b600640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:06:50.185 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:50.183+0000 7f6a6b600640 -1 osd.7 90 *** Got signal Terminated *** 2024-09-21T03:06:50.185 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:50.183+0000 7f6a6b600640 -1 osd.7 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:06:50.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:50 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:50.269+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:50.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:50 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:50.269+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:50.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:50 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:50.269+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:51.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:50 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:50.713+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:51.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:50 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:50.713+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:51.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:50 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:50.713+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:51.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:50 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:50.769+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:51.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:50 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:50.769+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:51.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:50 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:50.769+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:51.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:50 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:50.769+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:51.150 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:50.863+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:51.150 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:50.863+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:51.150 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:50.863+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:51.150 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:50.863+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:51.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:51 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:51.149+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:51.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:51 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:51.149+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:51.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:51 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:51.149+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:22.326973+0000 front 2024-09-21T03:06:22.326998+0000 (oldest deadline 2024-09-21T03:06:47.627143+0000) 2024-09-21T03:06:51.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:51 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:51.280+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:51.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:51 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:51.280+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:51.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:51 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:51.280+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:52.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:51 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:51.741+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:52.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:51 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:51.741+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:52.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:51 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:51.741+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:52.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:51 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:51.741+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:06:52.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:51 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:51.794+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:52.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:51 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:51.794+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:52.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:51 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:51.794+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:52.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:51 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:51.794+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:52.163 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:51 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:51.815+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:52.163 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:51 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:51.815+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:52.163 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:51 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:51.815+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:52.163 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:51 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:51.815+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:52.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:52.161+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:52.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:52.161+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:52.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:52.161+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:22.326973+0000 front 2024-09-21T03:06:22.326998+0000 (oldest deadline 2024-09-21T03:06:47.627143+0000) 2024-09-21T03:06:52.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:52.161+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.627594+0000 front 2024-09-21T03:06:27.627569+0000 (oldest deadline 2024-09-21T03:06:51.628414+0000) 2024-09-21T03:06:52.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:52 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:52.276+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:52.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:52 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:52.276+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:52.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:52 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:52.276+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:53.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:52 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:52.749+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:53.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:52 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:52.749+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:53.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:52 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:52.749+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:53.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:52 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:52.749+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:06:53.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:52 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:52.744+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:53.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:52 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:52.744+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:53.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:52 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:52.744+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:53.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:52 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:52.744+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:53.197 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:52.817+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:53.197 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:52.817+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:53.197 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:52.817+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:53.197 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:52.817+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:53.197 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:52 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:52.817+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:06:53.399 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:53 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:53.256+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:53.399 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:53 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:53.256+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:53.399 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:53 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:53.256+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:53.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:53.196+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:53.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:53.196+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:53.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:53.196+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:22.326973+0000 front 2024-09-21T03:06:22.326998+0000 (oldest deadline 2024-09-21T03:06:47.627143+0000) 2024-09-21T03:06:53.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:53.196+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.627594+0000 front 2024-09-21T03:06:27.627569+0000 (oldest deadline 2024-09-21T03:06:51.628414+0000) 2024-09-21T03:06:54.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:53 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:53.740+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:54.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:53 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:53.740+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:54.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:53 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:53.740+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:54.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:53 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:53.740+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:06:54.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:53 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:53.703+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:54.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:53 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:53.703+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:54.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:53 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:53.703+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:54.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:53 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:53.703+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:54.156 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:53.853+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:54.156 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:53.853+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:54.156 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:53.853+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:54.156 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:53.853+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:54.156 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:53 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:53.853+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:06:54.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:54.155+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:54.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:54.155+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:54.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:54.155+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:22.326973+0000 front 2024-09-21T03:06:22.326998+0000 (oldest deadline 2024-09-21T03:06:47.627143+0000) 2024-09-21T03:06:54.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:54.155+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.627594+0000 front 2024-09-21T03:06:27.627569+0000 (oldest deadline 2024-09-21T03:06:51.628414+0000) 2024-09-21T03:06:54.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:54.250+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:54.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:54.250+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:54.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:54.250+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:54.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:54.250+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:06:55.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:54.751+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:55.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:54.751+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:55.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:54.751+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:55.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:54 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:54.751+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:06:55.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:54 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:54.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:55.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:54 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:54.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:55.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:54 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:54.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:55.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:54 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:54.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:55.179 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:54.848+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:55.179 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:54.848+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:55.179 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:54.848+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:55.179 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:54.848+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:55.179 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:54.848+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:06:55.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:55.178+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:03.623955+0000 front 2024-09-21T03:06:03.623989+0000 (oldest deadline 2024-09-21T03:06:27.724143+0000) 2024-09-21T03:06:55.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:55.178+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:13.025207+0000 front 2024-09-21T03:06:13.025094+0000 (oldest deadline 2024-09-21T03:06:36.525239+0000) 2024-09-21T03:06:55.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:55.178+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:22.326973+0000 front 2024-09-21T03:06:22.326998+0000 (oldest deadline 2024-09-21T03:06:47.627143+0000) 2024-09-21T03:06:55.510 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7[50482]: 2024-09-21T03:06:55.178+0000 7f6a63be3640 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.627594+0000 front 2024-09-21T03:06:27.627569+0000 (oldest deadline 2024-09-21T03:06:51.628414+0000) 2024-09-21T03:06:55.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:55 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:55.282+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:55.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:55 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:55.282+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:55.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:55 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:55.282+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:55.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:55 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:55.282+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:06:55.882 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:55 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:55.621+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:55.882 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:55 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:55.621+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:55.882 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:55 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:55.621+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:55.883 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:55 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:55.621+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:55.998 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:55 smithi066 podman[71481]: 2024-09-21 03:06:55.717349676 +0000 UTC m=+5.658469395 container died a200deb3f2115806abb46a46bdec0d822e7f087fad39df249b220c58378852bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T03:06:55.998 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:55.874+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:55.998 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:55.874+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:55.998 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:55.874+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:55.998 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:55.874+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:55.998 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:55 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:55.874+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:06:56.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:55 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:55.723+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:56.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:55 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:55.723+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:56.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:55 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:55.723+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:56.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:55 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:55.723+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:06:56.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:56 smithi066 podman[71481]: 2024-09-21 03:06:56.087606994 +0000 UTC m=+6.028726708 container cleanup a200deb3f2115806abb46a46bdec0d822e7f087fad39df249b220c58378852bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:56.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:56 smithi066 bash[71481]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7 2024-09-21T03:06:56.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:56 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:56.292+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:56.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:56 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:56.292+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:56.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:56 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:56.292+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:56.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:56 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:56.292+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:06:56.714 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:56 smithi066 podman[71495]: 2024-09-21 03:06:56.347118984 +0000 UTC m=+0.624901350 container remove a200deb3f2115806abb46a46bdec0d822e7f087fad39df249b220c58378852bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T03:06:56.968 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:56 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:56.921+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:56.968 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:56 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:56.921+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:56.968 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:56 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:56.921+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:56.969 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:56 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:56.921+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:56.969 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:56 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:56.921+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:06:56.969 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:56 smithi066 podman[71585]: 2024-09-21 03:06:56.616782595 +0000 UTC m=+0.016578293 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:06:56.969 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:56 smithi066 podman[71585]: 2024-09-21 03:06:56.739523468 +0000 UTC m=+0.139319197 container create 814aa5e73de474764f93b438ff43eafd92a0fb40b7d1d465b819d0ded2924002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:06:56.969 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:56 smithi066 podman[71585]: 2024-09-21 03:06:56.967635838 +0000 UTC m=+0.367431530 container init 814aa5e73de474764f93b438ff43eafd92a0fb40b7d1d465b819d0ded2924002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T03:06:57.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:56 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:56.756+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:57.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:56 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:56.756+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:57.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:56 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:56.756+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:57.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:56 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:56.756+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:06:57.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:56 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:56.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:57.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:56 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:56.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:57.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:56 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:56.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:57.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:56 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:56.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:57.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:56 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:56.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:06:57.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:56 smithi066 podman[71585]: 2024-09-21 03:06:56.971987186 +0000 UTC m=+0.371782876 container start 814aa5e73de474764f93b438ff43eafd92a0fb40b7d1d465b819d0ded2924002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7-deactivate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-21T03:06:57.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:57 smithi066 podman[71585]: 2024-09-21 03:06:57.042151485 +0000 UTC m=+0.441947187 container attach 814aa5e73de474764f93b438ff43eafd92a0fb40b7d1d465b819d0ded2924002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:06:57.260 INFO:journalctl@ceph.osd.7.smithi066.stdout:Sep 21 03:06:57 smithi066 podman[71585]: 2024-09-21 03:06:57.167828451 +0000 UTC m=+0.567624151 container died 814aa5e73de474764f93b438ff43eafd92a0fb40b7d1d465b819d0ded2924002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T03:06:57.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:57 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:57.299+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:57.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:57 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:57.299+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:57.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:57 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:57.299+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:57.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:57 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:57.299+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:06:57.884 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.7.service' 2024-09-21T03:06:57.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:06:57.926 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T03:06:57.926 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-21T03:06:57.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.8 2024-09-21T03:06:58.124 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:57 smithi190 systemd[1]: Stopping Ceph osd.8 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:06:58.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:57 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:57.773+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:58.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:57 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:57.773+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:58.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:57 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:57.773+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:58.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:57 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:57.773+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:06:58.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:57 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:57.677+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:58.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:57 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:57.677+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:58.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:57 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:57.677+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:58.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:57 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:57.677+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:58.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:57 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:57.677+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:06:58.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:57 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:57.956+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:58.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:57 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:57.956+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:58.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:57 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:57.956+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:58.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:57 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:57.956+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:58.261 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:57 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:57.956+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:06:58.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:58.214+0000 7fd8602cf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:06:58.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:58.214+0000 7fd8602cf640 -1 osd.8 90 *** Got signal Terminated *** 2024-09-21T03:06:58.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:58.214+0000 7fd8602cf640 -1 osd.8 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:06:58.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:58.256+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:58.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:58.256+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:58.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:58.256+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:58.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:58.256+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:06:59.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:58.768+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:06:59.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:58.768+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:06:59.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:58.768+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:06:59.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:58 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:58.768+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:06:59.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:58 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:58.648+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:06:59.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:58 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:58.648+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:06:59.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:58 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:58.648+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:06:59.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:58 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:58.648+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:06:59.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:58 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:58.648+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:06:59.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:58 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:58.992+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:06:59.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:58 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:58.992+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:06:59.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:58 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:58.992+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:06:59.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:58 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:58.992+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:06:59.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:06:58 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:06:58.992+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:06:59.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:59 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:59.270+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:06:59.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:59 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:59.270+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:06:59.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:59 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:59.270+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:06:59.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:06:59 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:06:59.270+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:07:00.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:59 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:59.791+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:00.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:59 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:59.791+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:00.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:59 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:59.791+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:00.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:06:59 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:06:59.791+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:00.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:59 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:59.652+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:00.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:59 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:59.652+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:00.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:59 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:59.652+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:00.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:59 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:59.652+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:00.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:06:59 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:06:59.652+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:00.464 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.009+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:00.464 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.009+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:00.464 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.009+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:00.464 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.009+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:00.464 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.009+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:00.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:00 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:00.313+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:07:00.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:00 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:00.313+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:07:00.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:00 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:00.313+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:07:00.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:00 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:00.313+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:07:01.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:00 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:00.821+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:01.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:00 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:00.821+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:01.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:00 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:00.821+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:01.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:00 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:00.821+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:01.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:00 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:00.655+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:01.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:00 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:00.655+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:01.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:00 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:00.655+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:01.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:00 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:00.655+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:01.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:00 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:00.655+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:01.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.975+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:01.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.975+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:01.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.975+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:01.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.975+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:01.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:00 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:00.975+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:01.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:01.264+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:07:01.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:01.264+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:07:01.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:01.264+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:07:01.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:01.264+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:07:02.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:01.787+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:02.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:01.787+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:02.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:01.787+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:02.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:01.787+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:02.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:01 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:01.787+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:02.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:01 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:01.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:02.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:01 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:01.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:02.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:01 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:01.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:02.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:01 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:01.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:02.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:01 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:01.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:02.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:01 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:01.659+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:02.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:02 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:02.022+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:02.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:02 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:02.022+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:02.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:02 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:02.022+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:02.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:02 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:02.022+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:02.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:02 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:02.022+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:02.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:02.251+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.880352+0000 front 2024-09-21T03:06:04.880446+0000 (oldest deadline 2024-09-21T03:06:27.180533+0000) 2024-09-21T03:07:02.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:02.251+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.481300+0000 front 2024-09-21T03:06:12.481478+0000 (oldest deadline 2024-09-21T03:06:37.781549+0000) 2024-09-21T03:07:02.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:02.251+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:19.383082+0000 front 2024-09-21T03:06:19.383069+0000 (oldest deadline 2024-09-21T03:06:45.282977+0000) 2024-09-21T03:07:02.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8[51146]: 2024-09-21T03:07:02.251+0000 7fd85c8ba640 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:28.784119+0000 front 2024-09-21T03:06:28.784073+0000 (oldest deadline 2024-09-21T03:06:54.084203+0000) 2024-09-21T03:07:03.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:02.826+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:03.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:02.826+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:03.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:02.826+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:03.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:02.826+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:03.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:02.826+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:03.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:02 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:02.685+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:03.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:02 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:02.685+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:03.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:02 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:02.685+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:03.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:02 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:02.685+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:03.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:02 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:02.685+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:03.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:02 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:02.685+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:03.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:03 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:03.019+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:03.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:03 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:03.019+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:03.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:03 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:03.019+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:03.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:03 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:03.019+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:03.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:03 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:03.019+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:03.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:03 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:03.019+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:03.949 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:03 smithi190 podman[71865]: 2024-09-21 03:07:03.689498891 +0000 UTC m=+5.567636146 container died 4d428e62ab432db29ada15e17852131c430d9a4f306732caf2437dfb7e257744 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T03:07:03.949 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:03 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:03.796+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:03.949 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:03 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:03.796+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:03.949 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:03 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:03.796+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:03.949 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:03 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:03.796+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:03.949 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:03 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:03.796+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:04.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:03 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:03.699+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:04.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:03 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:03.699+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:04.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:03 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:03.699+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:04.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:03 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:03.699+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:04.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:03 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:03.699+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:04.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:03 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:03.699+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:04.341 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:04 smithi190 podman[71865]: 2024-09-21 03:07:04.047482354 +0000 UTC m=+5.925619608 container cleanup 4d428e62ab432db29ada15e17852131c430d9a4f306732caf2437dfb7e257744 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD) 2024-09-21T03:07:04.342 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:04 smithi190 bash[71865]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8 2024-09-21T03:07:04.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.018+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:04.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.018+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:04.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.018+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:04.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.018+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:04.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.018+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:04.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.018+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:04.624 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:04 smithi190 podman[71882]: 2024-09-21 03:07:04.340793309 +0000 UTC m=+0.652234848 container remove 4d428e62ab432db29ada15e17852131c430d9a4f306732caf2437dfb7e257744 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T03:07:04.911 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:04 smithi190 podman[71971]: 2024-09-21 03:07:04.5584037 +0000 UTC m=+0.015779689 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:07:04.911 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:04 smithi190 podman[71971]: 2024-09-21 03:07:04.666719698 +0000 UTC m=+0.124095689 container create e006a2e649fe67ed0dff293347077dfff92bc61a24b86e2b6aa3dd5feb12a71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8-deactivate, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:07:04.911 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:04 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:04.770+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:04.911 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:04 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:04.770+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:04.911 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:04 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:04.770+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:04.911 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:04 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:04.770+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:04.912 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:04 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:04.770+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:05.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:04.724+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:05.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:04.724+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:05.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:04.724+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:05.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:04.724+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:05.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:04.724+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:05.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:04 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:04.724+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:05.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.972+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:05.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.972+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:05.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.972+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:05.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.972+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:05.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.972+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:05.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:04 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:04.972+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:05.370 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:04 smithi190 podman[71971]: 2024-09-21 03:07:04.910619644 +0000 UTC m=+0.367995635 container init e006a2e649fe67ed0dff293347077dfff92bc61a24b86e2b6aa3dd5feb12a71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T03:07:05.370 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:04 smithi190 podman[71971]: 2024-09-21 03:07:04.913996097 +0000 UTC m=+0.371372095 container start e006a2e649fe67ed0dff293347077dfff92bc61a24b86e2b6aa3dd5feb12a71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T03:07:05.370 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:05 smithi190 podman[71971]: 2024-09-21 03:07:05.027447062 +0000 UTC m=+0.484823066 container attach e006a2e649fe67ed0dff293347077dfff92bc61a24b86e2b6aa3dd5feb12a71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:07:05.370 INFO:journalctl@ceph.osd.8.smithi190.stdout:Sep 21 03:07:05 smithi190 podman[71971]: 2024-09-21 03:07:05.096872014 +0000 UTC m=+0.554248011 container died e006a2e649fe67ed0dff293347077dfff92bc61a24b86e2b6aa3dd5feb12a71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-09-21T03:07:05.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.8.service' 2024-09-21T03:07:05.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:05.792 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-21T03:07:05.792 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-21T03:07:05.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.9 2024-09-21T03:07:06.097 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:05 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:05.728+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:06.097 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:05 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:05.728+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:06.097 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:05 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:05.728+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:06.097 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:05 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:05.728+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:06.097 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:05 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:05.728+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:06.097 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:05 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:05.728+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:06.097 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:05 smithi029 systemd[1]: Stopping Ceph osd.9 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:06.097 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.094+0000 7f7c11542640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:06.098 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.094+0000 7f7c11542640 -1 osd.9 90 *** Got signal Terminated *** 2024-09-21T03:07:06.098 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.094+0000 7f7c11542640 -1 osd.9 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:07:06.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:05 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:05.808+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:06.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:05 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:05.808+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:06.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:05 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:05.808+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:06.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:05 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:05.808+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:06.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:05 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:05.808+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:06.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:05 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:05.934+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:06.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:05 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:05.934+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:06.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:05 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:05.934+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:06.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:05 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:05.934+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:06.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:05 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:05.934+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:06.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:05 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:05.934+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:07.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:06 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:06.799+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:07.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:06 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:06.799+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:07.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:06 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:06.799+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:07.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:06 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:06.799+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:07.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:06 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:06.799+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:07.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.752+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:07.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.752+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:07.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.752+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:07.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.752+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:07.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.752+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:07.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:06 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:06.752+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:07.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:06 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:06.940+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:07.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:06 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:06.940+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:07.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:06 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:06.940+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:07.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:06 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:06.940+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:07.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:06 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:06.940+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:07.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:06 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:06.940+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:08.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:07 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:07.761+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:08.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:07 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:07.761+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:08.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:07 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:07.761+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:08.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:07 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:07.761+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:08.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:07 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:07.761+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:08.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:07 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:07.705+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:08.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:07 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:07.705+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:08.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:07 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:07.705+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:08.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:07 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:07.705+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:08.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:07 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:07.705+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:08.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:07 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:07.705+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:08.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:07.947+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:08.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:07.947+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:08.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:07.947+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:08.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:07.947+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:08.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:07.947+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:08.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:07.947+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:08.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:07 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:07.947+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:09.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:08 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:08.793+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:09.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:08 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:08.793+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:09.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:08 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:08.793+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:09.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:08 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:08.793+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:09.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:08 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:08.793+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:09.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:08 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:08.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:09.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:08 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:08.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:09.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:08 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:08.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:09.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:08 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:08.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:09.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:08 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:08.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:09.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:08 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:08.665+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:09.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:08 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:08.971+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:09.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:08 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:08.971+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:09.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:08 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:08.971+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:09.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:08 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:08.971+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:09.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:08 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:08.971+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:09.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:08 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:08.971+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:09.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:08 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:08.971+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:09.882 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:09 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:09.618+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:09.882 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:09 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:09.618+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:09.883 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:09 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:09.618+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:09.883 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:09 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:09.618+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:09.883 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:09 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:09.618+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:09.883 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:09 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:09.618+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:10.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:09 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:09.760+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:10.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:09 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:09.760+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:10.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:09 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:09.760+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:10.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:09 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:09.760+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:10.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:09 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:09.760+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:10.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:09 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:09.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:10.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:09 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:09.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:10.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:09 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:09.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:10.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:09 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:09.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:10.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:09 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:09.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:10.261 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:09 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:09.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:10.261 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:09 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:09.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:11.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:10 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:10.805+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:11.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:10 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:10.805+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:11.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:10 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:10.805+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:11.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:10 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:10.805+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:11.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:10 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:10.806+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:11.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:10 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:10.650+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:58.432465+0000 front 2024-09-21T03:05:58.432440+0000 (oldest deadline 2024-09-21T03:06:21.833504+0000) 2024-09-21T03:07:11.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:10 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:10.650+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:01.834070+0000 front 2024-09-21T03:06:01.834023+0000 (oldest deadline 2024-09-21T03:06:27.734194+0000) 2024-09-21T03:07:11.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:10 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:10.650+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:14.735927+0000 front 2024-09-21T03:06:14.736022+0000 (oldest deadline 2024-09-21T03:06:37.635831+0000) 2024-09-21T03:07:11.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:10 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:10.650+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.736995+0000 front 2024-09-21T03:06:21.737047+0000 (oldest deadline 2024-09-21T03:06:44.637105+0000) 2024-09-21T03:07:11.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:10 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:10.650+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:30.538335+0000 front 2024-09-21T03:06:30.538313+0000 (oldest deadline 2024-09-21T03:06:55.838141+0000) 2024-09-21T03:07:11.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:10 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9[65702]: 2024-09-21T03:07:10.650+0000 7f7c0db2d640 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.539978+0000 front 2024-09-21T03:06:38.540033+0000 (oldest deadline 2024-09-21T03:07:01.439941+0000) 2024-09-21T03:07:11.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:11 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:11.014+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:11.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:11 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:11.014+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:11.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:11 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:11.014+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:11.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:11 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:11.014+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:11.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:11 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:11.014+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:11.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:11 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:11.014+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:11.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:11 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:11.014+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:11.789 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:11 smithi029 podman[117612]: 2024-09-21 03:07:11.466017434 +0000 UTC m=+5.476205847 container died 72e6208f77f4099782e970e8880443faa70d54fb07e1e2871c1e42d9a439a7fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:07:12.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:11 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:11.763+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:12.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:11 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:11.763+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:12.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:11 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:11.763+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:12.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:11 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:11.763+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:12.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:11 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:11.763+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:12.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:11 smithi029 podman[117612]: 2024-09-21 03:07:11.964623375 +0000 UTC m=+5.974811776 container cleanup 72e6208f77f4099782e970e8880443faa70d54fb07e1e2871c1e42d9a439a7fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T03:07:12.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:11 smithi029 bash[117612]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9 2024-09-21T03:07:12.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:12 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:12.050+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:12.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:12 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:12.050+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:12.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:12 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:12.050+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:12.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:12 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:12.050+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:12.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:12 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:12.050+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:12.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:12 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:12.050+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:12.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:12 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:12.050+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:12.578 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:12 smithi029 podman[117630]: 2024-09-21 03:07:12.260560007 +0000 UTC m=+0.783110985 container remove 72e6208f77f4099782e970e8880443faa70d54fb07e1e2871c1e42d9a439a7fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:07:12.850 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:12 smithi029 podman[117717]: 2024-09-21 03:07:12.480993516 +0000 UTC m=+0.016406931 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:07:12.850 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:12 smithi029 podman[117717]: 2024-09-21 03:07:12.603628971 +0000 UTC m=+0.139042413 container create 46cc1c4dd7eac48498f2202d19adb00868624103338091ed034a3908abf0ac3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T03:07:12.850 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:12 smithi029 podman[117717]: 2024-09-21 03:07:12.849443904 +0000 UTC m=+0.384857311 container init 46cc1c4dd7eac48498f2202d19adb00868624103338091ed034a3908abf0ac3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T03:07:13.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:12 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:12.797+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:13.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:12 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:12.797+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:13.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:12 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:12.797+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:13.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:12 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:12.797+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:13.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:12 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:12.797+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:13.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:12 smithi029 podman[117717]: 2024-09-21 03:07:12.852774171 +0000 UTC m=+0.388187570 container start 46cc1c4dd7eac48498f2202d19adb00868624103338091ed034a3908abf0ac3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9-deactivate, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T03:07:13.132 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:12 smithi029 podman[117717]: 2024-09-21 03:07:12.949720725 +0000 UTC m=+0.485134137 container attach 46cc1c4dd7eac48498f2202d19adb00868624103338091ed034a3908abf0ac3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T03:07:13.133 INFO:journalctl@ceph.osd.9.smithi029.stdout:Sep 21 03:07:13 smithi029 podman[117717]: 2024-09-21 03:07:13.046450801 +0000 UTC m=+0.581864213 container died 46cc1c4dd7eac48498f2202d19adb00868624103338091ed034a3908abf0ac3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-9-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2) 2024-09-21T03:07:13.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:13 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:13.062+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:13.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:13 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:13.062+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:13.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:13 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:13.062+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:13.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:13 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:13.062+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:13.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:13 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:13.062+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:13.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:13 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:13.062+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:13.511 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:13 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:13.062+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:13.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.9.service' 2024-09-21T03:07:13.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:13.788 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-21T03:07:13.788 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-21T03:07:13.788 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.10 2024-09-21T03:07:14.094 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:13 smithi066 systemd[1]: Stopping Ceph osd.10 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:14.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:13 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:13.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:14.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:13 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:13.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:14.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:13 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:13.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:14.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:13 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:13.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:14.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:13 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:13.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:14.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:13 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:13.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:14.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:14.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:14.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:14.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:14.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:14.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:14.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:14.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.098+0000 7f923c2d3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:14.511 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.098+0000 7f923c2d3640 -1 osd.10 90 *** Got signal Terminated *** 2024-09-21T03:07:14.511 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:14 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:14.098+0000 7f923c2d3640 -1 osd.10 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:07:15.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:14.772+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:15.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:14.772+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:15.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:14.772+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:15.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:14.772+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:15.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:14.772+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:15.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:14.772+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:15.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:15 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:15.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:15.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:15 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:15.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:15.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:15 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:15.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:15.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:15 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:15.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:15.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:15 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:15.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:15.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:15 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:15.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:15.511 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:15 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:15.093+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:16.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:15 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:15.802+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:16.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:15 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:15.802+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:16.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:15 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:15.802+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:16.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:15 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:15.802+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:16.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:15 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:15.802+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:16.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:15 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:15.802+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:16.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.044+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:16.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.044+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:16.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.044+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:16.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.044+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:16.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.044+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:16.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.044+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:16.511 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.044+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:17.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:16 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:16.818+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:17.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:16 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:16.818+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:17.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:16 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:16.818+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:17.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:16 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:16.818+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:17.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:16 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:16.818+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:17.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:16 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:16.818+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:17.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:17.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:17.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:17.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:17.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:17.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:17.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:16 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:16.996+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:18.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:17 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:17.825+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:18.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:17 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:17.825+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:18.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:17 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:17.825+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:18.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:17 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:17.825+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:18.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:17 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:17.825+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:18.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:17 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:17.825+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:18.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:18 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:18.017+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:18.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:18 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:18.017+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:18.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:18 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:18.017+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:18.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:18 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:18.017+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:18.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:18 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:18.017+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:18.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:18 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:18.017+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:18.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:18 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:18.017+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:19.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:18 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:18.839+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:19.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:18 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:18.839+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:19.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:18 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:18.839+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:19.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:18 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:18.839+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:19.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:18 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:18.839+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:19.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:18 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:18.839+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:19.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:19 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:19.055+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:55.569428+0000 front 2024-09-21T03:05:55.569292+0000 (oldest deadline 2024-09-21T03:06:19.669331+0000) 2024-09-21T03:07:19.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:19 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:19.055+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:04.871542+0000 front 2024-09-21T03:06:04.871569+0000 (oldest deadline 2024-09-21T03:06:27.771712+0000) 2024-09-21T03:07:19.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:19 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:19.055+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:12.472831+0000 front 2024-09-21T03:06:12.472853+0000 (oldest deadline 2024-09-21T03:06:39.473561+0000) 2024-09-21T03:07:19.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:19 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:19.055+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-09-21T03:06:21.174900+0000 front 2024-09-21T03:06:21.174824+0000 (oldest deadline 2024-09-21T03:06:46.474941+0000) 2024-09-21T03:07:19.510 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:19 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:19.055+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:26.475254+0000 front 2024-09-21T03:06:26.475288+0000 (oldest deadline 2024-09-21T03:06:52.275792+0000) 2024-09-21T03:07:19.511 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:19 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:19.055+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:36.977101+0000 front 2024-09-21T03:06:36.977030+0000 (oldest deadline 2024-09-21T03:07:02.177387+0000) 2024-09-21T03:07:19.511 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:19 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10[56682]: 2024-09-21T03:07:19.055+0000 7f92388be640 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:42.678288+0000 front 2024-09-21T03:06:42.678227+0000 (oldest deadline 2024-09-21T03:07:07.378238+0000) 2024-09-21T03:07:19.867 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:19 smithi066 podman[71699]: 2024-09-21 03:07:19.582882196 +0000 UTC m=+5.579107313 container died 6f1b78f955d12be60555e53293d2eec7c929d9b99ebdcd0cd17069633002bf1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T03:07:20.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:19 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:19.847+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:20.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:19 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:19.847+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:20.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:19 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:19.847+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:20.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:19 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:19.847+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:20.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:19 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:19.847+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:20.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:19 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:19.847+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:20.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:19 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:19.847+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6820 osd.7 since back 2024-09-21T03:06:53.118248+0000 front 2024-09-21T03:06:53.118190+0000 (oldest deadline 2024-09-21T03:07:18.918325+0000) 2024-09-21T03:07:20.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:20 smithi066 podman[71699]: 2024-09-21 03:07:20.098404207 +0000 UTC m=+6.094629315 container cleanup 6f1b78f955d12be60555e53293d2eec7c929d9b99ebdcd0cd17069633002bf1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T03:07:20.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:20 smithi066 bash[71699]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10 2024-09-21T03:07:20.622 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:20 smithi066 podman[71715]: 2024-09-21 03:07:20.29131365 +0000 UTC m=+0.700017012 container remove 6f1b78f955d12be60555e53293d2eec7c929d9b99ebdcd0cd17069633002bf1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T03:07:20.970 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:20 smithi066 podman[71804]: 2024-09-21 03:07:20.523911833 +0000 UTC m=+0.016227805 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:07:20.971 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:20 smithi066 podman[71804]: 2024-09-21 03:07:20.64204921 +0000 UTC m=+0.134365163 container create dbcf428512b1cbb6b8fdf69d127efa97682bec5ccf608a7d9647bcd0bac3abd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-21T03:07:20.971 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:20 smithi066 podman[71804]: 2024-09-21 03:07:20.852715419 +0000 UTC m=+0.345031385 container init dbcf428512b1cbb6b8fdf69d127efa97682bec5ccf608a7d9647bcd0bac3abd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T03:07:20.971 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:20 smithi066 podman[71804]: 2024-09-21 03:07:20.856003614 +0000 UTC m=+0.348319575 container start dbcf428512b1cbb6b8fdf69d127efa97682bec5ccf608a7d9647bcd0bac3abd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T03:07:20.971 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:20 smithi066 podman[71804]: 2024-09-21 03:07:20.969216542 +0000 UTC m=+0.461532510 container attach dbcf428512b1cbb6b8fdf69d127efa97682bec5ccf608a7d9647bcd0bac3abd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T03:07:21.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:20.822+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:21.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:20.822+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:21.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:20.822+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:21.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:20.822+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:21.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:20.822+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:21.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:20.822+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:21.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:20 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:20.822+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6820 osd.7 since back 2024-09-21T03:06:53.118248+0000 front 2024-09-21T03:06:53.118190+0000 (oldest deadline 2024-09-21T03:07:18.918325+0000) 2024-09-21T03:07:21.260 INFO:journalctl@ceph.osd.10.smithi066.stdout:Sep 21 03:07:21 smithi066 podman[71804]: 2024-09-21 03:07:21.049858292 +0000 UTC m=+0.542174245 container died dbcf428512b1cbb6b8fdf69d127efa97682bec5ccf608a7d9647bcd0bac3abd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-10-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T03:07:21.758 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.10.service' 2024-09-21T03:07:21.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:21.799 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-21T03:07:21.799 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-21T03:07:21.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.11 2024-09-21T03:07:22.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:21.804+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:22.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:21.804+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:22.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:21.804+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:22.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:21.804+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:22.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:21.804+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:22.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:21.804+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:22.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:21 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:21.804+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6820 osd.7 since back 2024-09-21T03:06:53.118248+0000 front 2024-09-21T03:06:53.118190+0000 (oldest deadline 2024-09-21T03:07:18.918325+0000) 2024-09-21T03:07:22.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:21 smithi190 systemd[1]: Stopping Ceph osd.11 for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:22.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.133+0000 7fb61c0fb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:22.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.133+0000 7fb61c0fb640 -1 osd.11 90 *** Got signal Terminated *** 2024-09-21T03:07:22.624 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.133+0000 7fb61c0fb640 -1 osd.11 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T03:07:23.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.795+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:23.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.795+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:23.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.795+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:23.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.795+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:23.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.795+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:23.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.795+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:23.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:22 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:22.795+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6820 osd.7 since back 2024-09-21T03:06:53.118248+0000 front 2024-09-21T03:06:53.118190+0000 (oldest deadline 2024-09-21T03:07:18.918325+0000) 2024-09-21T03:07:24.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:23.800+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:24.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:23.801+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:24.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:23.801+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:24.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:23.801+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:24.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:23.801+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:24.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:23.801+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:24.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:23 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:23.801+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6820 osd.7 since back 2024-09-21T03:06:53.118248+0000 front 2024-09-21T03:06:53.118190+0000 (oldest deadline 2024-09-21T03:07:18.918325+0000) 2024-09-21T03:07:25.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:24.758+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:25.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:24.758+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:25.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:24.758+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:25.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:24.758+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:25.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:24.758+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:25.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:24.758+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:25.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:24 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:24.758+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6820 osd.7 since back 2024-09-21T03:06:53.118248+0000 front 2024-09-21T03:06:53.118190+0000 (oldest deadline 2024-09-21T03:07:18.918325+0000) 2024-09-21T03:07:26.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:25.771+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:26.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:25.771+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:26.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:25.771+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:26.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:25.771+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:26.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:25.771+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:26.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:25.771+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:26.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:25.771+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6820 osd.7 since back 2024-09-21T03:06:53.118248+0000 front 2024-09-21T03:06:53.118190+0000 (oldest deadline 2024-09-21T03:07:18.918325+0000) 2024-09-21T03:07:26.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:25 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:25.771+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6820 osd.8 since back 2024-09-21T03:07:00.019135+0000 front 2024-09-21T03:07:00.019163+0000 (oldest deadline 2024-09-21T03:07:25.319130+0000) 2024-09-21T03:07:27.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:26.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-21T03:05:57.109785+0000 front 2024-09-21T03:05:57.109834+0000 (oldest deadline 2024-09-21T03:06:20.009917+0000) 2024-09-21T03:07:27.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:26.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6804 osd.1 since back 2024-09-21T03:06:05.311393+0000 front 2024-09-21T03:06:05.311134+0000 (oldest deadline 2024-09-21T03:06:30.011124+0000) 2024-09-21T03:07:27.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:26.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-09-21T03:06:10.011440+0000 front 2024-09-21T03:06:10.011460+0000 (oldest deadline 2024-09-21T03:06:38.212186+0000) 2024-09-21T03:07:27.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:26.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6812 osd.4 since back 2024-09-21T03:06:27.013658+0000 front 2024-09-21T03:06:27.013677+0000 (oldest deadline 2024-09-21T03:06:51.713392+0000) 2024-09-21T03:07:27.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:26.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-09-21T03:06:38.615647+0000 front 2024-09-21T03:06:38.615705+0000 (oldest deadline 2024-09-21T03:07:00.915710+0000) 2024-09-21T03:07:27.124 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:26.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-09-21T03:06:46.117161+0000 front 2024-09-21T03:06:46.117195+0000 (oldest deadline 2024-09-21T03:07:13.117544+0000) 2024-09-21T03:07:27.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:26.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.66:6820 osd.7 since back 2024-09-21T03:06:53.118248+0000 front 2024-09-21T03:06:53.118190+0000 (oldest deadline 2024-09-21T03:07:18.918325+0000) 2024-09-21T03:07:27.125 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:26 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11[57772]: 2024-09-21T03:07:26.784+0000 7fb6186e6640 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.190:6820 osd.8 since back 2024-09-21T03:07:00.019135+0000 front 2024-09-21T03:07:00.019163+0000 (oldest deadline 2024-09-21T03:07:25.319130+0000) 2024-09-21T03:07:27.968 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:27 smithi190 podman[72087]: 2024-09-21 03:07:27.678692861 +0000 UTC m=+5.679784953 container died 7f999657ec193e865c1e3f1150b5afe701d46eca61f1ee378fd5ab66590d5381 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-21T03:07:28.302 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:28 smithi190 podman[72087]: 2024-09-21 03:07:28.058666369 +0000 UTC m=+6.059758459 container cleanup 7f999657ec193e865c1e3f1150b5afe701d46eca61f1ee378fd5ab66590d5381 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:07:28.302 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:28 smithi190 bash[72087]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11 2024-09-21T03:07:28.607 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:28 smithi190 podman[72116]: 2024-09-21 03:07:28.301396751 +0000 UTC m=+0.615069360 container remove 7f999657ec193e865c1e3f1150b5afe701d46eca61f1ee378fd5ab66590d5381 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T03:07:28.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:28 smithi190 podman[72204]: 2024-09-21 03:07:28.509675322 +0000 UTC m=+0.015466457 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T03:07:28.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:28 smithi190 podman[72204]: 2024-09-21 03:07:28.62722306 +0000 UTC m=+0.133014179 container create 296a23206d6c8c02b964f889afa227944670ecaf56f2afbc22b102a4e0b51e46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-09-21T03:07:28.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:28 smithi190 podman[72204]: 2024-09-21 03:07:28.846569983 +0000 UTC m=+0.352361111 container init 296a23206d6c8c02b964f889afa227944670ecaf56f2afbc22b102a4e0b51e46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T03:07:28.874 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:28 smithi190 podman[72204]: 2024-09-21 03:07:28.849915035 +0000 UTC m=+0.355706163 container start 296a23206d6c8c02b964f889afa227944670ecaf56f2afbc22b102a4e0b51e46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T03:07:29.325 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:28 smithi190 podman[72204]: 2024-09-21 03:07:28.929948311 +0000 UTC m=+0.435739453 container attach 296a23206d6c8c02b964f889afa227944670ecaf56f2afbc22b102a4e0b51e46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:07:29.325 INFO:journalctl@ceph.osd.11.smithi190.stdout:Sep 21 03:07:29 smithi190 podman[72204]: 2024-09-21 03:07:29.041706731 +0000 UTC m=+0.547497855 container died 296a23206d6c8c02b964f889afa227944670ecaf56f2afbc22b102a4e0b51e46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-osd-11-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:07:29.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@osd.11.service' 2024-09-21T03:07:29.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:29.656 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-21T03:07:29.656 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-21T03:07:29.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.a 2024-09-21T03:07:29.974 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:07:29 smithi029 systemd[1]: Stopping Ceph mds.a for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:30.382 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:07:29 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a[72143]: 2024-09-21T03:07:29.972+0000 7fef510b0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:30.382 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:07:29 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a[72143]: 2024-09-21T03:07:29.972+0000 7fef510b0640 -1 mds.a *** got signal Terminated *** 2024-09-21T03:07:33.667 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:07:33 smithi029 podman[117831]: 2024-09-21 03:07:33.399979578 +0000 UTC m=+3.544904516 container died 53953a68aa93f7be5d00b7c8f1b74097316ea7fd466ee48535d6ce3372f4b74a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True) 2024-09-21T03:07:34.077 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:07:33 smithi029 podman[117831]: 2024-09-21 03:07:33.766118214 +0000 UTC m=+3.911043191 container cleanup 53953a68aa93f7be5d00b7c8f1b74097316ea7fd466ee48535d6ce3372f4b74a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T03:07:34.078 INFO:journalctl@ceph.mds.a.smithi029.stdout:Sep 21 03:07:33 smithi029 bash[117831]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-a 2024-09-21T03:07:34.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.a.service' 2024-09-21T03:07:34.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:34.242 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-21T03:07:34.243 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-21T03:07:34.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.d 2024-09-21T03:07:34.382 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:34 smithi029 systemd[1]: Stopping Ceph mds.d for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:34.882 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:34 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-d[71096]: 2024-09-21T03:07:34.565+0000 7f4c4308f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:34.883 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:34 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-d[71096]: 2024-09-21T03:07:34.565+0000 7f4c4308f640 -1 mds.d *** got signal Terminated *** 2024-09-21T03:07:37.277 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:36 smithi029 podman[117954]: 2024-09-21 03:07:36.954228518 +0000 UTC m=+2.491427151 container died a77e9981ac91fb72f566b34f78f43d8c402adf674bc8ce058835a887f5d4387f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-d, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T03:07:37.604 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:37 smithi029 podman[117954]: 2024-09-21 03:07:37.392106475 +0000 UTC m=+2.929305099 container cleanup a77e9981ac91fb72f566b34f78f43d8c402adf674bc8ce058835a887f5d4387f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T03:07:37.604 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:37 smithi029 bash[117954]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-d 2024-09-21T03:07:37.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.d.service' 2024-09-21T03:07:37.854 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:37 smithi029 podman[117968]: 2024-09-21 03:07:37.603278602 +0000 UTC m=+0.646201437 container remove a77e9981ac91fb72f566b34f78f43d8c402adf674bc8ce058835a887f5d4387f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-d, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T03:07:37.854 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:37 smithi029 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.d.service: Deactivated successfully. 2024-09-21T03:07:37.855 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:37 smithi029 systemd[1]: Stopped Ceph mds.d for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T03:07:37.855 INFO:journalctl@ceph.mds.d.smithi029.stdout:Sep 21 03:07:37 smithi029 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.d.service: Consumed 1.285s CPU time. 2024-09-21T03:07:37.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:37.867 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-21T03:07:37.867 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-21T03:07:37.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.g 2024-09-21T03:07:38.132 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:07:37 smithi029 systemd[1]: Stopping Ceph mds.g for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:38.632 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:07:38 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g[71834]: 2024-09-21T03:07:38.215+0000 7f1492f45640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:38.632 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:07:38 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g[71834]: 2024-09-21T03:07:38.215+0000 7f1492f45640 -1 mds.g *** got signal Terminated *** 2024-09-21T03:07:41.632 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:07:41 smithi029 podman[118075]: 2024-09-21 03:07:41.379175213 +0000 UTC m=+3.283050503 container died 23c9bd050707f68df772a36f0502c3f0f22afdb0df7f1a36ea038bad848adf14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:07:42.021 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:07:41 smithi029 podman[118075]: 2024-09-21 03:07:41.785070149 +0000 UTC m=+3.688945439 container cleanup 23c9bd050707f68df772a36f0502c3f0f22afdb0df7f1a36ea038bad848adf14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T03:07:42.022 INFO:journalctl@ceph.mds.g.smithi029.stdout:Sep 21 03:07:41 smithi029 bash[118075]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-g 2024-09-21T03:07:42.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.g.service' 2024-09-21T03:07:42.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:42.204 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-21T03:07:42.205 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-21T03:07:42.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.j 2024-09-21T03:07:42.632 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:42 smithi029 systemd[1]: Stopping Ceph mds.j for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:42.632 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j[70717]: 2024-09-21T03:07:42.525+0000 7f18dae4a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:42.632 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:42 smithi029 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j[70717]: 2024-09-21T03:07:42.525+0000 7f18dae4a640 -1 mds.j *** got signal Terminated *** 2024-09-21T03:07:46.062 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:45 smithi029 podman[118194]: 2024-09-21 03:07:45.703500187 +0000 UTC m=+3.280564675 container died a110a80bdb156d1c8ba01ea86a242860a31a592abf9dab37b0d26c47e294df07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T03:07:46.366 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:46 smithi029 podman[118194]: 2024-09-21 03:07:46.14503155 +0000 UTC m=+3.722096024 container cleanup a110a80bdb156d1c8ba01ea86a242860a31a592abf9dab37b0d26c47e294df07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T03:07:46.366 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:46 smithi029 bash[118194]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j 2024-09-21T03:07:46.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.j.service' 2024-09-21T03:07:46.620 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:46 smithi029 podman[118218]: 2024-09-21 03:07:46.365376364 +0000 UTC m=+0.638511972 container remove a110a80bdb156d1c8ba01ea86a242860a31a592abf9dab37b0d26c47e294df07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-j, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T03:07:46.620 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:46 smithi029 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.j.service: Deactivated successfully. 2024-09-21T03:07:46.621 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:46 smithi029 systemd[1]: Stopped Ceph mds.j for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T03:07:46.621 INFO:journalctl@ceph.mds.j.smithi029.stdout:Sep 21 03:07:46 smithi029 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.j.service: Consumed 1.281s CPU time. 2024-09-21T03:07:46.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:46.634 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-21T03:07:46.634 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-21T03:07:46.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.b 2024-09-21T03:07:46.926 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:07:46 smithi066 systemd[1]: Stopping Ceph mds.b for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:47.260 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:07:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b[61527]: 2024-09-21T03:07:46.923+0000 7f72862e1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:47.260 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:07:46 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b[61527]: 2024-09-21T03:07:46.923+0000 7f72862e1640 -1 mds.b *** got signal Terminated *** 2024-09-21T03:07:49.567 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:07:49 smithi066 podman[71924]: 2024-09-21 03:07:49.311947295 +0000 UTC m=+2.474810100 container died 5e59c4f1839e56cc4973652b988f1fd96e5f06722d215ccb6818eb69b594eb9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T03:07:49.959 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:07:49 smithi066 podman[71924]: 2024-09-21 03:07:49.657703606 +0000 UTC m=+2.820566409 container cleanup 5e59c4f1839e56cc4973652b988f1fd96e5f06722d215ccb6818eb69b594eb9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T03:07:49.960 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:07:49 smithi066 bash[71924]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b 2024-09-21T03:07:50.227 INFO:journalctl@ceph.mds.b.smithi066.stdout:Sep 21 03:07:49 smithi066 podman[71939]: 2024-09-21 03:07:49.958930733 +0000 UTC m=+0.642483592 container remove 5e59c4f1839e56cc4973652b988f1fd96e5f06722d215ccb6818eb69b594eb9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T03:07:50.242 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.b.service' 2024-09-21T03:07:50.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:50.278 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-21T03:07:50.278 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-21T03:07:50.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.e 2024-09-21T03:07:50.509 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:07:50 smithi066 systemd[1]: Stopping Ceph mds.e for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:51.010 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:07:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: 2024-09-21T03:07:50.602+0000 7f5d844c9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:51.010 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:07:50 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e[60980]: 2024-09-21T03:07:50.602+0000 7f5d844c9640 -1 mds.e *** got signal Terminated *** 2024-09-21T03:07:53.704 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:07:53 smithi066 podman[72044]: 2024-09-21 03:07:53.316185668 +0000 UTC m=+2.810498421 container died 00e9dcb7646e829478b2fe8fe4ecbd822835e4a9e7e3042526060f1f908026b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2) 2024-09-21T03:07:54.010 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:07:53 smithi066 podman[72044]: 2024-09-21 03:07:53.794752777 +0000 UTC m=+3.289065571 container cleanup 00e9dcb7646e829478b2fe8fe4ecbd822835e4a9e7e3042526060f1f908026b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T03:07:54.010 INFO:journalctl@ceph.mds.e.smithi066.stdout:Sep 21 03:07:53 smithi066 bash[72044]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-e 2024-09-21T03:07:54.232 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.e.service' 2024-09-21T03:07:54.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:54.266 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-21T03:07:54.266 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-21T03:07:54.266 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.h 2024-09-21T03:07:54.583 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:07:54 smithi066 systemd[1]: Stopping Ceph mds.h for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:55.010 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:07:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-h[61255]: 2024-09-21T03:07:54.581+0000 7fdb18e36640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:55.010 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:07:54 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-h[61255]: 2024-09-21T03:07:54.581+0000 7fdb18e36640 -1 mds.h *** got signal Terminated *** 2024-09-21T03:07:57.750 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:07:57 smithi066 podman[72165]: 2024-09-21 03:07:57.311624209 +0000 UTC m=+2.828596579 container died 51d18469cafeb8a1f951837ecc7a57858efffa2da11a8d0d5606e60bab4b9fdc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-h, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:07:58.010 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:07:57 smithi066 podman[72165]: 2024-09-21 03:07:57.823737524 +0000 UTC m=+3.340709883 container cleanup 51d18469cafeb8a1f951837ecc7a57858efffa2da11a8d0d5606e60bab4b9fdc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T03:07:58.010 INFO:journalctl@ceph.mds.h.smithi066.stdout:Sep 21 03:07:57 smithi066 bash[72165]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-h 2024-09-21T03:07:58.208 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.h.service' 2024-09-21T03:07:58.241 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:07:58.241 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-21T03:07:58.241 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-21T03:07:58.241 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.k 2024-09-21T03:07:58.760 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:07:58 smithi066 systemd[1]: Stopping Ceph mds.k for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:07:58.760 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:07:58 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k[60698]: 2024-09-21T03:07:58.568+0000 7f6476c95640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:07:58.760 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:07:58 smithi066 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k[60698]: 2024-09-21T03:07:58.568+0000 7f6476c95640 -1 mds.k *** got signal Terminated *** 2024-09-21T03:08:01.587 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:08:01 smithi066 podman[72288]: 2024-09-21 03:08:01.313965521 +0000 UTC m=+2.847813512 container died 328ec19fddf2132b2b468281bc5e45a2576cc433e5088acbe7e767a09fed6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T03:08:01.929 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:08:01 smithi066 podman[72288]: 2024-09-21 03:08:01.668865337 +0000 UTC m=+3.202713316 container cleanup 328ec19fddf2132b2b468281bc5e45a2576cc433e5088acbe7e767a09fed6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T03:08:01.929 INFO:journalctl@ceph.mds.k.smithi066.stdout:Sep 21 03:08:01 smithi066 bash[72288]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-k 2024-09-21T03:08:02.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.k.service' 2024-09-21T03:08:02.134 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:08:02.134 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-21T03:08:02.134 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-21T03:08:02.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.c 2024-09-21T03:08:02.501 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:08:02 smithi190 systemd[1]: Stopping Ceph mds.c for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:08:02.764 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:08:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c[62423]: 2024-09-21T03:08:02.499+0000 7fdddd8e5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:08:02.764 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:08:02 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c[62423]: 2024-09-21T03:08:02.499+0000 7fdddd8e5640 -1 mds.c *** got signal Terminated *** 2024-09-21T03:08:05.624 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:08:05 smithi190 podman[72306]: 2024-09-21 03:08:05.337819311 +0000 UTC m=+3.014112184 container died 783b37f837dc826a97638b02cba5ba98f22419457bb6c6a879d456c4fcab42eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T03:08:05.968 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:08:05 smithi190 podman[72306]: 2024-09-21 03:08:05.741198548 +0000 UTC m=+3.417491430 container cleanup 783b37f837dc826a97638b02cba5ba98f22419457bb6c6a879d456c4fcab42eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T03:08:05.968 INFO:journalctl@ceph.mds.c.smithi190.stdout:Sep 21 03:08:05 smithi190 bash[72306]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-c 2024-09-21T03:08:06.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.c.service' 2024-09-21T03:08:06.157 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:08:06.158 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-21T03:08:06.158 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-21T03:08:06.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.f 2024-09-21T03:08:06.471 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:06 smithi190 systemd[1]: Stopping Ceph mds.f for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:08:06.873 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:06 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f[62151]: 2024-09-21T03:08:06.470+0000 7fdcf7b32640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:08:06.874 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:06 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f[62151]: 2024-09-21T03:08:06.470+0000 7fdcf7b32640 -1 mds.f *** got signal Terminated *** 2024-09-21T03:08:09.605 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:09 smithi190 podman[72439]: 2024-09-21 03:08:09.346369465 +0000 UTC m=+2.963763838 container died a231aab13ca903f0db603b1da8a98321d76231ec2218709ae2a31287da5b0e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T03:08:09.872 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:09 smithi190 podman[72439]: 2024-09-21 03:08:09.678228452 +0000 UTC m=+3.295622890 container cleanup a231aab13ca903f0db603b1da8a98321d76231ec2218709ae2a31287da5b0e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T03:08:09.872 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:09 smithi190 bash[72439]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f 2024-09-21T03:08:10.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.f.service' 2024-09-21T03:08:10.123 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:09 smithi190 podman[72451]: 2024-09-21 03:08:09.871060953 +0000 UTC m=+0.527153723 container remove a231aab13ca903f0db603b1da8a98321d76231ec2218709ae2a31287da5b0e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-f, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T03:08:10.124 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:10 smithi190 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.f.service: Deactivated successfully. 2024-09-21T03:08:10.124 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:10 smithi190 systemd[1]: Stopped Ceph mds.f for e1049baa-77c0-11ef-baf5-efdc52797490. 2024-09-21T03:08:10.124 INFO:journalctl@ceph.mds.f.smithi190.stdout:Sep 21 03:08:10 smithi190 systemd[1]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.f.service: Consumed 4min 38.612s CPU time. 2024-09-21T03:08:10.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:08:10.143 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-21T03:08:10.143 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-21T03:08:10.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.i 2024-09-21T03:08:10.466 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:08:10 smithi190 systemd[1]: Stopping Ceph mds.i for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:08:10.874 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:08:10 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: 2024-09-21T03:08:10.465+0000 7fc2798e2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:08:10.874 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:08:10 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i[62695]: 2024-09-21T03:08:10.465+0000 7fc2798e2640 -1 mds.i *** got signal Terminated *** 2024-09-21T03:08:13.624 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:08:13 smithi190 podman[72559]: 2024-09-21 03:08:13.341603989 +0000 UTC m=+2.986666865 container died bc6ea4992bafb0117474ec924701e072345834330b93b20c6d28e27f492b943b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T03:08:13.992 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:08:13 smithi190 podman[72559]: 2024-09-21 03:08:13.698438009 +0000 UTC m=+3.343500884 container cleanup bc6ea4992bafb0117474ec924701e072345834330b93b20c6d28e27f492b943b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T03:08:13.992 INFO:journalctl@ceph.mds.i.smithi190.stdout:Sep 21 03:08:13 smithi190 bash[72559]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-i 2024-09-21T03:08:14.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.i.service' 2024-09-21T03:08:14.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:08:14.175 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-21T03:08:14.175 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-21T03:08:14.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.l 2024-09-21T03:08:14.624 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:08:14 smithi190 systemd[1]: Stopping Ceph mds.l for e1049baa-77c0-11ef-baf5-efdc52797490... 2024-09-21T03:08:14.624 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:08:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l[61877]: 2024-09-21T03:08:14.485+0000 7f37ea97a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T03:08:14.624 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:08:14 smithi190 ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l[61877]: 2024-09-21T03:08:14.485+0000 7f37ea97a640 -1 mds.l *** got signal Terminated *** 2024-09-21T03:08:17.713 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:08:17 smithi190 podman[72680]: 2024-09-21 03:08:17.438533952 +0000 UTC m=+3.051748445 container died 7019d1c01fae166566e17ca56bbfc5ca988a59e3ec7d375e9664618cd9c18fa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T03:08:18.070 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:08:17 smithi190 podman[72680]: 2024-09-21 03:08:17.785429575 +0000 UTC m=+3.398644059 container cleanup 7019d1c01fae166566e17ca56bbfc5ca988a59e3ec7d375e9664618cd9c18fa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T03:08:18.070 INFO:journalctl@ceph.mds.l.smithi190.stdout:Sep 21 03:08:17 smithi190 bash[72680]: ceph-e1049baa-77c0-11ef-baf5-efdc52797490-mds-l 2024-09-21T03:08:18.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e1049baa-77c0-11ef-baf5-efdc52797490@mds.l.service' 2024-09-21T03:08:18.249 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T03:08:18.249 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-21T03:08:18.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e1049baa-77c0-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T03:08:18.463 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T03:08:21.033 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e1049baa-77c0-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T03:08:21.248 INFO:teuthology.orchestra.run.smithi066.stdout:Deleting cluster with fsid: e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T03:08:23.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e1049baa-77c0-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T03:08:23.948 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T03:08:26.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T03:08:26.527 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T03:08:26.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T03:08:26.587 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T03:08:26.589 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi029/crash 2024-09-21T03:08:26.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash -- . 2024-09-21T03:08:26.639 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T03:08:26.639 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-09-21T03:08:26.641 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi066/crash 2024-09-21T03:08:26.642 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash -- . 2024-09-21T03:08:26.744 INFO:teuthology.orchestra.run.smithi066.stderr:tar: /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T03:08:26.745 INFO:teuthology.orchestra.run.smithi066.stderr:tar: Error is not recoverable: exiting now 2024-09-21T03:08:26.747 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi190/crash 2024-09-21T03:08:26.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash -- . 2024-09-21T03:08:26.786 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T03:08:26.786 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-09-21T03:08:26.787 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T03:08:26.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-21T03:08:26.926 INFO:tasks.cephadm:Compressing logs... 2024-09-21T03:08:26.926 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T03:08:26.932 DEBUG:teuthology.orchestra.run.smithi066:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T03:08:26.934 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T03:08:26.957 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T03:08:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-09-21T03:08:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-21T03:08:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.log: /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.9.log 2024-09-21T03:08:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T03:08:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T03:08:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.a.log 2024-09-21T03:08:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T03:08:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.a.log 2024-09-21T03:08:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.g.log 2024-09-21T03:08:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-21T03:08:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log 2024-09-21T03:08:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.d.log 2024-09-21T03:08:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log: /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.x.log 2024-09-21T03:08:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.j.log 2024-09-21T03:08:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-21T03:08:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/cephadm.log: 87.0% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.10.log: No such file or directory 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.b.log 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.k.log 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-21T03:08:26.966 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T03:08:26.967 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T03:08:26.969 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log: 90.4% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T03:08:26.969 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T03:08:26.970 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.y.log 2024-09-21T03:08:26.970 INFO:teuthology.orchestra.run.smithi029.stderr: 95.1% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.d.log.gz 2024-09-21T03:08:26.971 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T03:08:26.971 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log 2024-09-21T03:08:26.972 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T03:08:26.972 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T03:08:26.972 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.i.log 2024-09-21T03:08:26.972 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.c.log 2024-09-21T03:08:26.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T03:08:26.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.11.log 2024-09-21T03:08:26.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-21T03:08:26.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-21T03:08:26.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log 2024-09-21T03:08:26.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.l.log 2024-09-21T03:08:26.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.8.log 2024-09-21T03:08:26.976 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.z.log 2024-09-21T03:08:26.976 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.c.log 2024-09-21T03:08:26.976 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.f.log 2024-09-21T03:08:26.976 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T03:08:26.976 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.h.log 2024-09-21T03:08:26.980 INFO:teuthology.orchestra.run.smithi066.stderr: 90.9% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T03:08:26.983 INFO:teuthology.orchestra.run.smithi066.stderr: 93.2% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T03:08:26.988 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log: 87.0% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T03:08:26.988 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.b.log 2024-09-21T03:08:26.989 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.e.log 2024-09-21T03:08:27.009 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T03:08:27.015 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.e.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T03:08:27.038 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.f.log: /var/log/ceph/cephadm.log: 83.4% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T03:08:27.051 INFO:teuthology.orchestra.run.smithi190.stderr: 87.0% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T03:08:27.193 INFO:teuthology.orchestra.run.smithi029.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-09-21T03:08:27.221 INFO:teuthology.orchestra.run.smithi190.stderr: 93.3% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.z.log.gz 2024-09-21T03:08:27.229 INFO:teuthology.orchestra.run.smithi190.stderr: 90.9% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T03:08:27.231 INFO:teuthology.orchestra.run.smithi029.stderr: 90.3% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T03:08:27.253 INFO:teuthology.orchestra.run.smithi066.stderr: 94.1% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.k.log.gz 2024-09-21T03:08:27.275 INFO:teuthology.orchestra.run.smithi029.stderr: 95.1% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.j.log.gz 2024-09-21T03:08:27.361 INFO:teuthology.orchestra.run.smithi029.stderr: 83.1% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T03:08:27.466 INFO:teuthology.orchestra.run.smithi190.stderr: 90.5% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T03:08:27.470 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T03:08:27.610 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T03:08:27.695 INFO:teuthology.orchestra.run.smithi029.stderr: 94.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T03:08:27.779 INFO:teuthology.orchestra.run.smithi066.stderr: 91.6% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.b.log.gz 2024-09-21T03:08:30.908 INFO:teuthology.orchestra.run.smithi066.stderr: 93.8% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.h.log.gz 2024-09-21T03:08:33.543 INFO:teuthology.orchestra.run.smithi029.stderr: 93.9% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.g.log.gz 2024-09-21T03:08:33.656 INFO:teuthology.orchestra.run.smithi029.stderr: 90.5% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T03:08:35.489 INFO:teuthology.orchestra.run.smithi066.stderr: 94.8% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.e.log.gz 2024-09-21T03:08:39.919 INFO:teuthology.orchestra.run.smithi190.stderr: 91.6% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.c.log.gz 2024-09-21T03:08:47.271 INFO:teuthology.orchestra.run.smithi066.stderr: 91.7% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-21T03:08:50.517 INFO:teuthology.orchestra.run.smithi029.stderr: 91.6% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-21T03:08:53.304 INFO:teuthology.orchestra.run.smithi066.stderr: 91.3% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.10.log.gz 2024-09-21T03:08:54.630 INFO:teuthology.orchestra.run.smithi029.stderr: 91.3% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-21T03:08:56.593 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mon.a.log.gz 2024-09-21T03:08:57.165 INFO:teuthology.orchestra.run.smithi066.stderr: 91.3% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-21T03:08:58.001 INFO:teuthology.orchestra.run.smithi066.stderr: 91.3% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-21T03:09:12.370 INFO:teuthology.orchestra.run.smithi066.stderr: 95.0% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.b.log.gz 2024-09-21T03:09:12.372 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-09-21T03:09:12.372 INFO:teuthology.orchestra.run.smithi066.stderr:real 0m45.422s 2024-09-21T03:09:12.372 INFO:teuthology.orchestra.run.smithi066.stderr:user 2m41.660s 2024-09-21T03:09:12.372 INFO:teuthology.orchestra.run.smithi066.stderr:sys 0m4.567s 2024-09-21T03:09:12.510 INFO:teuthology.orchestra.run.smithi029.stderr: 95.2% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.a.log.gz 2024-09-21T03:09:30.835 INFO:teuthology.orchestra.run.smithi190.stderr: 95.1% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.l.log.gz 2024-09-21T03:09:53.577 INFO:teuthology.orchestra.run.smithi190.stderr: 93.8% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.c.log.gz 2024-09-21T03:10:54.624 INFO:teuthology.orchestra.run.smithi190.stderr: 91.2% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-21T03:11:32.601 INFO:teuthology.orchestra.run.smithi029.stderr: 91.2% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-21T03:11:41.385 INFO:teuthology.orchestra.run.smithi029.stderr: 91.2% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.9.log.gz 2024-09-21T03:12:20.189 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.11.log.gz 2024-09-21T03:13:17.086 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.8.log.gz 2024-09-21T03:15:23.056 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-21T03:17:40.512 INFO:teuthology.orchestra.run.smithi029.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-09-21T03:17:40.514 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-21T03:17:40.514 INFO:teuthology.orchestra.run.smithi029.stderr:real 9m13.567s 2024-09-21T03:17:40.514 INFO:teuthology.orchestra.run.smithi029.stderr:user 6m10.181s 2024-09-21T03:17:40.514 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m15.877s 2024-09-21T03:21:11.627 INFO:teuthology.orchestra.run.smithi190.stderr: 94.5% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.f.log.gz 2024-09-21T03:26:51.208 INFO:teuthology.orchestra.run.smithi190.stderr: 94.8% -- replaced with /var/log/ceph/e1049baa-77c0-11ef-baf5-efdc52797490/ceph-mds.i.log.gz 2024-09-21T03:26:51.210 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-21T03:26:51.210 INFO:teuthology.orchestra.run.smithi190.stderr:real 18m24.261s 2024-09-21T03:26:51.210 INFO:teuthology.orchestra.run.smithi190.stderr:user 8m52.046s 2024-09-21T03:26:51.210 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m25.466s 2024-09-21T03:26:51.211 INFO:tasks.cephadm:Archiving logs... 2024-09-21T03:26:51.211 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi029/log 2024-09-21T03:26:51.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T03:27:46.830 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi066/log 2024-09-21T03:27:46.831 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T03:28:07.971 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi190/log 2024-09-21T03:28:07.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T03:29:21.693 INFO:tasks.cephadm:Removing cluster... 2024-09-21T03:29:21.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e1049baa-77c0-11ef-baf5-efdc52797490 --force 2024-09-21T03:29:21.920 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T03:29:22.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e1049baa-77c0-11ef-baf5-efdc52797490 --force 2024-09-21T03:29:22.510 INFO:teuthology.orchestra.run.smithi066.stdout:Deleting cluster with fsid: e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T03:29:22.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e1049baa-77c0-11ef-baf5-efdc52797490 --force 2024-09-21T03:29:23.499 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: e1049baa-77c0-11ef-baf5-efdc52797490 2024-09-21T03:29:24.152 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T03:29:24.153 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T03:29:24.173 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T03:29:24.193 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T03:29:24.209 INFO:tasks.cephadm:Teardown complete 2024-09-21T03:29:24.209 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T03:29:24.222 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T03:29:24.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T03:29:24.225 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T03:29:24.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T03:29:24.356 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T03:29:24.368 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T03:29:24.368 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T03:29:24.371 DEBUG:teuthology.orchestra.run.smithi066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T03:29:24.374 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T03:29:24.386 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-09-21T03:29:24.388 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-09-21T03:29:24.389 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-09-21T03:29:24.394 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T03:29:24.394 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-09-21T03:29:24.394 INFO:teuthology.orchestra.run.smithi066.stdout:^* hv01.front.sepia.ceph.com 3 6 377 11 -235us[ -258us] +/- 44ms 2024-09-21T03:29:24.394 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 17 +422us[ +399us] +/- 70ms 2024-09-21T03:29:24.394 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -9208ns[ -32us] +/- 75ms 2024-09-21T03:29:24.394 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T03:29:24.408 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T03:29:24.408 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-09-21T03:29:24.409 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 -234us[ -222us] +/- 44ms 2024-09-21T03:29:24.409 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 21 +430us[ +443us] +/- 70ms 2024-09-21T03:29:24.409 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -4237ns[-4237ns] +/- 75ms 2024-09-21T03:29:24.409 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T03:29:24.417 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T03:29:24.418 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-09-21T03:29:24.418 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 -255us[ -278us] +/- 44ms 2024-09-21T03:29:24.418 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 +406us[ +406us] +/- 70ms 2024-09-21T03:29:24.418 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -20us[ -20us] +/- 75ms 2024-09-21T03:29:24.418 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T03:29:24.418 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T03:29:24.427 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T03:29:24.428 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T03:29:24.442 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T03:29:24.603 DEBUG:teuthology.orchestra.run.smithi066:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T03:29:24.679 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T03:29:24.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T03:29:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:29:24.829 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T03:29:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:29:24.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T03:29:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:29:24.890 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T03:29:24.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T03:29:24.952 INFO:teuthology.task.internal:Duration was 4137.645382 seconds 2024-09-21T03:29:24.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T03:29:24.967 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T03:29:24.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T03:29:24.970 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T03:29:24.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T03:29:25.018 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T03:29:25.027 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T03:29:25.029 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T03:29:25.438 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T03:29:25.439 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T03:29:25.439 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T03:29:25.464 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi066.front.sepia.ceph.com 2024-09-21T03:29:25.465 DEBUG:teuthology.orchestra.run.smithi066:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T03:29:25.497 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-09-21T03:29:25.497 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T03:29:25.521 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T03:29:25.521 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T03:29:25.524 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T03:29:25.542 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T03:29:25.587 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T03:29:25.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T03:29:25.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T03:29:25.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T03:29:26.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T03:29:26.638 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T03:29:26.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T03:29:26.666 DEBUG:teuthology.orchestra.run.smithi066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T03:29:26.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T03:29:26.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T03:29:26.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T03:29:26.767 DEBUG:teuthology.orchestra.run.smithi066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T03:29:26.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T03:29:26.796 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = core 2024-09-21T03:29:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-09-21T03:29:26.809 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-09-21T03:29:26.830 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T03:29:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:29:26.878 DEBUG:teuthology.orchestra.run.smithi066:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T03:29:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:29:26.895 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T03:29:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:29:26.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T03:29:26.923 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T03:29:26.925 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi029 2024-09-21T03:29:26.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T03:29:26.985 DEBUG:teuthology.misc:Transferring archived files from smithi066:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi066 2024-09-21T03:29:26.985 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T03:29:27.024 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913420/remote/smithi190 2024-09-21T03:29:27.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T03:29:27.061 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T03:29:27.061 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T03:29:27.064 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T03:29:27.066 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T03:29:27.112 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T03:29:27.125 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T03:29:27.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T03:29:27.139 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T03:29:27.139 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T03:29:27.142 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T03:29:27.144 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T03:29:27.155 INFO:teuthology.orchestra.run.smithi029.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 03:29 /home/ubuntu/cephtest 2024-09-21T03:29:27.158 INFO:teuthology.orchestra.run.smithi066.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 03:29 /home/ubuntu/cephtest 2024-09-21T03:29:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 03:29 /home/ubuntu/cephtest 2024-09-21T03:29:27.169 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T03:29:27.182 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T03:29:27.210 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 4137.645382404327 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-09-21T03:29:27.210 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T03:29:27.304 INFO:teuthology.run:pass